Skip to content
This repository was archived by the owner on Apr 7, 2026. It is now read-only.
This repository was archived by the owner on Apr 7, 2026. It is now read-only.

feat: probar experiment — ML training experiment tracker with early stopping #32

Description

@noahgift

Problem

ML training exploration in bashrs (Shell Safety Classifier) consumed significant GPU time:

  • 6 training runs on RTX 4090
  • 3 kill criteria triggered
  • Peaked at 32% accuracy
  • HuggingFace model ultimately deleted
  • Root cause (data distribution mismatch) identified late

Proposed Solution

probar experiment init "SSC v11 training"
probar experiment run --config train.toml --budget 2h-gpu
probar data-audit training_data.jsonl        # Pre-flight data quality check
probar experiment status                      # Live training dashboard
probar experiment compare run-1 run-6        # Compare runs

Requirements

  1. Pre-flight data audit: Check label distribution, detect class imbalance, validate domain coverage before burning GPU time
  2. Budget gates: Set GPU-hour or cost limits per experiment
  3. Early stopping: Monitor validation loss curve, stop when plateau detected (save remaining GPU budget)
  4. Experiment tracking: Store hyperparameters, metrics, and artifacts per run
  5. Kill criteria evaluation: Check accuracy thresholds at intermediate checkpoints, not just at the end
  6. Cost reporting: Track actual GPU-hours and estimated cost per experiment

Acceptance Criteria

  • probar data-audit detects class imbalance >3:1
  • probar data-audit detects domain mismatch between train/eval
  • Budget gate prevents exceeding GPU-hour limit
  • Early stopping triggers on validation loss plateau (patience=3)
  • Kill criteria checked at each checkpoint, not just end of training
  • Experiment comparison shows metric diff across runs

Cross-Reference

Filed from paiml-mcp-agent-toolkit#258 as part of sovereign stack tooling gap analysis.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions