Model Regression Detection System

A CI/CD-style quality gate for LLM features. It runs a feature (a prompt + model) over a fixed golden dataset on every prompt/model change, compares the result against the previously-accepted version, and blocks the change if quality regressed — before bad outputs reach users.

Most teams ship prompt changes blind. This project is about what happens after you decide to change a prompt: does the change quietly make things worse?

---

What it does

flowchart LR
    P["prompt.txt + model"] --> R
    G[("golden.jsonl<br/>15 cases")] --> R
    R["run<br/>(call model per case)"] --> S["score<br/>deterministic gate<br/>+ judge (advisory)"]
    S --> C{"compare vs<br/>baseline.json"}
    C --> V[["PASS / REGRESSION / INCONCLUSIVE"]]

Loading

1. Run the feature over every golden case, collecting outputs (+ cost & latency).
2. Score each output: cheap deterministic checks are the gate; an optional LLM judge is advisory (a soft quality score that never flips the gate).
3. Compare the run against a promoted baseline (the last accepted version).
4. Decide: PASS, REGRESSION (blocks), or INCONCLUSIVE (too little signal).

The whole engine is a single readable file — regdet.py (~300 lines, no framework). Modules/CI/report get split out only when one file actually hurts.

---

The part that matters: honest statistics

The naive version of this project gates on "the score dropped." That's wrong. On ~15–30 cases, one or two failures can just be model noise (LLMs are non-deterministic — the same input can give different outputs). Gating on a raw number produces a flaky check teams rip out within a week.

This engine gates like an experiment:

flowchart TD
    Start["new run vs baseline"] --> Infra{"infra errors?<br/>(5xx / timeout)"}
    Infra -- yes --> INFRA[["INFRA · exit 3"]]
    Infra -- no --> Hard{"critical-slice flip<br/>pass → fail?"}
    Hard -- yes --> REG[["REGRESSION · exit 1"]]
    Hard -- no --> NoReg{"any pass → fail?"}
    NoReg -- no --> PASS[["PASS · exit 0"]]
    NoReg -- yes --> Noise{"drop ≤ baseline's<br/>own measured noise?"}
    Noise -- yes --> PASS
    Noise -- no --> Sig{"McNemar p < 0.05<br/>and drop ≥ 5%?"}
    Sig -- yes --> REG
    Sig -- no --> INC[["INCONCLUSIVE · exit 2"]]

Loading

• Hard-block: a single break in a critical slice (invalid output format, or a lost safety refusal) blocks immediately — regardless of statistics.
• McNemar's test: a paired test on per-case pass/fail changes. It asks whether the change is asymmetric beyond chance, not whether a mean wiggled.
• Noise band: the baseline stores its own run-to-run variance (measured by re-running it K times at promotion). A drop within that band is noise, not a regression.
• INCONCLUSIVE: when the drop looks real but the sample is too small to prove it isn't noise, the engine says so — instead of a fake green or a false alarm.

Talking point: "3 failures out of 15 is McNemar p = 0.25 — not significant. I don't pretend it is. Critical breaks hard-block; everything else is tested against the baseline's measured noise, and when I'm underpowered I return INCONCLUSIVE."

---

Measured demo

The bundled feature is structured extraction (a support message → JSON with intent, product, amount, urgency; garbage input → {"refused": true}).

Introduce a change that only breaks the critical slices, and the gate reacts selectively — a single bad format is enough to block:

xychart-beta
    title "Per-slice pass-rate — a change that breaks only the critical slices"
    x-axis [extraction, edge, format, refusal]
    y-axis "pass-rate %" 0 --> 100
    bar [100, 100, 0, 0]

Loading

Verified end-to-end (real claude-haiku-4-5, actual numbers):

Scenario	Command	Verdict	Exit	Cost
Healthy prompt	run --provider anthropic	PASS (100% all slices)	0	$0.0047
Weakened prompt	run --provider anthropic --prompt prompt_weak.txt	REGRESSION (hard-block, McNemar p≈0.000)	1	$0.0101
Offline, no key	run --provider mock	PASS	0	$0
Simulated regression	run --provider mock --mock-bad	REGRESSION	1	$0
Unit tests	pytest	12 passed	0	$0

What the gate prints

A healthy change — the gate stays green:

$ python regdet.py run --provider mock

=== per-slice pass-rate ===
  edge           100%
  extraction     100%
  format         100%  (critical)
  refusal        100%  (critical)
  OVERALL        100%
  cost $0.0000   infra-errors 0

=== detector ===
  drop +0%   noise-band +/-0%   McNemar p=1.000

  VERDICT: PASS  (no regressions)

A change that breaks the critical slices — blocked, with the exact failing cases:

$ python regdet.py run --provider mock --mock-bad

=== per-slice pass-rate ===
  edge           100%
  extraction     100%
  format           0%  (critical)
  refusal          0%  (critical)
  OVERALL         60%
  cost $0.0000   infra-errors 0

=== detector ===
  drop +40%   noise-band +/-0%   McNemar p=0.031
  regressed (pass -> fail):
    - fmt1 (format) [CRITICAL]
    - fmt2 (format) [CRITICAL]
    - fmt3 (format) [CRITICAL]
    - ref1 (refusal) [CRITICAL]
    - ref2 (refusal) [CRITICAL]
    - ref3 (refusal) [CRITICAL]

  VERDICT: REGRESSION  (hard-block on critical slice)

$ echo $?          # non-zero exit -> CI check goes red -> merge blocked
1

---

Quickstart

pip install -r requirements.txt

# 1) Offline, no API key — proves the whole pipeline works deterministically
python regdet.py promote --provider mock     # set the baseline
python regdet.py run     --provider mock      # -> PASS  (exit 0)
python regdet.py run     --provider mock --mock-bad   # -> REGRESSION (exit 1)

# 2) Real model — put your key in a gitignored .env first:
#    ANTHROPIC_API_KEY=sk-ant-...
python regdet.py run --provider anthropic --model claude-haiku-4-5

# 3) See a real regression get caught
python regdet.py run --provider anthropic --prompt prompt_weak.txt   # -> REGRESSION

# 4) Optional advisory LLM judge (costs tokens)
python regdet.py run --provider anthropic --judge --judge-model claude-opus-4-8

# tests (no key)
pytest -q

---

The golden dataset

15 hand-curated cases in golden.jsonl, stratified into slices so a regression can be localized, not just detected:

Slice	What it checks	Critical?
extraction	correct fields extracted from a well-formed message	no
edge	missing info → null (no hallucinated values)	no
format	output is valid JSON per the schema	yes (hard-block)
refusal	garbage/empty input → correct refusal	yes (hard-block)

Each case declares its own checks. Golden data is plain JSONL in git on purpose — when the quality bar moves, it shows up in a reviewable diff (not hidden in a database).

Scoring: gate vs advisory

Check type	Role	Cost
valid_json, json_field, is_refusal, regex, contains	gate — decides pass/fail	free, deterministic
judge	advisory — soft 1–5 quality score, never flips the gate	one model call (opt-in via --judge)

The judge uses structured output (a JSON schema) for a stable verdict, an optional self-consistency median (--judge-samples), and a pinned model — because current Opus models don't expose a temperature knob, so judge stability is engineered, not assumed.

---

Verdicts & exit codes

Verdict	Exit	Meaning
PASS	0	quality held (or the drop is within measured noise)
REGRESSION	1	critical break, or a statistically significant drop
INCONCLUSIVE	2	drop looks real but the sample is underpowered — add cases / raise K
INFRA	3	provider errors (5xx / rate-limit) — not a quality verdict

Distinguishing an infra failure from a quality failure is deliberate: a flaky API must never masquerade as a regression.

---

CI gate (.github/workflows/regression-gate.yml)

Triggers on PRs that touch the prompt, model, dataset, or harness:

• test — runs pytest (no key).
• gate — runs a deterministic mock smoke (always, no secret), then the real Claude quality gate. A REGRESSION exits non-zero → the check goes red → merge is blocked (with branch protection).

Security note: the keyed Claude gate runs on same-repo PRs only (head.repo.full_name == github.repository). Forked PRs don't receive secrets by design, so the API key is never exposed to untrusted code — forks still get the mock smoke.

---

Design decisions (and what's intentionally not here)

• One file first (Karpathy-style). No premature providers/, scorers/, loaders/ packages. Split when it hurts, not before.
• Golden stays JSONL in git, not a vector database — the reviewable PR diff is the feature.
• Deliberately excluded: a reranker, model quantization, and a full retrieval stack. There's nothing to rerank or quantize in a regression detector; adding them would be résumé-padding, not engineering. An embedding-similarity / drift scorer is a plausible future plugin, not a core need.
• Multi-format ingestion (Docling) is a lazy, side-of-core option — the core is format-agnostic and only ever sees normalized text.

---

Repo layout

regdet.py         # the whole engine (provider, scorers, judge, detector, CLI)
prompt.txt        # the feature prompt (editing it = a "PR" the gate evaluates)
prompt_weak.txt   # a deliberately weakened prompt, for the regression demo
golden.jsonl      # 15 golden cases across 4 slices
baseline.json     # the promoted "known-good" version (committed, reviewable)
test_regdet.py    # 12 offline unit tests (scorers, McNemar, detector, e2e-mock)
requirements.txt
.github/workflows/regression-gate.yml

Roadmap

• Promote the baseline on the real provider (measure real noise band).
• Subtler regression fixtures to exercise the INCONCLUSIVE path on live data.
• Slack alert + a static HTML diff report as CI artifacts.
• Optional Docling loader for document-input features; optional embedding-drift scorer.

License

MIT.