eval-auditor

A meta-eval: an eval that grades the quality of other evals. Point it at a benchmark and it prints a benchmark health report — per-dimension scores, letter grades, and prioritized, copy-pasteable fixes.

Why this exists

Whoever writes the benchmark writes the leaderboard. Every "model X beats model Y" claim, every ship/no-ship decision, every research result rests on an eval that some human built — and a bad eval doesn't announce itself. It silently misleads everyone downstream: a saturated benchmark where the best and worst models both score ~95% can't rank anything, but its numbers still look authoritative. A benchmark with the answer leaked into the prompt rewards copying over reasoning. Duplicate items quietly reweight the metric. A tiny test set produces confidence intervals wide enough to swallow the difference you're trying to measure.

Most teams QA their model and never QA their benchmark. eval-auditor makes benchmark quality a measurable, gradeable artifact — the same way a linter makes code quality one.

What it audits

Each dimension returns a score in [0, 1], a letter grade, the raw metrics, and human-readable warnings. Dimensions that don't apply to a given eval are skipped, not scored zero.

Dimension	Question it answers	How
discrimination	Does the eval separate a strong system from a weak one?	Mean gap and Cohen's d between the best/worst baseline (takes the min — you need both a real gap and non-overlapping distributions).
difficulty_spread	Is there a spread of difficulty, or is it all-easy / all-impossible?	Per-item difficulty = 1 − mean score across systems; flags saturated (~1.0) and impossible (~0.0) items.
redundancy	Are items near-duplicates?	Token-Jaccard clustering (union-find); reports the duplicate fraction.
contamination_risk	Is the answer visible in the prompt? Is there a canary?	Token containment of the gold answer in the prompt text; flags a missing held-out marker.
label_balance	Are categorical labels balanced?	Normalized Shannon entropy of each label's class distribution.
statistical_power	Is n large enough to trust the number?	95% CI half-width on the headline (best-system) mean; warns below n=30.
judge_calibration	If an LLM judge scored items, does it track humans?	Pearson r between human_label and judge_score; warns below r=0.70.

The dimension scores are averaged into an overall health score and letter grade, and the weakest dimensions are surfaced first with concrete remediation.

Quickstart

pip install -e .
python examples/demo.py         # audits both bundled evals
eval-auditor data/weak_eval.jsonl          # audit your own eval
eval-auditor data/healthy_eval.jsonl --json # raw metrics as JSON

eval-auditor is fully offline and deterministic — every dimension is a statistic over the data you hand it, so tests and CI are green with no API key and no network.

The whole point: healthy vs. weak

The repo ships two sample evals — one well-built, one deliberately broken — and the demo audits both. The contrast is the pitch. This output is produced entirely by the bundled data (data/healthy_eval.jsonl, data/weak_eval.jsonl); the numbers below are computed by the auditor, not hand-written.

########## healthy_eval.jsonl ##########
============================================================
  BENCHMARK HEALTH REPORT   grade A   score 0.925   (12 items)
============================================================
dimension           score  grade  status
------------------  -----  -----  ------
discrimination      1.000  A      ok
difficulty_spread   0.917  A      ok
redundancy          1.000  A      ok
contamination_risk  1.000  A      ok
label_balance       1.000  A      ok
statistical_power   0.574  F      WARN
judge_calibration   0.983  A      ok

--- Warnings ---
  [!] [statistical_power] n=12 is small; the 95% CI on the suite mean is +/-0.064 (aim for n>=30 to call small deltas).

--- Prioritized fixes (weakest dimensions first) ---
  1. statistical_power (grade F): Collect more items: at n<30 the confidence interval on the suite score is too wide to call small model differences real.

########## weak_eval.jsonl ##########
============================================================
  BENCHMARK HEALTH REPORT   grade F   score 0.224   (12 items)
============================================================
dimension           score  grade  status
------------------  -----  -----  ------
discrimination      0.107  F      WARN
difficulty_spread   0.000  F      WARN
redundancy          0.500  F      WARN
contamination_risk  0.000  F      WARN
label_balance       0.000  F      WARN
statistical_power   0.962  A      WARN
judge_calibration   0.000  F      WARN

--- Warnings ---
  [!] [discrimination] best vs worst system differ by only 0.027 — the eval barely tells them apart (possible saturation).
  [!] [difficulty_spread] 12/12 items are saturated (every system ~1.0) — they add cost but no signal.
  [!] [difficulty_spread] difficulty std=0.008 is very low — items are near-identical in difficulty.
  [!] [redundancy] 50% of items are near-duplicates (e.g. w1~w2, w1~w3, w1~w4) — they inflate whatever they measure.
  [!] [contamination_risk] 100% of items leak their answer into the prompt (e.g. w1, w2, w3, w4) — models can score by copying, not reasoning.
  [!] [contamination_risk] no canary/held-out marker found — you cannot later detect if this eval leaked into training data.
  [!] [label_balance] label 'topic' is imbalanced (balance=0.00, majority class = 100%) — a constant baseline would look strong.
  [!] [statistical_power] n=12 is small; the 95% CI on the suite mean is +/-0.006 (aim for n>=30 to call small deltas).
  [!] [judge_calibration] judge vs human Pearson r=-0.62 is below 0.70 — decisions made on this judge are not backed by human agreement.

--- Prioritized fixes (weakest dimensions first) ---
  1. difficulty_spread (grade F): Retire saturated items ...
  2. contamination_risk (grade F): Remove the gold answer from the prompt text and add a held-out canary string ...
  3. label_balance (grade F): Rebalance classes ...
  4. judge_calibration (grade F): Validate the LLM judge against human labels ...
  5. discrimination (grade F): Add harder items that only strong systems get right ...
  6. redundancy (grade F): Deduplicate near-identical items ...

Read the nuance. The broken eval scores an A on statistical_power — because when every system gets ~0.96, the variance is tiny and the confidence interval is razor-thin. That's misleading precision: a very confident estimate of a number that can't tell any two models apart. A single blended "is it good?" score would hide that; grading each dimension separately exposes it. Conversely, the healthy eval takes an F on power purely because n=12 — a real, honest limitation of a 12-item demo set, not a construction flaw.

Data format

One JSONL row per eval item. Only id, text, and gold are required; the richer the row, the more dimensions apply.

{
  "id": "q1",
  "text": "What is the capital city of Australia?",
  "gold": "Canberra",
  "labels": {"topic": "geography", "canary": "EVAL-CANARY-7Q2"},
  "system_scores": {"strong_model": 0.95, "weak_model": 0.30},
  "human_label": 0.90,
  "judge_score": 0.85
}

• system_scores — per-item scores from ≥2 baseline systems (e.g. a strong and a weak model). This powers discrimination, difficulty, and statistical power, so an audit is far more useful with it.
• labels — optional categorical metadata for balance-checking. The reserved key canary holds a held-out contamination marker and is excluded from balance.
• human_label / judge_score — optional; supply both to calibrate an LLM judge.

How it works / design decisions

• Skip, don't zero. A dimension the eval can't support (e.g. only one baseline system) is reported as n/a and excluded from the average, so a sparse eval isn't unfairly punished for data it never claimed to have.
• Discrimination needs both a gap and an effect size. It takes the minimum of the normalized mean-gap and Cohen's d. A tiny-but-consistent gap can produce a huge d despite being practically meaningless — requiring both closes that loophole.
• Actionable over academic. Every failing dimension emits a concrete fix, not just a red number. The report is meant to change what you do next.
• Pure-Python stats. Pearson, Cohen's d, and normalized entropy are implemented from scratch (no numpy), so the package installs and runs anywhere and every test asserts on real, reproducible values.

Using a real model (optional)

The auditor never needs a model. The optional anthropic extra is reserved for a future "have Claude draft the fixes prose" feature; the grades themselves are always computed offline. See .env.example.

Roadmap

• Item-level flags (which specific items to cut) exported as a CSV.
• Bootstrap confidence intervals on the discrimination effect size.
• Semantic (embedding) redundancy in addition to lexical Jaccard.
• Pass/fail gating in CI: fail a PR that pushes an eval below a health threshold.
• HTML health card with per-dimension drill-down.

---

MIT © 2026 Nick Raddon