judge-probe

Instrument your LLM-as-judge before you trust it.

Most evals now use a language model as the grader ("LLM-as-judge"). But judges have well-documented biases: they favor whichever answer they see first, they reward longer answers, and they're swayed by markdown, confident tone, and citations — none of which should change the verdict. judge-probe is a Monte Carlo simulator for exactly this: it repeatedly samples your judge's verdict on the same comparisons while varying only those irrelevant attributes, measuring how much the verdict moves — and how much a simple mitigation removes.

The biases themselves aren't news — they're in the literature. The point of this tool is operational: don't trust the judge, instrument it. Run this before you rely on any LLM-as-judge eval and you get a one-screen report card with a PASS/WARN/FAIL per bias.

LLM-as-judge bias report card        judge: fake · sample · 16800 judgements · n=120

  Bias                                  Raw        After swap-avg   95% CI   Verdict
  Position — first-shown win advantage  +28.7 pp   +0.0 pp          ±1.4     FAIL → PASS
  Verbosity — longer-answer lift        -0.7 pp    -2.1 pp          ±2.4     PASS
  Formatting — markdown lift            -0.9 pp    -2.6 pp          ±2.4     PASS

• Raw = single-order judgements.
• After swap-avg = each pair judged in both orders, the verdict kept only when the two orders agree. This is a real recomputed score, not a suggestion — it shows how much swap-averaging actually removes. Here a strong position bias collapses to ~0 (it's a pure order effect), and the no-bias rows correctly stay PASS.

The card above (examples/report_card.png) was generated with the built-in fake judge (a known position bias injected) so it's reproducible with no API key — make example / the fake-judge command below regenerates it. Point it at a real Claude judge to score your own setup.

Install

git clone https://github.com/<you>/judge-probe && cd judge-probe
uv sync                       # or: pip install -e .
cp .env.example .env          # then put your key in .env (loaded automatically)
# or: export ANTHROPIC_API_KEY=...   — only needed for the real (anthropic) judge

Python ≥ 3.10. The default judge is Claude (claude-opus-4-8) via the official anthropic SDK; the backend is pluggable (see below).

Quickstart

# 1. Preview the cost first — no probe spend
judgeprobe estimate --dataset sample --n 20

# 2. Run the probe (makes the judge calls), write the evidence
judgeprobe probe --dataset sample --n 20 --out results.jsonl

# 3. Render the report card with the before/after mitigation columns
judgeprobe report results.jsonl --mitigate --out report_card.md --png report_card.png

No API key? Run it against the built-in fake judge with an injected bias — useful for trying the tool or regenerating the example card:

judgeprobe probe --judge fake --fake-position 0.4 --fake-verbosity 0.3 \
                 --dataset sample --n 25 --out results.jsonl
judgeprobe report results.jsonl --mitigate --png report_card.png

Point it at your own judge config

--dataset accepts the bundled sample, or a path to your own .csv / .jsonl with columns prompt, answer_a, answer_b (optional id, gold):

judgeprobe probe --dataset my_pairs.csv --model claude-sonnet-4-6 --n 30 --out r.jsonl

Select which biases to probe with --biases position,verbosity,formatting,tone,citations (default: position,verbosity,formatting). Use --batch in estimate to price the run at the 50%-off Batches rate.

What it measures

Bias	What varies	Score axis
Position	answer order (A/B vs B/A)	first-shown win advantage
Verbosity	one answer padded longer, no new info	win-rate lift for the longer side
Formatting	one answer re-dressed in markdown	win-rate lift for the formatted side
Tone	confident framing added to one answer	win-rate lift
Citations	citation-styled lines appended	win-rate lift

Each comparison runs in both presentation orders, with the decorated side alternating A/B across trials so an answer's intrinsic quality cancels out — the lift between the baseline and decorated arms is attributable to the decoration, not to A-vs-B. Every score carries a 95% confidence interval (Wilson for the single-proportion position score, normal-approximation for the decoration lifts), so a bias measured on too few trials reads with a wide interval rather than false precision.

The mitigation (--mitigate)

The one mitigation built in is swap-averaging: judge each pair both ways and only count a win when the two orders agree (a flip becomes "no decision"). --mitigate re-runs the metrics on those swap-averaged decisions and reports the residual beside the raw score:

• Position residual is 0 by construction — once you average both orders, "first shown" is no longer defined, so the order effect is provably gone. The meaningful post-mitigation diagnostic is the consistency rate (how often the verdict was order-stable to begin with), reported under the table.
• Verbosity / formatting residuals are the lift that survives averaging — a real content preference the judge holds regardless of order.

This is the staff-vs-mid move: measure the bias, apply the fix, and report what's left — not "we found bias, so we swapped models."

Methodology

The full experimental design, the call-count math, the exact metrics and how the confidence intervals are computed (Wilson vs. difference-of-proportions), what n actually samples (Monte Carlo, not bootstrap), and the known statistical caveats live in docs/METHODOLOGY.md.

Output

• report_card.md / report_card.png — the scored card (the screenshot).
• results.jsonl — one row per judgement (trial, bias, arm, order, winner, reason): the raw evidence the report is computed from. Re-run report on it with different thresholds or --mitigate on/off without re-spending on the judge.

Architecture

src/judgeprobe/
  judge.py          Judge interface + AnthropicJudge (Claude) + FakeJudge (tests/demo)
  perturbations.py  the irrelevant-attribute transforms
  dataset.py        Trial model + sample / CSV / JSONL loaders
  probe.py          experiment design: arms × both orders × N reps → records
  metrics.py        per-bias scores + Wilson CIs
  mitigate.py       swap-averaging → residual
  report.py         markdown + PNG report card
  cli.py            estimate / probe / report

Adding a new backend (OpenAI, Gemini, a local model) is implementing one class — the Judge interface is compare(prompt, answer_a, answer_b) -> Verdict.

Caveats / honest limits

• Self-preference bias is not in v1. A judge rating its own model family higher needs ≥2 model families to measure cleanly; Judge.family is already carried for it, but the v1 backend is Claude-only. Don't read this tool's output as a self-preference check.
• The bundled dataset uses similar-quality answer pairs on technical questions, so a fair judge should land near 50/50 and any systematic preference is bias. If you bring pairs with a real quality gap, "first-shown win advantage" mixes bias with genuine discrimination — read it alongside the flip rate.
• gold labels aren't used yet; judge-vs-human agreement (Cohen's κ) is a planned addition for deciding when no mitigation will save a judge.
• Swap-averaging is one mitigation. Reference/rubric anchoring and panel-of-judges aggregation are the natural v2.

Tests

uv run pytest

The suite runs entirely on FakeJudge (no API, no key) and pins the two claims the tool rests on: the metrics recover an injected bias, and swap-averaging collapses a pure position bias toward zero while a genuine verbosity bias survives.

License

MIT.