Most systems generate answers. This one judges them. It fans an LLM output out to
several competing critics that independently score accuracy / consistency /
completeness with quotable callouts, then synthesizes their critiques into one
confidence-scored verdict — ACCEPT / REVISE / REJECT, with an ESCALATE flag
when the jury genuinely cannot decide.
A jury only beats a single judge to the extent its jurors fail independently. So this system does not assume decorrelation — it measures it, and discounts its own confidence when the critics turn out to agree for the wrong reasons.
flowchart LR
IN["task + context<br/>+ LLM output"] --> F(( ))
F --> C1["factual<br/><i>haiku</i>"]
F --> C2["logical<br/><i>sonnet</i>"]
F --> C3["adversary<br/><i>opus</i>"]
C1 --> S["synthesize<br/><b>deterministic code</b>"]
C2 --> S
C3 --> S
S --> V["ACCEPT / REVISE / REJECT<br/>+ confidence + callouts"]
S --> E["ESCALATE<br/>→ human"]
V --> A[("audit trail<br/>JSON + SQLite")]
Every critic runs all three detectors — the lens only shifts sensitivity, not capability. Synthesis is deterministic code, never an LLM: otherwise there is no auditable aggregation and no offline evaluation.
flowchart TD
L["a critic scores this dimension low"] --> Q{"can it quote<br/>the span?"}
Q -- "no" --> O["opinion → outvoted by the median"]
Q -- "yes, and the source refutes it" --> C[["contradiction → the low score wins outright"]]
Q -- "yes, but merely ungrounded" --> U["unsupported → capped at REVISE, never REJECT"]
C --> R{"2+ critics<br/>same finding?"}
R -- yes --> H[["hard REJECT"]]
R -- no --> ESC["REVISE + escalate to a human"]
A low score with a quotable span is evidence; without one it is an opinion. Evidence
overrides the vote, opinion gets outvoted. Median aggregation — the obvious choice —
would discard exactly the minority finding this product exists to surface. And absence of
proof is not proof of error, so an ungrounded claim can only ever reach REVISE.
One critic out of three overrides two. logical is the only lens that hunts polarity
flips; it finds one, and it can quote it — so the verdict moves despite two clean 5s:
$ python arbitrate.py judge --case con_02 --provider mock
critic model acc con cmp ms status
factual claude-haiku-4-5 5 5 5 0.3 ok
logical claude-sonnet-5 5 3 5 0.1 ok
adversary claude-opus-5 5 5 5 0.1 ok
AGGREGATE 5 3 5
callouts:
[1/3] unconfirmed consistency contradicted 'required' here but 'optional' there
> Note that for high-value orders a signature is optional if nobody is home.
VERDICT: REVISE confidence 0.5 (flip-margin 0, agreement 0.833)
ESCALATE -> human review: critics disagree and no finding is corroborated
When all three corroborate, it hard-blocks — and each callout names the exact rival figure from the source:
$ python arbitrate.py judge --case multi_02 --provider mock
callouts:
[3/3] CORROBORATED accuracy contradicted output says 45, context says 30
> - Refunds are accepted within 45 days of delivery
[3/3] CORROBORATED accuracy contradicted output says 60, context says 14
> - Elsewhere we note the free trial runs for 60 days instead
[3/3] CORROBORATED consistency contradicted says 14 here and 60 there
> - Elsewhere we note the free trial runs for 60 days instead
VERDICT: REJECT confidence 0.95 (flip-margin 2, agreement 1.0)
hard block: a corroborated, quoted contradiction # exit code 2 — gates a pipeline
$ python arbitrate.py eval --provider mock
=== eval on 24 cases ===
detection precision 1.00 (13/13) [0.77, 1.00]
detection recall 0.81 (13/16) [0.57, 0.93]
recall (string-detectable defects only) 1.00 (12/12) [0.76, 1.00]
^ the gap between these two recalls is the measured value of a real LLM jury
exact-verdict acc. 0.83 (20/24) [0.64, 0.93]
confusion TP 13 FP 0 FN 3 TN 8
escalate rate 0.21 corroboration rate 0.75
risk-coverage coverage 0.79 (19/24) accuracy@coverage 0.84 (16/19)
calibration ECE 0.113 (bootstrap 95% CI [0.006, 0.256]) mean conf 0.721
discrimination conf|correct 0.735 vs conf|wrong 0.65
ablation (does the jury beat one critic?)
jury precision recall accuracy
factual 1.00 0.69 0.54
logical 1.00 0.62 0.50
adversary 1.00 0.75 0.54
factual+adversary 1.00 0.81 0.83
factual+logical+adversary 1.00 0.81 0.83
If an offline confusion matrix comes out perfect, the eval is broken. So the dataset
plants defects the string detectors provably cannot see — "about a month and a half"
against a source that says 30 days, "ships in 3 days … arrives early next month", a
task that asks why and an answer that only says what. They are labelled
detectable: "semantic", and they are the 3 misses above.
It also plants hard negatives that look defective: 1,200 vs "twelve hundred",
"not required" vs "optional", "list up to 5" answered with 3. Those are what keep
precision honest — a naive detector flags all three.
| value | |
|---|---|
| recall on defects a string detector can reach | 1.00 (12/12) |
| recall overall | 0.81 (13/16) |
| the gap | exactly what a real LLM jury has to earn |
Labels were hand-written before the detectors, and the synthesis layer is tested on injected score vectors that never touch a detector — so the confusion matrix is not a tautology.
Offline the critics are deterministic string detectors that differ by sensitivity
threshold. The jury beats the best single critic (0.83 vs 0.54 exact-verdict accuracy),
and that gap is partly designed in. The offline numbers prove the plumbing — routing,
quote validation, dedupe, corroboration, aggregation — and act as a regression fixture.
The claim that a diverse jury catches more real defects is only established with
--provider anthropic.
confidence = 0.34 + 0.61 × (0.6 × flip-margin/2 + 0.4 × agreement) → range [0.34, 0.95],
never 1.0.
- flip-margin = how many single-critic ±1 changes it takes to flip the verdict. Distance-to-a-boundary is meaningless on an integer grid; "this verdict survives any one critic being off by a point" is something a reviewer can check.
- It is computed after the verdict and never feeds back into it — so tuning the weights cannot leak into the decision.
- Caps: escalating → ≤ 0.50; unanimous-but-unevidenced → ≤ 0.70 (that pattern is what correlated error looks like).
evalexits non-zero if mean confidence exceeds accuracy by more than 0.15, or if the escalate rate leaves[0.05, 0.40](a flag that fires on everything or nothing is decoration).
A bug this caught in its own design. The first version was more confident when it was wrong (0.81 vs 0.76) — a jury outvoting a lone dissenter looks unanimous. The fix is a rule, not a fudge: accepting despite a finding nobody corroborated now escalates. Discrimination flipped to 0.735 vs 0.65, and
accuracy@coverage(0.84) rose above overall accuracy (0.83), which is how you show the flag earns its keep.
Agreement between critics only means something if each critic is individually stable —
otherwise "consensus" is measuring dice, ESCALATE fires on noise, and confidence is
calibrated against sampling variance instead of correctness.
The obvious way to get that stability is temperature=0. It no longer exists. The
sampling parameters were removed on the current frontier models — sending temperature,
top_p, or top_k to Sonnet 5 or Opus 5 is a 400. The modern control is
output_config.effort, and it is held constant across the jury on purpose, so the
only deliberate variation is design (model tier × lens), never sampling noise.
effort is not uniformly available either — it errors on Haiku 4.5 while being the
depth control on Sonnet 5 / Opus 5. That is a real per-model capability difference, so
supports_effort is a field on the critic spec and _complete() only sends the parameter
where the model accepts it. Note also that none of this buys bit-determinism (batching,
fp, routing), which is why the real-provider path still repeats and reports variance.
Two bugs this project shipped with, both found by actually running the jury:
- A latent 400. The first roster pinned
claude-opus-4-8and passedtemperature. Offline everything was green — the mock provider ignores both. The first live call would have failed on a parameter that had been removed from the API. - Corroboration silently under-counted. Callouts were merged on a model-supplied
key. The offline detectors emit deterministic keys, so merging looked perfect; real critics invent different ids for the same finding, so one 3/3 corroboration fragmented into three 1/3 rows and the hard block nearly didn't fire. Fixed by joining on the verified quote — the one identifier every critic must agree on to be talking about the same span, and one that is checked against the output rather than invented.
Both are the same lesson, and it is the reason the offline eval is a fixture and not a claim: a mock that is deterministic where production is not will hide exactly the failures that only production has.
pip install -r requirements.txt
# offline, no key
python arbitrate.py demo # a clean case, then a broken one
python arbitrate.py judge --case con_02 # evidence beats votes
python arbitrate.py judge --case multi_02 # corroborated -> hard REJECT (exit 2)
python arbitrate.py eval # metrics, ablation, calibration gates
python arbitrate.py replay # regression over cases.jsonl
# judge an arbitrary output
python arbitrate.py judge --task "..." --context "..." --output "..."
# real jury of Claude tiers (key in a gitignored .env: ANTHROPIC_API_KEY=sk-ant-...)
python arbitrate.py judge --case multi_02 --provider anthropic
# serving + the verdict explorer
uvicorn app:app --reload
# GET /healthz | GET /critics | POST /arbitrate | GET /runs | GET /verdict/{run_id} | POST /replay
pytest -q # 28 offline testsLangGraph earns its keep with cycles, checkpointing, and human-in-the-loop interrupts.
Arbitration is a static two-layer DAG: fan out to N critics, reduce to one verdict. The
graph is a threadpool map plus a function call. Taking langgraph → langchain-core →
langsmith to wrap two nodes is ceremony.
What I give up by hand-rolling (worth stating plainly): checkpoint/resume, so a jury
that dies at critic 7 of 9 restarts from zero; free LangSmith trace integration; streaming
intermediate state to the UI. The seam is kept — run_critics and synthesize are pure
functions over a state dict, which is the LangGraph node signature:
# Equivalent LangGraph topology — documented, not taken as a dependency.
# class S(TypedDict):
# draft: str
# critiques: Annotated[list[dict], operator.add] # reducer merges parallel writes
# verdict: dict
#
# def fan_out(s: S): # dynamic fan-out: one Send per critic spec
# return [Send("critic", {"draft": s["draft"], "spec": c}) for c in CRITICS]
#
# g = StateGraph(S)
# g.add_node("critic", run_one_critic) # == arbitrate.run_one_critic
# g.add_node("synthesize", synthesize) # == arbitrate.synthesize
# g.add_conditional_edges(START, fan_out, ["critic"])
# g.add_edge("critic", "synthesize") # implicit join: waits for every Send
# g.add_edge("synthesize", END)
# app = g.compile() # .compile(checkpointer=...) buys resumeThe moment a contested callout needs to go back to the jury, or a long run needs to survive a crash, that migration pays for itself — and it is a mechanical wrap, not a rewrite.
The brief called for OpenAI + Anthropic + Ollama. Only an Anthropic key was available, so this runs a jury of Claude tiers under different lenses. That is deliberately not a cross-vendor jury: same-family models share pretraining and alignment, so their errors are correlated, and correlated jurors overstate confidence. Rather than assume decorrelation, the system measures agreement and discounts confidence when a unanimous verdict has no evidence behind it.
provider is a field on the critic spec, and _complete() is the only provider-aware
function in the codebase — the OpenAI/Ollama branches are written, reuse the same
schema dict, and are labelled UNTESTED rather than pretended to work.
Schemas — Pydantic is the single source of truth; _strict() converts its JSON Schema
into strict JSON Schema. This is not cosmetic: Pydantic omits additionalProperties, and
ConfigDict(extra="forbid") does not propagate into nested models or $defs, so the
root looks correct and the first real API call fails. Scores are Literal[1..5] (an enum)
because numeric constraints like ge/le are silently unsupported by strict mode.
Quote validation — a free hallucination check on the critic itself: if a callout's quote is not verbatim in the output, it is demoted to an opinion. A critic that cannot quote the text it is complaining about does not get to override the vote.
Fan-out — ThreadPoolExecutor with executor.map, not as_completed: completion
order would make the audit artifact non-reproducible under real network jitter while
looking fine offline. A dead juror is recorded, never fatal — 2-of-3 is still a verdict.
Audit trail — one inspectable JSON artifact per run plus a SQLite row per critique;
/replay turns stored runs into a regression harness.
Serving — FastAPI, with a Pydantic request model on /arbitrate so a missing field is
a 422, not a 500. The verdict explorer is server-rendered HTML with every
interpolation escaped — the output under review arrives from an untrusted request body,
so naive interpolation would be stored XSS.
arbitrate.py # schemas, detectors, jury fan-out, synthesis, confidence, eval, CLI
app.py # FastAPI + the server-rendered verdict explorer
cases.jsonl # 24 hand-labelled cases (8 clean incl. 4 hard negatives, 16 defective)
test_arbitrate.py # 28 offline tests (synthesis tested on injected vectors; API via TestClient)
requirements.txt
.github/workflows/ci.yml
- Deterministic synthesis. An LLM meta-critic writing the verdict would destroy both auditability and the offline eval. An LLM narrative would stay advisory, never flipping the verdict.
- No multi-round debate. Letting critics read each other doubles cost and destroys the independence the whole design rests on — it amplifies correlated error. Considered and rejected on purpose.
- No reliability-weighted critics. Three weights cannot be fitted on 24 cases. The per-critic numbers are computed and printed, then deliberately not used.
- No
instructor— native constrained decoding cannot emit non-conforming JSON, so it would be a wrapper around a wrapper. (Worth saying: constrained decoding guarantees schema conformance, not semantic correctness — which is exactly why a jury exists.) - No React/Streamlit — a second server or a build step to render one page.
- Cohen's κ between critics on the real jury (if κ > 0.9 the jury is one critic in a trenchcoat — that number should be published either way).
- The cost-matched baseline as a first-class command: 3 samples of one model vs 3 different models on the same call budget — the experiment that tests the diversity thesis rather than the plumbing.
- Fuzzier span matching for corroboration: critics that quote overlapping-but-unequal spans of the same sentence still cluster separately (visible as leftover 1/3 rows in the live run above).
- Reliability-weighted aggregation once N ≳ 200 labelled cases exist.
- Real OpenAI/Ollama critics, so "multi-provider" becomes a measured claim.
MIT.