A cost-quality Pareto study of multi-agent LLM debate vs. adaptive routing. Router v3 achieves 91.8% aggregate accuracy at $0.0016/query — matching Claude Haiku 4.5 on cost, beating every fixed baseline on 3 of 5 benchmarks, and staying within noise of frontier models like Claude Opus 4.7 (94.2% on GPQA-Diamond) and GPT-5.5 (93.6%) at roughly 1/17× the cost per query.
Technical findings · Decisions log (68+ ADRs) · Live demo — coming soon
The initial hypothesis was straightforward: four role-specialized debater LLMs (Researcher, Reasoner, Critic, Skeptic) arguing over three rounds, followed by a Judge synthesizing the transcript, should systematically beat a single strong model across a mix of reasoning, factual, code, and open-ended tasks. The pitch was "four heads are better than one, and you can measure by how much."
The pitch didn't survive the data. A 3,375-query evaluation across five benchmarks and six configurations showed the full council (c7) reaching 68.6% aggregate accuracy at $0.019/query, while a single Claude Haiku 4.5 (c10) reached 77.5% at $0.0015/query — 9pp better accuracy at one-twelfth the cost. Multi-agent debate was Pareto-dominated on the cost-quality plane.
But the per-benchmark decomposition told a different story: no single config won everything. Council wins on adversarial-misconception TruthfulQA items. Kimi K2.6 dominates GPQA. Claude Haiku owns HumanEval. The routing problem is genuinely multi-class. The final system is a learned router — XGBoost over 14 pre-inference features — that picks per query. It matches Haiku on average cost while beating every fixed config on 3 of 5 benchmarks.
- Router v3 hits 91.8% [85.9, 97.6] at $0.0016/query — inside c10's 95% CI at comparable cost, above every fixed baseline on TruthfulQA, HumanEval, and MT-Bench.
- Multi-agent debate is Pareto-dominated at aggregate. c7 (68.6%) sits behind both c5 (78.9%) and c10 (77.5%) on accuracy while costing 3× to 13× more.
- On GPQA-Diamond the router reaches 92.7% (via c5 routing). Vendor-reported Claude Opus 4.7 (94.2%, primary source: Anthropic system card) and GPT-5.5 (93.6%, third-party aggregation) sit inside the router's CI — at ~17× and ~20× the cost respectively.
- Debate catastrophically fails on direct-generation tasks. c7 scores 53.5% on HumanEval vs. c10's 89.0% — a 35.5pp gap. Same story on AlpacaEval: 60.0% vs. 91.0%.
- The Judge synthesis is worth 15pp over majority voting. c7 = 68.6% vs. c6 (same council, no Judge) = 53.2%. The Judge does real synthesis, not vote-tallying.
- Debate earns its cost on a narrow, mechanistically defined subset. On six specific TruthfulQA questions, c7 answers correctly and c5 does not — all share the property that the intuitive answer is confidently wrong. The Skeptic role is what surfaces the correct answer against those defaults.
flowchart LR
Q[User query] --> R{Router<br/>XGBoost v3}
R -->|adaptive misconception<br/>signal| C[Council]
R -->|coding / factual| S[Single strong model]
C --> A1[Researcher<br/>Llama 4 Maverick]
C --> A2[Reasoner<br/>DeepSeek V4]
C --> A3[Critic<br/>GLM-4.7]
C --> A4[Skeptic<br/>Qwen 3.5]
A1 & A2 & A3 & A4 --> D[3 debate rounds<br/>with convergence tracking]
D --> J[Judge<br/>Kimi K2.6]
J --> Ans[Final answer]
S --> Ans
style R fill:#C2410C,color:#fff
style J fill:#1A1A1A,color:#fff
style Ans fill:#166534,color:#fff
- 5 benchmarks — TruthfulQA (adversarial), GPQA-Diamond (graduate science), MT-Bench (open-ended), HumanEval (code), AlpacaEval (instruction following).
- 3,375 graded queries across six fixed configurations plus the learned router; total spend ≈$31.50.
- 6 configs + router — c2 (DeepSeek V4 solo), c5 (Kimi K2.6 solo, same model as council Judge), c6 (council, majority vote, no Judge — ablation), c7 (council + Judge), c9 (GPT-4o mini anchor), c10 (Claude Haiku 4.5 anchor).
- Evaluation — exact-match letter extraction for GPQA-Diamond, code-execution sandbox for HumanEval, Claude Haiku 4.5 as LLM-as-judge rubric grader for TruthfulQA / MT-Bench / AlpacaEval. 95% CIs are bootstrap with 1,000 resamples.
Four debaters stream in parallel with live convergence tracking, cost meter, and router decision.
Router picks per-query. Here the coding classifier tags the query and picks c5_judge_alone (single strong model); SHAP explains the choice.
Session memory: the follow-up "What are the tradeoffs?" retrieves the earlier React SSR debate and grounds the answer in it, rather than answering generic-tradeoffs.
PDF retrieval: attach a paper, ask a question, get an answer with page-anchored citations — [filename.pdf, p. 1].
| Config | n (graded) | Accuracy | 95% CI | $/query |
|---|---|---|---|---|
| router v3 (this work) | 85 | 91.8% | [85.9, 97.6] | $0.00160 |
| c5_judge_alone | 660 | 78.9% | [75.8, 82.0] | $0.00627 |
| c10_claude_haiku | 480 | 77.5% | [73.8, 81.2] | $0.00145 |
| c2_reasoner_alone | 805 | 71.3% | [68.2, 74.4] | $0.000436 |
| c7_council_judge | 478 | 68.6% | [64.4, 72.8] | $0.01937 |
| c9_gpt4o_mini | 480 | 65.6% | [61.5, 69.6] | $0.000118 |
| c6_council_majority | 472 | 53.2% | [49.1, 57.4] | $0.01104 |
The Pareto frontier is c9 → c2 → c10 → c5. Both council configurations (c6, c7) are strictly Pareto-dominated. The c5–c10 gap of 1.4pp sits inside both configs' CIs, meaning they are statistically indistinguishable at aggregate — c10 costs 4.3× less. The router sits above every fixed baseline at c10-comparable cost, driven by benchmark-conditional routing (see below).
Aggregate reporting hides the whole point. Look per-benchmark:
| Benchmark | c2 | c5 | c6 | c7 | c9 | c10 |
|---|---|---|---|---|---|---|
| truthfulqa | 64.8 | 77.5 | 69.0 | 72.0 | 54.0 | 68.0 |
| gpqa | 76.0 | 92.7 * | 65.2 | 75.8 | 38.0 | 54.0 |
| humaneval | 68.1 | 56.1 | 47.0 | 53.5 | 74.0 | 89.0 |
| mtbench | 65.0 | 91.3 | 40.0 | 85.0 | 78.8 | 90.0 |
| alpacaeval | 78.4 | 91.0 | 43.0 | 60.0 | 86.0 | 89.0 |
* c5 GPQA graded only 123/198 attempts — 37.9% ungraded because Kimi K2.6 produced prose without an extractable answer letter. See Finding 6.
The rows disagree. Kimi K2.6 (c5) crushes GPQA-Diamond at 92.7% while collapsing on HumanEval (56.1%, vs. c10 at 89.0%). Claude Haiku (c10) leads HumanEval by a wide margin but sits at 68.0% on TruthfulQA — a benchmark c5 leads. Council + Judge (c7) is inside 6pp of the best config on MT-Bench but 35pp behind c10 on HumanEval. No single config wins everything, and the gap on the losing benchmarks is much larger than the gap on the winning ones. That's the routing signal.
The reliability-blended router v3 (α=0.6, λ=10) recovers per-benchmark winners:
| Benchmark | router v1 (λ=10) | router v3 blended | Δ (pp) |
|---|---|---|---|
| truthfulqa | 0.50 | 0.90 | +40 |
| gpqa | 0.67 | 0.67 | 0 |
| humaneval | 0.94 | 1.00 | +6 |
| mtbench | 0.94 | 1.00 | +6 |
| alpacaeval | 0.95 | 0.95 | 0 |
Label function: α · 1{correct} + (1 − α) · benchmark_accuracy[c][b] − λ · cost. See ADR-060.
The full comparison table with citations lives on the /benchmarks page. Verbatim from that page:
| Model | Era | $/M in/out | TruthQA | GPQA-D | MT-B | HumEval | AlpEval | $/query |
|---|---|---|---|---|---|---|---|---|
| Claude Opus 4.7 | Anthropic · Apr 2026 | $5 / $25 | n/p | 94.2¹ | n/p | n/p | n/p | $0.0275² |
| Claude Opus 4.6 | Anthropic · Feb 2026 | $5 / $25 | 77.8³ ⚠ | n/p | n/p | 92.8³ ⚠ | n/p | $0.0275² |
| Claude Sonnet 4.6 | Anthropic · 2026 | $3 / $15 | n/p | 74.1⁴ ⚠ | n/p | n/p | n/p | $0.0165² |
| GPT-4o | OpenAI · 2024 | $2.50 / $10 | n/p | 53.6⁵ ‡⚠ | n/p | 90.2⁵ ⚠ | n/p | $0.0113² |
| GPT-4o mini | OpenAI · 2024 | $0.15 / $0.60 | n/p | 40.0⁶ | n/p | 87.2⁶ | n/p | $0.0007² |
| GPT-5.2 | OpenAI · Dec 2025 | $1.75 / $14 | n/p | 92.4⁷ ⚠ | n/p | n/p | n/p | $0.0149² |
| GPT-5.5 | OpenAI · Apr 2026 | $5 / $30 | n/p | 93.6⁸ ⚠ | n/p | n/p | n/p | $0.0325² |
| router v3 (this work) | adaptive · Phase 3C | routed | 77.5⁹ | 92.7⁹ | 91.3⁹ | 89.0⁹ | 91.0⁹ | $0.0016⁹ |
n/p = not published. ⚠ = third-party aggregation, not independently verified against a primary source (vendor system card or release-page body). ‡ = GPQA (5-shot), not GPQA-Diamond specifically. $/query assumes 500 input + 1000 output tokens.
Honest assessment. On GPQA-Diamond, the router achieves 92.7% (via c5, n=85). Vendor-reported scores for frontier models — Claude Opus 4.7 at 94.2% (primary: Anthropic system card, April 2026) and GPT-5.5 at 93.6% (third-party aggregation of OpenAI release material; OpenAI's own release page does not quote a GPQA-Diamond number) — sit within the noise of the router's result on that benchmark, but the methodology (judge model, prompt template, sampling temperature) is not disclosed to the level that would make this a strict head-to-head. The comparison is directional. The cost delta — roughly 1/17× vs. Opus 4.7 and 1/20× vs. GPT-5.5 — is directly observable from published API pricing. On HumanEval the router (89.0% via c10 Claude Haiku 4.5) is within CI of GPT-4o's reported 90.2% at ~7× lower cost. On the older benchmarks (TruthfulQA, MT-Bench, AlpacaEval) the frontier tier stopped publishing scores, so no direct comparison is possible above the same-cost tier.
Vendor sources 1–9 with primary/aggregator classifications are enumerated in frontend/app/benchmarks/page.tsx and rendered on the live page.
Full analysis in docs/observations/2026-06-30-phase3-complete-findings.md.
- Multi-agent debate is Pareto-dominated by single models at aggregate. c7 at 68.6% costs 13× more than c10 at 77.5% — no accuracy-cost tradeoff makes c7 rational for average-case use. Reframes Phase 4: the routing question is not "debate or not" but "which of six configs for this query."
- Debate provides differentiated value on adversarial misconception queries. On six specific TruthfulQA questions, c7 answers correctly and c5 does not. All share the structural property that the intuitive folk answer is confidently wrong. The Skeptic role is what surfaces the correct answer against those defaults. Head-to-head on the 100 shared items: c5 wins 12, c7 wins 6.
- The Judge synthesis is worth 15pp over majority voting. c7 vs. c6 (same council, majority-vote instead of Judge synthesis) = 68.6% vs. 53.2%. The Judge reads the full transcript and synthesizes across positions; majority-vote averages, and the average is often worse than any individual answer. On MT-Bench the gap is 45pp.
- Claude Haiku 4.5 is the surprise cost-quality winner. c10 achieves 77.5% at $0.00145/query — statistically indistinguishable from c5 (78.9%) at 4.3× lower cost. Modern proprietary "cheap" models appear to subsume much of what the council achieves through debate.
- Multi-agent debate catastrophically fails on direct-generation tasks. HumanEval gap: c10 89% vs. c7 53.5% = 35.5pp. AlpacaEval gap: c5 91% vs. c7 60% = 31pp. Debate produces over-elaboration under adversarial critique; the code-execution grader does not reward hedged elaboration. This is a clean negative finding.
- c5 (Kimi K2.6) has a reliability tradeoff on GPQA. 92.7% headline accuracy on 123/198 attempts — 37.9% of transcripts were ungraded because Kimi produced prose without an extractable answer letter. If ungraded rows are treated as incorrect (pessimistic bound), effective accuracy drops to ~62%. Benchmark reports should always cite the gradeability rate alongside the accuracy rate.
- Python 3.12+
- Node 20+ and pnpm
- Docker (Postgres 16 + pgvector, Redis 7)
- OpenRouter API key (all inference routes through OpenRouter)
- Optional: HuggingFace token — required to load gated benchmark datasets (GPQA-Diamond, AlpacaEval)
docker compose up -d postgres redis
cd backend
uv sync
cp .env.example .env
# Edit .env — at minimum set OPENROUTER_API_KEY and SECRET_KEY.
# Generate a secret key with: openssl rand -hex 32
uv run alembic upgrade head
uv run uvicorn app.main:app --port 8000Health check: curl http://localhost:8000/health → {"status":"ok","checks":{"postgres":"ok","redis":"ok"}}.
cd frontend
pnpm install
pnpm devOpen http://localhost:3000, submit a query. Modes:
- Route — router picks the config per query (default; Phase 4).
- Council — always run c7 (4 debaters + Judge) regardless of router.
- Fast — always Claude Haiku 4.5 direct, no debate, no judge.
Try the seeded example queries on the landing page — each shows the config the router picks.
cd backend
# Pilot: 20 samples per config-benchmark cell, 9 configs, 5 benchmarks
uv run python -m scripts.run_evaluation \
--configs all --benchmarks all \
--sample-size 20 --concurrency 3 --budget-per-config 5.0 \
--tag pilot-verification
# Full: 200 samples on specific cells, resumable via --tag
uv run python -m scripts.run_evaluation \
--configs c7_council_judge,c10_claude_haiku_anchor \
--benchmarks truthfulqa,gpqa \
--sample-size 200 --concurrency 3 --budget-per-config 15.0 \
--tag my-eval-2026-07Results land in the evaluations table. Grading and Pareto plotting scripts live in backend/scripts/.
llm-council/
├── backend/
│ ├── app/
│ │ ├── agents/ # 4 debater roles + Judge + memory extractor
│ │ ├── orchestration/ # LangGraph state machine, convergence, prompt augmentation
│ │ ├── router/ # XGBoost router: features, embeddings, eval, service
│ │ ├── evaluation/ # Benchmark harness + graders (exact-match, code-exec, LLM judge)
│ │ ├── memory/ # Session memory (Phase 7A) — pgvector similarity search
│ │ ├── documents/ # PDF ingestion with page-tracked chunks (Phase 7B)
│ │ ├── inference/ # LiteLLM wrapper against OpenRouter
│ │ ├── db/ # SQLAlchemy models + Alembic migrations
│ │ └── api/ # FastAPI routes + WebSocket streaming
│ ├── scripts/ # Evaluation runner, grading, Pareto plotting
│ └── tests/ # 41 test files
├── frontend/
│ └── app/ # Next.js 14 App Router — landing, /debate, /benchmarks, /about, /how-it-works
├── docs/
│ ├── DECISIONS.md # 68+ ADRs — one file, chronological, every design decision
│ └── observations/ # Findings document (this file is public)
├── assets/ # README plots and screenshots
└── docker-compose.yml # Postgres + Redis for local dev
Python 3.12 · FastAPI · LangGraph · LiteLLM (OpenRouter) · Postgres 16 + pgvector · Redis 7 · XGBoost 2 · sentence-transformers · Next.js 14 · TypeScript · Tailwind · WebSockets · Playwright (for the screenshot capture that produced the images above).
- c5 GPQA reliability. 37.9% of Kimi K2.6 transcripts on GPQA-Diamond were ungraded. The router's 92.7% GPQA cell inherits this ceiling — the effective accuracy is somewhere between 62% (pessimistic) and 92.7% (optimistic). GPQA validation for the router uses n=12, so per-item swings dominate.
- AlpacaEval data-quality caveat. A large-scale audit (
docs/observations/2026-06-21-phase3b-audit.md) surfaced that 75% of AlpacaEval rows had missing context fields; models correctly flagged them as malformed and the judge scored the refusals as failures. Numbers are reported with this known caveat. - Costs are 2026 OpenRouter list prices. Provider prices shift; the numerical ratios above will drift.
- Per-benchmark λ scheduling was investigated and dropped. The v2 schedule router (per-benchmark cost weighting) was bit-equivalent to v1 λ=10 in aggregate; v3's blended label function (α · correctness + (1−α) · benchmark_accuracy − λ · cost) dominates it. See ADR-059.
- Open questions that Phase 3 could not answer — role-ablation (Skeptic-only removed), a code-specialist debater in the council, and whether the Pareto picture holds at 7B open-source model sizes rather than the frontier-adjacent proprietary tier used here.
Benchmarks. TruthfulQA · GPQA-Diamond · MT-Bench · HumanEval · AlpacaEval.
Models. Llama 4 Maverick (Meta) · DeepSeek V4 Flash (DeepSeek AI) · GLM-4.7 (Zhipu AI) · Qwen 3.5 (Alibaba) · Kimi K2.6 (Moonshot AI) · Claude Haiku 4.5 (Anthropic) · GPT-4o mini (OpenAI). All routed through OpenRouter.
Proprietary comparison sources. Anthropic Claude Opus 4.7 system card (April 2026); OpenAI pricing pages; third-party aggregators marked ⚠ in the comparison table. Full source list in frontend/app/benchmarks/page.tsx.
LLM judge. Claude Haiku 4.5 for TruthfulQA, MT-Bench, and AlpacaEval grading — chosen for family diversity from the council debaters and Judge (Kimi K2.6). Judge prompts underwent two rounds of blind calibration audits before the Phase 3C run.
See LICENSE (MIT).
Portfolio research project · Not affiliated with any of the model vendors mentioned.