A system where AI models audit each other's work. Three specialized critics — each running on a different model — independently evaluate any LLM output, and an adjudicator resolves their disagreements into a single confidence-scored verdict.
Most AI projects generate answers. This one catches bad answers.
Three critics fan out in parallel, fan in for disagreement detection, then either short-circuit to a clean pass or go to full adjudication.
┌──────────────────────┐
┌────▶│ Accuracy Critic │────┐ Groq · Llama 3.3 70B
│ └──────────────────────┘ │
│ ┌──────────────────────┐ │
START ──────┼────▶│ Logic Critic │────┼──▶ collect_and_detect
│ └──────────────────────┘ │ (disagreement detector)
│ ┌──────────────────────┐ │ │
└────▶│ Completeness Critic │────┘ │
└──────────────────────┘ │
▼
┌───────────────────────────┐
│ all critics clean? │
└───────────────────────────┘
│ │
yes │ │ no
▼ ▼
┌──────────────┐ ┌──────────────┐
│ short_circuit│ │ Adjudicator │
│ (10/10 pass) │ │ │
└──────────────┘ └──────────────┘
│ │
└────────┬────────┘
▼
END
If all three critics used the same model, they would share the same blind spots. Routing each critic through a different model is what makes the disagreements meaningful signal rather than noise.
| Critic | Primary | Fallback (if primary is down) |
|---|---|---|
| Factual Accuracy | Groq · llama-3.3-70b-versatile |
— |
| Logical Consistency | Google Gemini · gemini-2.0-flash |
Groq · openai/gpt-oss-120b |
| Completeness | NVIDIA NIM · meta/llama-3.1-8b-instruct |
Groq · llama-3.1-8b-instant |
| Adjudicator | Groq · llama-3.3-70b-versatile |
— |
All models are open-source/open-weight and available on free tiers.
Provider fallback: if a primary provider's key is missing or its API is
unreachable, that critic reroutes to a different model on a working provider
rather than collapsing the system to a single model. The fallback is recorded
on the report and surfaced in the UI — never hidden. If fallback is disabled
(ENABLE_PROVIDER_FALLBACK=false), the dimension degrades instead: the verdict
is still produced from the remaining critics, with a note that the dimension
has lower confidence.
Requires Python 3.11+ and uv.
# 1. Install dependencies
uv sync
# 2. Configure keys
cp .env.example .env
# Edit .env and add at minimum a GROQ_API_KEY (free: https://console.groq.com/keys)
# 3. Start the API (terminal 1)
uv run uvicorn app.api:app --reload
# 4. Start the UI (terminal 2)
uv run streamlit run streamlit_app.pyThe UI opens at http://localhost:8501, the API docs at
http://127.0.0.1:8000/docs.
Only GROQ_API_KEY is required — the other two critics fall back to Groq-hosted
models if GEMINI_API_KEY / NVIDIA_API_KEY are absent.
A basic two-container setup — one for the FastAPI backend, one for the Streamlit UI — wired together on a shared network.
# 1. Configure keys (same .env as above)
cp .env.example .env
# Edit .env and add at minimum a GROQ_API_KEY
# 2. Build and start both services
docker compose up --build- API:
http://localhost:8000/docs - UI:
http://localhost:8501
The UI talks to the API over the internal Docker network (http://api:8000),
not localhost. The SQLite database is written to ./data/arbitration.db on
the host via a mounted volume, so arbitration history survives
docker compose down / rebuilds.
Stop everything with docker compose down (add -v to also drop the network;
the ./data volume is a host bind mount so it isn't removed either way).
Interactive OpenAPI docs at /docs once the server is running.
| Method | Endpoint | Purpose |
|---|---|---|
POST |
/v1/arbitrate |
Arbitrate a single LLM output |
POST |
/v1/arbitrate/batch |
Arbitrate multiple outputs |
GET |
/v1/arbitrations/{id} |
Retrieve a past verdict |
GET |
/v1/arbitrations |
List past verdicts (paginated) |
GET |
/v1/analytics |
Aggregate stats on critic behaviour |
GET |
/health |
Liveness check |
curl -X POST http://127.0.0.1:8000/v1/arbitrate \
-H "Content-Type: application/json" \
-d '{
"original_prompt": "What is the capital of France?",
"llm_output": "The capital of France is Berlin."
}'Every arbitration is persisted to SQLite as a full audit trail — the original input, all three critic reports, detected disagreements, and the final verdict.
Security note: the API has no authentication and permissive CORS. It is built for local/portfolio use. Add an auth layer before exposing it publicly.
Three pages:
- Arbitrate — submit an output, get the verdict with the original text annotated by issue (red = confirmed by the adjudicator, yellow = raised but not confirmed, severity colour-coded), plus a side-by-side critic comparison panel that highlights agreement vs. disagreement.
- Batch Mode — submit multiple outputs, results in a sortable table.
- History & Analytics — aggregate stats and a browsable archive of past verdicts.
Four cases covering distinct failure modes:
uv run python scripts/run_test_cases.py # all four, persisted
uv run python scripts/run_test_cases.py --case 2 # one case
uv run python scripts/run_test_cases.py --no-persistVerified results:
| Case | Verdict | Key signal |
|---|---|---|
| 1. Factually incorrect (planted errors) | 1/10 | Accuracy 1/5 with 4 issues; logic scored 5/5 — the error is factual, not logical |
| 2. Logically flawed argument | 1/10 | Logic 1/5 — reversed causation and a dismissed constraint |
| 3. Technically answers, misses the point | 1/10 | Completeness 1/5 — true statements, wrong question |
| 4. Genuinely good response | 10/10 | All critics clean → short-circuited, adjudicator skipped |
Case 1 is the interesting one: the per-dimension scores diverge sharply (accuracy 1/5, logic 5/5), which is exactly the signal a single-model self-evaluation would miss.
uv run pytest tests -q51 tests, no API keys required — providers are stubbed.
| File | Covers |
|---|---|
test_graph.py |
Graph compiles; parallel fan-out; fan-in; conditional routing; short-circuit skips the adjudicator |
test_disagreement.py |
Score gaps, severity gaps, unique issues, degraded-report exclusion |
test_critics.py |
Graceful degradation, metadata stamping, fallback provenance |
test_llm_providers.py |
Fallback rerouting, fallback disable switch |
test_storage.py |
Persistence roundtrip, analytics aggregation |
test_api.py |
Every endpoint, error handling, OpenAPI spec |
test_streamlit_app.py |
UI renders headlessly via Streamlit's AppTest |
app/
config.py Env loading, model selection
models.py Pydantic schemas (the structured-output contract)
llm_providers.py Multi-provider factory + fallback logic
critics.py The three critic node functions
disagreement.py Disagreement detector
adjudicator.py Adjudicator prompt + verdict synthesis
graph.py LangGraph StateGraph orchestration
storage.py SQLite persistence + analytics
api.py FastAPI service
streamlit_app.py Verdict Explorer UI
scripts/
run_test_cases.py The four portfolio test cases
tests/ Test suite
docs/
PROJECT_SPEC.md Original build specification
Structured outputs. Every LLM call is validated against a Pydantic schema
via instructor. The pipeline stamps dimension, critic_name and
model_used itself rather than trusting the model to set them.
Native SDKs over LangChain wrappers. langchain-groq and
langchain-google-genai both pin langchain-core<0.4, which conflicts with
the langchain-core 1.x that current LangGraph needs. The critics use the
native groq, google-genai and openai SDKs wrapped with instructor
instead; LangGraph still drives orchestration.
JSON mode for fallback models. gpt-oss and llama-3.1-8b-instant are
more reliable with instructor's JSON mode than tool-calling mode for the
nested CriticReport schema. Qwen was tested and rejected — it fails
nested-schema validation.
Short-circuit is conservative. A clean pass requires all three critics to score ≥4 with zero issues and no critic to have degraded. A degraded critic never yields a clean bill of health, because absence of evidence is not evidence of absence.
- Deployment