groundtruth

Ask a question, get an answer with every claim traceable to a live source — not "trust me," a URL you can click and check yourself.

Python · Claude API (web search + web fetch server tools)

---

How it works

question
   │
   ▼
plan          Claude breaks it into 2-4 independently-researchable sub-questions
   │
   ▼
research      one Claude call per sub-question, using live web_search + web_fetch —
   │          citations extracted from the actual search results, not invented
   ▼
synthesize    Claude combines the research notes into one answer; every factual
   │          claim is paired with the source URL that supports it
   ▼
report        { answer, citations: [{claim, url}] }

Three separate, inspectable Claude calls rather than one opaque tool-use loop. That's deliberate — it means the orchestration code owns real decisions (how many sub-questions to spawn, what counts as a usable research note, how a claim gets matched to its source) instead of leaving all of it to a single black-box request.

Web search and fetch run as Anthropic server-side tools — no separate search API key to provision, no scraping code to maintain.

Setup

Requires Python 3.11+ and an Anthropic API key.

cp .env.example .env      # then add your key to .env
python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -r requirements.txt

Usage

python -m groundtruth.cli "What changed in the EU AI Act enforcement timeline in 2026?"

Progress streams to stderr as it plans and researches; the final answer and its citations print to stdout.

Tests

pytest

12 tests, all against a mocked client — they exercise sub-question parsing and fallback behavior, citation extraction and de-duplication, and the plan → research → synthesize call ordering. No API key or network access needed to run them.

Accuracy eval

python eval/eval_research.py

A 12-question golden set with known, stable facts (plus one deliberately time-sensitive question — "which country has the largest population" — to check the agent actually searches rather than answering from stale training data). Scores fact_accuracy (did the answer contain the right fact) and citation_rate (did it cite a source at all).

This does not run on every push. Unlike a fully offline eval, this one makes real, billed API calls and hits the live web — gating CI on that would be flaky (source pages change) and expensive for no real benefit. It's wired as a workflow_dispatch job instead: run it manually from the Actions tab, or add a schedule trigger if you want a periodic accuracy check. Requires an ANTHROPIC_API_KEY repository secret to run in CI.

Design notes

Why decompose into sub-questions at all? A single web_search call often returns a shallow, one-source answer. Forcing the model to plan first consistently surfaces the follow-up angle a one-shot search misses (e.g. "who leads it now" as well as "when was it founded").

Why claude-sonnet-5 by default, not Opus? A single query costs a plan call plus one research call per sub-question plus a synthesis call — 4-6 calls per question. Sonnet is the right cost/quality tradeoff for this call volume; pass --model claude-opus-4-8 for harder questions.

Why citations are extracted from tool results, not asked for as free text. Asking the model to "include citations" invites plausible-looking but made-up URLs. Citations here come only from web_search_tool_result blocks Claude actually received — a claim can only be cited to a page that was genuinely returned by a search.

Limitations

• No cross-checking between sub-questions yet — if two sources disagree, the synthesis step may not catch it. Noted as a possible extension, not solved.
• The golden set is small (12 questions) and knowingly includes one time-sensitive fact that will need updating eventually.
• No conversation memory — each question is a fresh run.

License

MIT