A benchmark harness that measures how RAG pipeline choices affect answer quality, cost and latency — on a human-verified gold set, with confidence intervals and a validated judge.
Status: v0.1 — building. The harness is under construction. Every number below fills in from a committed run against a gold set verified by hand, or it stays empty. Methodology is written up in METHODOLOGY.md before the numbers exist, so it cannot be retrofitted to whatever they turn out to be.
pending — Stage 1 lands first (retrieval metrics, no LLM calls).
| Configuration | recall@10 | faithfulness | hallucination rate | $/query | p95 latency |
|---|---|---|---|---|---|
| pending |
Judge validation: pending — Cohen's kappa against hand labels. If it comes in below 0.6 it gets published anyway, and every generation metric here is labelled indicative rather than decisive.
Almost every published RAG comparison has the same three holes: no lexical baseline, so BM25 never gets the chance to win the numeric queries it usually wins; gold data anchored to chunk ids, which makes the chunking comparison circular; and an LLM judge nobody checked against a human, which makes every quality number an assertion rather than a measurement.
This project is those three holes, filled in. The deliverable is the table, not the chatbot.
Scope discipline is part of the design.
- Not a chat UI. A CLI and a static report. No conversational interface.
- Not a RAG framework. It does not compete with LlamaIndex. It evaluates pipelines.
- Not a production service. No auth, no multi-tenancy, no streaming.
- Not agentic RAG. Single-turn question answering only.
Three ideas do most of the work.
Gold data is anchored to document character spans, never to chunk ids. Chunk boundaries change with every configuration, so grading a strategy against its own boundaries would let each one mark its own homework. A retrieved chunk counts as relevant when its span overlaps a gold span by more than half.
Hallucination is measured on questions that have no answer. 20% of the gold set is unanswerable by construction, including near-misses where a similar fact exists in the corpus but not the one asked for. On those, any substantive answer is fabrication — there is no retrieval excuse available.
The judge is validated before its scores are trusted. Cohen's kappa against hand-labelled samples, published either way.
make install # uv sync
make verify # ruff + mypy --strict + pytestRunning the sweep additionally needs local embedding models and an API key:
uv sync --group ml
export ANTHROPIC_API_KEY=...
ragbench corpus fetch --tickers-file configs/tickers.txt
ragbench corpus parse
ragbench run retrieval --config configs/stage1/ --gold data/gold/gold_v1.jsonlEvery command takes --sample N for development, and every stage prints an estimated
token spend and waits for confirmation before spending anything.
| SPEC.md | The full specification. This repo is v0.1 of it — see Scope below. |
| METHODOLOGY.md | Gold set construction, known biases, threshold choices, what this cannot tell you. |
| RESULTS.md | Full tables, confidence intervals, judge validation, cost ledger. |
| ARCHITECTURE.md | Decisions and the alternatives they beat. |
| AGENTS.md | Working conventions. |
SPEC.md describes a 250-question gold set across 25–40 filings. This version
runs a smaller instrument end to end so the numbers are real today; the reduction is in
n, not in method. The exact counts are stated in RESULTS.md beside every
table, and the confidence intervals are wide accordingly.
MIT — see LICENSE.