A CI gate that measures RAG retrieval quality (recall@10, MRR, nDCG) against a labeled query set and fails the build on regression, in 2.6 seconds at 20k documents with zero external calls.
- RAG changes ship on vibes: someone edits chunking or swaps a retriever, spot-checks five queries, and merges. This gate scores every PR against a labeled set and blocks measurable regressions.
- Embedding APIs in CI cost money, need secrets, and add network flake. The default backend is deterministic and offline; the measured gate runtime is 1.3s at 2k docs and 2.6s at 20k.
- Metric drops from dataset edits masquerade as pipeline regressions. The baseline store fingerprints the corpus and labels, and refuses to compare across dataset changes.
Teams that ship RAG features usually have strong test coverage for code paths and none for retrieval quality. The failure mode is quiet: a chunking tweak that helps long documents hurts short ones, recall@10 drops for a slice of queries, and nobody notices until users complain. Catching it manually means an engineer re-running a query list and eyeballing results per PR, which does not survive contact with a real release cadence.
rag-eval-gate makes retrieval quality a build artifact. It embeds the corpus, retrieves top-k for every labeled query, computes recall@10, MRR@10, and nDCG@10, and compares them to a stored baseline for the same (dataset, embedder, k) configuration. If any metric drops more than the configured tolerance (default 0.02 absolute), the build fails with a structured log line naming the metric, the baseline, and the drop. Baselines are updated deliberately through a snapshot command, never implicitly.
On the included seeded corpus, the measured gate evaluates 200 labeled queries over 2,000 documents in 1.3 seconds and 500 queries over 20,000 documents in 2.6 seconds, running entirely inside the CI container. The same codebase serves retrieval over HTTP for load testing and staging checks; under Locust the 2k-document service sustained 358 requests per second at 75 concurrent users with a p95 of 61 ms (full matrix below).
flowchart LR
subgraph inputs [Committed inputs]
C[corpus.jsonl]
L[labels.jsonl]
B[(baselines.db)]
end
subgraph gate [rag-eval-gate run]
V{Label validation}
E[Embedder: tfidf / bm25 / api]
R[Top-k retriever]
M[Metrics: recall@k, MRR, nDCG]
F{Fingerprint match?}
G{Drop > tolerance?}
end
C --> V
L --> V
V -- malformed: exit 2 --> X2[fail build]
V --> E --> R --> M --> G
B --> F
C -. sha256 .-> F
F -- dataset changed: exit 1 --> X1[fail build]
F --> G
G -- regression: exit 1 --> X1
G -- pass: exit 0 --> OK[merge]
Failure boundaries are explicit: malformed labels exit 2 before any evaluation, a changed dataset fingerprint exits 1 before any comparison, and a metric regression exits 1 with per-metric log lines.
| Technology | Role in this project | Why chosen here |
|---|---|---|
| Python 3.11+ | Everything | Single-language repo keeps the CI image small |
| scikit-learn TF-IDF | Default CI embedder | Deterministic per corpus, so metric diffs attribute to the PR, not embedding noise (ADR-0001) |
| rank-bm25 | Second retriever | Lets the report compare lexical strategies with numbers instead of habit |
| SciPy sparse (CSR) | Vector representation | Dense TF-IDF at 20k docs x 50k vocab is a 7.45 GiB allocation; sparse is 2.9 MB (see war story) |
| SQLite | Baseline snapshots | One committed file, atomic upserts, no service to run in CI (ADR-0002) |
| Flask + gunicorn | Retrieval HTTP service | Gives Locust a real service so latency numbers are HTTP-measured, not function-call-measured |
| Locust | Load testing | Scriptable traffic with realistic sampled queries, CSV export committed to benchmark/results/ |
| GitHub Actions | CI | The gate runs against its own committed baseline on every push |
Prerequisites: Python 3.11+, or Docker.
git clone https://github.com/NavyasriAmand/rag-eval-gate.git
cd rag-eval-gate
pip install -e ".[dev]"
# Run the evaluation and print metrics
rag-eval-gate report --corpus data/corpus_2k.jsonl --labels data/labels_2k.jsonl
# Store a baseline, then gate against it
rag-eval-gate snapshot --corpus data/corpus_2k.jsonl --labels data/labels_2k.jsonl
rag-eval-gate run --corpus data/corpus_2k.jsonl --labels data/labels_2k.jsonl
# Tests
pytest --cov=rag_eval_gate
# Retrieval service (or: docker compose up -d)
gunicorn --bind 0.0.0.0:8080 "rag_eval_gate.service:create_app()"
curl -s -X POST localhost:8080/retrieve -H 'Content-Type: application/json' \
-d '{"query": "kafka rebalancing failover", "top_k": 5}'Gating your own pipeline: export your corpus and labels as JSONL ({"id", "text"} and {"id", "text", "relevant": [doc_ids]}), snapshot once, add rag-eval-gate run to CI. The included workflow in .github/workflows/ci.yml is a working example: it gates this repo with its own committed baseline.
Both retrievers, evaluated on the committed seeded corpus (k=10):
| Corpus | Embedder | recall@10 | MRR@10 | nDCG@10 | Gate wall clock |
|---|---|---|---|---|---|
| 2k docs / 200 queries | tfidf | 0.400 | 0.730 | 0.432 | 1.3 s |
| 2k docs / 200 queries | bm25 | 0.426 | 0.703 | 0.447 | n/a (report mode) |
| 20k docs / 500 queries | tfidf | 0.607 | 0.841 | 0.631 | 2.6 s |
These are lexical retrievers on a synthetic corpus: mid-range scores are expected and useful, since a gate needs headroom in both directions to detect movement.
Methodology: Locust drives POST /retrieve with queries sampled from the corpus generator's template pools (not a single cached string), 25 s per run after a 10/s ramp, against gunicorn (2 workers, 4 threads) in a Linux container. Raw CSV exports are in benchmark/results/; the table is generated from them by benchmark/summarize.py.
xychart-beta
title "POST /retrieve latency, 2k-doc corpus (ms)"
x-axis "concurrent users" [5, 25, 75]
y-axis "latency (ms)" 0 --> 90
line "p50" [4, 4, 32]
line "p95" [7, 10, 61]
line "p99" [11, 15, 77]
| Corpus | Users | Requests | Failures | RPS | p50 (ms) | p95 (ms) | p99 (ms) |
|---|---|---|---|---|---|---|---|
| 2k docs | 5 | 852 | 0 | 35.5 | 4 | 7 | 11 |
| 2k docs | 25 | 4241 | 0 | 169.7 | 4 | 10 | 15 |
| 2k docs | 75 | 8657 | 0 | 357.7 | 32 | 61 | 77 |
| 20k docs | 5 | 788 | 0 | 32.8 | 9 | 19 | 30 |
| 20k docs | 25 | 3395 | 0 | 141.1 | 27 | 61 | 78 |
| 20k docs | 75 | 3857 | 0 | 156.6 | 240 | 330 | 370 |
Where it degrades: the 20k corpus saturates around 25 users (RPS stops scaling, 141 to 157), and at 75 users queueing dominates: p50 jumps to 240 ms. The cause is brute-force cosine work growing linearly with corpus size while worker capacity stays fixed. That is acceptable for the gate (batch, 2.6 s total) and for staging checks; ADR-0002 documents the trigger for adding an ANN index to a real serving path.
- ADR-0001: Deterministic offline embedders as the default CI backend
- ADR-0002: Brute-force cosine and SQLite baselines over a vector database (the boring choice, defended with this repo's measurements)
LLM-judged answer faithfulness (does the generated answer follow from the retrieved context). Retrieval metrics are the highest-signal, lowest-cost gate, so they ship first. Trigger for adding faithfulness: retrieval metrics stable for a quarter and a budgeted API key for CI, at which point it lands as a third metric family behind the same baseline interface. Also out of scope: graded (non-binary) relevance labels, until a labeled set that actually has grades exists.
- No secrets in the default path: the CI gate makes zero network calls. The optional API backend reads
OPENAI_API_KEYfrom the environment only, never from config files; production deployments should inject it from a secret manager (GitHub Actions secrets, Vault, AWS Secrets Manager). - Logging is structured JSON and logs query metadata (request id, top_k, latency), never document text, so corpus content does not leak into log pipelines.
- The service validates and bounds all input: query length capped at 2,000 characters, top_k capped at 100, non-JSON bodies rejected with 400.
- Dependencies are minimal by design (numpy, scipy via scikit-learn, rank-bm25, flask); fewer packages, smaller audit surface.
| Failure | Detection | Behavior | Recovery |
|---|---|---|---|
| Corpus or label file malformed or missing | JSONL parse + referential check before evaluation | Exit 2, log names the file and line; build fails fast | Fix the data, no state to clean up |
| Labels reference deleted doc ids | Pre-evaluation referential check | Exit 2 with the offending query id and doc ids | Regenerate or repair labels |
| Corpus edited without re-snapshotting | SHA-256 fingerprint mismatch vs baseline | Exit 1, refuses to compare across datasets | Deliberate snapshot run in the same PR |
| Baseline missing for a new configuration | Keyed lookup returns nothing | Exit 1 with instructions to snapshot | Run snapshot once |
| Retrieval quality regresses | Per-metric drop vs tolerance | Exit 1, one log line per regressed metric with baseline, current, drop | Fix the pipeline change or consciously re-snapshot |
| Service gets oversized or malformed requests | Input validation on every field | 400 with a specific error, request never reaches the retriever | n/a |
| Baseline DB corrupted | SQLite open or query error surfaces immediately | Gate crashes loudly rather than passing silently | Restore the committed db from git |
The TF-IDF embedder originally converted scikit-learn's sparse transform output to a dense array before retrieval. Every test passed, the 2k and 20k benchmarks ran clean, and the code shipped through several commits.
The masking factor was the synthetic corpus: its template-generated vocabulary is a few thousand terms, so the dense matrix stayed small. Probing the design before writing these docs, I re-ran the embedder against a realistic vocabulary (20k documents sampled from a 60k-word lexicon) and it crashed immediately: MemoryError: Unable to allocate 7.45 GiB for an array with shape (20000, 50000). Two compounding mistakes: densifying a matrix whose rows are ~99.9% zeros, and doing it via todense(), which materializes float64 before the intended float32 cast, doubling the allocation.
The fix (commit) keeps the pipeline sparse end to end: embedders return CSR matrices, row normalization uses scikit-learn's sparse-safe normalize, and the only densification left is the (n_queries, n_docs) score matrix, which is bounded by the query batch rather than the vocabulary (500 x 20,000 is 40 MB). The same failing repro now embeds the full corpus into 2.9 MB of sparse data with peak RSS of 393 MB. The satisfying part: the gate validated its own refactor. Running rag-eval-gate run against the pre-fix baseline returned identical metrics and exit 0, which is precisely the regression-detection job the tool exists to do. Two regression tests now pin the sparse contract so a future contributor cannot reintroduce the densification.
- JSON export of baselines alongside the SQLite file, so baseline changes are human-readable in PR diffs.
- Per-query score dumps on failure, so a regressed metric comes with the ten queries that moved most.
- A
comparesubcommand that evaluates two embedders side by side and prints a significance-aware delta table. - Slice metrics (per topic or per document length bucket) to catch regressions that average out globally.
- First metric to watch in real use: gate wall-clock time as the corpus grows, since it is the CI-minutes cost and the earliest signal that the exact-search decision needs revisiting.