RAG Evaluation Lab

Benchmark chunking and retrieval strategies for Retrieval-Augmented Generation — with reproducible, measurable metrics instead of vibes.

Most RAG demos stop at "it answers questions". This project treats RAG as an engineering problem: every design decision (chunking strategy, retriever type, top-k) is expressed as a versioned experiment config, and all experiments are evaluated on the same dataset with both deterministic retrieval metrics and LLM-as-a-judge quality metrics.

What this project demonstrates

• Retrieval engineering — three chunking strategies (fixed, recursive, sentence-window / small-to-big) and three retrievers (dense; hybrid dense+BM25 with Reciprocal Rank Fusion; two-stage retrieval with a cross-encoder re-ranker), all behind clean interfaces.
• Evaluation as a first-class citizen — hit rate, MRR, faithfulness, answer relevance, context precision and context recall, aggregated into a comparable benchmark table.
• Reproducibility — experiments are YAML files, each writing into its own Qdrant collection; judges run at temperature 0 with structured output.
• Production-minded structure — typed data models, provider-agnostic LLM client, Docker-based infrastructure, unit tests that run without any external services.

Architecture

                       ┌───────────────────────────────────────────┐
                       │            experiments/*.yaml             │
                       │  (chunker, retriever, top_k, embedder)    │
                       └────────────────────┬──────────────────────┘
                                            │ one pipeline per config
                                            ▼
 data/corpus/*.md ──► Chunker ──► Embedder ──► Qdrant (1 collection/exp)
                      fixed |      sentence-        │
                  recursive |      transformers     │  dense search
             sentence-window                        ▼
                                            Retriever (dense | hybrid+RRF
                                                       | + cross-encoder rerank)
                                                    │  top-k chunks
                                                    ▼
 data/eval/questions.json ──► question ──► Generator (LLM) ──► answer
                                    │                             │
                                    ▼                             ▼
                          ┌─────────────────────────────────────────┐
                          │            Evaluation                   │
                          │  deterministic: hit_rate@k, MRR         │
                          │  LLM-judge:     faithfulness,           │
                          │                 answer relevance,       │
                          │                 context precision,      │
                          │                 context recall          │
                          └───────────────────┬─────────────────────┘
                                              ▼
                            results/summary.md + per_sample_results.csv

Quickstart

Requirements: Python 3.10+, Docker, an Anthropic or OpenAI API key (used for answer generation and the LLM judge; embeddings run locally).

# 1. Install
pip install -r requirements.txt && pip install -e .

# 2. Configure
cp .env.example .env        # add your API key

# 3. Start Qdrant
docker compose up -d        # dashboard: http://localhost:6333/dashboard

# 4. Run the unit tests (no services / keys required)
pytest -v

# 5. Ask a single question against one configuration
python scripts/ask.py experiments/recursive_hybrid.yaml "How does RRF work?"

# 6. Run the full benchmark (all configs x all eval questions)
python scripts/run_benchmark.py

The benchmark writes:

• results/per_sample_results.csv — every (experiment, question) pair with all metrics, for debugging individual failures.
• results/summary.md — the aggregated comparison table, e.g.:

experiment	hit_rate	mrr	faithfulness	answer_relevance	context_precision	context_recall	latency_s
fixed_dense	...	...	...	...	...	...	...
recursive_dense	...	...	...	...	...	...	...
recursive_hybrid	...	...	...	...	...	...	...
recursive_rerank	...	...	...	...	...	...	...
sentence_window_dense	...	...	...	...	...	...	...

(Run the benchmark to fill in real numbers for your corpus — that is the whole point.)

Metrics explained

Metric	Type	Question it answers
hit_rate@k	deterministic	Did we retrieve at least one chunk from the expected document?
MRR	deterministic	How early is the first relevant chunk ranked?
faithfulness	LLM judge	What fraction of the answer's claims are supported by the context? (low = hallucination)
answer_relevance	LLM judge	Does the answer actually address the question?
context_precision	LLM judge	How much of the retrieved context is relevant? (low = noisy retrieval)
context_recall	LLM judge	How much of the ground truth is covered by the retrieved context? (low = retrieval misses)

Separating retrieval metrics from generation metrics is the key idea: a low faithfulness score with high context recall points at the generator, while high faithfulness with low context recall points at retrieval.

Adding an experiment

Create a YAML file in experiments/ — no code changes needed:

name: recursive_large_k         # becomes the Qdrant collection name
chunker: recursive              # fixed | recursive | sentence_window
chunker_params:
  max_chars: 800
retriever: hybrid               # dense | hybrid | rerank
top_k: 8
# embedding_model: BAAI/bge-base-en-v1.5   # optional override

Two-stage retrieval (retriever: rerank) wraps any base retriever with a cross-encoder re-ranking stage:

retriever: rerank
retriever_params:
  base: hybrid                  # stage 1: dense | hybrid
  candidate_pool: 20            # stage 1 fetches 20 candidates
  rerank_model: cross-encoder/ms-marco-MiniLM-L-6-v2
top_k: 4                        # stage 2 keeps the best 4

The cross-encoder runs locally on CPU (like the embeddings) and is only applied to the candidate pool, never the whole corpus.

Using your own data

1. Drop .md / .txt files into data/corpus/.
2. Write questions with ground-truth answers into data/eval/questions.json (see data/eval/README.md for the schema and tips).
3. Re-run python scripts/run_benchmark.py.

Project structure

rag-eval-lab/
├── src/rag_eval/
│   ├── config.py               # env-based settings (pydantic-settings)
│   ├── models.py               # shared typed data models
│   ├── pipeline.py             # experiment config + pipeline assembly
│   ├── ingestion/
│   │   ├── loader.py           # corpus loading
│   │   └── chunking.py         # fixed / recursive / sentence-window
│   ├── retrieval/
│   │   ├── embedder.py         # sentence-transformers wrapper
│   │   ├── vector_store.py     # Qdrant (collection per experiment)
│   │   ├── retriever.py        # dense + hybrid (BM25 + RRF)
│   │   └── reranker.py         # two-stage retrieval (cross-encoder)
│   ├── generation/
│   │   ├── llm.py              # provider-agnostic LLM client
│   │   └── generator.py        # grounded answer generation
│   └── evaluation/
│       ├── metrics.py          # deterministic + LLM-judge metrics
│       ├── dataset.py          # eval dataset loading
│       └── runner.py           # benchmark orchestration + reports
├── experiments/                # one YAML per RAG configuration
├── scripts/                    # ingest / ask / run_benchmark CLIs
├── data/
│   ├── corpus/                 # source documents (sample included)
│   └── eval/questions.json     # evaluation dataset (sample included)
├── tests/                      # unit tests (run without Qdrant or API keys)
├── docker-compose.yml          # local Qdrant
└── docs/DESIGN_DECISIONS.md    # why things are built the way they are

Design decisions

The reasoning behind the architecture (collection-per-experiment, RRF over score normalisation, custom judge vs. RAGAS, etc.) is documented in docs/DESIGN_DECISIONS.md.

Limitations & possible extensions

• Judge noise — LLM-judged metrics are noisy per sample; run enough questions (30+) before trusting differences smaller than ~0.05.
• Query rewriting / HyDE — a natural next experiment dimension in front of the retriever; the pipeline factory makes this a small addition.
• Cost tracking — token accounting per experiment would make the quality/cost trade-off explicit.
• CI — unit tests are service-free by design; a GitHub Actions workflow running pytest is a 10-line addition.

License

MIT