This project extends the InsureLLM knowledge base from Week 5 to build a production-quality Retrieval-Augmented Generation system. The goal is to systematically improve upon the baseline implementation provided by Ed Donner across six evaluation metrics.
In the root directory:
lab.ipynb is a walk-through notebook
cd implementation and then uv run ingest.py to ingest data
From project root, uv run app.py to run the Q&A Chatbot
From project root, uv run evaluator.py to run the evaluation
In the implementation directory:
answer.py is the module that answers a user's question. You can change or rewrite fetch_context() and answer_question()
ingest.py is the module that loads in the data. You can change any of this!
In the evaluation directory:
Private code that runs the evaluation on test data. Don't change this!
- Work through the lab to understand the current state and ingest data
- Reimplement
ingest.pyandanswer.pywith your ideas - Beat Ed and beat the other teams!
Good luck!
The baseline system embeds document chunks with all-MiniLM-L6-v2 and stores them in a Chroma vector store. At query time, the three nearest chunks are retrieved by cosine similarity and passed to gpt-4.1-nano for answer generation. This approach scores MRR 0.72, nDCG 0.74, and completeness 3.53 out of 5 across 150 test questions. The completeness deficit is most pronounced on questions that require synthesising information across multiple documents, which the baseline retriever handles poorly because it encodes the query and each document independently.
A cross encoder reads the query and a candidate document together, making its relevance judgement more accurate than the baseline bi encoder, which reads them independently. I widened the candidate pool to 15 chunks and reranked them with cross-encoder/ms-marco-MiniLM-L-6-v2 before passing the top 3 to the language model. MRR improved from 0.72 to 0.88 and keyword coverage rose by 9 percentage points. Completeness improved only marginally, indicating the remaining gap was about which documents were retrieved rather than how they are ranked. From here, my battle was to improve the completeness score while keeping other metrics from dropping significantly.
Dense vector search captures semantic meaning well but can miss documents containing the exact terms used in a query. BM25 keyword search has the opposite strength. Running both retrievers in parallel and merging their candidates captures both signals. I loaded all chunks into a BM25 index, retrieved 20 candidates from each retriever independently, merged and deduplicated, then reranked with the cross encoder and passed the top 5 to the language model. Retrieval precision improved further but completeness remained flat, confirming the bottleneck had shifted from ranking quality to coverage.
A single user question often has multiple implicit aspects, and retrieving on the literal phrasing alone may miss relevant documents for some of them. I prompted gpt-4.1-nano to generate three diverse sub-queries per question, ran retrieval for each, merged the candidate pools, and reranked the combined set. The context window was also increased from 5 to 7 documents. Completeness improved by 0.15 points, the largest single gain from any retrieval-only change to that point.
Short chunks are retrieved precisely but often lack the surrounding detail the language model needs to give a complete answer. Small to Big retrieval resolves this by using small chunks for search and then replacing each retrieved chunk with a merged window of its neighbouring chunks before passing the context to the model. Each of the top 5 retrieved chunks was expanded to include two neighbours on either side, producing approximately 5 000 characters of context per slot. All four metrics reached their best values to that point, confirming this technique effectively breaks the precision and recall tradeoff.
I tested a 12-layer cross encoder (ms-marco-MiniLM-L-12-v2) against the 6-layer baseline in an otherwise identical pipeline. Retrieval scores improved marginally, but completeness and accuracy both declined, likely because the larger model overfit to ranking signals that do not translate to answer quality on this corpus. The 6-layer model was retained.
RAPTOR builds a tree of summaries over the corpus: one summary per source document and one broad roster per knowledge-base category. The intent was to give the retriever access to presynthesised, category-wide context for holistic questions. We ingested 76 per-document summaries and 4 category rosters alongside the 395 original chunks. Retrieval metrics reached their highest values in the study when RAPTOR nodes competed freely with expanded leaf chunks. Completeness, however, fell sharply because each RAPTOR summary is shorter and less detailed than a Small to Big expanded chunk, so it displaces richer context from the limited context window. No configuration of RAPTOR simultaneously improved both retrieval metrics and completeness.
Reciprocal Rank Fusion scores each document by its rank in both the BM25 and dense result lists, promoting documents that appear near the top of both. This produces a more principled merged candidate pool than simple concatenation and deduplication. In practice, the cross encoder reranking step dominates and the upstream merge strategy has negligible impact on final results. RRF with seven context slots performed equivalently to the Small to Big baseline on all answer quality metrics, confirming that further retrieval tuning alone cannot move the needle on completeness.
Seven successive retrieval interventions all plateaued at a completeness of approximately 3.88.
I then configured gpt-4.1-nano to handle sub-query expansion because it is fast and cheap and reasoning adds no benefit for that task, while deepseek/deepseek-r1 generates the final answer at temperature 0.6, the value recommended by DeepSeek for reasoning tasks. Four configurations were evaluated with the RRF retrieval stack held constant, as shown in Table 2.
Table 2. DeepSeek-R1 configuration ablation. k is the number of retrieved context documents, w is the Small to Big window radius, and N is the number of sub-queries.
| Configuration | MRR | nDCG | Accuracy | Completeness | Relevance |
|---|---|---|---|---|---|
| gpt-4.1-nano, k=5, w=2, N=3 (prior best) | 0.9449 | 0.9210 | 4.753 | 3.880 | 4.987 |
| DeepSeek-R1, k=5, w=2, N=3 | 0.9449 | 0.9210 | 4.800 | 3.973 | 4.967 |
| DeepSeek-R1, k=7, w=2, N=4 (active) | 0.9455 | 0.9033 | 4.853 | 4.047 | 4.967 |
| DeepSeek-R1, k=5, w=2, N=5 | 0.9446 | 0.9197 | 4.807 | 3.960 | 4.940 |
| DeepSeek-R1, k=5, w=3, N=3 | 0.9449 | 0.9218 | 4.727 | 3.993 | 4.953 |
Even with only five context slots and three sub-queries, DeepSeek-R1 raised completeness from 3.88 to 3.97. Increasing to seven slots with four sub-queries pushed completeness to 4.05, the highest value recorded across all experiments. The wider window of three neighbours produced the best retrieval score but reduced accuracy, as the extra context introduced noise for precise factual questions. The active configuration is k=7, w=2, N=4.
Table 1 traces the evolution of all six metrics across each intervention. Retrieval metrics (MRR, nDCG, Coverage) improve steadily through the first six interventions, reaching their ceiling with RAPTOR. Answer quality metrics (Accuracy, Completeness, Relevance) improve through Intervention 4 and then plateau near a completeness of 3.88 for all subsequent retrieval-only changes. Switching the answer model to DeepSeek-R1 in Intervention 8 produces the largest single gain in completeness of any step in the study, confirming that the bottleneck was in answer generation rather than retrieval.
Table 1. Evaluation results across all pipeline configurations (150 test questions).
| Pipeline | MRR | nDCG | Coverage | Accuracy | Completeness | Relevance |
|---|---|---|---|---|---|---|
| Baseline (Ed Donner) | 0.723 | 0.739 | 80.8% | 4.07 | 3.53 | 4.80 |
| Cross encoder reranking | 0.882 | 0.882 | 90.0% | 4.36 | 3.59 | 4.85 |
| Hybrid BM25 and dense retrieval | 0.932 | 0.916 | 95.5% | 4.45 | 3.58 | 4.87 |
| Multi-query expansion, N=3, k=7 | 0.937 | 0.908 | 96.7% | 4.58 | 3.73 | 4.95 |
| Small to Big expansion, w=2, k=5 | 0.945 | 0.921 | 97.2% | 4.75 | 3.88 | 4.99 |
| Cross encoder L-12 upgrade | 0.946 | 0.924 | 97.2% | 4.73 | 3.80 | 4.99 |
| RAPTOR hierarchical summaries | 0.958 | 0.949 | 98.1% | 4.71 | 3.53 | 4.96 |
| RRF hybrid search, k=5 | 0.944 | 0.919 | 97.2% | 4.67 | 3.87 | 4.94 |
| DeepSeek-R1, k=7, w=2, N=4 (active) | 0.946 | 0.903 | 97.5% | 4.85 | 4.05 | 4.97 |
Active pipeline: The bi encoder all-MiniLM-L6-v2 embeds queries and documents. An RRF hybrid retriever combines BM25 and dense search, drawing 30 candidates from each. gpt-4.1-nano generates four sub-queries per question. The cross encoder ms-marco-MiniLM-L-6-v2 reranks the merged candidate pool and selects seven documents. Each document is expanded to a window-2 Small to Big context and passed to deepseek/deepseek-r1 at temperature 0.6 for final answer generation.