Short rationale for the non-obvious choices in this codebase. Interviewers ask "why did you do it this way?" — this document is the answer.
Different chunking strategies produce incompatible chunk sets. Mixing them in one collection would make results impossible to attribute. A collection per experiment (named after the config) keeps runs isolated, reproducible and independently inspectable in the Qdrant dashboard.
RAGAS implements the same metric ideas, and using it would be perfectly
valid. Implementing faithfulness / relevance / precision / recall directly
(each ~1 structured judge call) was chosen because (a) it shows how these
metrics actually work rather than hiding them behind a library, (b) it
removes a heavy dependency chain, and (c) the judge prompts stay fully
inspectable and versioned with the code. Swapping in RAGAS later would only
touch evaluation/metrics.py.
BM25 scores and cosine similarities live on different, corpus-dependent
scales. Normalising them (min-max, z-score) is brittle and needs tuning.
RRF uses only ranks — score = Σ 1/(k + rank) — which is scale-free,
parameter-light (k=60 is a robust default) and a well-established baseline
for hybrid search.
Embedding a single sentence gives a sharp, low-noise vector for matching,
but a lone sentence is often insufficient context for generation. Storing
the surrounding window as the payload decouples the matching unit from
the context unit. The embed_text metadata field carries this through
the vector store without special-casing the rest of the pipeline.
Evaluation must be reproducible. Temperature 0 plus strict JSON output (defensively parsed) makes judge runs stable enough to compare experiments. The parser tolerates markdown fences and surrounding prose because even well-prompted models occasionally produce them.
Hit rate and MRR cost nothing, are perfectly reproducible, and localise
failures: if hit rate is low, no judged metric needs interpreting — the
retriever simply missed. Judged metrics add nuance only once retrieval
fundamentally works. Samples without expected_doc_ids yield NaN for the
deterministic metrics and are excluded from their averages.
Embeddings (sentence-transformers, CPU) are free and offline, so indexing
experiments cost nothing to iterate on. Only generation and judging call a
paid API. The LLMClient abstraction supports Anthropic and OpenAI —
pointing it at a local OpenAI-compatible server (e.g. Ollama) is a config
change.
Tests cover the pure logic (chunking invariants, RRF fusion, judge output parsing) and mock the dense retriever where needed. They run in CI without Qdrant or API keys, which keeps the feedback loop fast and free.
RerankingRetriever composes an arbitrary base retriever (dense or
hybrid) with a cross-encoder stage instead of duplicating retrieval logic.
This keeps re-ranking an orthogonal experiment dimension: any stage-1
strategy can be combined with it via YAML. The stage-1 pool is always
widened to at least top_k, otherwise re-ranking could never change the
final result set. The cross-encoder itself sits behind a two-method
Scorer protocol, so unit tests inject a fake scorer and verify the
reordering logic without loading torch.