Skip to content

Project_Goal.md

Latest commit

 

History

History
340 lines (245 loc) · 6.6 KB

Project_Goal.md

File metadata and controls

340 lines (245 loc) · 6.6 KB
Raw

rag-retrieval-benchmark

Project Goal

rag-retrieval-benchmark is a production-style retrieval evaluation lab for Retrieval-Augmented Generation systems.

The goal of this project is to compare different retrieval strategies and measure which method retrieves the most relevant documents for a given query.

This repository focuses only on the retrieval layer of a RAG pipeline.

It does not generate final answers. It does not call an LLM for response generation. It evaluates retrieval quality before generation happens.

Why This Project Matters

Many RAG systems fail because retrieval is weak, not because the language model is weak.

If the wrong chunks are retrieved, the generation layer will produce irrelevant, incomplete, or hallucinated answers.

This project demonstrates how to evaluate and improve retrieval instead of blindly using vector search.

Dataset

This project uses the BEIR SciFact dataset.

SciFact is a compact public information retrieval dataset suitable for testing RAG retrieval pipelines.

The dataset has three required parts:

corpus  → documents to retrieve from
queries → search questions / claims
qrels   → ground-truth relevance labels

Recommended local structure:

data/
├── raw/
│   ├── corpus/
│   │   └── corpus.parquet
│   ├── queries/
│   │   └── queries.parquet
│   └── qrels/
│       └── test.tsv
│
├── processed/
│   ├── corpus.jsonl
│   ├── queries.jsonl
│   └── qrels.jsonl
│
└── reports/
    └── retrieval_benchmark.md

The corpus should come from:

BeIR/scifact - corpus

The queries should come from:

BeIR/scifact - queries

The relevance labels should come from:

BeIR/scifact-qrels

Core Scope

This project should benchmark these retrieval methods:

dense retrieval
BM25 retrieval
hybrid retrieval
hybrid retrieval with Reciprocal Rank Fusion
metadata-filtered retrieval
query rewriting before retrieval

Retrieval Methods

1. Dense Retrieval

Dense retrieval embeds queries and documents into vectors and retrieves documents by vector similarity.

This is the baseline semantic search method.

Recommended stack:

sentence-transformers
BAAI/bge-small-en-v1.5
Qdrant

2. BM25 Retrieval

BM25 is the keyword-based baseline.

It is useful for exact terms, scientific phrases, acronyms, and domain-specific vocabulary.

Recommended stack:

rank-bm25

3. Hybrid Retrieval

Hybrid retrieval combines BM25 and dense retrieval.

The goal is to use both exact keyword matching and semantic similarity.

Supported merge methods:

simple merge
score-based merge
Reciprocal Rank Fusion

4. Reciprocal Rank Fusion

RRF combines ranked results from multiple retrievers without requiring their scores to be directly comparable.

This is the main hybrid ranking method for the project.

5. Metadata Filtering

The retrieval system should support filtering by metadata fields such as:

document_id
title
source
dataset_name

This is useful for testing retrieval under scoped search conditions.

6. Query Rewriting

Query rewriting creates a better search query before retrieval.

The first implementation can be simple and rule-based.

LLM-based query rewriting can be added later, but it is not required for the first version.

Evaluation Metrics

This project should report retrieval quality using:

Recall@k
MRR
Hit Rate@k
Average retrieval latency

Recommended k values:

k = 1, 3, 5, 10

Expected Benchmark Output

The project should generate a report comparing retrieval methods.

Example report table:

Method               Recall@5    MRR      Avg Latency
Dense Retrieval       0.72       0.61     180ms
BM25                  0.64       0.54      60ms
Hybrid                0.79       0.67     220ms
Hybrid + RRF          0.83       0.70     230ms
Metadata + Hybrid     0.85       0.72     200ms

API Scope

The repository should expose simple retrieval and evaluation endpoints.

Recommended endpoints:

GET  /health
POST /retrieve
POST /evaluate
GET  /reports/latest

POST /retrieve

Input:

{
  "query": "Does this scientific claim have supporting evidence?",
  "method": "hybrid_rrf",
  "top_k": 5
}

Output:

{
  "query": "...",
  "method": "hybrid_rrf",
  "top_k": 5,
  "results": [
    {
      "document_id": "...",
      "title": "...",
      "score": 0.87,
      "rank": 1
    }
  ],
  "latency_ms": 230
}

POST /evaluate

Runs benchmark evaluation against the SciFact qrels.

Output:

{
  "method": "hybrid_rrf",
  "recall_at_5": 0.83,
  "mrr": 0.70,
  "average_latency_ms": 230
}

Technology Stack

Python 3.11+
FastAPI
Qdrant
sentence-transformers
BAAI/bge-small-en-v1.5
rank-bm25
pandas
pyarrow
pydantic-settings
pytest
ruff
Docker
Docker Compose
GitHub Actions

Production-Style Requirements

This project should include:

clean config management
Docker Compose
structured logging
benchmark reports
unit tests
smoke tests
Makefile commands
GitHub Actions
clear README

Suggested Makefile Commands

make download-data
make prepare-data
make up
make index
make retrieve
make evaluate
make test
make lint

What This Project Does Not Do

This project does not generate final RAG answers.

It does not implement citation-aware answer generation.

It does not implement observability dashboards.

Those belong to:

production-rag-answering-api

This project only answers one question:

Which retrieval strategy finds the right documents most effectively?

Success Criteria

The project is complete when:

SciFact corpus, queries, and qrels are loaded correctly
dense retrieval works
BM25 retrieval works
hybrid retrieval works
RRF works
Recall@k and MRR are calculated
latency is measured
benchmark report is generated
API endpoints work
Docker Compose runs the service
tests pass
README explains the retrieval comparison clearly

Branding Message

This repository demonstrates retrieval engineering for RAG systems.

It shows how to benchmark dense search, keyword search, hybrid retrieval, metadata filtering, and query rewriting using real relevance labels instead of subjective manual testing.