Find the best chunking, embedding, and retrieval strategy for your RAG pipeline — in minutes, not days.
RAG BenchKit is an open-source evaluation toolkit that benchmarks every combination of chunking strategies, embedding models, and retrieval methods against your documents — and visualizes the results in a clean dashboard. No boilerplate, no notebooks, no guesswork.
Most RAG teams pick chunking and embedding settings once and never revisit them. This tool makes it trivial to find out if that was the right call.
You give it documents and queries with ground-truth labels. It runs every combination you configure and tells you which one wins — with Recall@K, Precision@K, MRR, NDCG, MAP, and Hit Rate.
Fixed Size ─┐
Recursive ─┤ × MiniLM ─┐ ┌─ Dense (FAISS)
Semantic ─┤ BGE ─┤ → eval ─┤ Sparse (BM25)
Doc-Aware ─┘ OpenAI ─┘ └─ Hybrid (RRF)
Cohere
| 4 Chunking Strategies | Fixed Size, Recursive, Semantic (sentence similarity), Document-Aware (markdown/code) |
| 5 Embedding Models | MiniLM, BGE Small (local/free), OpenAI Small, OpenAI Large, Cohere |
| 3 Retrieval Methods | Dense (FAISS cosine), Sparse (BM25), Hybrid (Reciprocal Rank Fusion) |
| 6 IR Metrics | Precision@K, Recall@K, MRR, NDCG@K, MAP@K, Hit Rate@K |
| Visual Dashboard | Leaderboard, heatmaps, ranked charts, per-query drill-down |
| Sample Data | 10 Python tutorial docs + 15 queries — run a demo in 30 seconds |
| Export | Download results as CSV or JSON |
git clone https://github.com/sausi-7/rag-benchkit.git
cd rag-benchkit# Create venv
python3 -m venv .venv
# Activate — macOS/Linux
source .venv/bin/activate
# Activate — Windows (PowerShell)
.venv\Scripts\Activate.ps1For running the app:
pip install .For development (includes pytest, ruff):
pip install -e ".[dev]"streamlit run app.pyThen open the browser, click Load Sample Data, select your strategies, and hit Run Benchmark.
Requirements: Python 3.9+. Local embeddings (MiniLM, BGE) run on CPU — no GPU needed. API keys for OpenAI/Cohere are entered in the sidebar.
Documents — upload .txt or .md files. Each file = one document. The filename (without extension) becomes the doc_id.
Queries — upload a JSON file:
[
{
"query_id": "q01",
"query": "How does Python handle memory management?",
"relevant_doc_ids": ["doc_01_python_basics"]
}
]relevant_doc_ids must match document filenames (without extension). These are your ground-truth labels — the tool measures how well each pipeline retrieves them.
| Tab | What it shows |
|---|---|
| Leaderboard | All combinations ranked by Recall@K with color-coded metric columns |
| Heatmaps | Chunker × Embedder performance matrix, one heatmap per retrieval method |
| Charts | Ranked bar chart of all configs, multi-metric comparison, radar profile of best config |
| Per-Query Details | Hit/miss breakdown for each query on a selected configuration |
rag-benchkit/
├── app.py # Streamlit UI
├── pyproject.toml # Package config and dependencies
├── src/rag_benchkit/
│ ├── chunkers.py # 4 chunking strategies
│ ├── embedders.py # 5 embedding models
│ ├── retrievers.py # 3 retrieval methods
│ ├── metrics.py # 6 evaluation metrics
│ ├── runner.py # Benchmark orchestration
│ └── sample_data/ # Built-in demo corpus
│ ├── corpus/ # 10 .txt documents
│ └── queries.json # 15 queries with ground truth
└── tests/ # Pytest test suite
For architecture details, implementation notes, and how to extend the system, see README_TECHNICAL.md.
# Make sure dev dependencies are installed
pip install -e ".[dev]"
pytest tests/We welcome contributions — new chunkers, embedders, retrieval methods, metrics, UI improvements, and more. See CONTRIBUTING.md to get started.
MIT — use it, fork it, build on it.