graph-vector-retrieval-benchmark

HippoRAG 2 (graph) vs. ChromaDB (vector) retrieval, benchmarked head-to-head on multi-hop QA. Real recall numbers, real significance tests, real cost.

Research question

Under what conditions does a lean graph-based retrieval layer (HippoRAG 2: OpenIE triple extraction + Personalized PageRank) beat vector-based retrieval on multi-hop QA — and at what indexing cost?

• SQ1 — Retrieval quality. recall@k by dataset and question difficulty, tested for significance (paired Wilcoxon, Holm-corrected, effect size, bootstrap CI).
• SQ2 — Graph structure. Connectivity of the extracted graph, triple-extraction fidelity against gold triples, and whether it exposes an inspectable reasoning path that vector retrieval can't.
• SQ3 — Cost/benefit. Real indexing cost vs. the vector baseline, and when the gain justifies it.

Answer-generation quality (RAGAS) is out of scope — this is a retrieval benchmark.

Pipelines

Pipeline	Retrieval	Store	Indexing cost
Vector RAG (baseline)	Cosine similarity, bge-large-en-v1.5	ChromaDB	€0, local
HippoRAG 2 (graph)	OpenIE extraction → Personalized PageRank	in-memory igraph	~€13 (3 datasets)

No Microsoft GraphRAG, LightRAG, or hybrid fusion — lean graph only. No LangChain either (you don't need 40 abstraction layers to call .similarity_search()). HippoRAG 2 is reimplemented from scratch here, not the hipporag PyPI package — its CUDA deps don't resolve on Apple Silicon.

flowchart LR
    subgraph VectorRAG["Vector RAG (baseline)"]
        A1[Corpus] --> B1["bge-large-en-v1.5"]
        B1 --> C1[(ChromaDB)]
        C1 -->|cosine top-k| D1[Passages]
    end
    subgraph HippoRAG2["HippoRAG 2 (graph)"]
        A2[Corpus] --> B2[OpenIE extraction]
        B2 --> C2[(in-memory igraph)]
        C2 -->|Personalized PageRank| D2[Passages]
    end

Loading

Results

N=250 questions per dataset, pooled-distractor corpora (HippoRAG §3.1 method), seed 20260712. Full breakdown in the Studio dashboard.

recall@10, all questions:

Dataset	Vector RAG	HippoRAG 2	Δ	p (Holm)
2WikiMultiHopQA	0.807	0.803	−0.004	0.558 (n.s.)
MuSiQue	0.685	0.496	−0.189	<0.001
HotpotQA	0.970	0.868	−0.102	<0.001

Vector RAG wins on average, every dataset. But the average hides the interesting part:

recall@10, genuinely multi-hop questions only:

Dataset	Hard questions	Vector	HippoRAG 2	Δ	p (Holm)	Result
2WikiMultiHopQA	53 / 250	0.571	0.646	+0.075	0.025	Graph wins
MuSiQue	51 / 250	0.407	0.294	−0.113	0.009	Vector wins
HotpotQA	197 / 250	0.967	0.850	−0.117	<0.001	Vector wins

On 2WikiMultiHopQA's genuinely multi-hop questions, the graph wins, significantly. On MuSiQue and HotpotQA it doesn't — the disadvantage gets worse on the hard subset. Graph retrieval pays off for identifiable multi-hop cases in some corpora, not as a blanket upgrade. Root-cause diagnosis: results/hard_subset_failure_diagnosis.json.

Indexing cost (glm-5.2 OpenIE, reasoning tokens disabled):

Dataset	HippoRAG 2	Vector RAG
2WikiMultiHopQA	€2.66	€0
MuSiQue	€6.00	€0
HotpotQA	€4.34	€0
Total	€13.01	€0

glm-5.2 reasons by default, silently — 6.6× the tokens, for triple extraction that didn't ask for it. Disabled it. Ask us how we found out.

Studio dashboard

Local FastAPI + vanilla-JS viewer over the result files. No database.

• Dashboard — both pipelines' metrics side by side, cost, gold-evidence check.
• Questions — per-question recall; click one for a split-screen retrieval comparison, gold passages highlighted.

Graph explorer	Vector space

• Graph explorer — the HippoRAG 2 entity graph, force-directed, searchable, PNG export.
• Vector space — indexed chunks projected to 2D with UMAP.

Run it: uvicorn studio.backend.app:create_app --factory — see studio/README.md.

Datasets

Three public multi-hop QA benchmarks. Not redistributed here — fetch from the source.

• 2WikiMultiHopQA — ships native gold triples. Apache 2.0. Ho et al., 2020. Hugging Face
• MuSiQue — hardest, least "leaky" of the three. CC BY 4.0. Trivedi et al., 2022. GitHub
• HotpotQA — reference dataset; some questions are single-hop-resolvable. CC BY-SA 4.0. Yang et al., 2018. Project page

Corpus construction and dataset selection reasoning: docs/datasets.md, docs/dataset-evaluation.md, docs/retrieval-corpus-methodology.md.

Quickstart

git clone https://github.com/matteo-ise/graph-vector-retrieval-benchmark
cd graph-vector-retrieval-benchmark

conda create -n benchmark python=3.11 && conda activate benchmark
pip install -e ".[dev,studio]"

pytest -q

# rerun the deep-dive analysis over all three datasets
python scripts/run_deep_dive_analysis.py \
    --run-ids track_b_2wikimultihopqa_n250_v1 track_b_musique_n250_v1 track_b_hotpotqa_n250_v1

# explore results in the Studio dashboard
uvicorn studio.backend.app:create_app --factory

Structure

src/graph_vector_retrieval_benchmark/
├── pipelines/
│   ├── vector_rag/        # chunking, bge-large embedding, ChromaDB store
│   └── hipporag2/         # OpenIE extraction, graph builder, PPR retrieval
├── analysis/deep_dive.py  # significance tests, hard-subset, cost/benefit
├── pooling/                # pooled-distractor corpus construction
├── data_screening/         # corpus validation before indexing
├── loaders/                 # one loader per dataset
├── corpus_io/                # jsonl read/write helpers
└── evaluation/                # recall@k
scripts/                       # one script per pipeline step + analysis runners
studio/                         # FastAPI + JS dashboard
results/track_b_<dataset>_n250_v1/  # the canonical run per dataset
tests/                           # pytest, TDD throughout
docs/                             # dataset selection, corpus methodology

Adapting this

Each pipeline writes the same on-disk schema (results/<run_id>/<pipeline>/metrics.json, retrieval_results.jsonl), which analysis/deep_dive.py and the Studio dashboard read. A new pipeline needs to produce that schema; a new dataset needs one loader following the pattern in loaders/.

Notes

• results/track_b_*_v1/ (without _n250_) are legacy N=1000 runs, methodologically inconsistent, excluded from the dashboard. Only *_n250_v1 is canonical.
• Every number here traces to a file under results/.

Citation

See CITATION.cff, or GitHub's "Cite this repository" button. Companion codebase to a Bachelor's thesis, cited separately in that file.

License

MIT