questions.json is a JSON list of evaluation samples:
{
"question": "What does the faithfulness metric measure?",
"ground_truth": "Faithfulness measures the fraction of claims in the answer supported by the retrieved context; a low score indicates hallucination.",
"expected_doc_ids": ["rag_evaluation"]
}| Field | Required | Used by |
|---|---|---|
question |
yes | everything |
ground_truth |
yes | context recall (LLM judge) |
expected_doc_ids |
no | hit rate & MRR (deterministic). File stem(s) of the corpus document(s) containing the answer. Omit if unknown — the sample is then skipped for these two metrics. |
- Cover every corpus document with at least one question, otherwise the benchmark cannot detect retrieval blind spots.
- Mix paraphrase questions (test dense retrieval) with exact-term questions containing rare tokens like acronyms (test BM25 / hybrid).
- Include at least one unanswerable question if you want to test that the generator refuses instead of hallucinating.
- 30+ questions make LLM-judged metric differences meaningful; the included 10 are a starter set for the sample corpus.