feat(eval): add DeepEval providers

configs/eval-dataset-deepeval.yml

@@ -0,0 +1,14 @@
+dataset_generation:
+  provider: deepeval
+  options:
+    model: gpt-4o-mini
+    max_goldens_per_context: 1
+    limit_chunks: 50
+    min_context_chars: 120
+    include_expected_output: true
+    max_concurrent: 10
+
+llm:
+  provider: openai
+  model: gpt-4o-mini
+  temperature: 0.3

docs/deepeval.md

@@ -0,0 +1,91 @@
+# DeepEval evaluation and synthesis
+
+DeepEval is available as an optional dev evaluation provider alongside the
+existing RAGAS provider. Use it when you want one framework for both RAG
+component scoring and synthetic golden generation from prepared xrag Chunks.
+
+## Why DeepEval for xrag
+
+The DeepEval RAG guide recommends evaluating retrieval and generation together:
+retrieval quality is measured with contextual precision, recall, and relevancy;
+generation quality is measured with faithfulness and answer relevancy. Its
+Synthesizer can generate goldens from prepared contexts, which maps cleanly to
+xrag's Chunk artifacts because xrag already controls parsing, chunk metadata,
+and Retrieval Text.
+
+## Installation
+
+DeepEval is an optional tool dependency. Install it in your active dev
+environment before selecting the provider:
+
+```bash
+uv pip install deepeval
+```
+
+## Full-eval provider
+
+Set `evaluation.provider: deepeval` in any RAG config and keep `--full-eval`
+on the eval command:
+
+```yaml
+evaluation:
+  provider: deepeval
+  options:
+    judge_model: gpt-4o-mini
+    metrics:
+      - faithfulness
+      - answer_relevancy
+      - context_precision
+      - context_recall
+      - context_relevancy
+    threshold: 0.7
+    include_reason: true
+```
+
+Supported metric names:
+
+- `faithfulness`
+- `answer_relevancy`
+- `contextual_precision` or `context_precision`
+- `contextual_recall` or `context_recall`
+- `contextual_relevancy` or `context_relevancy`
+
+The eval runner still writes the same artifacts as RAGAS runs:
+`eval_results.json`, `per_question.jsonl`, `summary.txt`, and the append-only
+`experiments/eval/experiments.jsonl`. Deterministic retrieval metrics remain the
+benchmark gate; DeepEval metrics are added only for `--full-eval` runs.
+
+## Synthetic data provider
+
+Use `configs/eval-dataset-deepeval.yml` to generate DeepEval goldens from xrag
+Chunk contexts:
+
+```bash
+uv run python -m xrag.cli eval create-dataset \
+  --input data/dev/ir_document/latest/chunks.json \
+  --config configs/eval-dataset-deepeval.yml \
+  --output-dir data/artifacts/ir_document/eval-deepeval
+```
+
+The provider calls `Synthesizer.generate_goldens_from_contexts()` rather than
+`generate_goldens_from_docs()` because xrag already parsed documents into
+canonical Chunks with `doc_id`, `chunk_id`, page, and source-element metadata.
+Each generated golden is converted back into the existing `testset.jsonl` schema
+with the source chunk pinned as `evidence_chunk_ids`.
+
+Important options:
+
+- `model`: DeepEval synthesizer model.
+- `limit_chunks`: maximum number of chunks to submit.
+- `min_context_chars`: skips tiny chunks that do not provide enough signal.
+- `max_goldens_per_context`: generated questions per chunk context.
+- `include_expected_output`: keep enabled for RAG scoring.
+- `max_concurrent`: reduce if the model provider rate-limits.
+
+## Benchmark discipline
+
+DeepEval synthetic rows are LLM-generated, but they are not a replacement for the
+canonical curated/LLM xrag benchmark unless explicitly promoted. After any eval
+run, keep following the repository benchmark rules: compile the benchmark,
+append a new `docs/BENCHMARK.md` row, and update key findings and phase gate
+status when there is new evidence.

scripts/eval/README.md

@@ -78,5 +78,5 @@ the per-metric tolerance bands exits non-zero.
 
 ```bash
 scripts/eval/verify_chunk_schema.sh              # deterministic-only (~12 min)
-scripts/eval/verify_chunk_schema.sh --full-eval  # include RAGAS judge (~50 min)
+scripts/eval/verify_chunk_schema.sh --full-eval  # include configured judge metrics (~50 min)
 ```

tests/unit/test_deepeval_eval.py

@@ -0,0 +1,102 @@
+from __future__ import annotations
+
+import sys
+import types
+from dataclasses import dataclass
+from typing import Any
+
+import pytest
+
+from xrag.config.models import ProviderConfig
+
+
+class _FakeMetric:
+    def __init__(self, **kwargs: Any) -> None:
+        self.kwargs = kwargs
+        self.score: float | None = None
+        self.reason = "grounded"
+
+    def measure(self, test_case: Any) -> None:
+        self.score = 1.0 if test_case.actual_output else 0.0
+
+
+def _install_fake_deepeval(monkeypatch: pytest.MonkeyPatch) -> None:
+    deepeval = types.ModuleType("deepeval")
+    metrics = types.ModuleType("deepeval.metrics")
+    test_case_mod = types.ModuleType("deepeval.test_case")
+
+    @dataclass
+    class LLMTestCase:
+        input: str
+        actual_output: str
+        expected_output: str = ""
+        retrieval_context: list[str] | None = None
+
+    for name in (
+        "AnswerRelevancyMetric",
+        "ContextualPrecisionMetric",
+        "ContextualRecallMetric",
+        "ContextualRelevancyMetric",
+        "FaithfulnessMetric",
+    ):
+        setattr(metrics, name, _FakeMetric)
+    test_case_mod.LLMTestCase = LLMTestCase
+
+    monkeypatch.setitem(sys.modules, "deepeval", deepeval)
+    monkeypatch.setitem(sys.modules, "deepeval.metrics", metrics)
+    monkeypatch.setitem(sys.modules, "deepeval.test_case", test_case_mod)
+    sys.modules.pop("tools.evaluation.deepeval_eval", None)
+
+
+def test_build_evaluator_returns_deepeval_provider(monkeypatch: pytest.MonkeyPatch) -> None:
+    """Factory lazily builds the DeepEval provider."""
+    _install_fake_deepeval(monkeypatch)
+
+    from tools.evaluation.deepeval_eval import DeepEvalEvaluator
+    from tools.evaluation.factory import build_evaluator
+
+    evaluator = build_evaluator(ProviderConfig(provider="deepeval", options={}))
+
+    assert isinstance(evaluator, DeepEvalEvaluator)
+
+
+def test_deepeval_evaluator_scores_and_aliases_metrics(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """DeepEval scores use xrag's existing aggregate/per-question contract."""
+    _install_fake_deepeval(monkeypatch)
+
+    from tools.evaluation.deepeval_eval import DeepEvalEvaluator
+
+    evaluator = DeepEvalEvaluator()
+    output = evaluator.evaluate(
+        [
+            {
+                "query": "What is revenue?",
+                "answer": "$10M",
+                "contexts": ["Revenue was $10M."],
+                "ground_truth": "$10M",
+            }
+        ],
+        {
+            "metrics": ["faithfulness", "contextual_precision"],
+            "judge_model": "gpt-4o-mini",
+            "include_reason": True,
+        },
+    )
+
+    assert output["scores"] == {"faithfulness": 1.0, "context_precision": 1.0}
+    assert output["per_question"][0]["query"] == "What is revenue?"
+    assert output["per_question"][0]["context_precision_reason"] == "grounded"
+
+
+def test_deepeval_evaluator_rejects_unknown_metric(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """Unknown metric names fail fast before any judge calls are made."""
+    _install_fake_deepeval(monkeypatch)
+
+    from tools.evaluation.deepeval_eval import DeepEvalEvaluator
+
+    with pytest.raises(ValueError, match="unknown deepeval metrics"):
+        DeepEvalEvaluator().evaluate([], {"metrics": ["bogus"]})

tests/unit/test_deepeval_synthesizer.py

@@ -0,0 +1,114 @@
+from __future__ import annotations
+
+import sys
+import types
+from dataclasses import dataclass
+from typing import Any
+
+import pytest
+
+from xrag.config.models import ProviderConfig
+
+
+@dataclass
+class _Golden:
+    input: str
+    expected_output: str
+    source_file: str
+
+
+class _FakeSynthesizer:
+    last_kwargs: dict[str, Any] = {}
+    last_contexts: list[list[str]] = []
+
+    def __init__(self, **kwargs: Any) -> None:
+        type(self).last_kwargs = kwargs
+
+    def generate_goldens_from_contexts(self, **kwargs: Any) -> list[_Golden]:
+        type(self).last_contexts = kwargs["contexts"]
+        return [
+            _Golden(
+                input="What was reported?",
+                expected_output="Revenue was $10M.",
+                source_file="doc:chunk:1",
+            )
+        ]
+
+
+def _install_fake_deepeval_synthesizer(monkeypatch: pytest.MonkeyPatch) -> None:
+    deepeval = types.ModuleType("deepeval")
+    synthesizer_mod = types.ModuleType("deepeval.synthesizer")
+    synthesizer_mod.Synthesizer = _FakeSynthesizer
+    monkeypatch.setitem(sys.modules, "deepeval", deepeval)
+    monkeypatch.setitem(sys.modules, "deepeval.synthesizer", synthesizer_mod)
+    sys.modules.pop("tools.dataset_generation.deepeval_synthesizer", None)
+
+
+def test_build_dataset_generator_returns_deepeval_provider(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """Dataset generator factory exposes DeepEval synthesis."""
+    _install_fake_deepeval_synthesizer(monkeypatch)
+
+    from tools.dataset_generation.deepeval_synthesizer import (
+        DeepEvalSynthesizerDatasetGenerator,
+    )
+    from tools.dataset_generation.factory import build_dataset_generator
+
+    generator = build_dataset_generator(ProviderConfig(provider="deepeval", options={}))
+
+    assert isinstance(generator, DeepEvalSynthesizerDatasetGenerator)
+
+
+def test_deepeval_synthesizer_maps_goldens_to_testset_schema(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """DeepEval goldens are converted into xrag testset rows with pinned evidence."""
+    _install_fake_deepeval_synthesizer(monkeypatch)
+
+    from tools.dataset_generation.deepeval_synthesizer import (
+        DeepEvalSynthesizerDatasetGenerator,
+    )
+
+    chunks = [
+        {
+            "chunk_id": "doc:chunk:1",
+            "doc_id": "doc",
+            "chunk_strategy": "section_table",
+            "content_with_weight": "Revenue was $10M. " * 10,
+            "source_element_ids": ["el1"],
+            "page_number": 3,
+        }
+    ]
+    rows = DeepEvalSynthesizerDatasetGenerator().run(
+        chunks,
+        {
+            "model": "gpt-4o-mini",
+            "limit_chunks": 1,
+            "min_context_chars": 10,
+            "max_goldens_per_context": 1,
+        },
+    )
+
+    assert _FakeSynthesizer.last_kwargs["model"] == "gpt-4o-mini"
+    assert _FakeSynthesizer.last_contexts == [[chunks[0]["content_with_weight"].strip()]]
+    assert rows == [
+        {
+            "example_id": "doc:qa:0001",
+            "question": "What was reported?",
+            "answer": "Revenue was $10M.",
+            "answer_type": "string",
+            "reasoning_type": "deepeval_synthetic",
+            "difficulty": "medium",
+            "doc_id": "doc",
+            "chunk_strategy": "section_table",
+            "evidence_chunk_ids": ["doc:chunk:1"],
+            "evidence_element_ids": ["el1"],
+            "page_numbers": [3],
+            "metadata": {
+                "generator_provider": "deepeval",
+                "category": "deepeval_synthetic",
+                "validation_status": "valid",
+            },
+        }
+    ]