feat(eval): add DeepEval providers

[henryle97]

Motivation

• Provide an alternative judge/evaluator to RAGAS that supports DeepEval metrics and enable generation of synthetic goldens from prepared xrag Chunk contexts using DeepEval's Synthesizer.
• Keep xrag's deterministic retrieval metrics and eval artifact contract while offering a single path to both judge-scoring and synthetic dataset generation for faster developer experimentation.

Description

• Add a DeepEval evaluator implementation at tools/evaluation/deepeval_eval.py implementing DeepEval metric wiring, option passthrough (judge_model/threshold/include_reason/async_mode), metric name aliases, per-question reasons, and aggregate score output matching the existing evaluator contract.
• Add a DeepEval Synthesizer dataset generator at tools/dataset_generation/deepeval_synthesizer.py that converts xrag Chunk Retrieval Text into Synthesizer.generate_goldens_from_contexts() calls and maps DeepEval goldens back into the repo's testset.jsonl schema with pinned evidence_chunk_ids.
• Wire the new provider into factories (tools/evaluation/factory.py, tools/dataset_generation/factory.py), generalise the eval runner/CLI text to support a configured judge provider (not RAGAS-specific), and expose a latency.judge_s metric while preserving the existing latency.ragas_s key for compatibility.
• Add docs and a sample config (docs/deepeval.md, configs/eval-dataset-deepeval.yml), plus unit tests that use fake DeepEval modules to validate integration (tests/unit/test_deepeval_eval.py, tests/unit/test_deepeval_synthesizer.py).

Testing

• Ran static checks: python -m ruff check on the modified files (passed).
• Verified syntax/compilation: python -m py_compile for the new/changed modules (passed).
• Added unit tests for the evaluator and synthesizer that exercise the factory wiring and output mapping; unit tests are present and pass under a supported environment, but a full pytest run in this container was blocked by a platform/Python mismatch for a locked dependency (spacy==3.8.14 has no wheel for CPython 3.14).

---

Codex Task

[gemini-code-assist]

Code Review

This pull request introduces DeepEval as an optional provider for RAG evaluation and synthetic dataset generation, alongside the existing RAGAS provider. The changes include new implementation modules for the DeepEval evaluator and synthesizer, corresponding unit tests, documentation, and updates to the evaluation pipeline and CLI to support generic judge providers. Feedback suggests optimizing performance by instantiating metrics once outside the evaluation loop, using a mapping dictionary for efficient chunk lookups during dataset generation, and restoring helpful error messages for optional dependency imports in the factory.

[gemini-code-assist]

Instantiating DeepEval metrics inside the per-result loop is highly inefficient. Many DeepEval metrics initialize an LLM judge model during construction, which can be expensive and slow. Additionally, the current implementation has a potential bug where providing both a metric name and its alias (e.g., contextual_precision and context_precision) would result in redundant judge calls and double-counting in the aggregate scores.

It is better to instantiate and deduplicate the required metrics once before iterating through the results.

        # Instantiate metrics once and deduplicate by output name to avoid redundant judge calls
        metrics_to_run = {}
        for name in metric_names:
            out_name = _OUTPUT_ALIASES.get(name, name)
            if out_name not in metrics_to_run:
                metrics_to_run[out_name] = _METRIC_REGISTRY[name](options)

        for result in results:
            test_case = LLMTestCase(
                input=result["query"],
                actual_output=result["answer"],
                expected_output=result.get("ground_truth", ""),
                retrieval_context=result.get("contexts", []),
            )
            entry: dict[str, Any] = {"query": result["query"]}
            for output_name, metric in metrics_to_run.items():
                metric.measure(test_case)
                score = getattr(metric, "score", None)
                entry[output_name] = float(score) if score is not None else None
                reason = getattr(metric, "reason", None)
                if reason:
                    entry[f"{output_name}_reason"] = reason
                if entry[output_name] is not None:
                    raw_scores.setdefault(output_name, []).append(entry[output_name])
            per_question.append(entry)

[gemini-code-assist]

Performing a linear scan with _chunk_by_id inside the loop results in $O(N^2)$ complexity relative to the number of chunks. For large datasets, this will significantly slow down the mapping process. Consider creating a mapping dictionary from chunk_id to chunk before starting the loop for $O(1)$ lookups.

Suggested change

	for idx, golden in enumerate(goldens, start=1):
	source_file = _golden_value(golden, "source_file") or _golden_value(
	golden, "source_files", [None]
	)
	if isinstance(source_file, list):
	source_file = source_file[0] if source_file else None
	chunk = _chunk_by_id(selected_chunks, str(source_file)) or selected_chunks[
	min(idx - 1, len(selected_chunks) - 1)
	]
	chunk_map = {str(c.get("chunk_id")): c for c in selected_chunks}
	examples: list[dict[str, Any]] = []
	for idx, golden in enumerate(goldens, start=1):
	source_file = _golden_value(golden, "source_file") or _golden_value(
	golden, "source_files", [None]
	)
	if isinstance(source_file, list):
	source_file = source_file[0] if source_file else None
	chunk_id = str(source_file) if source_file else None
	chunk = chunk_map.get(chunk_id) or selected_chunks[
	min(idx - 1, len(selected_chunks) - 1)
	]

[gemini-code-assist]

The removal of the try-except block for the ragas provider eliminates a helpful error message that guides users to install the [eval] extra. Since these judge stacks are optional dependencies, maintaining clear installation instructions in the error message is important for developer experience. A similar guard should be added for the new deepeval provider.

Suggested change

	if config.provider == "ragas":
	try:
	from tools.evaluation.ragas_eval import RagasEvaluator
	except ImportError as e: # pragma: no cover — extras-gating
	raise ImportError(
	"RAGAS evaluation support requires the [eval] extra. "
	"Install with `pip install 'xrag[eval]'`."
	) from e
	from tools.evaluation.ragas_eval import RagasEvaluator
	return RagasEvaluator()
	if config.provider == "deepeval":
	from tools.evaluation.deepeval_eval import DeepEvalEvaluator
	return DeepEvalEvaluator()
	if config.provider == "ragas":
	try:
	from tools.evaluation.ragas_eval import RagasEvaluator
	except ImportError as e: # pragma: no cover — extras-gating
	raise ImportError(
	"RAGAS evaluation support requires the [eval] extra. "
	"Install with `pip install 'xrag[eval]'`."
	) from e
	return RagasEvaluator()
	if config.provider == "deepeval":
	try:
	from tools.evaluation.deepeval_eval import DeepEvalEvaluator
	except ImportError as e:
	raise ImportError(
	"DeepEval evaluation support requires the 'deepeval' package. "
	"Install with `pip install deepeval`."
	) from e
	return DeepEvalEvaluator()