Proposal: rag-eval extension — automated RAG pipeline evaluation on top of Langfuse

[Aymenec-212]

Hello everyone! I'm a junior data scientist / ML engineer and I've been looking at ODS with an eye toward contributing something in my lane. Before writing any code I wanted to check whether this is wanted and, if so, what shape you'd want it in.

The gap

ODS ships a full RAG path (Qdrant + TEI embeddings + Open WebUI) and Langfuse for observability, but there's currently no way to answer "is my retrieval actually any good?" Tracing tells you what happened; it doesn't tell you whether the right chunks came back or whether the answer was grounded in them. For a self-hosted stack, I think this matters more than in a hosted one, when someone swaps the embedding model, changes chunk size, or moves to a different hardware tier, they have no signal on whether retrieval quality just got better or worse.

Proposed shape

An opt-in extension under extensions/services/rag-eval/ (manifest + compose), running a Python async worker. No new UI to start, it reads traces from Langfuse and writes results back as Langfuse scores, so everything shows up where operators already look.

Two modes, because the metrics split cleanly:

1. Online (trace-driven, reference-free), runs continuously over live traces:

• Groundedness / faithfulness, context relevance, answer relevance
• Cheap non-LLM signals: retrieval score distribution, chunk redundancy, context utilization rate, empty-retrieval rate

2. Offline (golden set, reference-based), batch run, triggered manually or on config change:

• MRR, recall@k, hit rate, nDCG@k, context precision/recall
• Golden set generated from the user's own indexed documents, then editable, so no external dataset is required

This is deliberately not LLM-as-judge-only. Judge quality collapses on lower hardware tiers, and IR metrics give a stable, cheap, deterministic signal that works even on the CPU fallback path.

Thanks for building this.