Evidence-driven RAG optimization for question answering

[subbaksh]

Problem Statement

CAIPE has a RAG system that supports document ingestion, hybrid search, vector storage, structured entity ingestion, graph storage, ontology generation, and MCP tools for search and graph exploration.

The open question is how to improve question-answering quality while controlling latency and model cost.

The project should evaluate RAG behavior across three dimensions:

• Accuracy: Are the retrieved sources relevant, complete, and sufficient to answer the question?
• Latency: How long do retrieval, reranking, graph traversal, and generation take?
• Cost: What is the cost of embeddings, reranking, generation, memory extraction, graph construction, and storage?

The goal is not to assume that any specific technique is better. The goal is to measure the current system, explore targeted improvements, and recommend which approaches are worth productizing.

Current System and Capabilities

Relevant repository areas:

• ai_platform_engineering/knowledge_bases/rag/server: query APIs, MCP tools, and hybrid search orchestration.
• ai_platform_engineering/knowledge_bases/rag/common: shared models, embedding factory, graph abstractions, and ingestor APIs.
• ai_platform_engineering/knowledge_bases/rag/ingestors: source-specific ingestion into LangChain Document objects and structured entities.
• ai_platform_engineering/knowledge_bases/rag/agent_ontology: ontology generation, relation evaluation, and BM25 entity search.

Current capabilities include:

• Text ingestion from multiple data sources.
• Dense retrieval through Milvus vector search.
• Sparse retrieval through BM25-style keyword search.
• Hybrid retrieval with configurable dense/sparse weighting.
• Structured data ingestion into Neo4j.
• Graph exploration through ontology and data graph tools.
• MCP access for search, document fetch, and graph traversal.

Things to Explore

1. RAG Evaluation

Before changing retrieval strategies, create a repeatable way to evaluate the system.

Start with a small golden question set that includes:

• Representative questions.
• Expected answers where available.
• Source-of-truth citations.
• Question categories such as factual lookup, troubleshooting, how-to, dependency, topology, multi-hop, global summary, and visual/diagram-dependent questions.

Evaluate:

• Retrieval quality: Recall@k, Precision@k, MRR, nDCG@k, context precision, context recall.
• Answer quality: correctness, faithfulness, relevance, citation quality, refusal quality.
• Operational behavior: p50/p95 latency, token usage, embedding cost, generation cost, reranker cost, index size.
• Failure causes: missing data, poor retrieval, ranking issues, generation issues, citation issues.

Frameworks to start with:

• In-repo eval runner: a lightweight Python runner that reads JSONL/CSV questions, calls the RAG endpoint or MCP tools, stores retrieved contexts and generated answers, and writes comparable run reports.
• Ragas: useful for context precision, context recall, faithfulness, answer relevance, and multimodal relevance.
• DeepEval: useful for test-style contextual precision, contextual recall, contextual relevancy, answer relevancy, and faithfulness checks.

The project should evaluate Ragas and DeepEval against the same sample questions and recommend one framework for CAIPE to standardize on.

Stretch:

• CAIPE Admin UI extension: explore whether the evaluation framework could be surfaced in the admin experience.
  • Operators/admins should be able to launch a one-time evaluation of their current search settings against a curated dataset.
  • Operators/admins should be able to compare evaluation runs after changing search weights, embedding models, chunking, or tool settings.
  • Operators/admins should be able to see a live quality view based on user feedback, trace scores, and evaluation results.
  • This should be treated as a stretch PoC, not a required production UI.

2. Retrieval, Agent Tooling, and Multimodal Optimization

The current system already supports hybrid dense plus sparse search and MCP tools for search, document fetch, and graph traversal. This track should evaluate not only what gets retrieved, but also how AI agents use retrieved evidence and tools to produce grounded answers.

Explore:

• Hybrid retrieval tuning: dense/sparse weighting, top-k values, reranker parameters, chunking strategy, metadata filters, result handling, and BM25 tokenization.
• Embedding model comparison: compare available text embedding model families by accuracy, latency, cost, and deployability.
• Agent tool-use optimization: evaluate how agents use search, fetch_document, graph tools, filters, citations, and follow-up fetches. Check which LLM models use the tools better.
• Snippet versus fetch behavior: determine whether search snippets provide enough context or whether agents need better fetch patterns.
• Evidence packaging: evaluate how retrieved text, metadata, citations, image references, and graph context should be passed to the final answering model.
• Multimodal retrieval: evaluate Google multimodal embeddings for image/text retrieval, with OCR and captioning as baselines for other model families.
• Multimodal reasoning: explore adding retrieved images directly into the LLM context so the model can reason over the original visual evidence, not only OCR or captions.
• Audio/video as stretch: transcribe audio, extract video transcripts, sample keyframes, and link generated text or visual evidence back to the original media.

Embedding models to compare should stay within the model families we have access to:

• Current configured embedding model as the baseline.
• Azure/OpenAI text embeddings.
• Google text embedding models.
• Free or OSS local embeddings.
• Ollama/local embeddings where local-first operation matters.

3. Graph Memory

Explore whether CAIPE should learn durable knowledge from interactions and store it in an evolving graph memory.

This is different from retrieval over a static documentation corpus. The memory graph would be built from things the system observes over time, such as conversations, tool calls, incident investigations, troubleshooting sessions, feedback, structured operational events, and Langfuse traces.

CAIPE already has a lot of Langfuse traces, and those traces are a strong candidate event source for a live knowledge graph. A trace can capture the user/session, input, output, observations, tool calls, model calls, latency, costs, scores, errors, and metadata. That makes it possible to learn not just facts, but also how work actually happened.

Explore as an evaluation/PoC, not a production implementation:

• CAIPE integration evaluation: identifying how graph memory could fit with the current RAG stack, structured graph data, Langfuse traces, and MCP tools.
• Langfuse trace-to-knowledge-graph prototype: taking a small set of Langfuse traces and generating a knowledge graph from them.
• Trace ingestion options: comparing Langfuse API access, webhook flow, and scheduled incremental pulls.
• Memory extraction: deciding what facts, entities, relationships, events, and procedures should be written to memory.
• Temporal memory: tracking when facts were learned, when they were true, when they changed, and what evidence supports them.
• Memory retrieval: combining semantic search, keyword search, graph traversal, timestamps, and provenance.
• Production considerations: documenting what conflict handling, access control, freshness, and governance would require if this moved beyond PoC.

For the Langfuse trace-to-knowledge-graph prototype, explore:

• Trace-to-episode mapping: treat each trace, span, generation, tool call, and score as an episode that may produce memory.
• Entity extraction: identify users, teams, agents, tools, services, repositories, tickets, incidents, clusters, namespaces, deployments, and knowledge-base documents.
• Relationship extraction: capture relations such as asked_about, used_tool, called_agent, resolved_by, failed_on, depends_on, mentions, cites, and followed_up_from.
• Provenance: store links back to the Langfuse trace ID and observation ID for every extracted fact.
• Freshness: use trace timestamps and simple watermarks to show how the graph could be incrementally updated.
• Serving path sketch: show how graph memory could eventually be exposed through MCP tools so agents can ask what was learned from prior interactions.

State-of-the-art systems to explore:

• Graphiti: open-source temporal knowledge graph memory for AI agents, built around incremental updates, changing facts, provenance, and hybrid graph/vector/full-text retrieval. Graphiti is the open-source engine behind Zep's managed context graph platform.
• Mem0 Graph Memory: open-source graph-backed memory that extracts entities and relationships from memory writes and combines graph context with vector retrieval.
• Cognee: open-source memory platform that turns documents, tickets, transcripts, and other context into graph-plus-vector memory for agents.

Expected Deliverables

The project should produce:

• Evaluation framework recommendation and baseline: representative question set, answer labels where available, source links, question tags, repeatable evaluation runner, Ragas versus DeepEval recommendation, baseline report, and failure-mode breakdown. (Report/Recommendation)
• Retrieval, agent tooling, and multimodal optimization report: hybrid RAG settings, embedding model comparison, BM25/tokenization findings, agent tool-use recommendations, image-to-LLM reasoning findings, multimodal findings, audio/video stretch findings, latency/cost tradeoffs, and recommended retrieval configuration. (Report/Recommendation + Configuration)
• Graph memory PoC and CAIPE integration evaluation: assessment of how trace-derived graph memory could fit with CAIPE's RAG, Neo4j, Langfuse, and MCP architecture, plus a prototype that converts a small set of Langfuse traces into a knowledge graph with trace-backed provenance. (Python PoC)

[subbaksh]

Architecture diagram example that needs embedding: https://help.splunk.com/en/splunk-cloud-platform/get-started/splunk-validated-architectures/splunk-cloud-platform/splunk-cloud-platform-experiences