Add trace visualization for RAG pipelines

[rchojn]

Problem

SafeReader has a complex RAG pipeline (retrieval → embedding → LLM → response), but there's no way to visualize the flow and debug bottlenecks.

Proposed Solution

Add trace visualization to GoEvals dashboard showing:

• Sequential steps in RAG pipeline
• Timing breakdown per step
• Input/output at each stage
• Highlighting bottlenecks (slowest steps)

Competitive Context

• Langfuse: Full distributed tracing with spans
• Phoenix: OpenTelemetry-based instrumentation
• Helicone: Session visualization for agents

GoEvals approach: Lightweight trace visualization from JSONL metadata, no instrumentation library needed.

Implementation

1. Extend JSONL format to support trace metadata:

{
  "timestamp": "2025-10-26T14:30:00Z",
  "model": "gemma2:2b",
  "scores": {"combined": 0.85},
  "trace": {
    "steps": [
      {"name": "retrieval", "duration_ms": 120, "input": "query", "output": "5 chunks"},
      {"name": "embedding", "duration_ms": 45, "input": "5 chunks", "output": "vectors"},
      {"name": "llm_generation", "duration_ms": 1800, "input": "prompt+context", "output": "response"},
      {"name": "total", "duration_ms": 1965}
    ]
  }
}

2. Add trace visualization in UI:
   • Waterfall chart showing sequential steps
   • Percentage breakdown (retrieval: 6%, embedding: 2%, LLM: 92%)
   • Expand/collapse to see input/output

Value for SafeReader

• Debug slow queries (is it retrieval or LLM generation?)
• Optimize bottlenecks (e.g., reduce chunk count if embedding is slow)
• Compare traces across models (gemma2 vs llama3 pipeline timing)

Philosophy

• No instrumentation library - just log metadata to JSONL
• Stdlib only - parse and visualize in Go
• Local-first - no cloud tracing service needed
• Progressive enhancement - works with existing JSONL (trace is optional)

Success Metrics

• SafeReader can identify RAG bottlenecks in < 1 minute
• Trace visualization loads in < 50ms for 1000+ traces
• Zero external dependencies added