Add prompt comparison view (A/B testing)

[rchojn]

Problem

SafeReader uses different prompts for RAG queries, but there's no way to compare which prompts produce better results.

Proposed Solution

Add prompt comparison view:

• Side-by-side comparison of 2+ prompts
• Same test questions, different prompt versions
• Visual diff of responses
• Score comparison (which prompt wins?)

Competitive Context

• Langfuse: Prompt versioning and management
• Phoenix: Interactive playground for prompt testing
• Helicone: Prompt versioning using production data

GoEvals approach: Lightweight A/B comparison from JSONL, no prompt registry needed.

Implementation

1. Extend JSONL to include prompt metadata:

{
  "timestamp": "2025-10-26T14:30:00Z",
  "model": "gemma2:2b",
  "test_id": "eval_001",
  "prompt_version": "v1_concise",
  "question": "What is the capital of France?",
  "response": "The capital of France is Paris.",
  "scores": {"combined": 0.85}
}

2. Add comparison UI:
   • Filter by test_id (same question)
   • Group by prompt_version
   • Show side-by-side: Prompt v1 vs v2 vs v3
   • Highlight score differences

Value for SafeReader

• Optimize prompts: Which prompt template produces best RAG answers?
• A/B testing: Test "concise" vs "detailed" prompt styles
• Regression detection: Did new prompt version degrade quality?

Example Use Case

SafeReader tests 3 prompt versions:

• v1_concise: "Answer briefly using context"
• v2_detailed: "Provide comprehensive answer with sources"
• v3_hybrid: "Balance brevity and detail"

GoEvals shows:

• v1_concise: Avg score 0.78, fast (800ms)
• v2_detailed: Avg score 0.91, slow (2100ms)
• v3_hybrid: Avg score 0.88, medium (1400ms)

Decision: Use v3_hybrid (good scores, acceptable speed)

Philosophy

• No prompt registry - metadata in JSONL is enough
• Stdlib only - comparison logic in Go
• Local-first - no cloud prompt management
• Progressive enhancement - prompt_version is optional field

Success Metrics

• SafeReader can compare 3 prompt versions in < 1 minute
• Comparison view loads in < 100ms for 100+ tests
• Clear visual diff shows which prompt performs better