Add dataset support for test sets and benchmarks

[rchojn]

Problem

SafeReader needs to run the same test questions repeatedly (regression testing), but there's no structured way to manage test datasets.

Proposed Solution

Add dataset management:

• Define test datasets (e.g., "insurance_qa_v1", "general_knowledge")
• Load dataset and run all questions through models
• Track dataset version history
• Compare results across dataset versions

Competitive Context

• Langfuse: Datasets for evaluation and fine-tuning
• Phoenix: Versioned datasets for experimentation
• OpenAI Evals: YAML-based test definitions

GoEvals approach: Simple JSONL datasets, no database needed.

Implementation

1. Dataset file format (datasets/insurance_qa_v1.jsonl):

{"id": "q1", "question": "What does comprehensive coverage include?", "expected": "Damage from theft, fire, vandalism..."}
{"id": "q2", "question": "What is a deductible?", "expected": "Amount paid before insurance coverage..."}

2. Run dataset through models:

# SafeReader exports results
./goevals datasets/insurance_qa_v1.jsonl evals/gemma2_run1.jsonl

# GoEvals shows:
# - Dataset: insurance_qa_v1 (15 questions)
# - Model: gemma2:2b
# - Avg score: 0.87
# - Questions with low scores: q5, q12

3. Dashboard features:
   • Dataset selector dropdown
   • Per-question breakdown (which questions are hardest?)
   • Regression detection (did scores drop from last run?)

Value for SafeReader

• Regression testing: Ensure new Ollama version doesn't break answers
• Model comparison: Run same 50 questions through gemma2, llama3, qwen
• Progress tracking: Are answers improving over time?
• Quality gates: Block deploy if avg score < 0.85

Example Workflow

# 1. Create test dataset
cat > datasets/insurance_qa_v1.jsonl <<DATASET
{"id": "q1", "question": "What is comprehensive coverage?", "expected": "..."}
{"id": "q2", "question": "What is a deductible?", "expected": "..."}
DATASET

# 2. SafeReader runs dataset through model, exports to JSONL
safereader eval --dataset datasets/insurance_qa_v1.jsonl --output evals/run_2025_10_26.jsonl

# 3. GoEvals visualizes results
./goevals datasets/insurance_qa_v1.jsonl evals/run_2025_10_26.jsonl

Philosophy

• JSONL datasets - no proprietary format
• Stdlib only - load and compare in Go
• Local-first - datasets are local files
• Git-friendly - datasets can be versioned in Git

Success Metrics

• SafeReader can define 50-question dataset in < 10 minutes
• GoEvals loads and compares 5 dataset runs in < 200ms
• Regression detection works without external tools

Labels