Add Multi-Judge Evaluation Support

[rchojn]

Background

GoEvals currently displays evaluation results from single-judge LLM-as-a-judge systems.

SafeReader (rchojn/safereader#15) is implementing a Panel of LLMs (PoLL) multi-judge evaluation system where 4 diverse models vote on each test case.

We need to extend GoEvals to visualize multi-judge results, showing:

• Individual judge scores
• Consensus mechanism
• Disagreement detection
• Cases needing human review

Proposed Solution

Extended JSONL Format

Support new judge_scores and consensus_metadata fields:

{
  "timestamp": "2025-12-14T15:30:00Z",
  "model": "llama3.2:3b",
  "scores": {
    "combined": 0.84,
    "factual_correctness": 0.85,
    "faithfulness": 0.90
  },
  "judge_scores": {
    "bielik_11b": {
      "factual": 0.90,
      "faithful": 0.95,
      "weight": 0.4
    },
    "gemma2_9b": {
      "factual": 0.85,
      "faithful": 0.88,
      "weight": 0.3
    },
    "llama_3b": {
      "factual": 0.80,
      "faithful": 0.85,
      "weight": 0.2
    },
    "qwen_7b": {
      "factual": 0.75,
      "faithful": 0.90,
      "weight": 0.1
    }
  },
  "consensus_metadata": {
    "disagreement": 0.06,
    "strategy": "weighted_average",
    "needs_review": false
  }
}

UI Changes

1. Test Modal - Judge Breakdown Section

Add new section showing individual judge scores:

┌─────────────────────────────────────────────┐
│ Consensus Score: 0.84                       │
│ Strategy: Weighted Average                  │
│ Disagreement: 0.06 (LOW) ✅                 │
│                                             │
│ Judge Breakdown:                            │
│ ┌─────────────────────────────────────┐    │
│ │ ✅ Bielik 11b    0.90  (weight: 0.4) │    │
│ │    Factual: 0.90, Faithful: 0.95     │    │
│ │                                      │    │
│ │ ✅ Gemma2 9b     0.85  (weight: 0.3) │    │
│ │    Factual: 0.85, Faithful: 0.88     │    │
│ │                                      │    │
│ │ ✅ Llama3.2 3b   0.80  (weight: 0.2) │    │
│ │    Factual: 0.80, Faithful: 0.85     │    │
│ │                                      │    │
│ │ ⚠️  Qwen2.5 7b    0.75  (weight: 0.1) │    │
│ │    Factual: 0.75, Faithful: 0.90     │    │
│ └─────────────────────────────────────┘    │
└─────────────────────────────────────────────┘

Color coding:

• Green ✅: Score ≥ 0.8
• Yellow ⚠️: Score 0.6-0.8
• Red ❌: Score < 0.6

2. Main Table - Disagreement Column

Add optional column showing disagreement metric:

| Test ID | Model | Score | Disagreement | Status |
|---------|-------|-------|--------------|--------|
| test_01 | llama | 0.84  | 0.06 (LOW)   | ✅     |
| test_02 | llama | 0.72  | 0.24 (HIGH)  | ⚠️     |

Implementation Plan

Phase 1: Backend Support (1 day)

File: main.go

type JudgeScore struct {
    JudgeName string             `json:"judge_name"`
    Scores    map[string]float64 `json:"scores"`
    Weight    float64            `json:"weight"`
}

type ConsensusMetadata struct {
    Disagreement float64 `json:"disagreement"`
    Strategy     string  `json:"strategy"`
    NeedsReview  bool    `json:"needs_review"`
}

type Eval struct {
    // ... existing fields ...
    JudgeScores        map[string]JudgeScore `json:"judge_scores,omitempty"`
    ConsensusMetadata  *ConsensusMetadata    `json:"consensus_metadata,omitempty"`
}

Phase 2: UI Updates (1-2 days)

Template changes:

• Detect judge_scores presence in JSONL
• Render judge breakdown in modal
• Add disagreement column to main table (optional, toggle-able)

Phase 3: Backward Compatibility (0.5 day)

Ensure GoEvals works with both:

• Old format (single judge, no judge_scores)
• New format (multi-judge with judge_scores)

Auto-detect which format and render accordingly.

Success Criteria

• Parse judge_scores and consensus_metadata from JSONL
• Display judge breakdown in test modal
• Show disagreement metric in table
• Backward compatible with existing JSONL files
• Update README with multi-judge example

Screenshots Needed

After implementation:

• Judge breakdown modal view
• Disagreement column in main table
• High-disagreement case example

Time Estimate

Total: 2-3 days

• Phase 1 (Backend): 1 day
• Phase 2 (UI): 1-2 days
• Phase 3 (Compat): 0.5 day

Related Issues

• SafeReader issue rchojn/safereader#15 - Multi-judge arena implementation

[rchojn]

This issue supports SafeReader's multi-judge arena implementation:

• rchojn/safereader#15 - Multi-Judge Arena for RAG Evaluation

SafeReader will output extended JSONL format with judge_scores that GoEvals will visualize.