Add Disagreement Analysis Dashboard

[rchojn]

Background

With multi-judge evaluation (see #16), we'll have cases where judges disagree significantly on test quality.

High disagreement cases are valuable:

• Identify edge cases the model struggles with
• Find ambiguous questions needing clarification
• Build training data for future judge fine-tuning
• Prioritize human review effort

We need a dashboard view to analyze judge disagreement patterns.

Proposed Solution

Feature 1: Disagreement Heatmap

Show which judges disagree most often:

┌─────────────────────────────────────────────┐
│ Judge Agreement Matrix                      │
│                                             │
│          Bielik  Gemma2  Llama  Qwen        │
│ Bielik     -     0.12    0.18   0.25        │
│ Gemma2    0.12    -      0.09   0.20        │
│ Llama     0.18   0.09     -     0.15        │
│ Qwen      0.25   0.20    0.15    -          │
│                                             │
│ Color: 🟢 <0.1  🟡 0.1-0.2  🔴 >0.2         │
│                                             │
│ Insight: Qwen disagrees most with Bielik    │
└─────────────────────────────────────────────┘

Calculation:

disagreement(judgeA, judgeB) = avg(|scoreA - scoreB|) across all tests

Feature 2: High-Disagreement Filter

Add filter to show only cases needing review:

┌─────────────────────────────────────────────┐
│ Filters                                     │
│ [ ] Show only high disagreement (>0.25)     │
│ [ ] Show only flagged for review            │
│                                             │
│ Results: 12 cases need review               │
└─────────────────────────────────────────────┘

Cases:
┌─────────────────────────────────────────────┐
│ Test: pytanie1_q1                           │
│ Model: llama3.2:3b                          │
│ Judges: [0.9, 0.3, 0.8, 0.7]                │
│ Disagreement: 0.24 (HIGH) ⚠️                │
│                                             │
│ [View Details] [Mark Reviewed] [Export]     │
└─────────────────────────────────────────────┘

Feature 3: Disagreement Statistics

Add summary stats to main dashboard:

┌─────────────────────────────────────────────┐
│ Evaluation Summary                          │
│                                             │
│ Total Tests: 54                             │
│ Avg Score: 0.82                             │
│ Avg Disagreement: 0.12 (LOW) ✅             │
│                                             │
│ Cases needing review: 12 (22%)              │
│ High disagreement (>0.25): 5 cases          │
│ Perfect agreement (<0.05): 28 cases         │
└─────────────────────────────────────────────┘

Implementation Plan

Phase 1: Data Aggregation (0.5 day)

Compute disagreement metrics:

• Per-test disagreement (std_dev of judge scores)
• Judge-pair disagreement matrix
• Overall statistics

Phase 2: Heatmap View (1 day)

New route: /disagreement

Render judge agreement matrix with:

• Color-coded cells
• Tooltips showing sample size
• Insights text (which judges disagree most)

Phase 3: Filter Integration (0.5 day)

Add filters to main table:

• Checkbox: "Show only high disagreement"
• Checkbox: "Show only flagged for review"
• Update table dynamically

Phase 4: Export Feature (0.5 day)

Export disagreement cases:

[Export High Disagreement Cases] → CSV/JSON

Useful for:

• Sharing with team for review
• Building training datasets
• Analysis in external tools

UI Mockup

New Navigation Tab

┌─────────────────────────────────────────────┐
│ [Overview] [Tests] [Disagreement] [About]   │
└─────────────────────────────────────────────┘

Disagreement Page Layout

┌─────────────────────────────────────────────┐
│ Judge Agreement Heatmap                      │
│ [Matrix visualization]                       │
│                                             │
│ High Disagreement Cases (12)                │
│ [Filtered table]                            │
│                                             │
│ [Export Cases (CSV)] [Export (JSON)]        │
└─────────────────────────────────────────────┘

Success Criteria

• Heatmap shows judge agreement patterns
• Filter shows only high-disagreement cases
• Export function generates CSV/JSON
• Statistics show avg disagreement + distribution
• Responsive design (works on laptop screens)

Technical Notes

HTML/CSS for heatmap:

<table class="heatmap">
  <tr>
    <td class="high">0.25</td>  <!-- red -->
    <td class="medium">0.15</td> <!-- yellow -->
    <td class="low">0.08</td>    <!-- green -->
  </tr>
</table>

<style>
.heatmap td.low { background: #10b981; }
.heatmap td.medium { background: #f59e0b; }
.heatmap td.high { background: #ef4444; }
</style>

Time Estimate

Total: 2.5 days

• Phase 1 (Data): 0.5 day
• Phase 2 (Heatmap): 1 day
• Phase 3 (Filters): 0.5 day
• Phase 4 (Export): 0.5 day

Related Issues

• Add Multi-Judge Evaluation Support #16 - Multi-judge support (prerequisite)
• SafeReader rchojn/safereader#15 - Provides multi-judge data

Future Enhancements

• Time-series view (disagreement over time)
• Judge performance tracking (which judge is most accurate)
• Human feedback integration (mark correct judge)

[rchojn]

Depends on #16 (multi-judge support).

Works with SafeReader rchojn/safereader#15 to analyze judge disagreement patterns and identify cases needing human review.