As a developer, I want to use a higher-capability model as the eval judge so that self-evaluation bias is eliminated

[kiyotis]

Situation

In the current benchmark setup, the answer-generation LLM (the system under test) and the judge LLM (the evaluator) are both Claude Sonnet 4.6 — a self-evaluation structure. Self-evaluation bias (a model rating its own output more favorably) is a known concern in LLM-as-Judge designs.

In the current results, the direction of error is "too strict (false positives)" rather than "too lenient (false negatives)", so the bias is unlikely to be the dominant factor today. However, an expert review of the benchmark design confirmed that using a higher-capability model as the judge is preferable (see benchmark-review.md).

Pain

The presence and magnitude of self-evaluation bias has not been quantitatively confirmed, leaving a structural question about evaluation reliability that cannot be dismissed without measurement.

Benefit

• The actual impact of self-evaluation bias is quantified
• The decision to change or keep the judge model is grounded in data, not assumption
• The evaluation setup can be moved to a more trustworthy configuration if warranted

Success Criteria

• False positive rates and false negative rates per metric are available for both the current judge model and a higher-capability alternative, enabling direct comparison
• An adoption decision (change / keep) for the judge model is recorded with the comparison data as its rationale

🤖 Generated with Claude Code