As a developer, I want to fix DeepEval false positives structurally so that benchmark NG results are trustworthy

[kiyotis]

Situation

Manual verification of the DeepEval benchmark (34 scenarios × 3 runs = 102 cases, #393) revealed high false positive rates across all three metrics.

• answer_relevancy: 60.9% false positive rate (14/23)
  • Reference-line cause (8 cases): 参照: lines (source metadata appended to answers) judged as irrelevant content
  • Supplemental-explanation cause (6 cases): supplemental explanations in the answer body judged as irrelevant (min score 0.77)
• faithfulness: 94.3% false positive rate (33/35)
  • Omission, paraphrasing, and expression splitting misclassified as contradictions
• answer_correctness: 88.9% false positive rate (8/9)
  • Pattern 1 (6 cases): synonym/code-example references not recognized as covering the fact
  • Pattern 2 (2 cases): OR conditions required to satisfy both sides

Pain

The high false positive rate makes most benchmark NG results require human review, making the operational cost of automated evaluation excessive. The intended workflow — "NG triggers human review" — cannot function at this rate.

Benefit

• Developers can trust benchmark NG results as genuine quality signals
• Operational cost of NG review is significantly reduced
• Threshold appropriateness can be accurately verified using post-fix scores

Success Criteria

• The answer_relevancy false positive rate is lower than before the fix, with the 8 reference-line cases no longer appearing as NG
• The faithfulness false positive rate is lower than before the fix, with omission/paraphrasing cases no longer appearing as NG
• The answer_correctness false positive rate is lower than before the fix, with synonym/code-example and OR-condition cases no longer appearing as NG
• No previously passing cases (true negatives) flip to NG after the changes
• The 1 known faithfulness false negative (run-2-qa-11, score 1.0) remains unaffected

🤖 Generated with Claude Code