As a developer, I want to collect production feedback so that benchmark blind spots are complemented by real usage data

[kiyotis]

Situation

The current benchmark covers automated evaluation during development and initial calibration (full manual verification). Two layers from the best-practice multi-layer quality monitoring model remain unimplemented (see benchmark-review.md):

• Periodic human sampling: expert review of representative samples (monthly/quarterly)
• Production feedback collection: collecting user ratings and failure reports

Additionally, a statistical caveat exists: with a sample size of 67–93 OK-rated cases per metric, the confidence interval for the false negative rate is wide. The claim "false negatives are near zero" cannot be made with statistical confidence at this scale.

Pain

There is no mechanism to complement the benchmark with real production quality signals. Quality degradation and unhandled use cases after pilot release cannot be detected. The statistical foundation of the false negative rate claim cannot be strengthened by benchmark expansion alone.

Benefit

• Real-world quality issues not caught by automated evaluation become visible
• The benchmark and production feedback together provide a more complete picture of nabledge-6 quality

Success Criteria

• Quality issues surfaced by production feedback or human sampling can be traced back to specific benchmark scenarios or gaps, enabling targeted improvement
• The false negative rate estimate is supported by a larger evidence base (automated benchmark + production data combined)

🤖 Generated with Claude Code