fix(consistency-checker): guard empty denominators - #1630
Conversation
Two consistency checks divided by a count that can be zero, so the checker raised an unhandled ZeroDivisionError and aborted on exactly the failed run it exists to flag: - MinValidAnnotatorItems.check divided by num_translated_annotations, which is 0 when an annotator produced no translated annotations (e.g. its API was down for the whole run). It now returns False (with a clear failure message) in that case, flagging the annotator instead of crashing. - HazardScoreIsFracSafe._get_frac_safe divided by num_scored, which is 0 when no items were measured for a hazard. It now returns nan so the equality check fails rather than crashing. Add regression tests for both. Fixes mlcommons#1629
|
MLCommons CLA bot: |
|
Thanks for taking an interest, @Abelo9996. Is this a problem you ran into while using ModelBench? If so, I'd like to know more about the circumstances. Thanks! |
What
Guard two empty-denominator divisions in the consistency checker that raise a
ZeroDivisionErroron a failed run. The exception propagates out ofConsistencyChecker.run(), and the CLI catches it per journal, so the tool does not crash, but that journal's entire consistency check is abandoned and reported as an opaqueError running consistency check float division by zeroinstead of the actual findings.Why
The consistency checker exists to flag broken runs, but two checks crash on exactly those runs:
MinValidAnnotatorItems.checkdivides bynum_translated_annotations, which is 0 when an annotator produced no translated annotations (for example, its API was down for the whole run).run_checks_for_rowcallscheck.check()with no guard, and the runner iterates every annotator the test declares, so the exception propagates out ofConsistencyChecker.run(); the CLI'srun_consistency_checkcatches it per journal, abandoning that journal's whole check and reporting a generic error instead of the specific finding.HazardScoreIsFracSafe._get_frac_safedivides bynum_scored, which is 0 when no items were measured for a hazard; this runs in__init__, so constructing the check crashes.Details and a standalone repro are in #1629.
Change
MinValidAnnotatorItems.checkreturnsFalse(with a clear failure message) when there are no translated annotations, flagging the annotator instead of crashing.HazardScoreIsFracSafe._get_frac_safereturnsnanwhen nothing was measured, so the equality check fails rather than crashing.This mirrors the empty-denominator guard already in flight in #1608.
Fixes #1629