Skip to content

fix(consistency-checker): guard empty denominators - #1630

Open
Abelo9996 wants to merge 1 commit into
mlcommons:mainfrom
Abelo9996:fix-consistency-checker-empty-denominators
Open

fix(consistency-checker): guard empty denominators#1630
Abelo9996 wants to merge 1 commit into
mlcommons:mainfrom
Abelo9996:fix-consistency-checker-empty-denominators

Conversation

@Abelo9996

@Abelo9996 Abelo9996 commented Aug 29, 2026

Copy link
Copy Markdown

What

Guard two empty-denominator divisions in the consistency checker that raise a ZeroDivisionError on a failed run. The exception propagates out of ConsistencyChecker.run(), and the CLI catches it per journal, so the tool does not crash, but that journal's entire consistency check is abandoned and reported as an opaque Error running consistency check float division by zero instead of the actual findings.

Why

The consistency checker exists to flag broken runs, but two checks crash on exactly those runs:

  • MinValidAnnotatorItems.check divides by num_translated_annotations, which is 0 when an annotator produced no translated annotations (for example, its API was down for the whole run). run_checks_for_row calls check.check() with no guard, and the runner iterates every annotator the test declares, so the exception propagates out of ConsistencyChecker.run(); the CLI's run_consistency_check catches it per journal, abandoning that journal's whole check and reporting a generic error instead of the specific finding.
  • HazardScoreIsFracSafe._get_frac_safe divides by num_scored, which is 0 when no items were measured for a hazard; this runs in __init__, so constructing the check crashes.

Details and a standalone repro are in #1629.

Change

  • MinValidAnnotatorItems.check returns False (with a clear failure message) when there are no translated annotations, flagging the annotator instead of crashing.
  • HazardScoreIsFracSafe._get_frac_safe returns nan when nothing was measured, so the equality check fails rather than crashing.
  • Regression tests for both.

This mirrors the empty-denominator guard already in flight in #1608.

Fixes #1629

Two consistency checks divided by a count that can be zero, so the checker
raised an unhandled ZeroDivisionError and aborted on exactly the failed run it
exists to flag:

- MinValidAnnotatorItems.check divided by num_translated_annotations, which is 0
  when an annotator produced no translated annotations (e.g. its API was down for
  the whole run). It now returns False (with a clear failure message) in that
  case, flagging the annotator instead of crashing.
- HazardScoreIsFracSafe._get_frac_safe divided by num_scored, which is 0 when no
  items were measured for a hazard. It now returns nan so the equality check
  fails rather than crashing.

Add regression tests for both.

Fixes mlcommons#1629
@Abelo9996
Abelo9996 requested a review from a team as a code owner August 29, 2026 23:47
@github-actions

Copy link
Copy Markdown

MLCommons CLA bot:
Thank you very much for your submission; we really appreciate it. Before we can accept your contribution,
we ask that you sign the MLCommons CLA (Apache 2). Please submit your GitHub ID to our onboarding form to initiate
authorization. If you are from a MLCommons member organization, we will request that you be added to the CLA.
If you are not from a member organization, we will email you a CLA to sign. For any questions, please contact
support@mlcommons.org.
0 out of 1 committers have signed the MLCommons CLA.
@Abelo9996
You can retrigger this bot by commenting recheck in this Pull Request

@wpietri

wpietri commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Thanks for taking an interest, @Abelo9996. Is this a problem you ran into while using ModelBench? If so, I'd like to know more about the circumstances. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Consistency checker crashes with ZeroDivisionError on empty denominators

2 participants