Skip to content

Consistency checker crashes with ZeroDivisionError on empty denominators #1629

Description

@Abelo9996

Describe the bug

Two consistency checks in src/modelbench/consistency_checker.py divide by a count that can be zero. When it is zero the check raises a ZeroDivisionError on exactly the degenerate or failed run it exists to flag. The exception propagates out of ConsistencyChecker.run() (no per-check guard); the CLI wrapper run_consistency_check then catches it per journal, so the tool does not crash, but that journal's entire consistency check is abandoned and surfaced as an opaque Error running consistency check float division by zero instead of the actual findings, and the run cannot be validated.

  1. MinValidAnnotatorItems.check (line 250):

    return float(self.num_invalid_annotations) / self.num_translated_annotations < self.INVALID_THRESHOLD

    num_translated_annotations is len(query("translated annotation", sut, test, annotator)), which is 0 when an annotator produced no translated annotations for a test and sut, for example when that annotator's API was down for the whole run (every attempt is logged as an exception and no translated annotation entry is written). ConsistencyChecker.run iterates every annotator the test declares (for annotator in self.annotators, where self.annotators = test_obj.get_annotators()), and run_checks_for_row calls check.check() with no guard, so the exception propagates out of ConsistencyChecker.run(). The CLI catches it per journal (except Exception: print("Error running consistency check", e)), which abandons that journal's whole check and reports a generic error rather than the specific finding.

  2. HazardScoreIsFracSafe._get_frac_safe (line 376):

    return float(num_safe) / num_scored

    num_scored is 0 when there are no measured item quality entries for the sut across the hazard's tests. This runs in __init__ (line 352), so constructing the check crashes.

Net effect: the tool whose job is to flag broken runs cannot validate exactly the broken run, and hides the specific problem (an annotator produced nothing) behind a generic Error running consistency check float division by zero.

To reproduce

Verified on modelbench 1.0.0. Pure logic, no model call:

import json, tempfile
from pathlib import Path
from modelbench.consistency_checker import JournalSearch, MinValidAnnotatorItems

entries = [
    {"message": "translated annotation", "sut": "S", "test": "T",
     "annotator": "A", "annotation": {"is_valid": True}},
    # no "translated annotation" entries for annotator "B" (its API was down all run)
]
jp = Path(tempfile.mkdtemp()) / "journal.jsonl"
jp.write_text("\n".join(json.dumps(e) for e in entries) + "\n")

search = JournalSearch(jp)
MinValidAnnotatorItems(search, sut="S", test="T", annotator="B").check()
# ZeroDivisionError: float division by zero

Expected behavior

An annotator that produced zero translated annotations (or a hazard with nothing scored) is itself a failed run: the check should fail and be reported as a warning, not crash the checker.

Suggested fix

Guard both empty denominators. For MinValidAnnotatorItems.check, return False (with a clear failure_message) when num_translated_annotations == 0. For HazardScoreIsFracSafe._get_frac_safe, return nan (so the equality check fails) when num_scored == 0. This mirrors the empty-measurement guard already in flight in #1608. Happy to open a PR with this plus tests.

Version

modelbench 1.0.0.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions