fix(scorer): mark model_graded grade-parse failure as unscored (not INCORRECT)

[vladmesh]

This PR contains:

• New features
• Changes to dev-tools e.g. CI config / github tooling
• Docs
• Bug fixes
• Code refactor

What is the current behavior? (You can also link to an open issue here)

Closes #4026.

When the judge's output does not match DEFAULT_GRADE_PATTERN (or a user-supplied grade_pattern), the else branch in model_graded_qa / model_graded_fact (src/inspect_ai/scorer/_model.py) returns Score(value=INCORRECT, ...). This conflates "the judge said the answer was incorrect" with "we couldn't extract a verdict from the judge", making the two indistinguishable in the score distribution. When small judge models emit variants like GRID: C, ANSWER: C or **Answer: C** instead of GRADE: C, those samples are silently scored INCORRECT even when the judge clearly decided the answer was correct, so accuracy is understated with no signal that anything went wrong.

What is the new behavior?

The parse-failure branch now returns Score.unscored(...) instead of Score(value=INCORRECT, ...), tagged with metadata["unscored_reason"] = "grade_parse_failure".

This follows the layering distinction @MattFisher raised in the issue and PR threads. pattern() and math() score the output of the Model Under Test, so NOANSWER is the right sentinel there. model_graded_* is different: the grader model is scoring instrumentation, not the thing under test. A grade-parse failure is the instrument failing, not the subject declining to answer, so recording it as NOANSWER would still attribute a "no answer" to the subject that it may not have produced. Score.unscored() (NaN) is the framework's existing sentinel for "a scorer cannot produce a value for this sample": the sample is dropped from aggregate metrics and surfaced separately as an unscored sample, with unscored_reason available for downstream filtering and for reporting the judge-parse-failure rate.

This supersedes the original approach in this PR, which returned NOANSWER. See the #4026 thread for the discussion that moved it to unscored().

Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)

Yes, a metric-level one. Previously judge-parse failures were counted as INCORRECT (mapped to 0.0 by value_to_float), so they stayed in the denominator. They are now unscored (NaN), excluded from aggregate metrics and reported separately as unscored samples. Evals that previously hit judge-parse failures will see those samples leave the denominator, so reported accuracy can move (typically up, since genuinely-correct-but-unparseable grades are no longer counted as wrong). Any code that inspects the raw Score.value for these samples needs to handle NaN rather than "I".

Tests

• tests/scorer/test_model_graded.py::test_grade_parse_failure_is_unscored — parametrised over four representative malformed grader outputs (GRID: C, ANSWER: C, **Answer: C**, and a completion with no grade marker); asserts Score.value is NaN and metadata["unscored_reason"] == "grade_parse_failure".
• tests/scorer/test_model_graded.py::test_matched_grade_resolves_to_value — symmetry check that a parseable GRADE: C / GRADE: I / GRADE: P still resolves to CORRECT / INCORRECT / PARTIAL, so the matched-grade branch is undisturbed.
• test_model_graded_answer_set_on_grade_parse_failure (from model_graded_fact: answer field empty when judge GRADE parsing fails despite correct completion #4025) updated to assert the parse-failure sample is unscored (NaN) while Score.answer still carries the subject completion.

Other information:

• math() in _math.py has a sibling parse-failure case, handled separately in fix(scorer): tag math() answer-extraction failures with reason metadata (#4026) #4091. That one stays NOANSWER, since it scores the Model Under Test's own output, per the same layering distinction.
• @MattFisher also suggested surfacing the judge-failure rate more prominently (an analogue of --score-on-error). That is orthogonal to this fix and could be a follow-up; unscored_reason gives it something to key on.
• A companion observability fix (Score.answer populated in this same branch) is tracked in model_graded_fact: answer field empty when judge GRADE parsing fails despite correct completion #4025 / fix(scorer): set Score.answer on model_graded grade-parse failure #4039.

[avalyset]

Change looks right to me. Returning NOANSWER from the parse-failure branch lines up with the existing pattern() convention in _pattern.py:101, and the distinction between "judge couldn't be parsed" and "judge said incorrect" is exactly what the aggregate needs to stay visible — losing it silently understates accuracy in a way that's hard to notice downstream.

One non-blocking observation on the test side: the parse-failure path is well covered by the four parametrised cases, but there's no companion assertion that a valid GRADE: I still resolves to INCORRECT. Adding that would make the suite symmetric and catch any future drift in the matched-grade branch. Happy either way — fine to leave as-is.

[avalyset]

Thanks for the parametrized test cases and the symmetry assertion — that closes the observability gap cleanly. Good call flagging _math.py for a follow-up; the pattern is identical and antnewman's audit on #4026 backs that scope. This looks ready from my end.

[MattFisher]

We're currently addressing similar circumstances in inspect_evals, and I'd suggest this could be a good place to use Score.unscored().

Use this when a scorer cannot produce a value for a sample but you still want to record context (answer, explanation, metadata).

[avalyset]

The unscored() suggestion is worth weighing carefully here, because it and NOANSWER resolve two different situations and the distinction matters for what the metric reports.

unscored() sets value to NaN, which is filtered out upstream (in _eval/task/results.py, before the metric runs) and reported separately as unscored_samples — so it's excluded from both numerator and denominator. NOANSWER maps to 0 in value_to_float and stays in the denominator as a counted, non-answered outcome.

For a grader that fails to emit a parseable verdict, that difference is the whole question: unscored() removes the observation from the aggregate, which hides the grader-failure rate; NOANSWER keeps it visible as a counted miss. In a safety-eval setting, surfacing how often the grader itself fails seems like the property you want to preserve, not drop. That's also consistent with _pattern.py, which already returns NOANSWER on a non-match rather than excluding the sample.

unscored() looks like the right tool for the case where no score is meaningfully possible (e.g. _multi.py when all sub-scorers abstain), rather than for a grader parse failure where the sample did get graded and the grader simply returned something unparseable.

[vladmesh]

Thanks @MattFisher. unscored() is the right call when no score is possible. Here I still lean NOANSWER: it keeps the parse-failed sample in the denominator as a non-correct outcome, distinguishable from a real INCORRECT, whereas unscored() drops it from the rate. For a safety eval I'd rather a judge-parse-failure rate stay visible than have samples quietly leave the denominator. It also keeps the scorers aligned: pattern() already returns NOANSWER _pattern.py:101) and #4091 did the same for math(). Also, this doesn't move headline accuracy, since value_to_float maps both to 0.0; the win is they stop being indistinguishable downstream.
I'd really love to hear the maintainers' take here

[MattFisher]

We need to be a bit careful here - the pattern scorer returns NO_ANSWER when the output of the Model Under Test (MUT) doesn't conform to the requested pattern, whereas this PR is dealing with the output of the Grader Model not conforming to the requested pattern.

In this case, the Grader model is effectively part of the scoring instrumentation, and a failure by the instrumentation should not be recorded as "No answer" from the MUT - the MUT may have returned a correct or incorrect answer or no answer at all, but the scoring instrumentation failed.

4091's proposed change to math() is also around the MUT output, not the grader (and is not yet accepted).

If the crucial use case here is the judge failure rate not being sufficiently obvious when unscored is used, perhaps we can find a way to surface that information more prominently? E.g. the --score-on-error option changes how samples that raise errors are scored, maybe something analogous could be done for judge failures?

[MattFisher]

For reference, we're doing a bunch of analysis on this issue for inspect_evals that was prompted by a number of PRs that had different opinions - (#1681 swe_bench, #1679 kernelbench/osworld, #1655 vimgolf/agentharm, #1671 cybergym).

[vladmesh]

Thanks @MattFisher, agreed. Pushed a variant Score.unscored() tagged with unscored_reason="grade_parse_failure", so it leaves the rate but stays visible. Heads up that this shifts the accuracy denominator vs today.