fix(scorer): tag math() answer-extraction failures with reason metadata (#4026)

[antnewman]

Sister fix to #4048 (model_graded_qa / model_graded_fact), addressing the same silent-failure family flagged in #4026. Per the discussion there, this is the second-PR follow-up I committed to picking up.

Note: this PR originally returned NOANSWER on parse failure. Following the review discussion below with @MattFisher and @dragonstyle, it now implements the emerging convention instead: the value stays INCORRECT and the failure mode is recorded in metadata["reason"]. The title and description have been updated to match.

Problem

math() in src/inspect_ai/scorer/_math.py returns Score(value=INCORRECT, answer="None") when extract_answer() returns None (no parseable mathematical expression in the model output). This conflates:

• "the model answered, and was wrong"
• "the model didn't produce a parseable answer at all"

In analysis the two become indistinguishable: a sample where the model failed to format a math answer is silently recorded as an ordinary wrong answer (with the literal string "None" as its extracted answer), hiding the parse-failure rate entirely.

Fix

On parse failure, keep value=INCORRECT — a format violation is the model failing the task, and must not inflate accuracy by leaving the denominator — and tag the failure mode:

Score(
    value=INCORRECT,
    answer=None,
    explanation=f"No mathematical expression could be extracted from the model output: {completion}",
    metadata={"reason": "invalid_response_format"},
)

This follows @MattFisher's proposed abnormal-score vocabulary in the review discussion (invalid_response_format for model-under-test format violations), consistent with the metadata={"reason": ...} pattern already used in the docs' unscored example and adopted grader-side in UKGovernmentBEIS/inspect_evals#1933.

Also fixes the latent answer="None" artefact: on parse failure answer is now None rather than the stringified None.

Genuine wrong-answer cases — where extract_answer() succeeds but check_answers() returns False — are unchanged: plain INCORRECT, no reason tag.

Tests

• New: test_parse_failure_tags_reason_metadata — parametrised over 4 representative parse-fail completions (no numeric content, words only, empty completion, ambiguous equals); asserts INCORRECT + metadata={"reason": "invalid_response_format"} + answer is None.
• New: test_parse_success_with_wrong_value_has_no_reason — a parseable-but-wrong answer (\boxed{5} vs target 42) stays plain INCORRECT with no metadata.
• Updated: test_infinite_float_does_not_crash — extract_answer returns None for infinity, so the parse-fail branch applies (intent of the test, "no crash on inf", unchanged).

All 43 tests pass locally (Python 3.13, Windows).

Relationship to other work

• model_graded_qa / model_graded_fact silently default to INCORRECT when the judge's verdict doesn't match the grade regex #4026 (the issue): vladmesh's analysis covers both _model.py and _math.py as the two clear instances of this silent-failure pattern.
• fix(scorer): mark model_graded grade-parse failure as unscored (not INCORRECT) #4048 (vladmesh's PR, merged, shipped in 0.3.245): the model-graded scorers route grader-parse failures to Score.unscored() with metadata["unscored_reason"] — a different key than this PR; see the review discussion for the standardisation question.
• fix(scorers): route grader-instrument failures to unscored() (#1535) inspect_evals#1933: adopts metadata["reason"] with the grader-side vocabulary.

This PR contains:

• New features
• Changes to dev-tools e.g. CI config / github tooling
• Docs
• Bug fixes
• Code refactor

Does this PR introduce a breaking change?

The headline value is unchanged (INCORRECT before and after), so accuracy() / mean() and value-distribution dashboards are unaffected. The only observable changes are Score.metadata gaining {"reason": "invalid_response_format"} on parse failures, and Score.answer becoming None instead of the literal string "None".

[MattFisher]

On further analysis, I'm not sure whether this is the right approach, though it's something of a judgement call.
The pattern scorer previously returned INCORRECT when the pattern couldn't be found, but was only recently updated in April to return NOANSWER instead.

I think this was a mistake, and should potentially be reverted. My reasoning follows.

NOANSWER is documented as the "Value to assign for no answer or refusal to answer", which to me does not include answers in the incorrect format. The reason for the evaluation is to assess whether the model is able to provide the correct answer to a question, and the outcomes should correspond to: "tried and succeeded", "tried and failed", and "didn't try".

I think it's helpful to look at examples. For the question "What is 1+1? Answer with just the number", and a regex pattern of "\d+" we could have answers like:

1. "2": Model answered in the requested format, correctly => CORRECT
2. "3": Model answered in the requested format, incorrectly => INCORRECT
3. "The answer is two": model answered in a non-conforming format, correctly => ?
4. "The answer is 3": model answered in a non-conforming format, incorrectly => ?
5. "The Eiffel tower": model answered in a non-conforming format, with an answer that is outside the class of possibly-correct answers => ?
6. "I feel happy today": model replied in a non-conforming format, something that is obviously not an attempted answer => ?
7. "I cannot answer that": model refused to answer but still returned a response => ?
8. None + stop_reason="content_filter" (model refused to answer) => NOANSWER

Cases 1, 2, and 8 are clear-cut and easy to distinguish. Cases 3-7 are less clear, as they are non-trivial to distinguish from each other, and should ideally have different outputs. 3, 4 and 5 are "tried and failed", 6 and 7 are "didn't try".

If we can identify that a response like "I cannot answer that" is a refusal, it should definitely be marked as NOANSWER, but I do not know how consistent this kind of refusal response is across models/providers, and whether they give a stop_reason.

It's non-trivial to distinguish between cases 3-7, particularly when using pattern-based scorers. They're all seen as a regex failure-to-match. If we use a model-based graders, it becomes possible to distinguish between them, but that increases the expense and duration of the eval.

My thinking is that if the model is unable to follow the instructions in order to produce an answer in the requested format, the model has failed and should be marked as INCORRECT. We should try to separate genuine refusals to answer where we have a definitive signal (like a stop_reason), but absent that signal we should default to marking the response as INCORRECT and store a reason in the metadata, e.g. something like the following, applied consistently across the codebase:

return Score(
    value=INCORRECT,
    explanation=f"Scoring pattern not matched in output: {state.output.completion}",
    metadata={'reason': NO_MATCH},
)

@dragonstyle do you have thoughts here?

[antnewman]

Thanks @MattFisher, this is a fair challenge, and having re-read it I think you're right about math specifically.

The thing I'd add is that "parse failure" hasn't meant the same thing in all the scorers this has been applied to. In pattern/match/math we're parsing the model-under-test's own output, so failing to parse really is the model not producing a usable answer — your cases 3–6, which as you say you can't tell apart without a grader anyway. I don't have a good argument for calling that NOANSWER instead of INCORRECT.

Where I'd still gently push back is the model-graded scorers (#4048), because there the thing being parsed isn't the answer, it's the grader's verdict. When a small judge writes GRID: C instead of GRADE: C, the model under test may well have been right, and scoring it INCORRECT punishes the sample for the grader's formatting rather than the model's performance. That's the case #4026 originally started from, and it feels different in kind from a model that just couldn't follow the answer format itself.

So my honest read is that the answer probably isn't uniform: INCORRECT for the answer-extraction scorers (this PR included), with NOANSWER kept for genuine refusals and for grader-side parse failures where there's a signal.

Either way I'm sold on your metadata idea. Even the inf cases in math — where the model did answer and we just can't represent it — are worth a recorded reason rather than a bare verdict.

Happy to switch this PR to INCORRECT + metadata, and if it's useful I'll take a pass at making the three answer-extraction scorers consistent (including #3628). @dragonstyle, what's your preference on the standard?

[MattFisher]

Where I'd still gently push back is the model-graded scorers (#4048), because there the thing being parsed isn't the answer, it's the grader's verdict. When a small judge writes GRID: C instead of GRADE: C, the model under test may well have been right, and scoring it INCORRECT punishes the sample for the grader's formatting rather than the model's performance. That's the case #4026 originally started from, and it feels different in kind from a model that just couldn't follow the answer format itself.

I agree completely - an unparseable output from a grade model should not result in INCORRECT, it should be Score.unscored(), which is where we landed for #4048 👍🏻

[dragonstyle]

I think one other class of example that might be useful to consider is:

Prompt: What is 1+1? Answer with ANSWER: followed by a number.
Pattern: ANSWER:\s*(\d+)

1. The model answers ANSWER: 2. CORRECT
2. The model answers ANSWER: 3. INCORRECT
3. The model answers ANSWER: The Eiffel Tower. INCORRECT
4. The model answers The Eiffel Tower. ?
5. The model answers 2. ?

Especially in case 5, the answer doesn't seem incorrect though the format wasn't correct. It's not explicitly a refusal to answer, though perhaps failing to conform to the requested format could be interpreted as a refusal?

The other question that occurs to me is whether it is more useful to distinguish between answers that are definitively right, definitely wrong, and wrong for some other reason (e.g. refusal, wrong format, etc..). In terms of metrics these all compute to the same value, but simply marking a failure to answer properly as wrong might have the effect of hiding the failure (masking it as just a wrong answer).

[antnewman]

That case 5 is the one that gets me too — 2 on its own really is the right answer, just not in the shape we asked for, so calling it flatly incorrect feels a bit unfair to the model even though it didn't follow the instruction. I don't think there's a clean value that captures "right, but wrong format" without a grader in the loop.

Your second point is the one I keep coming back to: whatever single value we pick, it collapses "wrong answer", "wrong format", and "refused" into the same number, and that hides useful signal. So maybe the value and the reason are two separate questions.

Where I've landed, trying to hold both your and @MattFisher's points: keep the value as INCORRECT for these answer-extraction cases (so accuracy isn't quietly inflated by treating format failures as non-answers), but attach a metadata reason so the failure mode stays recoverable in analysis — something like:

Score(
    value=INCORRECT,
    explanation=f"Scoring pattern not matched in output: {state.output.completion}",
    metadata={"reason": NO_MATCH},
)

That way the headline metric behaves the way Matt argues it should, but anyone digging into a run can still separate "genuinely wrong" from "couldn't parse the format" from "refused" — you're not forced to choose one meaning for the value.

If that's the direction you're happy with, I'll align math to it here and I'm glad to do a follow-up making pattern/match consistent (and reconciling #3628). The one I'd keep on the NOANSWER side is the model-graded scorers (#4048) — there the thing we're parsing is the grader's verdict, not the model's answer, so a parse failure genuinely isn't the model under test failing.

What standard would you like to settle on?

[MattFisher]

I agree something like metadata={"reason": NO_MATCH}, is probably the best approach at this point, and the key and possible values should probably be standardised for model-graded scorers as well so that any grader that returns Score.unscored() should also include metadata={"reason": GRADER_FAILURE}, or something along those lines. Maybe more verbose constants like COULD_NOT_PARSE_RESPONSE or INVALID_RESPONSE_FORMAT for the model output and INVALID_GRADER_RESPONSE_FORMAT for the grader?

The recent change here to _model_graded_qa_single includes "unscored_reason": "grade_parse_failure" in the metadata, so this should perhaps be updated to whatever we land on. I lean towards using the same key for both with descriptive values, but there could be reasons to use different keys that I'm not anticipating.

[MattFisher]

I'm fleshing out my suggestion here and will link to this comment from other places.

I think our best bet is introducing ScoreReason, analogous to StopReason, and storing it in metadata["reason"] for all abnormal scores. Score.explanation can still hold the human-readable explanation.

Situation	Score value	metadata["reason"]
MUT output unparseable / format violation	INCORRECT	"invalid_response_format"
MUT refused / empty completion	NOANSWER	"refusal" / "no_response"
Grader verdict unparseable (after bounded retry)	Score.unscored()	"grader_parse_failure"
Grader refused	Score.unscored()	"grader_refusal"
Grader made no tool call (schema scorers)	Score.unscored()	"grader_no_tool_call"
Grader payload failed schema / bad grade field	Score.unscored()	"grader_schema_mismatch" / "grader_invalid_grade"

If "reason" seems like too generic a word that other evals might already be using, "score_reason" is an alternative.

[MattFisher]

See also UKGovernmentBEIS/inspect_evals#1933

[MattFisher]

Just noticed the unscored example in the docs sets metadata={"reason": "timeout"}

[antnewman]

The ScoreReason table is exactly the shape I was hoping this would settle into — thanks for doing the legwork, and the docs' metadata={"reason": "timeout"} example plus inspect_evals#1933 make a pretty compelling case that "reason" is already the de-facto key. I've updated this PR to implement your table's row for the MUT-format-violation case: value stays INCORRECT, metadata={"reason": "invalid_response_format"}, and the parse-failure branch no longer records the literal string "None" as the extracted answer (a small latent bug regardless of which design won). Title and description updated to match — 43aaa2b9d.

Two things I ran into while verifying the design that seem worth having on the table before the vocabulary ossifies:

Score edits silently destroy metadata-carried reasons. apply_score_edit in log/_score.py assigns score.metadata = edit.metadata wholesale when the edit supplies metadata — it replaces rather than merges. So a viewer score-edit that sets any metadata at all wipes the reason (and grading, and everything else the scorer recorded). The original is preserved in history, but anything consuming metadata["reason"] for analysis reads the current dict and quietly loses the signal. That's not an argument against metadata["reason"] — the convention is clearly winning, and this PR follows it — but it's a real leak in the pipeline the vocabulary will flow through. A merge-instead-of-replace on the edit path (or preserving framework keys) would close it. Happy to file that separately if you agree it's a bug.

On StopReason as the precedent — worth noting it's actually a typed field (stop_reason on ChatCompletionChoice), not metadata, with a companion StopDetails model for the free-form part. I don't think that changes the answer here (a typed Score.reason would drag the log schema and the ts-mono generated types along with it, which is a much bigger change than this vocabulary needs today), but it does mean promoting metadata["reason"] to a typed field later stays open as a clean follow-up once the values settle — the same closed-Literal-plus-migration pattern stop_reason already uses for evolving old logs.

On unifying the key: #4048's unscored_reason="grade_parse_failure" went out in 0.3.245, so the free-rename window I'd have liked has just closed. Still, one release old is the cheapest it will ever be — if you want everything on "reason" (which I'd also lean towards, with the descriptive values you suggest carrying the grader/MUT distinction), I'm glad to do that small PR, including whatever compatibility shim you'd want for logs written by 0.3.245.

One tiny vocabulary note: the docs example uses "timeout", which isn't in your table — might be worth the table (or eventual constants) having an explicit escape hatch for eval-authored values, the way StopReason keeps "unknown".

[MattFisher]

I think it would be worth setting answer=state.output.completion here, as the scorer docs say:

If you are extracting an answer from within a completion (e.g. looking for text using a regex pattern, looking at the beginning or end of the completion, etc.) you should strive to always return an answer as part of your Score, as this makes it much easier to understand the details of scoring when viewing the eval log file.

and in that case it wouldn't need to be in explanation as well.

Suggested change

	return Score(
	value=INCORRECT,
	answer=None,
	explanation="No mathematical expression could be extracted "
	+ f"from the model output: {state.output.completion}",
	metadata={"reason": "invalid_response_format"},
	)
	return Score(
	value=INCORRECT,
	answer=state.output.completion,
	explanation="No mathematical expression could be extracted from the model output",
	metadata={"reason": "invalid_response_format"},
	)

[antnewman]

Applied in 5b12409dd — thanks for the docs pointer, I had missed that guidance. Completion surfaced as answer, explanation trimmed to the static message, and the test now asserts the round-trip.