docs: add Scoring Policy page (score vs raise vs unscored)

[MattFisher]

Summary

This adds a new Scoring Policy page to the Scoring section of the docs, covering a question that keeps coming up for eval authors: when something goes wrong on a single sample, should the scorer return a Score, raise, or return Score.unscored()? The three outcomes route to different framework machinery and affect the metric denominator differently, and until now the guidance for choosing between them lived in scattered PR threads rather than the docs.

Alongside the new page:

• custom-scorers.qmd: the model_graded_qa example's grade-parse-failure branch now returns Score.unscored() instead of INCORRECT, matching what the real scorer has done since fix(scorer): mark model_graded grade-parse failure as unscored (not INCORRECT) #4048 (shipped in 0.3.246). The old example was teaching the pattern that fix(scorer): mark model_graded grade-parse failure as unscored (not INCORRECT) #4048 removed.
• _builtin-scorers.md: documents what the text-matching scorers return when the output doesn't match: match()/includes()/exact() score INCORRECT, while pattern()/answer() score NOANSWER when the pattern doesn't match at all (see the question below).
• Cross-references stitched through model-graded.qmd, handling-errors.qmd, and metrics.qmd anchors so the run-level options and the authoring decision point at each other.

How this came about

Over the past couple of months a cluster of inspect_evals PRs all hit the same ambiguity from different angles: contributors converting scorer raises into Score(0.0) to stop errors from eating samples (inspect_evals#1681, #1679, #1671, #1845), and model-graded scorers silently charging grader parse failures to the model under test (#4026, fixed by #4048). Each individual change was well-intentioned, but collectively they were making evals behave inconsistently: scoring an infrastructure failure 0.0 deflates accuracy and silently opts the sample out of retry_on_error, while raising on a genuine model failure inflates accuracy by shrinking the denominator.

We wrote the contributor-facing version of this guidance for inspect_evals in inspect_evals#1931, but the underlying semantics are Inspect's, not inspect_evals', so the canonical explanation belongs here. The new page is deliberately concept-first (what scoring measures and why the distinction exists) before it gets prescriptive.

Questions for maintainers

I'd genuinely value your opinion on three things before this merges, since the page currently takes positions on all of them.

1. The "only non-deterministic instruments get unscored" bright line. The page draws the raise-vs-unscored line as: deterministic machinery (sandbox exec, file I/O, your own runner, the grader API call) either works or is broken, so its failures should always raise; Score.unscored() is reserved for instruments that are inherently stochastic (a grader model emitting an unparseable verdict after bounded retries, a content filter on a fixed prompt). The rationale is that unscoring a deterministic failure hides a bug, whereas raising keeps it retryable and visible against fail_on_error. This line has worked well in practice across the inspect_evals cluster, but it's a policy statement in framework docs, so it should have your endorsement (or amendment) rather than just mine.

2. pattern() returning NOANSWER on no-match, and whether to revisit #3628. In April, #3628 changed pattern() (and by extension answer()) to return NOANSWER instead of INCORRECT when the regex doesn't match. The new docs describe this behaviour neutrally, but it's worth deciding deliberately, because a failed extraction conflates two cases: the model produced no answer (where NOANSWER fits), and the model produced an answer in the wrong format (an instruction-following failure, which reads as INCORRECT, and is the common case in practice). It's also now inconsistent with match()/includes()/exact(), which score the equivalent miss INCORRECT. Options as I see them:

• revert pattern() to INCORRECT (my lean, argued in more detail on #4091, with metadata["reason"] preserving the "didn't parse" signal for anyone who wants to partition on it);
• keep NOANSWER and accept the inconsistency;
• or introduce a distinct sentinel for "extraction failed" that is neither of the above, though note NOANSWER's labelling value is already undermined by the default mean epoch reducer collapsing it to 0.0 before metrics run, and a new sentinel would inherit the same problem.

Whichever way this lands, the docs change is one bullet in _builtin-scorers.md.

3. Upstreaming ScoreReason. The docs recommend tagging every abnormal score with a machine-readable metadata["reason"] (e.g. invalid_response_format, refusal, grader_parse_failure, grader_refusal, grader_no_tool_call) so failures are filterable in the viewer and in dataframes, with a grader_ prefix separating instrument failures from model failures. inspect_evals is already standardising on this: inspect_evals#1933 introduces a ScoreReason Literal plus constants in inspect_evals/utils, and #4048 already ships "grade_parse_failure" (under the key unscored_reason, which we'd align). I think the type wants to live in inspect_ai.scorer, analogous to StopReason (proposed shape in this #4091 comment). I can send that as a small follow-up PR if you're amenable; the docs deliberately describe it as a convention rather than API until then.

Related in-flight work

For the fuller picture, the pieces of this effort currently open:

• #4415: makes score_on_error scores actually contribute to metrics, per the existing docs (fixes #4412). Note the worked example on the new page describes score_on_error's documented behaviour, so this page and Include errored-but-scored samples in metrics for score_on_error #4415 should land in the same release.
• #4481: surfacing scored/errored/unscored counts next to headline metrics, so a run with a shrunken denominator can't read as a clean 100%.
• #4491: NaN inside a list-valued Score crashes the eval at sample-logging time (found while stress-testing the NaN handling this page documents).
• #4506: docstring fix noting that Docker exec pre-truncates over-limit output rather than raising OutputLimitExceededError.
• #4091: the math()/NOANSWER discussion where the ScoreReason shape is being worked out.

Checklist

• Docs-only; no behaviour change.
• New page wired into _quarto.yml (Scoring section) and the scoring.qmd landing table.
• All five touched pages render cleanly with quarto render (1.9.38 via the doc extra).

🤖 Generated with Claude Code