`score_on_error` scores errored samples in the log but excludes them from metrics (contradicts docs)

[MattFisher]

Summary

With score_on_error=True, an errored sample is scored (the scorer runs on the partial TaskState) and that score is written to the sample in the log, but the score is not included in metric computation or the scored_samples denominator. The docs state the opposite:

docs/handling-errors.qmd (line 109): "Each errored sample is recorded with both its error … and its scores (so the sample contributes to metrics)."

So either the implementation or the documentation is wrong. This matters because score_on_error is the documented mechanism for making "the model errored" a scoreable outcome, but it doesn't currently change the metrics.

Present since the feature landed in #3814

Reproduction

"""score_on_error writes a score to the errored sample but excludes it from metrics.

2 samples. Sample 1 solves fine and scores CORRECT. Sample 2's solver raises;
score_on_error re-runs the scorer on the partial state, which scores it INCORRECT.
If that score counted, accuracy would be 1/2 = 0.5. It reports 1.0.
"""

from inspect_ai import Task, eval
from inspect_ai.dataset import Sample
from inspect_ai.scorer import CORRECT, INCORRECT, Score, Target, accuracy, scorer
from inspect_ai.solver import Generate, TaskState, solver


@solver
def solver_that_raises_on_sample_2():
    async def solve(state: TaskState, generate: Generate) -> TaskState:
        if state.sample_id == 2:
            raise RuntimeError("boom")
        state.output.completion = "answered"
        return state

    return solve


@scorer(metrics=[accuracy()])
def score_by_completion():
    async def score(state: TaskState, target: Target) -> Score:
        # CORRECT if the solver produced output, INCORRECT on partial state
        return Score(value=CORRECT if state.output.completion else INCORRECT)

    return score


log = eval(
    Task(
        dataset=[Sample(id=1, input="x"), Sample(id=2, input="x")],
        solver=solver_that_raises_on_sample_2(),
        scorer=score_by_completion(),
    ),
    model="mockllm/model",
    score_on_error=True,
    fail_on_error=False,
    display="none",
)[0]

score = log.results.scores[0]
sample_2 = next(s for s in log.samples if s.id == 2)

print("accuracy       :", score.metrics["accuracy"].value)  # 1.0  (expected 0.5)
print("scored_samples :", score.scored_samples)             # 1    (expected 2)
print("sample 2 score :", sample_2.scores["score_by_completion"].value)  # 'I'
print("sample 2 error :", sample_2.error is not None)        # True

Expected vs. actual

	accuracy	scored_samples	sample 2 in log
Expected (per docs)	0.5	2	score I + error
Actual	1.0	1	score I + error

accuracy       : 1.0
scored_samples : 1
sample 2 score : I
sample 2 error : True

Sample 2 is recorded with value="I" and an error, but accuracy is 1.0 (not 0.5) and scored_samples is 1. The score_on_error score is written to the sample log but never reaches the denominator.

Root cause

The score is computed and written to the sample, but the per-sample coroutine's return value (which is what feeds metrics) is None for any errored sample:

• Scoring runs for errored samples under score_on_error (run.py:1505-1509), and the score is stored on the sample that gets logged.
• But when error is not None and the raise is suppressed (score_on_error), raise_error is None, so the coroutine falls through to the final else: return None (run.py:1801-1806).
• completed_scores is the list passed to eval_results() / metric computation; it only keeps dict return values, filtering out the None (run.py:761-765).

So the score lands in the sample log via the logging path, but not in the metrics via the return-value path.

This looks like an oversight in #3814: the return None for errored samples predates that PR and wasn't revisited when score_on_error added partial-state scoring plus docs promising metric contribution.

Proposed resolution

Two coherent options; a maintainer decision is needed on which is intended:

1. Make it match the docs (include in metrics). When score_on_error produced a score, return that score dict from the coroutine (or otherwise route it into completed_scores) so it enters the denominator. This is what the docs currently promise and what makes score_on_error useful as "count the error as an outcome." Note this changes reported numbers for anyone already using the flag.

2. Keep the exclusion, fix the docs. If errored samples should deliberately stay out of the denominator (visible-but-uncounted), update docs/handling-errors.qmd:109 to say the score is recorded on the sample for inspection but is not included in metrics, and point authors at returning a Score (or Score.unscored()) if they want the sample counted/excluded explicitly.

[goktugozkanmd]

This intersects with the fix in PR #4458 (exclude unscored samples from accuracy/mean denominator). The core issue is that score_on_error treats the error as a separate pipeline path — the scorer runs but its output never enters the metric accumulator because the metric loop skips errored samples.

Two possible approaches:

1. Fix the metric loop to include scored-but-errored samples when score_on_error is True (match the docs).
2. Or align the docs to match current behaviour and add a separate flag (e.g. include_errored_in_metrics) for explicit opt-in.

For clinical safety evaluations, approach 1 is safer — hiding an errored sample that was scored masks the model's failure-mode distribution. The clinician needs to see that the model errored consistently on high-acuity cases, not just that the error rate was X%.