Scoring metrics silently drop inconclusive/errored samples from the denominator and don't bound their range

[AUTHENSOR]

Summary

The accuracy/metric layer conflates several distinct outcomes — a sample that errored, a scorer that returned NaN/abstained, and a scorer that returned None — with "this sample does not exist," silently removing them from the metric denominator. Separately, value_to_float output is never range-checked, so a headline accuracy can exceed 1.0 / go negative / be inf; and mean() and accuracy() use incompatible value-to-float rules so mean() raises on the framework's own C/I/P labels. Net effect: a half-broken or abstained eval can report a clean, high accuracy with status="success" — the worst direction for a safety/capability eval.

---

1. Errored samples vanish from the accuracy denominator

When a sample errors and the error is not severe enough to trip fail_on_error, its score is dropped (run.py ~L1742-1745 returns None; ~L742-746 keeps only dict results; eval_results is called with samples=all but scores=completed; accuracy.py:33 divides by len(scores)). The denominator becomes "samples that happened to succeed."

from inspect_ai import Task, eval
from inspect_ai.dataset import Sample
from inspect_ai.scorer import Score, Target, scorer, accuracy, CORRECT
from inspect_ai.solver import TaskState, solver, Generate

@solver
def passthrough():
    async def solve(state, generate): return state
    return solve

@scorer(metrics=[accuracy()])
def flaky_scorer():
    async def score(state: TaskState, target: Target) -> Score:
        if int(state.metadata["idx"]) >= 5:        # the "hard"/adversarial half
            raise RuntimeError("judge/parse/sandbox error")
        return Score(value=CORRECT)
    return score

task = Task(
    dataset=[Sample(input=f"q{i}", target="x", metadata={"idx": i}) for i in range(10)],
    solver=passthrough(), scorer=flaky_scorer(),
)
log = eval(task, model="mockllm/model", fail_on_error=0.9, display="none")[0]
print(log.status, log.results.scores[0].metrics["accuracy"].value)  # success 1.0

Actual: status="success", accuracy=1.0 over the 5 survivors.
Expected (for discussion): errored samples should be visible in the headline — counted against the denominator, or the metric reported alongside an explicit "N errored / inconclusive" so a reader cannot mistake a half-failed run for 100%.

2. A scorer that returns NaN ("abstain/inconclusive") is dropped too

results.py ~L300-307 filters out any Score whose value is a float NaN as "unscored" — but NaN is reachable from the public API (Score(value=float("nan"))), the natural way a judge says "I couldn't decide." Those abstentions leave the denominator and inflate accuracy (3 correct, 2 incorrect, 5 abstained -> reported 3/5 = 0.6, not 0.3).

3. accuracy()/mean() never clamp -> headline > 100% / negative / inf

value_to_float passes numbers straight through (_metric.py:228-229) and accuracy.py:29-33 just sums and divides — no [0,1] clamp, no finiteness check. A scorer on a non-0..1 scale (a 0..10 rubric, a count) makes accuracy exceed 1; a single inf makes it inf (it also escapes the math.isnan-only unscored filter).

# per-sample values [8, 5, 10] -> accuracy = 7.67  (767%); [-1,-1,1] -> -0.33

Expected (for discussion): clamp value_to_float to [0,1] (or document that custom numeric scorers must define their own metric), and treat non-finite as unscored rather than a contributor.

4. mean() and accuracy() disagree; mean() raises on C/I/P

accuracy() uses value_to_float() ("C"->1.0); mean() uses Score.as_float() -> float(raw), and float("C") raises ValueError (mean.py:15, _metric.py:157). A scorer emitting the framework's own CORRECT constant is 1.0 under accuracy() but crashes mean(). Tasks that attach [accuracy(), mean(), stderr()] to a label-emitting scorer throw at metric time — the whole run's scores are lost.

from inspect_ai.scorer._metric import Score, value_to_float
vtf = value_to_float()
for v in ["C", "I", "P"]:
    try: m = Score(value=v).as_float()
    except Exception as e: m = f"{type(e).__name__}"
    print(v, "accuracy-path", vtf(v), "| mean-path", m)   # C 1.0 | mean-path ValueError

Expected: mean() should understand the same label vocabulary as accuracy() (or raise a clear error at construction when paired with a label scorer). This sub-item is an unambiguous bug and could ship as a small standalone PR independent of items 1-3.

---

Proposed direction (one coherent change)

Items 1, 2, and the inf part of 3 are one design question: inspect has no concept of an inconclusive outcome. Erroring / abstaining / out-of-range samples all collapse into "absent," silently. A single direction fixes all of them: keep inconclusive samples in the denominator (or report a first-class "inconclusive / errored" count next to every metric), clamp value_to_float to [0,1] and treat non-finite as inconclusive, and make mean()/accuracy() agree on the label vocabulary. Happy to implement once the intended semantics are agreed — and to split into focused PRs (item 4 first as a clear bug).

Environment

inspect_ai @ 8915598; Python 3.12+; reproduced with mockllm (no model provider required).

[HarperZ9]

The three problems here share a root shape worth naming: a measurement has three outcomes, not two. "Scored correct" and "scored incorrect" both presuppose that the measurement ran. Right now a measurement that didn't run — errored, abstained (NaN/None), or otherwise produced no usable value — gets folded into "this sample doesn't exist" and silently leaves the denominator. Abstention is data, not absence.

A shape that holds up well is to account for every sample in one of three explicit buckets, and never let bucket membership be implicit:

• scored — produced a usable value; only these contribute to the accuracy ratio (numerator and denominator).
• inconclusive — ran but abstained (NaN / None / explicit "don't know"); tracked and reported, excluded from the ratio but visible.
• errored — the measurement itself failed; tracked and reported separately.

Two invariants fall out of that:

1. Coverage becomes a first-class number. Report scored / total next to accuracy. "40% of samples scored, 95% of those correct" is a different result from "100% scored, 95% correct," and today they're indistinguishable in the headline.
2. The metric is range-bounded by construction. 0 <= accuracy <= 1 should be an assertion/clamp at the metric boundary, not an emergent property of value_to_float. A headline that can read inf or go negative is the metric layer trusting its inputs more than it should; an explicit bound turns a silent corruption into a loud failure.

The mean()-vs-accuracy() divergence on the C/I/P labels looks like the same theme one layer down: two value→float paths with different ideas of what a "valid" score is. Unifying them on one explicit value model — where "not a valid score" is representable rather than silently coercible — closes that too.

The worst case you describe (a half-broken eval reporting clean high accuracy with status="success") is really the symptom of "measurement failed" having no representation distinct from "measurement passed/failed." Give it one and the rest follow.

Happy to help turn this into a PR if a three-bucket accounting + boundary-clamp direction sounds right to you.

[MattFisher]

I agree that there is a gap in the visibility of how many samples were fully scored.

Currently, there are a couple of mechanisms that can be used to deal with issue 1:

• --fail-on-error can be set to something that will bound what the user considers a valid outcome. E.g. 0.1 will fail the run if more than 10% of the samples have errors.
• --score-on-error can be set to score any errored samples, and is intended to include them in the metrics (i.e. in the denominator), though there's currently a bug that prevents this.

See also discussion in #4091 and #4048

If the crucial use case here is the judge failure rate not being sufficiently obvious when unscored is used, perhaps we can find a way to surface that information more prominently? E.g. the --score-on-error option changes how samples that raise errors are scored, maybe something analogous could be done for judge failures?

[goktugozkanmd]

Klinik güvenlik eval'larında bu ayrım pratikte kritik. Bir model, hayatı tehdit eden ayırıcı tanı gerektiren en zor olgularda cevap vermeyi reddedip geri kalan olguların tamamını doğru yanıtlayabilir. Mevcut hesaplama bu reddedilen örnekleri paydadan sessizce çıkarırsa rapor 100% accuracy gösterebilir.

Bu sonuç sahada yanlış okunur. Başlık skoru, belirsizliği yöneten modeli zor klinik karar anında kaçan modelle aynı gösterir. --fail-on-error ve --score-on-error hata sınıfı için yardımcı olur, ama klinik eval tarafında abstention da ayrı bir güvenlik sinyali. Bu yüzden accuracy yanında scored / total, abstained / total ve errored / total görünür olmalı. Oran hesaplama aynı kalsa bile denominatör kapsamı sessiz kalmamalı. Aksi halde regresyon takibinde en riskli örneklerin kaybolduğunu geç fark ederiz.

[MattFisher]

Translation of previous comment according to Google Translate:

In clinical safety evaluations, this distinction is critical in practice. A model might refuse to answer the most difficult cases—those requiring the consideration of life-threatening differential diagnoses—while correctly answering all the remaining ones. If the current calculation silently excludes these refused samples from the denominator, the report could show 100% accuracy.

This result is liable to be misinterpreted in the field. The headline score makes a model that manages uncertainty appear identical to one that simply fails during critical clinical decision-making. While --fail-on-error and --score-on-error help address the error class, abstention itself serves as a distinct safety signal in clinical evaluations. Therefore, alongside accuracy, metrics such as scored/total, abstained/total, and errored/total should be visible. Even if the ratio calculation remains the same, the scope of the denominator must not be obscured; otherwise, we risk being late to notice the disappearance of the highest-risk samples when tracking performance regressions.

[goktugozkanmd]

PR #4458 addresses the metric-level half of this issue: accuracy() and mean() now skip NaN (unscored) samples instead of piping them through value_to_float() and producing NaN for the entire metric. Coverage (scored/total) is logged at INFO when below 100%.

Two gaps remain for a follow-up:

1. A three-bucket accounting layer (scored / inconclusive / errored) in the evaluation results, so coverage is a first-class number in the report — not just a log line.
2. An explicit ScoreStatus enum so that unscored has a richer representation than NaN-in-disguise.

[goktugozkanmd]

Following up on the remaining gaps from my last comment — the three-bucket accounting layer (#4286) and the monitor schema work (#4474 / #4456) share the same vocabulary naturally.

PR #4474 already defines a scoring_status: Literal["scored", "abstained", "unscored", "not_run", "errored"] on MonitorRecord. This maps directly to the three-bucket model @HarperZ9 described:

Bucket	scoring_status
scored	"scored"
inconclusive	"abstained", "unscored"
errored	"errored", "not_run"

The append-only record stream per attempt/sample already carries execution_status, scoring_status, status_reason, and epoch/attempt context — so coverage is observable at the record level, not reconstructed from metric denominators.

I will pick up the ScoreStatus enum as a direct follow-up after #4458 settles, aligned with this record shape so the metric layer and the monitor layer share the same vocabulary.

[MattFisher]

My suggested schema for recording the extra scoring information is here on #4091, intended to cover all abnormal scores. The simple place to put it is in the metadata, but there's also the option of adding an additional field to the Score model specifically for the machine-readable "reason", as opposed to the human-readable "explanation":

I think our best bet is introducing ScoreReason, analogous to StopReason, and storing it in metadata["reason"] for all abnormal scores. Score.explanation can still hold the human-readable explanation.

Situation	Score value	metadata["reason"]
MUT output unparseable / format violation	INCORRECT	"invalid_response_format"
MUT refused / empty completion	NOANSWER	"refusal" / "no_response"
Grader verdict unparseable (after bounded retry)	Score.unscored()	"grader_parse_failure"
Grader refused	Score.unscored()	"grader_refusal"
Grader made no tool call (schema scorers)	Score.unscored()	"grader_no_tool_call"
Grader payload failed schema / bad grade field	Score.unscored()	"grader_schema_mismatch" / "grader_invalid_grade"

If "reason" seems like too generic a word that other evals might already be using, "score_reason" is an alternative.

[MattFisher]

Note that I think the Score/Status/Reason vocabulary belongs in the scoring layer and shouldn't take a dependency on the monitor-schema proposal.