NaN in a list-valued `Score` crashes the eval at sample-logging time

[MattFisher]

Summary

A scorer that returns Score(value=[float("nan"), 1.0]) raises an error in the log-event validation step. After all samples complete, log-event validation fails and the eval finishes with status="error" and no results, reporting an opaque wall of pydantic errors (every event type in the union rejected). The identical value shaped as a dict ({"a": float("nan"), "b": 1.0}) works fine.

Found while stress-testing NaN handling around #4458/#4286. #3796 established NaN-at-root as the unscored sentinel for scalar, dict, and list-valued scorers — this is the one NaN placement that slips through: inside a list.

Reproduction

from inspect_ai import Task, eval
from inspect_ai.dataset import Sample
from inspect_ai.scorer import Score, Target, accuracy, scorer
from inspect_ai.solver import solver

@solver
def passthrough():
    async def solve(state, generate):
        state.output.completion = "done"
        return state
    return solve

@scorer(metrics=[accuracy()])
def listy():
    async def score(state, target):
        return Score(value=[float("nan"), 1.0])
    return score

log = eval(
    Task(dataset=[Sample(input="q", target="x")], solver=passthrough(), scorer=listy()),
    model="mockllm/model",
    display="none",
)[0]
print(log.status)   # "error"
print(log.results)  # None
print(log.error.message.splitlines()[0])
# 75 validation errors for list[union[SampleInitEvent,SampleLimitEvent,...]]

Reproduced on 0.3.232; the type definitions involved are unchanged on current main.

Root cause

The failure point (from log.error.traceback) is sample logging, not scoring:

_eval/task/run.py             log_sample
log/_recorders/streaming.py   materialize_streaming_sample
event/_validate.py            validate_events        # TypeAdapter(list[Event]).validate_python

Three facts compose:

1. Pydantic's JSON serialization dumps NaN as null (same for root, dict, and list placements — verified: ScoreEvent(score=Score(value=[nan, 1.0]), ...).model_dump_json() contains "value":[null,1.0]). The sample's events pass through this serialization on their way into the streaming sample buffer.
2. Value's branches are asymmetric (scorer/_metric.py): Mapping[str, str | int | float | bool | None] admits None, but Sequence[str | int | float | bool] does not.
3. When the completed sample is materialized from the buffer for logging, validate_events re-validates the buffered event dicts. The ScoreEvent now carries "value": [null, 1.0], the Sequence branch rejects None, and because validation tries the dict against all 23 event types in the union, the single bad event reports as 75 unrelated-looking errors. The dict shape survives the same round-trip ({"a": null} validates against the Mapping branch), which is why only the list placement crashes.

The trap for scorer authors: Score(value=[None, 1.0]) fails fast at construction with a clear message, but NaN is a perfectly valid float at construction time — it only becomes an invalid null at the serialization boundary, so the failure is deferred to end-of-run and the diagnostic points nowhere near the scorer.

Suggested fix

The minimal, symmetric fix is to align the Sequence branch with the Mapping branch: Sequence[str | int | float | bool | None]. That makes the round-trip lossy-but-stable (NaN → null), matching what the dict shape already does.

If NaN-in-container is instead considered invalid (as distinct from the sanctioned NaN-at-root sentinel from #3796), the alternative is to reject it at Score construction with a targeted error message — anything is better than the current deferred union-validation dump with loss of the whole run's results.

Related

• Allow NaN-at-root unscored sentinel for dict- and list-valued scorers #3796 — NaN-at-root unscored sentinel (works correctly; this is the uncovered placement)
• Scoring metrics silently drop inconclusive/errored samples from the denominator and don't bound their range #4286 / fix(scorer): exclude unscored samples from accuracy() and mean() denominator #4458 — NaN-vs-metrics discussion that prompted the stress test

[ebarkhordar]

Confirmed live on current main (8117bf2), and #4492's one-liner does fix it end to end. Two details worth having on the record, because the symptom quoted here and in #4492 no longer matches what main does.

Setup throughout: clean python:3.12-slim container, pip install -e . at 8117bf2, the repro script above verbatim.

1. The failure surfaces somewhere else now. The eval still ends with no results, but the top-line error is not a pydantic union wall:

STATUS: error | RESULTS: None
log.error.message: ValueError('Attempt to write to ZIP archive that was already closed')

The ValidationError is still the cause, it is just the chained exception now, and it lands at reduction validation rather than at log_sample/validate_events:

File "src/inspect_ai/log/_recorders/eval.py", line 1067, in _read_log_from_bytes
    EvalSampleReductions.model_validate(
pydantic_core._pydantic_core.ValidationError: 9 validation errors for EvalSampleReductions

During handling of the above exception, another exception occurred:

File "src/inspect_ai/log/_recorders/eval.py", line 962, in _zip_writestr
ValueError: Attempt to write to ZIP archive that was already closed

close() raises inside _read_log_from_bytes, then log_finish writes the summaries into the archive that just closed, and that second error is what reaches log.error.message. So the scorer author now gets a zip writer error, which is further from their scorer than the union dump was. Anyone re-testing #4492 against main should expect this rather than the 75 errors.

The errors still state the asymmetry exactly as diagnosed here:

samples.0.value.json-or-python[json=list[union[str,int,float,bool]],...].0.float
  Input should be a valid number [type=float_type, input_value=None, input_type=NoneType]
samples.0.value.dict[str,nullable[union[str,int,float,bool]]]
  Input should be a valid dictionary [type=dict_type, input_value=[None, 1.0], input_type=list]

2. The fix resolves it rather than moving it. Applying only Sequence[str | int | float | bool | None], which is what #4492 does, in the same container (_metric.py sha256 bf6064e7 before, 8f8da9ed after, so I know which code ran):

STATUS: success
RESULTS: {'accuracy': EvalMetric(name='accuracy', value=0.0, ...)}
SAMPLE SCORE VALUE: [None, 1.0]

The eval completes and accuracy() consumes the None without a downstream crash. Whether 0.0 is the right metric semantics for a NaN inside a list is a separate call, and not one I have any standing to make.

One history note for that decision: the asymmetry looks inherited rather than chosen. #2109 only converted list/dict to Sequence/Mapping; the | None was already on the dict branch and already absent from the list branch before it.

Disclosure: AI-assisted (agent-run). Everything above was measured in the container described, not inferred.