Skip to content

fix(scorer): admit None in list-valued Score so NaN survives event logging (#4491)#4492

Open
Anai-Guo wants to merge 2 commits into
UKGovernmentBEIS:mainfrom
Anai-Guo:fix/nan-list-value-4491
Open

fix(scorer): admit None in list-valued Score so NaN survives event logging (#4491)#4492
Anai-Guo wants to merge 2 commits into
UKGovernmentBEIS:mainfrom
Anai-Guo:fix/nan-list-value-4491

Conversation

@Anai-Guo

Copy link
Copy Markdown
Contributor

Summary

Fixes #4491. A scorer that returns a list-valued Score containing NaN (e.g. Score(value=[float("nan"), 1.0])) crashes the run at end-of-eval sample logging, finishing with status="error" and no results — an opaque wall of pydantic union errors. The identical value as a dict ({"a": float("nan"), "b": 1.0}) works fine.

Root cause

The two container branches of Value in scorer/_metric.py are asymmetric:

Value = Union[
    str | int | float | bool,
    Sequence[str | int | float | bool],          # <- no None
    Mapping[str, str | int | float | bool | None],  # <- has None
]

Pydantic serializes NaN to JSON null for root, dict, and list placements. On its way into the streaming sample buffer a ScoreEvent is serialized, so [nan, 1.0] becomes [null, 1.0]. When the completed sample is materialized for logging, validate_events re-validates the buffered dicts; the Sequence branch rejects the null, and since the failing event is tried against all event types in the union it surfaces as ~75 unrelated-looking errors. The Mapping branch already admits None, which is why the dict shape survives the same round-trip.

Fix

Align the Sequence branch with the Mapping branch so the NaN → null round-trip is lossy-but-stable, matching what the dict shape already does and the sanctioned NaN-at-root sentinel from #3796:

    Sequence[str | int | float | bool | None],

One-line, symmetric, no behavior change for non-NaN values.

Test

Added test_list_score_with_nan_logs_without_error in tests/scorer/test_metric.py, mirroring the issue's reproduction: a list-valued scorer returning [nan, 1.0] now completes with status="success" and non-None results.

🤖 Generated with Claude Code

@Anai-Guo

Copy link
Copy Markdown
Contributor Author

Pushed the regenerated inspect-openapi.json (added null to the Score.value/EvalSampleScore.value/ScoreEdit.value sequence branch), so the check_openapi_drift.py step is now in sync with the Python Value type.

The remaining check-schema-and-types diff is the generated TypeScript (packages/inspect-common/src/types/generated.ts), which lives in the pinned meridianlabs-ai/ts-mono submodule — I can't push there from a fork. It just needs a pnpm --filter @tsmono/inspect-common types:generate + submodule bump on your side to pick up the nullable list item. Happy to adjust if you'd prefer a different approach.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

NaN in a list-valued Score crashes the eval at sample-logging time

2 participants