fix(scorer): admit None in list-valued Score so NaN survives event logging (#4491)#4492
Open
Anai-Guo wants to merge 2 commits into
Open
fix(scorer): admit None in list-valued Score so NaN survives event logging (#4491)#4492Anai-Guo wants to merge 2 commits into
Anai-Guo wants to merge 2 commits into
Conversation
Contributor
Author
|
Pushed the regenerated The remaining |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #4491. A scorer that returns a list-valued
ScorecontainingNaN(e.g.Score(value=[float("nan"), 1.0])) crashes the run at end-of-eval sample logging, finishing withstatus="error"and no results — an opaque wall of pydantic union errors. The identical value as a dict ({"a": float("nan"), "b": 1.0}) works fine.Root cause
The two container branches of
Valueinscorer/_metric.pyare asymmetric:Pydantic serializes
NaNto JSONnullfor root, dict, and list placements. On its way into the streaming sample buffer aScoreEventis serialized, so[nan, 1.0]becomes[null, 1.0]. When the completed sample is materialized for logging,validate_eventsre-validates the buffered dicts; theSequencebranch rejects thenull, and since the failing event is tried against all event types in the union it surfaces as ~75 unrelated-looking errors. TheMappingbranch already admitsNone, which is why the dict shape survives the same round-trip.Fix
Align the
Sequencebranch with theMappingbranch so the NaN →nullround-trip is lossy-but-stable, matching what the dict shape already does and the sanctioned NaN-at-root sentinel from #3796:One-line, symmetric, no behavior change for non-NaN values.
Test
Added
test_list_score_with_nan_logs_without_errorintests/scorer/test_metric.py, mirroring the issue's reproduction: a list-valued scorer returning[nan, 1.0]now completes withstatus="success"and non-Noneresults.🤖 Generated with Claude Code