Score scales for SelfCritique/LMAsJudge, symbolic auto-build, bump 0.9.014 - #98
Merged
Conversation
Add discretized score scales to the pydantic metrics backend next to the
existing 0.0..1.0 `Score`: `FineScore` (21 float levels), `Rating` (1..5),
`Rating10` (1..10) and `Rating20` (1..20), plus helpers to resolve a scale
from a class or name (`get_score_type`), serialize it, describe it for the
LM, and `normalize_score` any raw value back to 0.0..1.0.
`SelfCritique` and `LMAsJudge` gain a `score_type` argument: the generator
schema's `reward` property becomes an inline enum of the scale's members so
the model picks a value on that scale, and the emitted reward is normalized
to 0.0..1.0 so downstream rewards/metrics are unchanged. The default
instructions now spell out the grading scale in plain words, since the
output schema is not part of the prompt by default. Round-trips through
`get_config`/`from_config`, exported under `synalinks.{FineScore,Rating,...}`.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`Module._maybe_build` traced `compute_output_spec`/`call` on whatever the first call received. For an eager call that meant concrete `JsonDataModel`s, so modules wrapping a `Generator` (agents in particular) ran a real forward pass, i.e. throwaway LM requests, just to discover their output schema. Convert the arguments to `SymbolicDataModel` before tracing so auto-build is purely symbolic, as on the functional-API path. Two agent tests were only passing because the build silently consumed the first two scripted mock responses; they now script the real 3-call sequence. Add a regression test and drop the now-stale "throwaway LM calls" rationale from the subagent `Program` wrappers in DeepAgent/RLM. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SelfCritiqueandLMAsJudgevia a newscore_typeargument. Alongside the existing 0.0..1.0Score, the pydantic metrics backend now shipsFineScore(21 float levels),Rating(1..5),Rating10(1..10) andRating20(1..20), plusget_score_type/normalize_scorehelpers (exported assynalinks.Rating, etc.). The generator'srewardproperty becomes an inline enum of the scale, the default instructions spell out the scale in words, and the emitted reward is always normalized back to 0.0..1.0 so downstream rewards/metrics are unchanged. Round-trips throughget_config/from_config.Module._maybe_buildnow converts concreteJsonDataModelarguments toSymbolicDataModelbefore tracing, so modules wrapping aGenerator(agents in particular) stop issuing throwaway LM requests just to discover their output schema. Two agent tests were only passing because the build silently consumed their first two scripted mock responses; they now script the real 3-call sequence. Regression test added inmodule_test.py.uv.lockrefresh.Test plan
uv run pytest synalinks: 2215 passed, 9 skipped (full suite, pre default-instructions change)synalinks/src/modules/ttc,synalinks/src/rewards,backend/pydantic/metrics_test.py,modules/core/generator_test.py: 101 passed after the default-instructions changetest_auto_build_traces_on_symbolic_inputs_for_eager_callsverified to fail on the old_maybe_buildruff+black -l 90clean on all touched files🤖 Generated with Claude Code