eval: durable per-trial action log side-car (plan 0067) - #371
Merged
Conversation
Closes-Plan-For: #369 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every delivered trial now writes its executed (post-approval) action sequence to <log_dir>/actions/<run_stamp>/<trial>.jsonl — atomic temp+fsync+os.replace write, header line + one line per control step, default-on via store_actions=True on eval()/eval_set(), best-effort degrade (warning, no file, no pointer) on OSError or non-finite actions. The relative path rides record.metadata["actions"], now framework- reserved. tests/conftest.py grows a delta-based no-litter gate. Closes #369 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #369
What
Every delivered trial now writes its executed action sequence to a durable JSONL side-car, default-on:
Header line (run id, raw scene id, epoch, action dim, optional
dim_labels), then one line per control step with the post-approval action vector — the complete commanded trajectory, including the prefix of errored/cancelled trials and header-only files for zero-step and start-failed trials. The relative path lands inrecord.metadata["actions"](now framework-reserved, liketranscript/wire_capture).Why
Commanded actions previously reached disk through exactly one path: the RerunSink
.rrdtee (#340), which drop-sheds under pressure by contract.JsonLogSinkpersists no per-step data andFrameStoreis frames-only, so under viewer load a step's action could exist nowhere on disk — anyone reconstructing trajectories offline had to interpolate between LLM-call boundaries (the GPT-5.6 Sol eval report hit exactly this).Design (plan 0067, 7 critique rounds → READY)
run_stamp(R5 ownership).TrialRecord.steps— zero I/O in the control-rate loop, no shedding concern; partial records are already recovered on every error path.os.replace;allow_nan=Falseper the strict-JSON convention. Degrade is caught inside the helper: warning, no file, no pointer, eval status untouched.frames._safe; the header keeps the raw scene id.sinks=now also writeactions/beneathlog_dirunless they passstore_actions=False.tests/conftest.pygrows a delta-basedpytest_sessionstart/sessionfinishno-litter gate (fails only if the suite created repo-rootlogs/, then cleans it up).Testing
tests/test_eval_action_log.pycovering: happy path, clamped-action fidelity, errored/cancelled/start-failed/zero-step trials,epochs>1,_safeparity, all threedim_labelscases, non-finite degrade (end-to-end via NaN-tolerant embodiment + helper seam), I/O degrade, and behavioraleval_setforwarding.log_dir) updated so the suite leaves no checkout litter.Follow-up filed separately: #370 (agent plugin side-cars use raw scene ids).
🤖 Generated with Claude Code