Skip to content

eval: durable per-trial action log side-car (plan 0067) - #371

Merged
jeqcho merged 9 commits into
mainfrom
feat/action-log
Aug 11, 2026
Merged

eval: durable per-trial action log side-car (plan 0067)#371
jeqcho merged 9 commits into
mainfrom
feat/action-log

Conversation

@jeqcho

@jeqcho jeqcho commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

Closes #369

What

Every delivered trial now writes its executed action sequence to a durable JSONL side-car, default-on:

<log_dir>/actions/<run_stamp>/<sanitized_scene_id>-e<epoch>.jsonl

Header line (run id, raw scene id, epoch, action dim, optional dim_labels), then one line per control step with the post-approval action vector — the complete commanded trajectory, including the prefix of errored/cancelled trials and header-only files for zero-step and start-failed trials. The relative path lands in record.metadata["actions"] (now framework-reserved, like transcript/wire_capture).

Why

Commanded actions previously reached disk through exactly one path: the RerunSink .rrd tee (#340), which drop-sheds under pressure by contract. JsonLogSink persists no per-step data and FrameStore is frames-only, so under viewer load a step's action could exist nowhere on disk — anyone reconstructing trajectories offline had to interpolate between LLM-call boundaries (the GPT-5.6 Sol eval report hit exactly this).

Design (plan 0067, 7 critique rounds → READY)

  • Eval-owned, not a sink — mirrors the transcript side-car precedent; sinks never learn run_stamp (R5 ownership).
  • One-shot write at trial end from TrialRecord.steps — zero I/O in the control-rate loop, no shedding concern; partial records are already recovered on every error path.
  • Atomic: serialize fully in memory, temp + flush + fsync + os.replace; allow_nan=False per the strict-JSON convention. Degrade is caught inside the helper: warning, no file, no pointer, eval status untouched.
  • Filenames reuse frames._safe; the header keeps the raw scene id.
  • Behavior change (called out in CHANGELOG): callers passing custom sinks= now also write actions/ beneath log_dir unless they pass store_actions=False.
  • tests/conftest.py grows a delta-based pytest_sessionstart/sessionfinish no-litter gate (fails only if the suite created repo-root logs/, then cleans it up).

Testing

  • 15 new tests in tests/test_eval_action_log.py covering: happy path, clamped-action fidelity, errored/cancelled/start-failed/zero-step trials, epochs>1, _safe parity, all three dim_labels cases, non-finite degrade (end-to-end via NaN-tolerant embodiment + helper seam), I/O degrade, and behavioral eval_set forwarding.
  • 12 existing call sites (custom sinks, no log_dir) updated so the suite leaves no checkout litter.
  • Full suite: 1487 passed, 6 skipped. Coverage 100% (core scope). mypy --strict clean.

Follow-up filed separately: #370 (agent plugin side-cars use raw scene ids).

🤖 Generated with Claude Code

jeqcho and others added 9 commits August 11, 2026 11:37
Closes-Plan-For: #369

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every delivered trial now writes its executed (post-approval) action
sequence to <log_dir>/actions/<run_stamp>/<trial>.jsonl — atomic
temp+fsync+os.replace write, header line + one line per control step,
default-on via store_actions=True on eval()/eval_set(), best-effort
degrade (warning, no file, no pointer) on OSError or non-finite actions.
The relative path rides record.metadata["actions"], now framework-
reserved. tests/conftest.py grows a delta-based no-litter gate.

Closes #369

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@jeqcho
jeqcho merged commit 0160995 into main Aug 11, 2026
26 checks passed
@jeqcho
jeqcho deleted the feat/action-log branch August 11, 2026 20:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Per-step actions are never persisted durably — add a default-on per-trial action log

1 participant