feat(control): inspect ctl sample messages (design + implementation)#4499
Open
ransomr wants to merge 18 commits into
Open
feat(control): inspect ctl sample messages (design + implementation)#4499ransomr wants to merge 18 commits into
ransomr wants to merge 18 commits into
Conversation
Designs (does not implement) a per-sample conversation read — HTTP GET /evals/<id>/sample/messages and CLI `inspect ctl sample messages` — so agents can live-watch a sample's .messages, including samples still in the buffer and not yet written to the .eval log. Co-authored-by: Ransom Richardson <1209015+ransomr@users.noreply.github.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Name the new running-side plumbing: a live TaskState handle on ActiveSample, kept current by set_sample_state (the ContextVar is unreachable from the control server's task) - Correct the events-projection claims: default projection omits ModelEvent.input entirely; truncation covers completions and tool arguments/results - Reconcile --all with the selector-conventions rejection note - Align the wrapped comment in the CLI sketch Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…713-1347 # Conflicts: # design/control-channel.md
Add the sample-conversation snapshot read designed in this PR: a new GET /evals/<id>/sample/messages endpoint and `inspect ctl sample messages TASK SID [EPOCH]` CLI verb. Running samples serve their live TaskState.messages (via a new ActiveSample.live_state handle refreshed by set_sample_state, so buffered-but-unlogged samples work); terminal samples fall back through the recorder buffer to the on-disk log. Summary-projected by default with --tail/--all/--full, deliberately snapshot (no cursor). Co-authored-by: Ransom Richardson <1209015+ransomr@users.noreply.github.com>
mypy failed because inspect_ai.log._log does not explicitly export EvalError (it re-imports it from inspect_ai._util.error). Import both names from the public inspect_ai.log package instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ate --tail; uniform empty-page envelope Address review round 1 on the sample-messages read: - Chain/Plan now call set_sample_state after each solver step, so ActiveSample.live_state (and sample_state()) survive a solver replacing the TaskState object outright (e.g. returning a fork() result or a deepcopy). The @solver decorator's own solver-boundary refresh is dead code (instance-level __call__ patching is bypassed by type-level dunder lookup) — the design doc now records this. - inspect ctl sample messages --tail is validated (>= 1); a negative ?tail= on the raw endpoint clamps to an empty window instead of overshooting the slice. - The no-running-evals --json empty page now carries as_of (None) so the envelope shape stays uniform. - Updated the live_state comments in log/_samples.py and the design doc's "Data source" paragraph to describe the actual refresh mechanism. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…plit messages 404 by origin - Chain/Plan step-boundary set_sample_state now passes the superseded state, making the ActiveSample.live_state refresh a compare-and-swap: a Chain/Plan running inside a fork() subtask (whose copied context reaches the shared ActiveSample) threads a deepcopy lineage the handle never pointed at, so its refreshes can't hijack the sample's main conversation. Regression tests for both Chain and Plan branches; design doc updated. - inspect ctl sample messages now passes not_found_missing_route so a router 404 (older inspect without the endpoint) names version skew instead of misreporting "sample not found". - Drop redundant nested _structured_failures in _run_sample_messages (already wrapped by @_envelope_failures). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…l.qmd Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The example's footer (3 of 15 messages) is only producible with --tail 3; the bare command with the default tail of 20 would show all 15. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pass exclude_fields={events, store, output} to _full_sample and resolve
attachments in core mode (messages are covered; full additionally walks
the excluded events/model-calls), so a terminal 'sample messages' read
no longer deserializes and walks the whole transcript.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…713-1347 # Conflicts: # CHANGELOG.md # src/inspect_ai/_control/server.py
…real-fork test - CAS-guard the @solver wrapper's set_sample_state refreshes (dormant today, but correct if the __call__ patching ever comes alive) - add test_fork_does_not_capture_live_state: real fork()/subtask mechanics, not the deepcopy+spawn simulation; verified it fails when the CAS guard is removed - assert log.status == "success" in eval-based fork tests so in-solver assertion failures can't pass silently as sample errors - document log/_samples.py as the process-wide running-sample registry (module docstring) and tighten the live_state field comment Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Drop the per-call-site explanations (Chain/Plan step loops, @solver wrapper) in favor of the full story on set_sample_state itself. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The table renderer expects the compact projection's flattened fields (unix-float timestamp, top-level tokens/stop_reason/completion), so raw events rendered as a mostly-empty table — less visible data with --full than without. Match the sample messages behavior: in terminal mode, --full pretty-prints the raw JSON (footer/cursor unchanged). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Closed
5 tasks
…713-1347 # Conflicts: # CHANGELOG.md
…713-1347 # Conflicts: # src/inspect_ai/_control/server.py
…713-1347 # Conflicts: # CHANGELOG.md # src/inspect_ai/_control/server.py # tests/_control/test_server.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR contains:
What is the current behavior? (You can also link to an open issue here)
Ref meridianlabs-ai#70
Agents live-watching a sample have no way to read its conversation (
.messages) — especially for samples only in the process buffer, not yet written to the.evallog. The closest surface isinspect ctl sample events, and reconstructing the conversation from the transcript firehose is difficult (messages ride insideModelEventpayloads, solvers can editstate.messageswithout emitting events, compaction rewrites history). A human just opensinspect view; an agent has no equivalent.Relatedly,
inspect ctl sample events --fullrendered less in the terminal than the default: the table renderer expects the compact projection's flattened fields, so raw events degraded to mostly-empty rows.What is the new behavior?
Designs and implements the sample-messages read:
GET /evals/<id>/sample/messages?sample_id=<sid>&epoch=<n>&tail=&full=(attempt-scoped, query-param sample addressing like the sibling reads).inspect ctl sample messages TASK SID [EPOCH]— a new verb in thesamplenoun group with--tail N(default: recent tail of 20, validated ≥ 1),--all,--full,--json;--all/--tailmutually exclusive.as_of/ resolved identity /status/count(uniform even on the no-running-evals empty page), withsample eventsstaying the incremental stream.--fullfor rawChatMessageJSON.samples/sample/events— running samples serve the liveTaskState.messagesvia a newActiveSample.live_statehandle (which is exactly why buffered-only samples work), terminal ones fall back through the recorder buffer to the log with attachments resolved. A newer CLI against an older process reports the missing route as version skew rather than "sample not found".set_sample_staterefreshesActiveSample.live_state, andChain/Plancallset_sample_stateafter each solver step so the handle (andsample_state()) survives a solver replacing theTaskStateobject outright (e.g. returning afork()result or a deepcopy). The step-boundary refresh is compare-and-swap guarded (the step passes the state it superseded) so aChain/Planrunning inside afork()subtask — which reaches the sharedActiveSamplethrough its copied context — can't capture the handle for its branch conversation. The@solverdecorator's own (dormant) solver-boundary refresh passes the same guard defensively. Rationale is consolidated inset_sample_state's docstring; a module docstring onlog/_samples.pydocumentsActiveSampleas the process-wide running-sample registry.sample events --fullfix: in terminal mode raw events are now pretty-printed as JSON (matchingsample messages --full) instead of being squeezed into the compact-fields summary table; the count/cursor footer is unchanged.The design doc's "Sample messages read" section (and its cross-referenced tables/lists) reflects the shipped behavior.
Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)
No.
Other information:
CHANGELOG entries under
## Unreleased. Tests: unit (tests/_control/test_messages.py), CLI incl. the events--fullrendering (test_ctl.py), server route (test_server.py), live integration (test_eval_set_integration.py),Chain/Planstate-threading + fork-isolation regression tests (tests/solver/test_chain.py), and a real-fork()live-state capture regression test (tests/solver/test_fork.py) verified to fail when the CAS guard is removed.Supersedes meridianlabs-ai#72.
🤖 Generated with Claude Code