Timeline: require a tool-calling loop for utility-agent classification#4516
Open
dragonstyle wants to merge 5 commits into
Open
Timeline: require a tool-calling loop for utility-agent classification#4516dragonstyle wants to merge 5 commits into
dragonstyle wants to merge 5 commits into
Conversation
The utility heuristics in timeline_build (foreign-prompt event wrapping and single-turn span classification) assumed a "primary trajectory" defined by the first system prompt seen. In plain workflows of generate() calls — e.g. a monitor making several checker calls with different prompts and no tools — this hid every call after the first prompt group as a utility agent. Both passes now only apply when the span actually contains an agentic (tool-calling) loop: _wrap_utility_events derives the primary prompt solely from the first tool-calling ModelEvent (no fallback to the first event), and _classify_utility_agents only demotes single-turn sub-spans whose parent runs a tool loop. Warmup-call wrapping is unchanged. Adds workflow_generate (mixed-prompt calls, no tools → nothing hidden) and bridge_utility (foreign-prompt call within a tool loop → still utility) scenarios. Mirrors the same fix in ts-mono's core.ts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Move the utility-classification test cases into JSON fixtures under
tests/timeline/fixtures/events/ ({description, events, expected}) so a
single definition drives both timeline implementations: the new
test_timeline_fixtures.py validates timeline_build() against them, and
ts-mono's fixtureTimeline.test.ts discovers the same directory to validate
buildTimeline(). The scenario-based duplicates (workflow_generate,
bridge_utility) are removed from generate.py/test_timeline.py, and two
additional cases cover single-turn sub-agent classification with and
without a parent tool-calling loop.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two fixes from code review: - _wrap_utility_events recursed into the synthetic wrapper spans it had just created; a wrapper's warmup call was wrapped again inside it, forever (RecursionError on any transcript with a cache-priming call). Recursion now visits only the span's original children. Also skips system-prompt extraction when there is no primary prompt to compare against. Adds a utility_warmup fixture covering warmup wrapping (which is prompt-independent and applies without an agentic loop). - log_viewer.yml gains a viewer-tests job running the inspect-components vitest suite: the fixture-driven timeline tests consume this repo's fixtures and are skipped in ts-mono's standalone CI, so without this job they ran nowhere. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Four parity fixes between timeline_build and ts-mono's buildTimeline, each now covered by a shared JSON fixture (or unit test where the fixture format can't express the case): - _get_system_prompt (classification) now delegates to _get_system_prompt_for_event, so billing-header normalization applies to span-level prompt comparison (previously raw prompts were compared, classifying Claude Code sub-agents differently than the viewer). - Prompt extraction returns None instead of "" when a prompt is empty after normalization (e.g. only the billing header line), keeping empty prompts out of comparisons and prompt inheritance. - _wrap_utility_events processes a branch's own direct events (branches are trajectories like any span), matching the TS port. - Wrapper spans for uuid-less (legacy) events get unique deterministic position-derived ids (utility-<span>-<index>) instead of id(item). Fixtures: utility_billing_header, utility_header_only_prompt, utility_branch_warmup (the fixture harness gains branch assertions). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ssification # Conflicts: # CHANGELOG.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR contains:
What is the current behavior? (You can also link to an open issue here)
Transcript timelines classify model calls as hidden "utility agents" using a weak heuristic: the first system prompt seen defines the "main trajectory", and any no-tool-call model event with a different prompt is wrapped as a utility span (hidden by default in the viewer). In a plain workflow of
generate()calls with mixed system prompts — monitors, judges, graders — this silently hides every call after the first prompt group (reported on Slack: 5 checker calls, only 2 rendered). There is also no indication anywhere in the UI that events were hidden.Additionally,
_wrap_utility_eventsrecursed into the synthetic wrapper spans it creates, producing infinite recursion (RecursionError) on any transcript containing a cache-warmup call (max_tokens=1, single-word user message).What is the new behavior?
_wrap_utility_eventsderives the primary prompt solely from the first tool-calling ModelEvent, and_classify_utility_agentsonly demotes single-turn sub-spans whose parent runs a tool loop. Workflows of plaingenerate()calls render every call.x-anthropic-billing-headerline; prompts that normalize to empty are treated as absent; branch direct content is processed like any span's; uuid-less wrapped events get unique deterministic wrapper ids.tests/timeline/fixtures/events/({description, events, expected}) and validated by bothtimeline_build()(newtests/timeline/test_timeline_fixtures.py) and ts-mono'sbuildTimeline()(its fixture suite discovers this directory). 8 fixtures cover the gate, warmup wrapping, billing-header normalization, empty prompts, and branch content.viewer-testsjob inlog_viewer.ymlruns the inspect-components vitest suite in this embedding — the fixture-driven TS tests are skipped in ts-mono's standalone CI, so previously they would have run nowhere.Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)
No API changes. Behavior change: transcripts whose "main loop" never calls tools (e.g. chat-only bridge sessions with interleaved extraction calls) no longer hide those extraction calls — with no tool loop the heuristic has no basis for declaring a primary trajectory, and showing everything is the safe default.
Other information:
Companion PRs:
The submodule pointer and regenerated dist are intentionally not included; they follow after the ts-mono PR merges. Verified all 50 of inspect_scout's fixtures pass against the new TS logic, and scout's full Python timeline suite (84 tests) passes against this branch.
🤖 Generated with Claude Code