Trace anomalies for stall diagnosis: design doc + --json on inspect trace anomalies/http#4524
Open
ransomr wants to merge 6 commits into
Open
Conversation
Designs the affordances requested in issue #81: --json on `inspect trace anomalies`, an `inspect ctl process anomalies` verb (client-side trace-file read, deliberately no HTTP endpoint so it works against a wedged process), and stall-site pointers. Design only — no implementation. Co-authored-by: Ransom Richardson <1209015+ransomr@users.noreply.github.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…otting - ctl process anomalies mirrors every inspect trace anomalies flag (--filter included) via a shared click decorator so they can't drift - Resolve the dead-pid open question: fall back to trace-<pid>.log or .log.gz for post-mortem reads; fix the rotation-vs-gzip conflation - Slot the work into phase 3 (current phase), --json first; remove the post-phase-4 deferral and add the verb to the CLI listings Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
First slice of the trace-log anomalies design (issue #81): both commands emit `{trace_file, as_of, ...}` envelopes per the agent output contract. Anomalies buckets (running/cancelled/errors/timeouts) are always present, rows are `{action, detail, start_time, duration, error?}` with running durations computed against `as_of`. Flags live in a shared `anomalies_options` decorator and the reconstruction in a shared `_trace_anomalies` function, ready for the planned `inspect ctl process anomalies` to reuse. Co-authored-by: Ransom Richardson <1209015+ransomr@users.noreply.github.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…crash - `inspect trace anomalies --json` now always populates the errors/timeouts buckets so an empty list always means "none occurred"; --all gates only the human rendering (design doc updated to match) - fix "anomolies" typo in the shared --all help text and rename the internal anomolies_command* functions - tolerate exit-side action records with no matching enter record (e.g. --filter matching only the exit side, or a truncated log) instead of crashing with RuntimeError Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR contains:
What is the current behavior? (You can also link to an open issue here)
Refs meridianlabs-ai#81 (the issue stays open for the remaining slices).
design/control-channel.mdhas no coverage of the trace log /inspect trace anomaliescontent, even though field use showed that telling an LLM agent about trace anomalies — what, if anything, is still running or hung — is a valuable part of the live-eval diagnosis loop. Todayinspect trace anomaliesandinspect trace httpemit only styled Rich tables (the worst case for an agent parser), nothing ininspect ctljoins to the trace subsystem, andanomalies --filtercrashes with aRuntimeErrorwhen the filter matches an action's completion record but not its start record.What is the new behavior?
Design doc. Adds a "Trace-log anomalies for stall diagnosis" section to
design/control-channel.md(slotted into phase 3, since it needs no server changes), designing:--jsononinspect trace anomaliesandhttp(implemented in this PR — see below).inspect ctl process anomalies [PID] [--filter TEXT] [--all] [--json]— a client-side trace-file read (deliberately no HTTP endpoint, so it works against exactly the busy/hung process it exists to diagnose), placed in theprocessnoun group since trace records carry no task/sample identity, with a dead-pid post-mortem fallback totrace-<pid>.log[.gz].sample show.Implementation (first slice).
--jsonon both commands:inspect trace anomalies --jsonemits a{trace_file, as_of, running, cancelled, errors, timeouts}envelope with per-action rows{action, detail, start_time, duration, error?};runningdurations are computed against the envelope'sas_of. All four buckets are always populated —--allgates only the human table — so an empty list always means "none occurred", never "not collected".inspect trace http --jsonemits a{trace_file, as_of, requests}envelope.--filter/--all/--json) are defined once as a shared click decorator, so the plannedctl process anomaliesmirrors them structurally rather than by convention.TraceAnomaliesNamedTuple (one reconstruction, two renderings), fixing the misspelledanomolies_*names along the way.anomalies --filterno longer raises when a completion record matches the filter but its start record doesn't (orphan completions are now skipped).Covered by new CLI tests (
tests/cli/test_cli_trace.py) and two CHANGELOG entries.Still to come (tracked in meridianlabs-ai#81):
inspect ctl process anomaliesand the stall-site pointers.Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)
No.
Other information:
None.
🤖 Generated with Claude Code