Trace anomalies for stall diagnosis: design doc + --json on inspect trace anomalies/http

CHANGELOG.md

@@ -15,6 +15,8 @@
 - Control Channel: `inspect ctl sample events` now accepts `--type all` (shell-safe spelling of `--type '*'`) and `--from-start` to read the full backlog from the first event.
 - Control Channel: `inspect ctl sample events` gains `--limit N` to cap the events returned per page (combinable with `--from-start`, `--tail`, or a cursor).
 - Control Channel: New `--json` flag for `inspect eval`, `inspect eval-set`, and `inspect eval-retry` emits machine-readable launch-handoff and `done` JSON lines on stdout, so agents can use `inspect ctl` immediately after launch instead of sleep-and-retrying.
+- CLI: `inspect trace anomalies` and `inspect trace http` gain a `--json` flag emitting machine-readable output for agents and shell pipelines; the JSON always includes error/timeout buckets (no `--all` needed).
+- CLI: `inspect trace anomalies` no longer errors when `--filter` matches an action's completion record but not its start record.
 - CLI: `--debug` attach diagnostics ("Waiting for debugger attach") now print to stderr, keeping stdout clean for machine-readable output such as `--json`.
 - Control Channel: `inspect ctl sample show` now reads the sample's summary and error detail in one atomic request, so its output can no longer mix data from different attempts.
 - Control Channel: `inspect ctl sample show` now reports a cancelled sample as `cancelled`/`pending` (matching `sample list`) instead of `error` with the cancellation as its error.

design/control-channel.md

@@ -8,7 +8,7 @@ This is **separate from** the [`agent-acp`](acp/agent-acp.md) work, even though
 
 The rudimentary control surface that fell out of the ACP work (per-sample cancellation, socket discovery via `--acp-server`) is a useful precedent but not the foundation — the control channel deserves its own protocol choice.
 
-> **Status (phases 1–2 shipped).** The read surface, per-sample events, and process keep-alive are implemented: the embedded FastAPI server on AF_UNIX, discovery, the `GET /tasks` / `GET /evals/<id>/samples` (with an `active_since` recency delta) / `GET /evals/<id>/sample` / `GET /evals/<id>/sample/events` read endpoints, `POST /release`, the `inspect ctl` CLI (organized into resource-noun groups — `ctl task` / `ctl sample` / `ctl config` / `ctl process`; see "CLI command hierarchy" below), and the `--ctl-server` flag (on by default; `false` disables, `keep` parks the process after the eval). **Phase 2** added the cursored-pull per-sample transcript `events` API plus the recency-delta filter on `samples`. **Phase 3** (in progress) adds the state-mutating directives — the log-flush directive (`ctl task log-flush`), the dynamic-config directive (`ctl config` — the `max_samples` / `max_sandboxes` / `max_subprocesses` / `max_connections` concurrency knobs plus the `log_buffer` / `log_shared` buffer params), and the cancel directives (`ctl task cancel` / `ctl sample cancel`) are shipped; adding a task to a running eval, then drain / requeue and the per-sample time/token/message limits follow; **phase 4** adds the push (SSE / `--follow`) shape, including the eval-wide fan-in. Much of the prose below describes the full target surface — see [Implementation](#implementation) for what's built vs planned, which is the source of truth for phasing.
+> **Status (phases 1–2 shipped).** The read surface, per-sample events, and process keep-alive are implemented: the embedded FastAPI server on AF_UNIX, discovery, the `GET /tasks` / `GET /evals/<id>/samples` (with an `active_since` recency delta) / `GET /evals/<id>/sample` / `GET /evals/<id>/sample/events` read endpoints, `POST /release`, the `inspect ctl` CLI (organized into resource-noun groups — `ctl task` / `ctl sample` / `ctl config` / `ctl process`; see "CLI command hierarchy" below), and the `--ctl-server` flag (on by default; `false` disables, `keep` parks the process after the eval). **Phase 2** added the cursored-pull per-sample transcript `events` API plus the recency-delta filter on `samples`. **Phase 3** (in progress) adds the state-mutating directives — the log-flush directive (`ctl task log-flush`), the dynamic-config directive (`ctl config` — the `max_samples` / `max_sandboxes` / `max_subprocesses` / `max_connections` concurrency knobs plus the `log_buffer` / `log_shared` buffer params), and the cancel directives (`ctl task cancel` / `ctl sample cancel`) are shipped; adding a task to a running eval, the trace-log anomalies affordances (`--json` on `inspect trace anomalies`, `ctl process anomalies`), then drain / requeue and the per-sample time/token/message limits follow; **phase 4** adds the push (SSE / `--follow`) shape, including the eval-wide fan-in. Much of the prose below describes the full target surface — see [Implementation](#implementation) for what's built vs planned, which is the source of truth for phasing.
 
 ## Goals
 
@@ -83,7 +83,7 @@ inspect ctl process list                    # running Inspect processes (pid / k
 inspect ctl process keep [PID]              # keep a running process alive after its eval finishes
 inspect ctl process release [PID]           # release a lingering keep-alive process
 
-# directives (phase 3; first slices shipped)
+# directives + diagnostics (phase 3; first slices shipped)
 inspect ctl config [TASK] [--max-samples N] [--max-connections N] [--max-sandboxes N]
                    [--max-subprocesses N] [--log-buffer N] [--log-shared S] [--model M] [--dry-run]
                                             # view / retune launch config mid-flight (shipped)
@@ -94,6 +94,8 @@ inspect ctl task add SPEC [--model M] [-T k=v] [--dry-run]  # add a task to a ru
 inspect ctl task drain TASK                 # stop accepting new samples; let in-flight finish
 inspect ctl sample requeue TASK SID EPOCH   # re-add a failed sample to the queue
 inspect ctl config TASK --time-limit N     # modify per-sample time/token/message limits (planned)
+inspect ctl process anomalies [PID] [--filter TEXT] [--all]  # in-flight / anomalous actions from the pid's trace file
+                                            #   (client-side read — works against a busy or hung process)
 ```
 
 Note the commands take a **task** selector (task-id prefix or task name), not a raw eval-id: a task id is stable across retries, whereas a per-attempt eval id is not. Process-lifecycle commands take a `PID` (optional when a single process is running).
@@ -141,6 +143,7 @@ inspect ctl
 ├── config [TASK] [...]         # was: limits + buffer; view/retune launch config, scope per knob
 ├── process                     # the running Inspect process itself (PID selector)
 │   ├── list                        # new: pids / keep-alive / hosted tasks (implied by bare `ctl process`)
+│   ├── anomalies [PID]             # planned: in-flight / anomalous actions from the trace file (client-side read)
 │   ├── keep [PID]                  # was: keep [--pid]
 │   └── release [PID]               # was: release [--pid]
 └── eval-set                    # later, with the eval-set surface
@@ -489,6 +492,36 @@ Open question: which knobs are realistically directable mid-flight without rearc
 
 Each is a small surgical addition to the eval runner; the control channel is the wire that triggers them.
 
+### Trace-log anomalies for stall diagnosis (planned)
+
+Field feedback from agent-driven eval babysitting (JJ, via issue #81): telling the LLM agent about `inspect trace anomalies` — what, if anything, is still running or hung in the process — was handy enough that it deserves first-class affordances rather than prompt lore. This section designs those affordances; the first slice — `--json` on `inspect trace anomalies` and `http` — has shipped, the rest is not implemented yet. Slotted into **phase 3** (the current phase) rather than after phase 4 — it needs no server changes, so it doesn't have to wait on the push work; the `ctl` verb and the stall-site pointers follow.
+
+**The gap.** The control channel's read surface answers *what* is stalled but not *why*. `sample list` carries `last_activity_at` / idle time; `sample events` shows the transcript tail — but both share a documented blind spot (see the `GET /evals/<id>/samples` note in Phase 1): a single in-flight operation emits no transcript event until it returns, so a sample idle for twenty minutes reads identically whether it's a long-but-healthy model call, a hung sandbox exec, or a wedged process. The trace log is exactly the layer below the transcript: every `trace_action` in the process (model generates, subprocess execs, sandbox operations, log writes, ...) logs `enter` / `exit` / `cancel` / `error` / `timeout` records to a per-process file, and the `inspect trace anomalies` reconstruction derives from them **what is still running right now** (entered, never exited — with live durations) plus what was cancelled, errored, or timed out. "Sample 5 idle 21m" plus "one `Model` action running 21m" is a diagnosis; either alone is a symptom.
+
+**What exists today** (outside `ctl`): the `inspect trace` command group — `list` (has `--json`), `dump` (JSON records, `--filter`), `http [--failed]`, and `anomalies [TRACE_FILE] [--filter] [--all]` (default buckets: running + cancelled actions; `--all` adds errors and timeouts). Trace files live at `<inspect_data_dir>/traces/trace-<pid>.log` — **pid-keyed, the same key as the control discovery files**. Two separate housekeeping mechanisms apply (easy to conflate): a keep-newest-10 **rotation** sweep at logger init deletes the oldest files, and an `atexit` hook **gzips** the file to `trace-<pid>.log.gz` on clean exit — so a live or hard-killed process leaves a bare `.log`, a cleanly-exited one a `.log.gz` (which `read_trace_file` reads transparently). Measured against the agent shape constraints, two gaps:
+
+- **No `--json` on `anomalies` (or `http`).** The anomalies output is a rich table exported *with ANSI styles* — the worst case for an agent parser. Constraint 1 says the JSON schema is the canonical shape and the human table a rendering of it; here there is no JSON shape at all. *(Closed: both commands now take `--json`, per the first affordance below.)*
+- **No join from the `ctl` surface.** Nothing in `inspect ctl` mentions traces: the agent has to know the subsystem exists, map its task to a pid, and map the pid to a file. Today that knowledge arrives via the agent's prompt ("try `inspect trace anomalies`"), which is exactly what surface-level discoverability (`--help`, teach-through-the-error) is supposed to replace.
+
+**Design shape.** Two decisions, then the affordances.
+
+1. **Client-side file read — deliberately no HTTP endpoint.** The prime scenario for anomalies is a process that is *unresponsive* — precisely the case where the control server (which shares the eval's loop; see "Busy is not absent") can't answer, so a `GET /trace/anomalies` would time out exactly when it matters most. The trace file needs no server participation: it's on local disk, pid-keyed, locatable from the same discovery files `ctl` already reads, and it outlives the process — a post-mortem "what was running when it died" works on a dead pid's file (as `.log` or, after a clean exit, `.log.gz`), which survives until the keep-newest-10 rotation sweep reaches it. So the affordance is CLI-side composition: resolve selector → pid → `trace-<pid>.log`, then run the same reconstruction `inspect trace anomalies` runs. Locality is not a new constraint — the CLI and the eval already share a machine by construction (AF_UNIX-only transport). If remote attach ever lands, the trace read is the one read that would need a server-side path; noted, not designed.
+
+2. **One reconstruction, two entry points.** The analysis stays in `inspect trace` (it isn't eval-specific — it works on any trace file, including rotated ones from long-dead processes); `ctl` adds a thin verb that resolves the selector and delegates to the same shared implementation function, so the two entry points can't drift into different answers.
+
+The affordances:
+
+- **`--json` on `inspect trace anomalies`** (the prerequisite for everything else; `http` gained it in the same pass). **Shipped.** An envelope, not bare buckets, per the agent output contract: `{"trace_file": ..., "as_of": <ts>, "running": [...], "cancelled": [...], "errors": [...], "timeouts": [...]}`, with per-action rows `{action, detail, start_time, duration, error?}`. `running` rows compute `duration` as now − `start_time` at read time (matching the human table). All four bucket keys always present **and always populated** — `--all` gates only the human rendering, so in the envelope an empty list always means "none occurred", never "not collected" (an agent diagnosing a stall must not read "not asked" as "no errors"). Empty lists, never presence-conditional (same rule as unconditional `task_id` on sample rows).
+- **`inspect ctl process anomalies [PID] [--filter TEXT] [--all] [--json]`.** Noun placement per the CLI hierarchy: the object is the **process** — trace files are per-pid, and `trace_action` records carry no task or sample identity (a process hosts an eval-set's many tasks plus shared machinery: log writes, sandbox pulls, the control service itself), so attributing actions to a task would be guesswork the surface shouldn't fake. Selector conventions apply as a *read*: no `PID` widens over every discovered live process, one section (one JSON object) per pid, each carrying `pid` unconditionally (outputs feed inputs). Because the verb is client-side composition over discovery + the trace file, it — unlike every other `ctl` read — **works against a busy or hung process**: it is the natural escalation path for the "pid N busy — try again shortly" outcome documented under "Busy is not absent". The verb mirrors **every** `inspect trace anomalies` flag — `--filter` (substring match on the message field), `--all` (human table shows running + cancelled by default, errors and timeouts opt-in; the JSON envelope always carries all four buckets), and the new `--json` — and the mirroring is structural, not conventional: the options are defined once as a shared click decorator applied to both commands (the same "one reconstruction" move as decision 2, at the flag layer), so the two entry points can't drift as flags are added.
+- **Dead-pid post-mortem fallback.** `ctl process anomalies <pid>` on a pid with *no* discovery file (process exited, discovery swept) falls back to the trace file directly when it can find one — matching both `trace-<pid>.log` (still-live or hard-killed process) and `trace-<pid>.log.gz` (clean exit gzips via the `atexit` hook; the shared reader handles `.gz` transparently). The pid → file mapping is trivial and the post-mortem is a real agent scenario: eval died overnight — what was in flight when it died? Only when neither file exists (swept by rotation) does the verb error, naming the path it looked for so the user can hunt for a copy and hand it to `inspect trace anomalies <file>` directly.
+- **Teach through the surface at the stall site.** Where the human rendering already shows a stall — a `sample list` idle column in the tens of minutes, the busy-skip stderr note — print the pointer: "idle 27m — `inspect ctl process anomalies <pid>` shows the process's in-flight actions". Same move as the group-level unknown-command handler: the surface teaches the escalation at the moment it's needed, instead of relying on the user (or the agent's system prompt) knowing the trace subsystem exists. On `--json` the hint is omitted (no prose inside envelopes); JSON consumers learn the verb from `--help`.
+
+**What this is not.** Not a new event stream — the trace file is already an append-only log with its own tooling (`trace dump --filter` remains the drill-down). Not a per-sample view — the records carry no sample identity, and if per-sample attribution ever matters the fix is upstream (stamp identity onto `trace_action` records), not heuristics in the reader. Not part of the HTTP API — deliberately, per decision 1. The `ctl` verb adds selector resolution and placement in the surface agents already discover; the analysis itself is unchanged.
+
+**Open questions.**
+
+- Should `sample show` on a long-idle sample *inline* the process's running actions (running the reconstruction server-side of the CLI, one fewer turn for the agent) rather than just printing the pointer? Inlining couples sample reads to trace-file parsing and re-raises the attribution problem (the actions shown belong to the process, not necessarily to that sample) — start with the pointer.
+
 ### Security model
 
 The control endpoint is default-on and unauthenticated. That's a deliberate trade-off — given the threat model it's the right one, but it deserves a written account.
@@ -746,6 +779,8 @@ Unlocks watchdog agents (cursored polling). Live-render TUIs follow once phase-4
 
 The first endpoints that **mutate the run**, each idempotent and supporting `?dry_run=true` / `--dry-run` from day one. Shipped so far: the log-flush and buffer-params directives, the concurrency-config directive (`max_samples` / `max_sandboxes` / `max_subprocesses` / `max_connections`) — surfaced through `ctl task log-flush` and `ctl config` — and the cancel directives (`ctl task cancel` / `ctl sample cancel`); adding a task to a running eval, then drain / requeue and the per-sample time/token/message limits follow. The Security model's "future hardening" (SO_PEERCRED UID check, self-targeting guard) lands with this phase, since it introduces the first state-mutating writes.
 
+This phase also carries the **trace-log anomalies affordances** (see "Trace-log anomalies for stall diagnosis") — not a directive, but scheduled here because the diagnosis need is current and the work touches no server code: `--json` on `inspect trace anomalies` (and `http`) first (shipped), then `inspect ctl process anomalies` (client-side trace-file read; deliberately no HTTP endpoint) and the stall-site pointers.
+
 #### Flush buffered samples + tune buffer params (shipped)
 
 Two small, naturally-idempotent buffer directives shipped ahead of the larger phase-3 work, motivated by long S3-backed runs whose completed samples sit in the local flush buffer (and aren't analyzable in the log) until enough queue up to trigger a write: