Extend Agent Trace Workbench

CHANGELOG.md

@@ -4,6 +4,27 @@ All notable changes to Agent Trace Workbench appear in this file.
 
 The version format follows a release cycle. A release adds one coherent capability to the workbench.
 
+## 1.10.0 - 2026-08-04
+
+### Added
+
+- A span detail panel on the run-level error timeline.
+- An Inspect button on each timeline event row that opens the full span record.
+- Clickable timeline markers that open the same span detail panel.
+- A detail panel that shows the span kind, status, offsets, duration, attributes, and tool call.
+- The detail panel shows the recorded arguments, result, outcome, and failure message.
+- A link in the detail panel that jumps to the matching span in the trace waterfall.
+- `GET /api/runs/{run_id}/spans/{span_id}` route that returns the full span record for scripts.
+- `atw span <run_id> <span_id>` command that prints the same span detail.
+- Deterministic tests for the span detail, the API route, the CLI command, and the run page panel.
+
+### Changed
+
+- Version numbers moved to 1.10.0.
+- The error timeline rows now carry an Inspect action beside the waterfall jump.
+- The timeline markers respond to click and keyboard focus.
+- The architecture now includes a span detail layer beside the run-level error timeline.
+
 ## 1.9.0 - 2026-08-04
 
 ### Added

README.md

@@ -28,6 +28,8 @@ Release 1.8 adds an agent comparison overlay to the failure trend. Choose a seco
 
 Release 1.9 adds a run-level error timeline to the run detail page. It marks each failed span at its offset from the run start. Read the same events from the API, the CLI, or a CSV file.
 
+Release 1.10 adds a span detail panel to the error timeline. Click a marker or an event row to open the full span record. Read the same record from the API or the CLI.
+
 ## Value
 
 Agent debugging needs evidence at tool boundaries.
@@ -70,7 +72,7 @@ SQLite runs in WAL mode with a busy timeout. Readers keep a committed snapshot.
 
 - `models.py` defines the portable trace contract.
 - `handlers.py` loads local handler config and applies side-effect guards.
-- `storage.py` owns the SQLite schema, WAL coordination, idempotent ingestion, and local annotations. It computes the review list, bulk labels, and the library report. It computes the failure trend, status breakdown, agent overlay, and run error timeline. It lists the runs for one day and enforces the retention cutoff. A cleanup log records each scheduled sweep.
+- `storage.py` owns the SQLite schema, WAL coordination, idempotent ingestion, and local annotations. It computes the review list, bulk labels, and the library report. It computes the failure trend, status breakdown, agent overlay, and run error timeline. It lists the runs for one day and enforces the retention cutoff. It returns the full record for one span. A cleanup log records each scheduled sweep.
 - `ingestion.py` watches JSON files and returns stable schema error reports.
 - `otlp.py` converts the OTLP JSON encoding to and from the trace contract.
 - `replay.py` runs guarded local handlers and records mismatches.
@@ -303,6 +305,59 @@ python -m agent_trace_workbench.cli timeline run-candidate-001 --format csv
 
 The run page lists each event below the chart. Click an event row to jump to the matching span in the trace waterfall.
 
+## Span detail
+
+Open one failed span from the error timeline.
+
+Click a marker on the chart or an Inspect button on an event row. The page opens a detail panel under the timeline. The panel shows the span kind, status, offsets, and duration. It shows the recorded attributes and tool call.
+
+Read the same record over the API.
+
+```powershell
+curl.exe "http://127.0.0.1:8000/api/runs/run-candidate-001/spans/span-tool-103"
+```
+
+The response carries the full span record.
+
+```json
+{
+  "run_id": "run-candidate-001",
+  "span_id": "span-tool-103",
+  "name": "reserve_inventory",
+  "kind": "tool",
+  "status": "error",
+  "sequence": 3,
+  "parent_span_id": "span-agent-101",
+  "start_offset_ms": 205.0,
+  "end_offset_ms": 260.0,
+  "duration_ms": 55.0,
+  "error": "reservation window expired",
+  "attributes": {
+    "tool.version": "fixture-2"
+  },
+  "tool_call": {
+    "name": "reserve_inventory",
+    "arguments": {
+      "sku": "lamp-01",
+      "quantity": 10
+    },
+    "result": null,
+    "outcome": "failure",
+    "error": "reservation window expired"
+  }
+}
+```
+
+The offsets count from the run start. They match the timeline markers. The `error` field carries the stable failure message.
+
+Use the CLI for scripts.
+
+```powershell
+python -m agent_trace_workbench.cli span run-candidate-001 span-tool-103
+```
+
+A missing run or span returns a 404. A missing CLI span exits with an error. The detail panel links back to the span in the trace waterfall.
+
 ## Failure trend
 
 The dashboard draws a daily failure line for the last 14 days.
@@ -1407,7 +1462,7 @@ curl.exe -X POST http://127.0.0.1:8000/api/traces `
 
 ## Test status
 
-The test suite covers the core flows. It covers storage, ingestion, replay, comparison, search, and annotations. It covers bulk labels, export, review, reports, retention, and scheduled cleanup. It covers the CLI, the API, collector export, and the server scheduler. It covers the dashboard trend, including the agent filter, window selector, day drill-down, status breakdown, overlay, and the run error timeline. The CSV exports have their own tests.
+The test suite covers the core flows. It covers storage, ingestion, replay, comparison, search, and annotations. It covers bulk labels, export, review, reports, retention, and scheduled cleanup. It covers the CLI, the API, collector export, and the server scheduler. It covers the dashboard trend, including the agent filter, window selector, day drill-down, status breakdown, overlay, and the run error timeline. It covers the span detail panel on the error timeline. The CSV exports have their own tests.
 
 Run the checks with these commands.
 
@@ -1418,7 +1473,7 @@ python scripts/check_requirements.py
 python -m compileall agent_trace_workbench tests
 ```
 
-Current verification passes 344 tests, Ruff lint, dependency checks, and Python compilation. CI installs from `requirements-lock.txt` and runs these checks on Python 3.11, 3.12, and 3.13 for every push and pull request.
+Current verification passes 357 tests, Ruff lint, dependency checks, and Python compilation. CI installs from `requirements-lock.txt` and runs these checks on Python 3.11, 3.12, and 3.13 for every push and pull request.
 
 ## Limitations
 
@@ -1492,6 +1547,12 @@ The timeline CSV lists one row per failed span. Clean runs produce only the head
 
 A tool call keeps its recorded error message. Other spans get a generated message.
 
+The span detail panel opens one span at a time. It shows the full record of that span only.
+
+The span detail offsets match the error timeline. They count from the recorded run start.
+
+The span detail panel loads over the API. It needs a running server to fetch a record.
+
 The cleanup history records policy and counts. It does not store the deleted traces.
 
 The report retention line counts runs under the current policy. It uses `older_than_days` from the request or the 30-day default.
@@ -1548,13 +1609,14 @@ The span exporter sends each workbench span as it ends. It does not batch spans.
 - Release 1.7 complete: add a status breakdown beside the daily failure line on the dashboard.
 - Release 1.8 complete: add an agent comparison overlay to the failure trend.
 - Release 1.9 complete: add a run-level error timeline to the run detail page.
-- Release 1.10: add a span detail panel to the run-level error timeline.
+- Release 1.10 complete: add a span detail panel to the run-level error timeline.
+- Release 1.11: add a run error summary to the dashboard trend drill-down.
 
 ## Repository map
 
 `fixtures/` contains meaningful baseline, candidate, and second-agent traces. It also contains a handler config and demo scripts.
 
-`tests/` contains deterministic tests for the core. It covers coordination, guards, search, annotations, OTLP, and export. It covers review, reports, retention cleanup, scheduled cleanup, and the server scheduler. It covers the failure trend, including the agent filter, window selector, day drill-down, status breakdown, overlay, and the run error timeline.
+`tests/` contains deterministic tests for the core. It covers coordination, guards, search, annotations, OTLP, and export. It covers review, reports, retention cleanup, scheduled cleanup, and the server scheduler. It covers the failure trend, including the agent filter, window selector, day drill-down, status breakdown, overlay, and the run error timeline. It covers the span detail panel on the error timeline.
 
 `static/` and `templates/` contain the presentation layer.
 

SECURITY.md

@@ -38,6 +38,6 @@ Pruning deletes runs from the local database. Preview with a dry run before you
 
 | Version | Supported |
 | ------- | --------- |
-| 1.2.x   | Yes       |
-| 1.1.x   | No        |
-| 1.0.x   | No        |
+| 1.10.x  | Yes       |
+| 1.9.x   | No        |
+| 1.8.x   | No        |

agent_trace_workbench/__init__.py

@@ -1,3 +1,3 @@
 """Agent Trace Workbench package."""
 
-__version__ = "1.9.0"
+__version__ = "1.10.0"

agent_trace_workbench/cli.py

@@ -102,6 +102,12 @@ def build_parser() -> argparse.ArgumentParser:
         help="Output format",
     )
 
+    span = subparsers.add_parser(
+        "span", help="Show the full detail of one recorded span"
+    )
+    span.add_argument("run_id")
+    span.add_argument("span_id")
+
     compare = subparsers.add_parser("compare", help="Compare two recorded runs")
     compare.add_argument("run_a")
     compare.add_argument("run_b")
@@ -399,6 +405,11 @@ def main() -> None:
             print(error_timeline_to_csv(timeline), end="")
         else:
             print(json.dumps(timeline, indent=2))
+    elif args.command == "span":
+        detail = store.span_detail(args.run_id, args.span_id)
+        if detail is None:
+            raise SystemExit(f"Span not found: {args.span_id}")
+        print(json.dumps(detail, indent=2))
     elif args.command == "search":
         print(json.dumps(store.search_runs(args.query, args.limit), indent=2))
     elif args.command == "comparisons":

agent_trace_workbench/main.py

@@ -535,6 +535,13 @@ def api_run_timeline(
             raise HTTPException(status_code=400, detail="format must be 'json' or 'csv'")
         return timeline
 
+    @app.get("/api/runs/{run_id}/spans/{span_id}")
+    def api_run_span_detail(run_id: str, span_id: str) -> dict[str, Any]:
+        detail = app.state.store.span_detail(run_id, span_id)
+        if detail is None:
+            raise HTTPException(status_code=404, detail=f"Span not found: {span_id}")
+        return detail
+
     @app.post("/api/traces", status_code=201)
     def api_ingest(trace: TraceDocument, request: Request) -> dict[str, Any]:
         source_name = request.headers.get("x-trace-source", "api.json")

agent_trace_workbench/storage.py

@@ -999,6 +999,43 @@ def error_timeline(self, run_id: str) -> dict[str, Any] | None:
             "events": events,
         }
 
+    def span_detail(self, run_id: str, span_id: str) -> dict[str, Any] | None:
+        """Return the full record of one span with run-relative offsets.
+
+        The detail carries the same fields as the error timeline event
+        plus the recorded attributes and tool call. A reviewer can read
+        one failed span without leaving the timeline. It also reports
+        the offsets from the run start and the stable failure message.
+        Returns None when the run or the span does not exist.
+        """
+
+        with traced_operation(
+            "storage.span_detail", {"run.id": run_id, "span.id": span_id}
+        ):
+            with self._connect() as connection:
+                run = connection.execute(
+                    "SELECT * FROM runs WHERE run_id = ?", (run_id,)
+                ).fetchone()
+                if run is None:
+                    return None
+                row = connection.execute(
+                    "SELECT * FROM spans WHERE run_id = ? AND span_id = ?",
+                    (run_id, span_id),
+                ).fetchone()
+                if row is None:
+                    return None
+        origin = ensure_utc(datetime.fromisoformat(run["started_at"]))
+        detail = _span_row(row)
+        detail["run_id"] = run_id
+        detail["start_offset_ms"] = round(
+            _offset_ms(origin, datetime.fromisoformat(row["start_time"])), 3
+        )
+        detail["end_offset_ms"] = round(
+            _offset_ms(origin, datetime.fromisoformat(row["end_time"])), 3
+        )
+        detail["error"] = _error_message(row)
+        return detail
+
     def update_annotations(
         self,
         run_id: str,

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "agent-trace-workbench"
-version = "1.9.0"
+version = "1.10.0"
 description = "A local workbench for recording, replaying, comparing, and inspecting agent traces."
 readme = "README.md"
 requires-python = ">=3.11"

static/app.js

@@ -199,4 +199,86 @@
             }
         });
     }
+
+    const detailPanel = document.querySelector("#span-detail-panel");
+    if (detailPanel) {
+        const body = detailPanel.querySelector("#span-detail-body");
+        const title = detailPanel.querySelector("#span-detail-name");
+        const runId = detailPanel.dataset.runId;
+        const openDetail = async (spanId) => {
+            title.textContent = "Loading span...";
+            detailPanel.hidden = false;
+            body.className = "span-detail-body";
+            body.textContent = "";
+            try {
+                const response = await fetch(
+                    `/api/runs/${encodeURIComponent(runId)}/spans/${encodeURIComponent(spanId)}`,
+                );
+                const detail = await response.json();
+                if (!response.ok) throw new Error(detail.detail || "Span could not be loaded.");
+                title.textContent = `${detail.name} · ${detail.span_id}`;
+                renderSpanDetail(body, detail);
+            } catch (error) {
+                body.className = "span-detail-body error";
+                body.textContent = error.message;
+            }
+        };
+        document.querySelectorAll("[data-span-id]").forEach((target) => {
+            target.addEventListener("click", () => openDetail(target.dataset.spanId));
+        });
+        document.querySelector("#span-detail-close").addEventListener("click", () => {
+            detailPanel.hidden = true;
+        });
+    }
+
+    const renderSpanDetail = (body, detail) => {
+        body.textContent = "";
+        const meta = document.createElement("p");
+        meta.className = "span-detail-meta";
+        meta.textContent = `${detail.kind} · ${detail.status} · ${detail.start_offset_ms} ms to ${detail.end_offset_ms} ms · ${detail.duration_ms} ms`;
+        body.append(meta);
+        if (detail.error) {
+            const box = document.createElement("div");
+            box.className = "failure-box";
+            const label = document.createElement("strong");
+            label.textContent = "Failure";
+            const message = document.createElement("span");
+            message.textContent = detail.error;
+            box.append(label, message);
+            body.append(box);
+        }
+        if (detail.tool_call) {
+            const grid = document.createElement("div");
+            grid.className = "tool-grid";
+            grid.append(spanJsonCell("Arguments", detail.tool_call.arguments));
+            grid.append(spanJsonCell("Result", detail.tool_call.result));
+            body.append(grid);
+        }
+        if (detail.attributes && Object.keys(detail.attributes).length) {
+            const attributes = document.createElement("details");
+            attributes.className = "attributes";
+            const summary = document.createElement("summary");
+            summary.textContent = "Span attributes";
+            const pre = document.createElement("pre");
+            pre.textContent = JSON.stringify(detail.attributes, null, 2);
+            attributes.append(summary, pre);
+            body.append(attributes);
+        }
+        const waterfall = document.createElement("a");
+        waterfall.className = "button button-quiet span-detail-link";
+        waterfall.href = `#span-${encodeURIComponent(detail.span_id)}`;
+        waterfall.textContent = "Open in waterfall";
+        body.append(waterfall);
+    };
+
+    const spanJsonCell = (label, value) => {
+        const cell = document.createElement("div");
+        const heading = document.createElement("span");
+        heading.className = "field-label";
+        heading.textContent = label;
+        const pre = document.createElement("pre");
+        pre.textContent = value === undefined || value === null ? "null" : JSON.stringify(value, null, 2);
+        cell.append(heading, pre);
+        return cell;
+    };
 })();

static/styles.css

@@ -122,10 +122,24 @@ select { min-height: 44px; padding: 0 12px; font-family: inherit; }
 .timeline-foot { display: flex; justify-content: flex-end; gap: 8px; margin-top: 16px; padding-top: 14px; border-top: 1px solid var(--line); }
 .timeline-foot .button { min-height: 36px; padding: 0 11px; font-size: 10px; }
 .timeline-list { margin-top: 18px; border-top: 1px solid var(--line); }
-.timeline-row { display: grid; grid-template-columns: 14px 1fr 2fr; align-items: center; gap: 12px; padding: 13px 0; border-bottom: 1px solid var(--line); }
-.timeline-row:hover strong, .timeline-row:hover .timeline-error { color: var(--blue); }
-.timeline-row strong { display: block; margin-bottom: 3px; color: var(--navy); font-size: 14px; }
-.timeline-row small { color: var(--muted); font: 10px ui-monospace, SFMono-Regular, Consolas, monospace; }
+.timeline-row { display: flex; align-items: center; gap: 12px; padding: 13px 0; border-bottom: 1px solid var(--line); }
+.timeline-row-main { display: grid; grid-template-columns: 14px 1fr 2fr; align-items: center; gap: 12px; color: inherit; }
+.timeline-row-main:hover strong, .timeline-row-main:hover .timeline-error { color: var(--blue); }
+.timeline-row-main strong { display: block; margin-bottom: 3px; color: var(--navy); font-size: 14px; }
+.timeline-row-main small { color: var(--muted); font: 10px ui-monospace, SFMono-Regular, Consolas, monospace; }
+.timeline-inspect { min-height: 30px; padding: 0 10px; font-size: 10px; }
+.timeline-event-click { cursor: pointer; }
+.timeline-event-click:focus-visible { outline: 0; }
+.timeline-event-click:focus-visible .timeline-dot, .timeline-event-click:hover .timeline-dot { fill: var(--navy); stroke-width: 2; }
+.span-detail-panel { margin-top: 18px; padding: 22px 24px; border: 1px solid var(--line); border-left: 3px solid var(--coral); background: #fff; }
+.span-detail-head { display: flex; align-items: flex-start; justify-content: space-between; gap: 16px; }
+.span-detail-head h3 { margin-top: 7px; color: var(--navy); font-size: 17px; }
+.span-detail-body { margin-top: 16px; }
+.span-detail-body.error { color: var(--coral); font-size: 12px; }
+.span-detail-meta { margin-bottom: 0; color: var(--muted); font: 10px ui-monospace, SFMono-Regular, Consolas, monospace; }
+.span-detail-body .failure-box { margin-top: 12px; }
+.span-detail-body .tool-grid { margin-top: 14px; }
+.span-detail-link { margin-top: 16px; min-height: 36px; padding: 0 11px; font-size: 10px; }
 .timeline-marker { width: 8px; height: 8px; border-radius: 50%; background: var(--coral); }
 .timeline-error { color: var(--coral); font-size: 12px; }
 .trace-list, .replay-list { border-top: 1px solid var(--line); }