DanielCuevas1208 · DanielCuevas1208 · Aug 4, 2026 · Aug 4, 2026
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -4,6 +4,29 @@ All notable changes to Agent Trace Workbench appear in this file.
 
 The version format follows a release cycle. A release adds one coherent capability to the workbench.
 
+## 1.9.0 - 2026-08-04
+
+### Added
+
+- A run-level error timeline on the run detail page.
+- A horizontal time axis that marks each failed span at its offset from the run start.
+- A marker chart that shows when the failures happened inside one run.
+- An event list below the chart with each span name, kind, offset, and failure message.
+- A clickable event row that jumps to the matching span in the trace waterfall.
+- `GET /api/runs/{run_id}/timeline` route that returns the failed spans for scripts.
+- `GET /api/runs/{run_id}/timeline?format=csv` route that returns the events as a CSV attachment.
+- `atw timeline <run_id>` command that prints the error timeline.
+- `atw timeline <run_id> --format csv` command that prints the events as CSV.
+- Run page JSON and CSV download links for the timeline panel.
+- Deterministic tests for the timeline, the API routes, the CLI options, the CSV export, and the run page panel.
+
+### Changed
+
+- Version numbers moved to 1.9.0.
+- The run detail page now shows when failures happened beside the trace waterfall.
+- The trace waterfall rows carry span IDs so the timeline can link to them.
+- The architecture now includes a run-level error timeline beside the trace inspection layer.
+
 ## 1.8.0 - 2026-08-04
 
 ### Added

diff --git a/README.md b/README.md
@@ -26,11 +26,13 @@ Release 1.7 adds a status breakdown beside the daily failure line. Each trend da
 
 Release 1.8 adds an agent comparison overlay to the failure trend. Choose a second agent on the dashboard and the chart draws its failure line beside the primary series. Read both series from the API, the CLI, or a CSV file.
 
+Release 1.9 adds a run-level error timeline to the run detail page. It marks each failed span at its offset from the run start. Read the same events from the API, the CLI, or a CSV file.
+
 ## Value
 
 Agent debugging needs evidence at tool boundaries.
 
-This workbench makes each boundary visible. It shows inputs, outputs, timing, attributes, and errors in one local record.
+This workbench makes each boundary visible. It shows inputs, outputs, timing, attributes, and errors in one local record. The error timeline marks where each failure happened.
 
 The design supports repeatable review. A replay runs a registered local handler when available. It uses the recorded result when no handler exists.
 
@@ -68,14 +70,14 @@ SQLite runs in WAL mode with a busy timeout. Readers keep a committed snapshot.
 
 - `models.py` defines the portable trace contract.
 - `handlers.py` loads local handler config and applies side-effect guards.
-- `storage.py` owns the SQLite schema, WAL coordination, idempotent ingestion, and local annotations. It also computes the review list, applies bulk labels, builds the library report, computes the daily failure trend, the status breakdown, and the agent comparison overlay, lists the runs for one day, and enforces the retention cutoff for cleanup. A cleanup log records each scheduled sweep.
+- `storage.py` owns the SQLite schema, WAL coordination, idempotent ingestion, and local annotations. It computes the review list, bulk labels, and the library report. It computes the failure trend, status breakdown, agent overlay, and run error timeline. It lists the runs for one day and enforces the retention cutoff. A cleanup log records each scheduled sweep.
 - `ingestion.py` watches JSON files and returns stable schema error reports.
 - `otlp.py` converts the OTLP JSON encoding to and from the trace contract.
 - `replay.py` runs guarded local handlers and records mismatches.
 - `compare.py` aligns tool calls by recorded position and reports field-level deltas.
-- `export.py` renders comparisons, run tool calls, library reports, failure trends, and day run lists as CSV files.
+- `export.py` renders comparisons, run tool calls, library reports, failure trends, error timelines, and day run lists as CSV files.
 - `collector.py` posts recorded runs to a local collector over OTLP HTTP JSON.
-- `main.py` serves the interface and the JSON API.
+- `main.py` serves the interface and the JSON API. It also shapes the dashboard charts and the run error timeline.
 - `scheduler.py` runs server-side retention sweeps on an interval.
 - `telemetry.py` creates OpenTelemetry spans and exports them locally.
 
@@ -112,7 +114,7 @@ uvicorn agent_trace_workbench.main:app --reload
 
 Open `http://127.0.0.1:8000` in a browser.
 
-The dashboard shows both runs. The candidate includes a reservation failure. The trend panel draws the daily failure rate for the recorded evidence.
+The dashboard shows both runs. The candidate includes a reservation failure. Its run page marks the failure on the error timeline. The trend panel draws the daily failure rate for the recorded evidence.
 
 Load the second agent to exercise the trend filter.
 
@@ -230,6 +232,77 @@ The response keeps only matching spans. The run metrics still show full totals.
 
 Use the drop-downs on a run page. Clear the filters to see every span.
 
+## Error timeline
+
+Open one run to see where its failures happened.
+
+The run page draws a horizontal time axis. Each failed span appears as a marker at its offset from the run start. A span counts as failed when its status is error or its tool call outcome is failure.
+
+Read the timeline over the API.
+
+```powershell
+curl.exe "http://127.0.0.1:8000/api/runs/run-candidate-001/timeline"
+```
+
+The response lists every failed span in recorded order.
+
+```json
+{
+  "run_id": "run-candidate-001",
+  "started_at": "2026-07-31T09:05:00+00:00",
+  "ended_at": "2026-07-31T09:05:00.280000+00:00",
+  "duration_ms": 280.0,
+  "error_count": 2,
+  "events": [
+    {
+      "span_id": "span-agent-101",
+      "name": "agent.run",
+      "kind": "agent",
+      "status": "error",
+      "start_offset_ms": 0.0,
+      "end_offset_ms": 280.0,
+      "duration_ms": 280.0,
+      "error": "agent.run ended with status error"
+    },
+    {
+      "span_id": "span-tool-103",
+      "name": "reserve_inventory",
+      "kind": "tool",
+      "status": "error",
+      "start_offset_ms": 205.0,
+      "end_offset_ms": 260.0,
+      "duration_ms": 55.0,
+      "error": "reservation window expired"
+    }
+  ]
+}
+```
+
+A tool call keeps its recorded error message. Other failed spans get a generated message.
+
+Download the timeline as a CSV document.
+
+```powershell
+curl.exe -o error-timeline.csv "http://127.0.0.1:8000/api/runs/run-candidate-001/timeline?format=csv"
+```
+
+The file lists one row per failed span.
+
+```text
+run_id,span_id,sequence,name,kind,status,start_offset_ms,end_offset_ms,duration_ms,error
+run-candidate-001,span-agent-101,0,agent.run,agent,error,0.0,280.0,280.0,agent.run ended with status error
+run-candidate-001,span-tool-103,3,reserve_inventory,tool,error,205.0,260.0,55.0,reservation window expired
+```
+
+Use the CLI for scripts.
+
+```powershell
+python -m agent_trace_workbench.cli timeline run-candidate-001
+python -m agent_trace_workbench.cli timeline run-candidate-001 --format csv
+```
+
+The run page lists each event below the chart. Click an event row to jump to the matching span in the trace waterfall.
+
 ## Failure trend
 
 The dashboard draws a daily failure line for the last 14 days.
@@ -1334,7 +1407,7 @@ curl.exe -X POST http://127.0.0.1:8000/api/traces `
 
 ## Test status
 
-The test suite covers the core flows. It covers storage, ingestion, replay, comparison, search, annotations, bulk labels, export, review, reports, retention cleanup, and scheduled cleanup. It covers the CLI, the API, collector export, the server scheduler, and the dashboard failure trend, including the agent filter, the window selector, the day drill-down, the status breakdown, the agent comparison overlay, and the CSV exports.
+The test suite covers the core flows. It covers storage, ingestion, replay, comparison, search, and annotations. It covers bulk labels, export, review, reports, retention, and scheduled cleanup. It covers the CLI, the API, collector export, and the server scheduler. It covers the dashboard trend, including the agent filter, window selector, day drill-down, status breakdown, overlay, and the run error timeline. The CSV exports have their own tests.
 
 Run the checks with these commands.
 
@@ -1345,7 +1418,7 @@ python scripts/check_requirements.py
 python -m compileall agent_trace_workbench tests
 ```
 
-Current verification passes 329 tests, Ruff lint, dependency checks, and Python compilation. CI installs from `requirements-lock.txt` and runs these checks on Python 3.11, 3.12, and 3.13 for every push and pull request.
+Current verification passes 344 tests, Ruff lint, dependency checks, and Python compilation. CI installs from `requirements-lock.txt` and runs these checks on Python 3.11, 3.12, and 3.13 for every push and pull request.
 
 ## Limitations
 
@@ -1409,6 +1482,16 @@ The day drill-down groups runs by the UTC calendar day they started. It ignores
 
 The day CSV repeats the active agent in every row. The all-agents view leaves that cell empty.
 
+The error timeline marks spans with an error status or a failed tool outcome.
+
+The timeline offsets count from the recorded run start. A span that starts before the run start clamps to zero.
+
+The timeline lists one marker per failed span. It does not mark recovered retries.
+
+The timeline CSV lists one row per failed span. Clean runs produce only the header row.
+
+A tool call keeps its recorded error message. Other spans get a generated message.
+
 The cleanup history records policy and counts. It does not store the deleted traces.
 
 The report retention line counts runs under the current policy. It uses `older_than_days` from the request or the 30-day default.
@@ -1464,13 +1547,14 @@ The span exporter sends each workbench span as it ends. It does not batch spans.
 - Release 1.6 complete: add a trend window selector and a per-day drill-down on the dashboard chart.
 - Release 1.7 complete: add a status breakdown beside the daily failure line on the dashboard.
 - Release 1.8 complete: add an agent comparison overlay to the failure trend.
-- Release 1.9: add a run-level error timeline to the run detail page.
+- Release 1.9 complete: add a run-level error timeline to the run detail page.
+- Release 1.10: add a span detail panel to the run-level error timeline.
 
 ## Repository map
 
 `fixtures/` contains meaningful baseline, candidate, and second-agent traces. It also contains a handler config and demo scripts.
 
-`tests/` contains deterministic tests for the core. It covers coordination, guards, search, annotations, OTLP, export, review, reports, retention cleanup, scheduled cleanup, the server scheduler, and the failure trend, including the agent filter, the window selector, the day drill-down, the status breakdown, the agent comparison overlay, and the CSV exports.
+`tests/` contains deterministic tests for the core. It covers coordination, guards, search, annotations, OTLP, and export. It covers review, reports, retention cleanup, scheduled cleanup, and the server scheduler. It covers the failure trend, including the agent filter, window selector, day drill-down, status breakdown, overlay, and the run error timeline.
 
 `static/` and `templates/` contain the presentation layer.
 

diff --git a/agent_trace_workbench/__init__.py b/agent_trace_workbench/__init__.py
@@ -1,3 +1,3 @@
 """Agent Trace Workbench package."""
 
-__version__ = "1.8.0"
+__version__ = "1.9.0"
diff --git a/agent_trace_workbench/cli.py b/agent_trace_workbench/cli.py
@@ -15,6 +15,7 @@
 from .export import (
     comparison_to_csv,
     day_runs_to_csv,
+    error_timeline_to_csv,
     report_to_csv,
     run_tools_to_csv,
     status_trend_to_csv,
@@ -90,6 +91,17 @@ def build_parser() -> argparse.ArgumentParser:
         help="Override the side-effect policy",
     )
 
+    timeline = subparsers.add_parser(
+        "timeline", help="Show the error timeline of one run"
+    )
+    timeline.add_argument("run_id")
+    timeline.add_argument(
+        "--format",
+        choices=["json", "csv"],
+        default="json",
+        help="Output format",
+    )
+
     compare = subparsers.add_parser("compare", help="Compare two recorded runs")
     compare.add_argument("run_a")
     compare.add_argument("run_b")
@@ -379,6 +391,14 @@ def main() -> None:
             print(comparison_to_csv(report), end="")
         else:
             print(json.dumps(report.as_dict(), indent=2))
+    elif args.command == "timeline":
+        timeline = store.error_timeline(args.run_id)
+        if timeline is None:
+            raise SystemExit(f"Run not found: {args.run_id}")
+        if args.format == "csv":
+            print(error_timeline_to_csv(timeline), end="")
+        else:
+            print(json.dumps(timeline, indent=2))
     elif args.command == "search":
         print(json.dumps(store.search_runs(args.query, args.limit), indent=2))
     elif args.command == "comparisons":

diff --git a/agent_trace_workbench/export.py b/agent_trace_workbench/export.py
@@ -91,6 +91,19 @@
     "label",
 ]
 
+_TIMELINE_HEADERS = [
+    "run_id",
+    "span_id",
+    "sequence",
+    "name",
+    "kind",
+    "status",
+    "start_offset_ms",
+    "end_offset_ms",
+    "duration_ms",
+    "error",
+]
+
 _SECTION_TOTAL = "total"
 _SECTION_SOURCE = "source"
 _SECTION_AGENT = "agent"
@@ -338,6 +351,38 @@ def status_trend_to_csv(
         return _to_csv(_STATUS_TREND_HEADERS, rows)
 
 
+def error_timeline_to_csv(timeline: dict[str, Any]) -> str:
+    """Render the error timeline of one run as a CSV document.
+
+    The document lists one row per failed span in recorded order. The
+    run_id cell repeats the target run, and the offsets keep their
+    millisecond precision so a spreadsheet can plot the markers.
+    """
+
+    run_id = timeline.get("run_id", "")
+    events = timeline.get("events", [])
+    with traced_operation(
+        "export.timeline_csv", {"run.id": run_id, "timeline.events": len(events)}
+    ):
+        rows: list[dict[str, Any]] = []
+        for event in events:
+            rows.append(
+                {
+                    "run_id": run_id,
+                    "span_id": event.get("span_id", ""),
+                    "sequence": _number(event.get("sequence")),
+                    "name": event.get("name", ""),
+                    "kind": event.get("kind", ""),
+                    "status": event.get("status", ""),
+                    "start_offset_ms": _number(event.get("start_offset_ms")),
+                    "end_offset_ms": _number(event.get("end_offset_ms")),
+                    "duration_ms": _number(event.get("duration_ms")),
+                    "error": event.get("error") or "",
+                }
+            )
+        return _to_csv(_TIMELINE_HEADERS, rows)
+
+
 def day_runs_to_csv(day: str, runs: list[dict[str, Any]], agent_name: str = "") -> str:
     """Render the runs that started on one day as a CSV document.
 

diff --git a/agent_trace_workbench/main.py b/agent_trace_workbench/main.py
@@ -22,6 +22,7 @@
 from .export import (
     comparison_to_csv,
     day_runs_to_csv,
+    error_timeline_to_csv,
     report_to_csv,
     run_tools_to_csv,
     status_trend_to_csv,
@@ -172,12 +173,15 @@ def run_detail(
         run = app.state.store.get_run(
             run_id, span_kind=kind, span_status=status, span_tool=tool
         )
+        timeline = app.state.store.error_timeline(run_id)
         return render_template(
             request,
             "run.html",
             {
                 "run": run,
                 "filters": filter_set,
+                "timeline": _error_timeline_chart(timeline) if timeline else None,
+                "timeline_csv_link": f"/api/runs/{run_id}/timeline?format=csv",
                 "store": app.state.store.store_info(),
                 "telemetry": _telemetry_info(),
                 "scheduler": _scheduler_status(app),
@@ -513,6 +517,24 @@ def api_run(
             app.state.store, run_id, span_kind=kind, span_status=status, span_tool=tool
         )
 
+    @app.get("/api/runs/{run_id}/timeline", response_model=None)
+    def api_run_timeline(
+        run_id: str,
+        export_format: str = Query(default="json", alias="format"),
+    ) -> Response | dict[str, Any]:
+        timeline = app.state.store.error_timeline(run_id)
+        if timeline is None:
+            raise HTTPException(status_code=404, detail=f"Run not found: {run_id}")
+        if export_format == "csv":
+            return _download_response(
+                error_timeline_to_csv(timeline),
+                f"{run_id}-error-timeline.csv",
+                "text/csv; charset=utf-8",
+            )
+        if export_format != "json":
+            raise HTTPException(status_code=400, detail="format must be 'json' or 'csv'")
+        return timeline
+
     @app.post("/api/traces", status_code=201)
     def api_ingest(trace: TraceDocument, request: Request) -> dict[str, Any]:
         source_name = request.headers.get("x-trace-source", "api.json")
@@ -1092,6 +1114,40 @@ def _day_csv_href(selected_agent: str, day: str) -> str:
     return f"/api/trend/{day}?{urlencode(params)}"
 
 
+def _error_timeline_chart(timeline: dict[str, Any]) -> dict[str, Any]:
+    """Shape an error timeline into SVG-ready data for the run page.
+
+    The helper maps each failed span to a marker on a fixed view box.
+    The marker position tracks the offset from the run start, so the
+    panel shows when the failures happened on one time axis. The return
+    value carries the marker geometry and the event details for the
+    list below the chart.
+    """
+
+    width = 680
+    height = 60
+    pad_x = 16
+    duration_ms = timeline.get("duration_ms", 0.0)
+    plot_width = width - 2 * pad_x
+    events = []
+    for event in timeline["events"]:
+        fraction = (
+            min(max(event["start_offset_ms"] / duration_ms, 0.0), 1.0)
+            if duration_ms > 0
+            else 0.0
+        )
+        x = round(pad_x + fraction * plot_width, 2)
+        events.append({**event, "x": x})
+    return {
+        "width": width,
+        "height": height,
+        "duration_ms": duration_ms,
+        "error_count": timeline.get("error_count", 0),
+        "has_errors": bool(events),
+        "events": events,
+    }
+
+
 def _span_filter_set(
     run: dict[str, Any],
     kind: str | None,