diff --git a/CHANGELOG.md b/CHANGELOG.md index 3737d43..6dd7a81 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,29 @@ All notable changes to Agent Trace Workbench appear in this file. The version format follows a release cycle. A release adds one coherent capability to the workbench. +## 1.9.0 - 2026-08-04 + +### Added + +- A run-level error timeline on the run detail page. +- A horizontal time axis that marks each failed span at its offset from the run start. +- A marker chart that shows when the failures happened inside one run. +- An event list below the chart with each span name, kind, offset, and failure message. +- A clickable event row that jumps to the matching span in the trace waterfall. +- `GET /api/runs/{run_id}/timeline` route that returns the failed spans for scripts. +- `GET /api/runs/{run_id}/timeline?format=csv` route that returns the events as a CSV attachment. +- `atw timeline ` command that prints the error timeline. +- `atw timeline --format csv` command that prints the events as CSV. +- Run page JSON and CSV download links for the timeline panel. +- Deterministic tests for the timeline, the API routes, the CLI options, the CSV export, and the run page panel. + +### Changed + +- Version numbers moved to 1.9.0. +- The run detail page now shows when failures happened beside the trace waterfall. +- The trace waterfall rows carry span IDs so the timeline can link to them. +- The architecture now includes a run-level error timeline beside the trace inspection layer. + ## 1.8.0 - 2026-08-04 ### Added diff --git a/README.md b/README.md index bd39824..69de39a 100644 --- a/README.md +++ b/README.md @@ -26,11 +26,13 @@ Release 1.7 adds a status breakdown beside the daily failure line. Each trend da Release 1.8 adds an agent comparison overlay to the failure trend. Choose a second agent on the dashboard and the chart draws its failure line beside the primary series. Read both series from the API, the CLI, or a CSV file. +Release 1.9 adds a run-level error timeline to the run detail page. It marks each failed span at its offset from the run start. Read the same events from the API, the CLI, or a CSV file. + ## Value Agent debugging needs evidence at tool boundaries. -This workbench makes each boundary visible. It shows inputs, outputs, timing, attributes, and errors in one local record. +This workbench makes each boundary visible. It shows inputs, outputs, timing, attributes, and errors in one local record. The error timeline marks where each failure happened. The design supports repeatable review. A replay runs a registered local handler when available. It uses the recorded result when no handler exists. @@ -68,14 +70,14 @@ SQLite runs in WAL mode with a busy timeout. Readers keep a committed snapshot. - `models.py` defines the portable trace contract. - `handlers.py` loads local handler config and applies side-effect guards. -- `storage.py` owns the SQLite schema, WAL coordination, idempotent ingestion, and local annotations. It also computes the review list, applies bulk labels, builds the library report, computes the daily failure trend, the status breakdown, and the agent comparison overlay, lists the runs for one day, and enforces the retention cutoff for cleanup. A cleanup log records each scheduled sweep. +- `storage.py` owns the SQLite schema, WAL coordination, idempotent ingestion, and local annotations. It computes the review list, bulk labels, and the library report. It computes the failure trend, status breakdown, agent overlay, and run error timeline. It lists the runs for one day and enforces the retention cutoff. A cleanup log records each scheduled sweep. - `ingestion.py` watches JSON files and returns stable schema error reports. - `otlp.py` converts the OTLP JSON encoding to and from the trace contract. - `replay.py` runs guarded local handlers and records mismatches. - `compare.py` aligns tool calls by recorded position and reports field-level deltas. -- `export.py` renders comparisons, run tool calls, library reports, failure trends, and day run lists as CSV files. +- `export.py` renders comparisons, run tool calls, library reports, failure trends, error timelines, and day run lists as CSV files. - `collector.py` posts recorded runs to a local collector over OTLP HTTP JSON. -- `main.py` serves the interface and the JSON API. +- `main.py` serves the interface and the JSON API. It also shapes the dashboard charts and the run error timeline. - `scheduler.py` runs server-side retention sweeps on an interval. - `telemetry.py` creates OpenTelemetry spans and exports them locally. @@ -112,7 +114,7 @@ uvicorn agent_trace_workbench.main:app --reload Open `http://127.0.0.1:8000` in a browser. -The dashboard shows both runs. The candidate includes a reservation failure. The trend panel draws the daily failure rate for the recorded evidence. +The dashboard shows both runs. The candidate includes a reservation failure. Its run page marks the failure on the error timeline. The trend panel draws the daily failure rate for the recorded evidence. Load the second agent to exercise the trend filter. @@ -230,6 +232,77 @@ The response keeps only matching spans. The run metrics still show full totals. Use the drop-downs on a run page. Clear the filters to see every span. +## Error timeline + +Open one run to see where its failures happened. + +The run page draws a horizontal time axis. Each failed span appears as a marker at its offset from the run start. A span counts as failed when its status is error or its tool call outcome is failure. + +Read the timeline over the API. + +```powershell +curl.exe "http://127.0.0.1:8000/api/runs/run-candidate-001/timeline" +``` + +The response lists every failed span in recorded order. + +```json +{ + "run_id": "run-candidate-001", + "started_at": "2026-07-31T09:05:00+00:00", + "ended_at": "2026-07-31T09:05:00.280000+00:00", + "duration_ms": 280.0, + "error_count": 2, + "events": [ + { + "span_id": "span-agent-101", + "name": "agent.run", + "kind": "agent", + "status": "error", + "start_offset_ms": 0.0, + "end_offset_ms": 280.0, + "duration_ms": 280.0, + "error": "agent.run ended with status error" + }, + { + "span_id": "span-tool-103", + "name": "reserve_inventory", + "kind": "tool", + "status": "error", + "start_offset_ms": 205.0, + "end_offset_ms": 260.0, + "duration_ms": 55.0, + "error": "reservation window expired" + } + ] +} +``` + +A tool call keeps its recorded error message. Other failed spans get a generated message. + +Download the timeline as a CSV document. + +```powershell +curl.exe -o error-timeline.csv "http://127.0.0.1:8000/api/runs/run-candidate-001/timeline?format=csv" +``` + +The file lists one row per failed span. + +```text +run_id,span_id,sequence,name,kind,status,start_offset_ms,end_offset_ms,duration_ms,error +run-candidate-001,span-agent-101,0,agent.run,agent,error,0.0,280.0,280.0,agent.run ended with status error +run-candidate-001,span-tool-103,3,reserve_inventory,tool,error,205.0,260.0,55.0,reservation window expired +``` + +Use the CLI for scripts. + +```powershell +python -m agent_trace_workbench.cli timeline run-candidate-001 +python -m agent_trace_workbench.cli timeline run-candidate-001 --format csv +``` + +The run page lists each event below the chart. Click an event row to jump to the matching span in the trace waterfall. + ## Failure trend The dashboard draws a daily failure line for the last 14 days. @@ -1334,7 +1407,7 @@ curl.exe -X POST http://127.0.0.1:8000/api/traces ` ## Test status -The test suite covers the core flows. It covers storage, ingestion, replay, comparison, search, annotations, bulk labels, export, review, reports, retention cleanup, and scheduled cleanup. It covers the CLI, the API, collector export, the server scheduler, and the dashboard failure trend, including the agent filter, the window selector, the day drill-down, the status breakdown, the agent comparison overlay, and the CSV exports. +The test suite covers the core flows. It covers storage, ingestion, replay, comparison, search, and annotations. It covers bulk labels, export, review, reports, retention, and scheduled cleanup. It covers the CLI, the API, collector export, and the server scheduler. It covers the dashboard trend, including the agent filter, window selector, day drill-down, status breakdown, overlay, and the run error timeline. The CSV exports have their own tests. Run the checks with these commands. @@ -1345,7 +1418,7 @@ python scripts/check_requirements.py python -m compileall agent_trace_workbench tests ``` -Current verification passes 329 tests, Ruff lint, dependency checks, and Python compilation. CI installs from `requirements-lock.txt` and runs these checks on Python 3.11, 3.12, and 3.13 for every push and pull request. +Current verification passes 344 tests, Ruff lint, dependency checks, and Python compilation. CI installs from `requirements-lock.txt` and runs these checks on Python 3.11, 3.12, and 3.13 for every push and pull request. ## Limitations @@ -1409,6 +1482,16 @@ The day drill-down groups runs by the UTC calendar day they started. It ignores The day CSV repeats the active agent in every row. The all-agents view leaves that cell empty. +The error timeline marks spans with an error status or a failed tool outcome. + +The timeline offsets count from the recorded run start. A span that starts before the run start clamps to zero. + +The timeline lists one marker per failed span. It does not mark recovered retries. + +The timeline CSV lists one row per failed span. Clean runs produce only the header row. + +A tool call keeps its recorded error message. Other spans get a generated message. + The cleanup history records policy and counts. It does not store the deleted traces. The report retention line counts runs under the current policy. It uses `older_than_days` from the request or the 30-day default. @@ -1464,13 +1547,14 @@ The span exporter sends each workbench span as it ends. It does not batch spans. - Release 1.6 complete: add a trend window selector and a per-day drill-down on the dashboard chart. - Release 1.7 complete: add a status breakdown beside the daily failure line on the dashboard. - Release 1.8 complete: add an agent comparison overlay to the failure trend. -- Release 1.9: add a run-level error timeline to the run detail page. +- Release 1.9 complete: add a run-level error timeline to the run detail page. +- Release 1.10: add a span detail panel to the run-level error timeline. ## Repository map `fixtures/` contains meaningful baseline, candidate, and second-agent traces. It also contains a handler config and demo scripts. -`tests/` contains deterministic tests for the core. It covers coordination, guards, search, annotations, OTLP, export, review, reports, retention cleanup, scheduled cleanup, the server scheduler, and the failure trend, including the agent filter, the window selector, the day drill-down, the status breakdown, the agent comparison overlay, and the CSV exports. +`tests/` contains deterministic tests for the core. It covers coordination, guards, search, annotations, OTLP, and export. It covers review, reports, retention cleanup, scheduled cleanup, and the server scheduler. It covers the failure trend, including the agent filter, window selector, day drill-down, status breakdown, overlay, and the run error timeline. `static/` and `templates/` contain the presentation layer. diff --git a/agent_trace_workbench/__init__.py b/agent_trace_workbench/__init__.py index 2cb3fbb..c0007ac 100644 --- a/agent_trace_workbench/__init__.py +++ b/agent_trace_workbench/__init__.py @@ -1,3 +1,3 @@ """Agent Trace Workbench package.""" -__version__ = "1.8.0" +__version__ = "1.9.0" diff --git a/agent_trace_workbench/cli.py b/agent_trace_workbench/cli.py index 3c13e6b..6a57203 100644 --- a/agent_trace_workbench/cli.py +++ b/agent_trace_workbench/cli.py @@ -15,6 +15,7 @@ from .export import ( comparison_to_csv, day_runs_to_csv, + error_timeline_to_csv, report_to_csv, run_tools_to_csv, status_trend_to_csv, @@ -90,6 +91,17 @@ def build_parser() -> argparse.ArgumentParser: help="Override the side-effect policy", ) + timeline = subparsers.add_parser( + "timeline", help="Show the error timeline of one run" + ) + timeline.add_argument("run_id") + timeline.add_argument( + "--format", + choices=["json", "csv"], + default="json", + help="Output format", + ) + compare = subparsers.add_parser("compare", help="Compare two recorded runs") compare.add_argument("run_a") compare.add_argument("run_b") @@ -379,6 +391,14 @@ def main() -> None: print(comparison_to_csv(report), end="") else: print(json.dumps(report.as_dict(), indent=2)) + elif args.command == "timeline": + timeline = store.error_timeline(args.run_id) + if timeline is None: + raise SystemExit(f"Run not found: {args.run_id}") + if args.format == "csv": + print(error_timeline_to_csv(timeline), end="") + else: + print(json.dumps(timeline, indent=2)) elif args.command == "search": print(json.dumps(store.search_runs(args.query, args.limit), indent=2)) elif args.command == "comparisons": diff --git a/agent_trace_workbench/export.py b/agent_trace_workbench/export.py index c7db478..0957fe1 100644 --- a/agent_trace_workbench/export.py +++ b/agent_trace_workbench/export.py @@ -91,6 +91,19 @@ "label", ] +_TIMELINE_HEADERS = [ + "run_id", + "span_id", + "sequence", + "name", + "kind", + "status", + "start_offset_ms", + "end_offset_ms", + "duration_ms", + "error", +] + _SECTION_TOTAL = "total" _SECTION_SOURCE = "source" _SECTION_AGENT = "agent" @@ -338,6 +351,38 @@ def status_trend_to_csv( return _to_csv(_STATUS_TREND_HEADERS, rows) +def error_timeline_to_csv(timeline: dict[str, Any]) -> str: + """Render the error timeline of one run as a CSV document. + + The document lists one row per failed span in recorded order. The + run_id cell repeats the target run, and the offsets keep their + millisecond precision so a spreadsheet can plot the markers. + """ + + run_id = timeline.get("run_id", "") + events = timeline.get("events", []) + with traced_operation( + "export.timeline_csv", {"run.id": run_id, "timeline.events": len(events)} + ): + rows: list[dict[str, Any]] = [] + for event in events: + rows.append( + { + "run_id": run_id, + "span_id": event.get("span_id", ""), + "sequence": _number(event.get("sequence")), + "name": event.get("name", ""), + "kind": event.get("kind", ""), + "status": event.get("status", ""), + "start_offset_ms": _number(event.get("start_offset_ms")), + "end_offset_ms": _number(event.get("end_offset_ms")), + "duration_ms": _number(event.get("duration_ms")), + "error": event.get("error") or "", + } + ) + return _to_csv(_TIMELINE_HEADERS, rows) + + def day_runs_to_csv(day: str, runs: list[dict[str, Any]], agent_name: str = "") -> str: """Render the runs that started on one day as a CSV document. diff --git a/agent_trace_workbench/main.py b/agent_trace_workbench/main.py index 1527652..87788fc 100644 --- a/agent_trace_workbench/main.py +++ b/agent_trace_workbench/main.py @@ -22,6 +22,7 @@ from .export import ( comparison_to_csv, day_runs_to_csv, + error_timeline_to_csv, report_to_csv, run_tools_to_csv, status_trend_to_csv, @@ -172,12 +173,15 @@ def run_detail( run = app.state.store.get_run( run_id, span_kind=kind, span_status=status, span_tool=tool ) + timeline = app.state.store.error_timeline(run_id) return render_template( request, "run.html", { "run": run, "filters": filter_set, + "timeline": _error_timeline_chart(timeline) if timeline else None, + "timeline_csv_link": f"/api/runs/{run_id}/timeline?format=csv", "store": app.state.store.store_info(), "telemetry": _telemetry_info(), "scheduler": _scheduler_status(app), @@ -513,6 +517,24 @@ def api_run( app.state.store, run_id, span_kind=kind, span_status=status, span_tool=tool ) + @app.get("/api/runs/{run_id}/timeline", response_model=None) + def api_run_timeline( + run_id: str, + export_format: str = Query(default="json", alias="format"), + ) -> Response | dict[str, Any]: + timeline = app.state.store.error_timeline(run_id) + if timeline is None: + raise HTTPException(status_code=404, detail=f"Run not found: {run_id}") + if export_format == "csv": + return _download_response( + error_timeline_to_csv(timeline), + f"{run_id}-error-timeline.csv", + "text/csv; charset=utf-8", + ) + if export_format != "json": + raise HTTPException(status_code=400, detail="format must be 'json' or 'csv'") + return timeline + @app.post("/api/traces", status_code=201) def api_ingest(trace: TraceDocument, request: Request) -> dict[str, Any]: source_name = request.headers.get("x-trace-source", "api.json") @@ -1092,6 +1114,40 @@ def _day_csv_href(selected_agent: str, day: str) -> str: return f"/api/trend/{day}?{urlencode(params)}" +def _error_timeline_chart(timeline: dict[str, Any]) -> dict[str, Any]: + """Shape an error timeline into SVG-ready data for the run page. + + The helper maps each failed span to a marker on a fixed view box. + The marker position tracks the offset from the run start, so the + panel shows when the failures happened on one time axis. The return + value carries the marker geometry and the event details for the + list below the chart. + """ + + width = 680 + height = 60 + pad_x = 16 + duration_ms = timeline.get("duration_ms", 0.0) + plot_width = width - 2 * pad_x + events = [] + for event in timeline["events"]: + fraction = ( + min(max(event["start_offset_ms"] / duration_ms, 0.0), 1.0) + if duration_ms > 0 + else 0.0 + ) + x = round(pad_x + fraction * plot_width, 2) + events.append({**event, "x": x}) + return { + "width": width, + "height": height, + "duration_ms": duration_ms, + "error_count": timeline.get("error_count", 0), + "has_errors": bool(events), + "events": events, + } + + def _span_filter_set( run: dict[str, Any], kind: str | None, diff --git a/agent_trace_workbench/storage.py b/agent_trace_workbench/storage.py index e0094da..fe15229 100644 --- a/agent_trace_workbench/storage.py +++ b/agent_trace_workbench/storage.py @@ -26,7 +26,7 @@ from typing import Any, Callable, TypeVar from uuid import uuid4 -from .models import TraceDocument +from .models import TraceDocument, ensure_utc from .telemetry import traced_operation _T = TypeVar("_T") @@ -944,6 +944,61 @@ def get_trace(self, run_id: str) -> TraceDocument | None: ).fetchone() return TraceDocument.model_validate_json(row["raw_json"]) if row else None + def error_timeline(self, run_id: str) -> dict[str, Any] | None: + """Return the failed spans of one run as a time-ordered timeline. + + The timeline places each failed span on the run time axis. It + reports the offset from the run start, so a reviewer sees when + the failures happened. A failed span has an error status or a + tool call with a failure outcome. The run detail page draws the + events as markers on one horizontal axis. + """ + + with traced_operation("storage.error_timeline", {"run.id": run_id}): + with self._connect() as connection: + run = connection.execute( + "SELECT * FROM runs WHERE run_id = ?", (run_id,) + ).fetchone() + if run is None: + return None + rows = connection.execute( + """ + SELECT * FROM spans WHERE run_id = ? + ORDER BY sequence_index IS NULL, sequence_index, start_time, span_id + """, + (run_id,), + ).fetchall() + origin = ensure_utc(datetime.fromisoformat(run["started_at"])) + events: list[dict[str, Any]] = [] + for row in rows: + if row["status"] != "error" and row["outcome"] != "failure": + continue + start_offset_ms = _offset_ms( + origin, datetime.fromisoformat(row["start_time"]) + ) + end_offset_ms = _offset_ms(origin, datetime.fromisoformat(row["end_time"])) + events.append( + { + "span_id": row["span_id"], + "sequence": row["sequence_index"], + "name": row["name"], + "kind": row["kind"], + "status": row["status"], + "start_offset_ms": round(start_offset_ms, 3), + "end_offset_ms": round(end_offset_ms, 3), + "duration_ms": row["duration_ms"], + "error": _error_message(row), + } + ) + return { + "run_id": run_id, + "started_at": run["started_at"], + "ended_at": run["ended_at"], + "duration_ms": run["duration_ms"], + "error_count": len(events), + "events": events, + } + def update_annotations( self, run_id: str, @@ -1125,6 +1180,30 @@ def _now_utc() -> datetime: return datetime.now(timezone.utc) +def _offset_ms(origin: datetime, instant: datetime) -> float: + """Return the millisecond offset of one instant after a run start. + + A span that starts before the recorded run start clamps to zero, + because a negative offset would draw a marker off the time axis. + """ + + return max((ensure_utc(instant) - ensure_utc(origin)).total_seconds() * 1000, 0.0) + + +def _error_message(row: sqlite3.Row) -> str: + """Return a stable failure message for one failed span. + + A tool call carries its recorded error. Every other failed span uses + a generated message, so the timeline always has text to show. + """ + + if row["error"]: + return row["error"] + if row["outcome"] == "failure": + return f"{row['name']} reported a failure outcome" + return f"{row['name']} ended with status error" + + def _retention_ids( connection: sqlite3.Connection, cutoff: datetime, diff --git a/pyproject.toml b/pyproject.toml index 81f1a2f..a9a2eb5 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "agent-trace-workbench" -version = "1.8.0" +version = "1.9.0" description = "A local workbench for recording, replaying, comparing, and inspecting agent traces." readme = "README.md" requires-python = ">=3.11" diff --git a/static/styles.css b/static/styles.css index c602f7a..1e1962d 100644 --- a/static/styles.css +++ b/static/styles.css @@ -111,6 +111,23 @@ select { min-height: 44px; padding: 0 12px; font-family: inherit; } .review-note { display: flex; flex-direction: column; gap: 5px; margin-top: 14px; padding: 13px; border-left: 3px solid var(--blue); background: rgba(44,111,159,.06); color: var(--navy); font-size: 13px; line-height: 1.55; } .review-note strong { font: 800 9px ui-monospace, SFMono-Regular, Consolas, monospace; letter-spacing: .1em; text-transform: uppercase; } .trace-section { padding-top: 62px; } +.timeline-section { padding: 20px 0 30px; } +.timeline-panel { padding: 24px 26px 20px; } +.timeline-chart { position: relative; } +.timeline-svg { display: block; width: 100%; height: 60px; overflow: visible; } +.timeline-axis { stroke: var(--line); stroke-width: 2; } +.timeline-stem { stroke: var(--coral); stroke-width: 1.5; } +.timeline-dot { fill: var(--coral); stroke: #fff; stroke-width: 1.4; } +.timeline-scale { display: flex; justify-content: space-between; margin-top: 4px; color: var(--muted); font-size: 10px; } +.timeline-foot { display: flex; justify-content: flex-end; gap: 8px; margin-top: 16px; padding-top: 14px; border-top: 1px solid var(--line); } +.timeline-foot .button { min-height: 36px; padding: 0 11px; font-size: 10px; } +.timeline-list { margin-top: 18px; border-top: 1px solid var(--line); } +.timeline-row { display: grid; grid-template-columns: 14px 1fr 2fr; align-items: center; gap: 12px; padding: 13px 0; border-bottom: 1px solid var(--line); } +.timeline-row:hover strong, .timeline-row:hover .timeline-error { color: var(--blue); } +.timeline-row strong { display: block; margin-bottom: 3px; color: var(--navy); font-size: 14px; } +.timeline-row small { color: var(--muted); font: 10px ui-monospace, SFMono-Regular, Consolas, monospace; } +.timeline-marker { width: 8px; height: 8px; border-radius: 50%; background: var(--coral); } +.timeline-error { color: var(--coral); font-size: 12px; } .trace-list, .replay-list { border-top: 1px solid var(--line); } .trace-row { display: grid; grid-template-columns: 27px 1fr; gap: 12px; padding: 20px 0 25px; } .trace-rail { position: relative; display: flex; justify-content: center; } diff --git a/templates/dashboard.html b/templates/dashboard.html index dbd5b16..df6f9ca 100644 --- a/templates/dashboard.html +++ b/templates/dashboard.html @@ -2,7 +2,7 @@ {% block title %}Runs | Agent Trace Workbench{% endblock %} {% block content %}
-
LOCAL OBSERVABILITY / RELEASE 1.8
+
LOCAL OBSERVABILITY / RELEASE 1.9

See what your agent did.
Replay why it did it.

Ingest local JSON traces, import OTLP exports, watch a folder, inspect every tool boundary, replay recorded work, compare runs, label evidence for review, and read a folder-level library report without a hosted service.

diff --git a/templates/run.html b/templates/run.html index a6da121..eb7794c 100644 --- a/templates/run.html +++ b/templates/run.html @@ -26,6 +26,44 @@

{{ run.agent_name }}

Tool calls{{ run.tool_count }}ordered by sequence
Trace ID{{ run.trace_id }}portable identity
+
+
+
ERROR TIMELINE

When did this run fail?

+ {% if timeline and timeline.error_count %}{{ timeline.error_count }} failed span{% if timeline.error_count != 1 %}s{% endif %} · {{ timeline.duration_ms|round(1) }} ms run{% else %}no failed spans{% endif %} +
+ {% if timeline and timeline.has_errors %} +
+ +
+ CSV + JSON +
+
+
+ {% for event in timeline.events %} + + + {{ event.name }}{{ event.kind }} · {{ event.start_offset_ms|round(1) }} ms in + {{ event.error }} + + {% endfor %} +
+ {% else %} +
✓

No failed spans

This run recorded no error spans, so the timeline stays empty.

+ {% endif %} +
REVIEW ANNOTATIONS

Label and notes

@@ -56,7 +94,7 @@

{{ run.agent_name }}

{% if run.spans %}
{% for span in run.spans %} -
+
{% if not loop.last %}{% endif %}
{{ span.kind }}

{{ span.name }}

{{ span.duration_ms|round(1) }} ms
diff --git a/tests/test_error_timeline.py b/tests/test_error_timeline.py new file mode 100644 index 0000000..51bf305 --- /dev/null +++ b/tests/test_error_timeline.py @@ -0,0 +1,251 @@ +"""Deterministic tests for the run-level error timeline.""" + +import csv +import io +import json +import sqlite3 + +import pytest +from fastapi.testclient import TestClient + +from agent_trace_workbench.cli import main +from agent_trace_workbench.export import error_timeline_to_csv +from agent_trace_workbench.main import create_app +from agent_trace_workbench.models import TraceDocument +from agent_trace_workbench.storage import TraceStore + + +def _read_csv(text: str) -> list[dict[str, str]]: + return list(csv.DictReader(io.StringIO(text))) + + +def test_error_timeline_lists_failed_spans_in_order(tmp_path, candidate): + store = TraceStore(tmp_path / "timeline.db") + store.ingest(candidate, "candidate.json") + + timeline = store.error_timeline(candidate.run_id) + + assert timeline["run_id"] == "run-candidate-001" + assert timeline["duration_ms"] == 280.0 + assert timeline["error_count"] == 2 + assert [event["span_id"] for event in timeline["events"]] == [ + "span-agent-101", + "span-tool-103", + ] + agent_event = timeline["events"][0] + assert agent_event["start_offset_ms"] == 0.0 + assert agent_event["end_offset_ms"] == 280.0 + assert agent_event["kind"] == "agent" + assert agent_event["error"] == "agent.run ended with status error" + tool_event = timeline["events"][1] + assert tool_event["start_offset_ms"] == 205.0 + assert tool_event["end_offset_ms"] == 260.0 + assert tool_event["kind"] == "tool" + assert tool_event["error"] == "reservation window expired" + + +def test_error_timeline_returns_empty_for_clean_run(tmp_path, baseline): + store = TraceStore(tmp_path / "timeline.db") + store.ingest(baseline, "baseline.json") + + timeline = store.error_timeline(baseline.run_id) + + assert timeline["error_count"] == 0 + assert timeline["events"] == [] + + +def test_error_timeline_returns_none_for_missing_run(tmp_path): + store = TraceStore(tmp_path / "timeline.db") + + assert store.error_timeline("not-here") is None + + +def test_error_timeline_uses_fallback_for_failure_without_error(tmp_path, baseline): + payload = baseline.as_jsonable() + payload["run_id"] = "run-fallback-001" + payload["ended_at"] = "2026-07-31T09:00:00.220000+00:00" + tool_spans = [span for span in payload["spans"] if span["kind"] == "tool"] + tool_spans.append( + { + "span_id": "span-tool-003", + "name": "reserve_inventory", + "kind": "tool", + "start_time": "2026-07-31T09:00:00.170000+00:00", + "end_time": "2026-07-31T09:00:00.200000+00:00", + "status": "error", + "sequence": 4, + "tool_call": { + "name": "reserve_inventory", + "arguments": {"sku": "lamp-01", "quantity": 10}, + "result": None, + "outcome": "failure", + "error": None, + }, + } + ) + payload["spans"] = tool_spans + store = TraceStore(tmp_path / "timeline.db") + store.ingest(TraceDocument.model_validate(payload), "fallback.json") + + timeline = store.error_timeline("run-fallback-001") + + assert timeline["error_count"] == 1 + assert timeline["events"][0]["error"] == "reserve_inventory reported a failure outcome" + + +def test_error_timeline_clamps_negative_offsets(tmp_path, candidate): + store = TraceStore(tmp_path / "timeline.db") + store.ingest(candidate, "candidate.json") + with sqlite3.connect(store.db_path) as connection: + connection.execute( + "UPDATE runs SET started_at = ? WHERE run_id = ?", + ("2026-07-31T09:05:00.300000+00:00", candidate.run_id), + ) + + timeline = store.error_timeline(candidate.run_id) + + assert timeline["error_count"] == 2 + assert all(event["start_offset_ms"] >= 0 for event in timeline["events"]) + assert all(event["end_offset_ms"] >= 0 for event in timeline["events"]) + + +def test_error_timeline_csv_has_headers_and_rows(tmp_path, candidate): + store = TraceStore(tmp_path / "timeline.db") + store.ingest(candidate, "candidate.json") + + rows = _read_csv(error_timeline_to_csv(store.error_timeline(candidate.run_id))) + + assert list(rows[0])[0] == "run_id" + assert len(rows) == 2 + assert rows[0]["name"] == "agent.run" + assert rows[0]["start_offset_ms"] == "0.0" + assert rows[1]["error"] == "reservation window expired" + assert rows[1]["run_id"] == "run-candidate-001" + + +def test_api_timeline_returns_events(tmp_path, candidate): + client = TestClient(create_app(tmp_path / "api.db")) + client.post("/api/traces", json=candidate.as_jsonable()) + + body = client.get("/api/runs/run-candidate-001/timeline").json() + + assert body["error_count"] == 2 + assert body["duration_ms"] == 280.0 + assert body["events"][-1]["error"] == "reservation window expired" + + +def test_api_timeline_csv_returns_attachment(tmp_path, candidate): + client = TestClient(create_app(tmp_path / "api.db")) + client.post("/api/traces", json=candidate.as_jsonable()) + + response = client.get( + "/api/runs/run-candidate-001/timeline", params={"format": "csv"} + ) + + assert response.status_code == 200 + assert response.headers["content-type"].startswith("text/csv") + assert response.headers["content-disposition"] == ( + 'attachment; filename="run-candidate-001-error-timeline.csv"' + ) + assert ( + response.text.splitlines()[0] + == "run_id,span_id,sequence,name,kind,status,start_offset_ms," + "end_offset_ms,duration_ms,error" + ) + + +def test_api_timeline_missing_run_returns_404(tmp_path): + client = TestClient(create_app(tmp_path / "api.db")) + + assert client.get("/api/runs/not-here/timeline").status_code == 404 + + +def test_api_timeline_rejects_unknown_format(tmp_path, candidate): + client = TestClient(create_app(tmp_path / "api.db")) + client.post("/api/traces", json=candidate.as_jsonable()) + + response = client.get( + "/api/runs/run-candidate-001/timeline", params={"format": "xml"} + ) + + assert response.status_code == 400 + + +def test_cli_timeline_prints_json(tmp_path, candidate, monkeypatch, capsys): + store = TraceStore(tmp_path / "cli.db") + store.ingest(candidate, "candidate.json") + + monkeypatch.setattr( + "sys.argv", + ["atw", "--db", str(tmp_path / "cli.db"), "timeline", "run-candidate-001"], + ) + main() + + timeline = json.loads(capsys.readouterr().out) + assert timeline["error_count"] == 2 + assert timeline["events"][-1]["error"] == "reservation window expired" + + +def test_cli_timeline_prints_csv(tmp_path, candidate, monkeypatch, capsys): + store = TraceStore(tmp_path / "cli.db") + store.ingest(candidate, "candidate.json") + + monkeypatch.setattr( + "sys.argv", + [ + "atw", + "--db", + str(tmp_path / "cli.db"), + "timeline", + "run-candidate-001", + "--format", + "csv", + ], + ) + main() + + out = capsys.readouterr().out + assert ( + out.splitlines()[0] + == "run_id,span_id,sequence,name,kind,status,start_offset_ms," + "end_offset_ms,duration_ms,error" + ) + assert out.splitlines()[1] == ( + "run-candidate-001,span-agent-101,0,agent.run,agent,error,0.0,280.0,280.0," + "agent.run ended with status error" + ) + + +def test_cli_timeline_missing_run_exits(tmp_path, monkeypatch): + monkeypatch.setattr( + "sys.argv", ["atw", "--db", str(tmp_path / "cli.db"), "timeline", "not-here"] + ) + + with pytest.raises(SystemExit, match="Run not found"): + main() + + +def test_run_page_shows_error_timeline(tmp_path, candidate): + client = TestClient(create_app(tmp_path / "api.db")) + client.post("/api/traces", json=candidate.as_jsonable()) + + page = client.get("/runs/run-candidate-001").text + + assert "ERROR TIMELINE" in page + assert "When did this run fail?" in page + assert "timeline-svg" in page + assert "2 failed spans" in page + assert "reservation window expired" in page + assert "/api/runs/run-candidate-001/timeline" in page + assert "#span-span-tool-103" in page + + +def test_run_page_timeline_empty_state(tmp_path, baseline): + client = TestClient(create_app(tmp_path / "api.db")) + client.post("/api/traces", json=baseline.as_jsonable()) + + page = client.get("/runs/run-baseline-001").text + + assert "ERROR TIMELINE" in page + assert "No failed spans" in page + assert "timeline-svg" not in page