diff --git a/CHANGELOG.md b/CHANGELOG.md index 2c1404a..cf35e73 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,26 @@ All notable changes to Agent Trace Workbench appear in this file. The version format follows a release cycle. A release adds one coherent capability to the workbench. +## 1.7.0 - 2026-08-04 + +### Added + +- A status breakdown beside the daily failure line on the dashboard. +- A stacked bar per trend day that shows the run status counts. +- A status legend with window totals for each recorded run status. +- `GET /api/trend/statuses` route that returns the daily status counts. +- `GET /api/trend/statuses?format=csv` route that returns the counts as a CSV attachment. +- `atw trend --statuses` command that prints the daily status counts. +- `atw trend --statuses --format csv` command that prints the counts as CSV. +- The dashboard status panel keeps the active agent and window in its export links. +- Deterministic tests for the status trend, the API routes, the CLI options, the CSV export, and the dashboard panel. + +### Changed + +- Version numbers moved to 1.7.0. +- The dashboard trend section now shows the status composition of each day. +- The architecture now includes a status breakdown beside the failure trend. + ## 1.6.0 - 2026-08-04 ### Added diff --git a/README.md b/README.md index 8b3a015..4fdeda4 100644 --- a/README.md +++ b/README.md @@ -22,6 +22,8 @@ Release 1.5 adds a CSV export for the failure trend and an agent-level trend fil Release 1.6 adds a trend window selector and a per-day drill-down on the dashboard. Choose 7, 14, 30, or 90 day views. Click a day to see the runs that started that day. +Release 1.7 adds a status breakdown beside the daily failure line. Each trend day shows a stacked bar of run status counts. Read the same counts from the API or the CLI. + ## Value Agent debugging needs evidence at tool boundaries. @@ -64,7 +66,7 @@ SQLite runs in WAL mode with a busy timeout. Readers keep a committed snapshot. - `models.py` defines the portable trace contract. - `handlers.py` loads local handler config and applies side-effect guards. -- `storage.py` owns the SQLite schema, WAL coordination, idempotent ingestion, and local annotations. It also computes the review list, applies bulk labels, builds the library report, computes the daily failure trend, lists the runs for one day, and enforces the retention cutoff for cleanup. A cleanup log records each scheduled sweep. +- `storage.py` owns the SQLite schema, WAL coordination, idempotent ingestion, and local annotations. It also computes the review list, applies bulk labels, builds the library report, computes the daily failure trend and the status breakdown, lists the runs for one day, and enforces the retention cutoff for cleanup. A cleanup log records each scheduled sweep. - `ingestion.py` watches JSON files and returns stable schema error reports. - `otlp.py` converts the OTLP JSON encoding to and from the trace contract. - `replay.py` runs guarded local handlers and records mismatches. @@ -408,6 +410,64 @@ python -m agent_trace_workbench.cli trend --day 2026-07-31 --format csv The day panel offers the same CSV download. A day outside the active window is ignored. The dashboard draws no panel for it. +## Status breakdown + +The dashboard shows which run statuses shape each trend day. + +Each day draws one stacked bar beside the failure line. The bar shows how many runs ended ok and how many failed. A legend lists the window totals for each status. + +Read the breakdown over the API. + +```powershell +curl.exe "http://127.0.0.1:8000/api/trend/statuses?days=14" +``` + +The response lists one bucket per day. Each bucket maps status names to run counts. + +```json +[ + { + "day": "2026-07-31", + "runs": 2, + "statuses": { + "ok": 1, + "error": 1 + } + } +] +``` + +Filter the breakdown to one agent. + +```powershell +curl.exe "http://127.0.0.1:8000/api/trend/statuses?agent=catalog-assistant" +``` + +The dashboard panel keeps the active agent and window. It follows the same trend filter. + +Download the breakdown as CSV. + +```powershell +curl.exe -o status-trend.csv "http://127.0.0.1:8000/api/trend/statuses?format=csv" +``` + +The file lists one row per status on a day. Empty days produce no rows. + +```text +day,agent_name,status,runs +2026-07-31,,ok,1 +2026-07-31,,error,1 +``` + +Use the CLI for scripts. + +```powershell +python -m agent_trace_workbench.cli trend --statuses +python -m agent_trace_workbench.cli trend --statuses --agent catalog-assistant --format csv +``` + +The panel JSON and CSV links keep the active agent and window. + ## Saved comparisons Save a comparison for later review. @@ -1203,7 +1263,7 @@ curl.exe -X POST http://127.0.0.1:8000/api/traces ` ## Test status -The test suite covers the core flows. It covers storage, ingestion, replay, comparison, search, annotations, bulk labels, export, review, reports, retention cleanup, and scheduled cleanup. It covers the CLI, the API, collector export, the server scheduler, and the dashboard failure trend, including the agent filter, the window selector, the day drill-down, and the CSV exports. +The test suite covers the core flows. It covers storage, ingestion, replay, comparison, search, annotations, bulk labels, export, review, reports, retention cleanup, and scheduled cleanup. It covers the CLI, the API, collector export, the server scheduler, and the dashboard failure trend, including the agent filter, the window selector, the day drill-down, the status breakdown, and the CSV exports. Run the checks with these commands. @@ -1214,7 +1274,7 @@ python scripts/check_requirements.py python -m compileall agent_trace_workbench tests ``` -Current verification passes 289 tests, Ruff lint, dependency checks, and Python compilation. CI installs from `requirements-lock.txt` and runs these checks on Python 3.11, 3.12, and 3.13 for every push and pull request. +Current verification passes 308 tests, Ruff lint, dependency checks, and Python compilation. CI installs from `requirements-lock.txt` and runs these checks on Python 3.11, 3.12, and 3.13 for every push and pull request. ## Limitations @@ -1250,6 +1310,14 @@ The failure trend groups by the calendar day a run started. It uses the UTC day. The trend counts a run by its recorded status. A run with any error span counts as a failure. +The status breakdown groups by the same UTC calendar day as the trend. + +The status breakdown counts runs by their recorded status. A run with an error span counts as an error run. + +The status breakdown draws one stacked bar per day. The bar height scales to the busiest day in the window. + +The status CSV lists one row per status present on a day. Empty days produce no rows. + The trend agent filter matches the exact recorded agent name. The trend CSV repeats the active agent in every row. The all-agents view leaves that cell empty. @@ -1313,13 +1381,14 @@ The span exporter sends each workbench span as it ends. It does not batch spans. - Release 1.4 complete: add a server-side sweep scheduler and a failure trend line on the dashboard. - Release 1.5 complete: add a CSV export for the failure trend and an agent-level trend filter on the dashboard. - Release 1.6 complete: add a trend window selector and a per-day drill-down on the dashboard chart. -- Release 1.7: add a status breakdown beside the daily failure line on the dashboard. +- Release 1.7 complete: add a status breakdown beside the daily failure line on the dashboard. +- Release 1.8: add an agent comparison overlay to the failure trend. ## Repository map `fixtures/` contains meaningful baseline, candidate, and second-agent traces. It also contains a handler config and demo scripts. -`tests/` contains deterministic tests for the core. It covers coordination, guards, search, annotations, OTLP, export, review, reports, retention cleanup, scheduled cleanup, the server scheduler, and the failure trend, including the agent filter, the window selector, the day drill-down, and the CSV exports. +`tests/` contains deterministic tests for the core. It covers coordination, guards, search, annotations, OTLP, export, review, reports, retention cleanup, scheduled cleanup, the server scheduler, and the failure trend, including the agent filter, the window selector, the day drill-down, the status breakdown, and the CSV exports. `static/` and `templates/` contain the presentation layer. diff --git a/agent_trace_workbench/__init__.py b/agent_trace_workbench/__init__.py index 88efe31..16839b5 100644 --- a/agent_trace_workbench/__init__.py +++ b/agent_trace_workbench/__init__.py @@ -1,3 +1,3 @@ """Agent Trace Workbench package.""" -__version__ = "1.6.0" +__version__ = "1.7.0" diff --git a/agent_trace_workbench/cli.py b/agent_trace_workbench/cli.py index 7553226..19270f0 100644 --- a/agent_trace_workbench/cli.py +++ b/agent_trace_workbench/cli.py @@ -17,6 +17,7 @@ day_runs_to_csv, report_to_csv, run_tools_to_csv, + status_trend_to_csv, trend_to_csv, ) from .handlers import ReplayPolicy, load_handler_config @@ -170,6 +171,11 @@ def build_parser() -> argparse.ArgumentParser: default=None, help="List the runs that started on one YYYY-MM-DD day", ) + trend.add_argument( + "--statuses", + action="store_true", + help="Show the per-day run status breakdown", + ) annotate = subparsers.add_parser( "annotate", help="Label a run and add local review notes" @@ -418,6 +424,12 @@ def main() -> None: ) elif args.days < 1 or args.days > 90: raise SystemExit("--days must be between 1 and 90") + elif args.statuses: + buckets = store.status_trend(args.days, agent_name=args.agent) + if args.format == "csv": + print(status_trend_to_csv(buckets, agent_name=args.agent or ""), end="") + else: + print(json.dumps(buckets, indent=2)) else: trend = store.failure_trend(args.days, agent_name=args.agent) if args.format == "csv": diff --git a/agent_trace_workbench/export.py b/agent_trace_workbench/export.py index cae93b9..fea702a 100644 --- a/agent_trace_workbench/export.py +++ b/agent_trace_workbench/export.py @@ -69,6 +69,8 @@ _TREND_HEADERS = ["day", "agent_name", "runs", "failures", "failure_rate"] +_STATUS_TREND_HEADERS = ["day", "agent_name", "status", "runs"] + _DAY_RUNS_HEADERS = [ "day", "run_id", @@ -264,6 +266,34 @@ def trend_to_csv(trend: list[dict[str, Any]], agent_name: str = "") -> str: return _to_csv(_TREND_HEADERS, rows) +def status_trend_to_csv( + trend: list[dict[str, Any]], agent_name: str = "" +) -> str: + """Render a daily run status breakdown as a CSV document. + + The document lists one row per status present on a day. The + agent_name cell repeats the active filter, so a filtered file stays + self-describing. Empty days produce no rows, because they carry no + status counts. + """ + + with traced_operation( + "export.status_trend_csv", {"trend.days": len(trend), "trend.agent": agent_name} + ): + rows: list[dict[str, Any]] = [] + for bucket in trend: + for status, count in sorted(bucket["statuses"].items()): + rows.append( + { + "day": bucket["day"], + "agent_name": agent_name, + "status": status, + "runs": count, + } + ) + return _to_csv(_STATUS_TREND_HEADERS, rows) + + def day_runs_to_csv(day: str, runs: list[dict[str, Any]], agent_name: str = "") -> str: """Render the runs that started on one day as a CSV document. diff --git a/agent_trace_workbench/main.py b/agent_trace_workbench/main.py index 7619058..0e905fa 100644 --- a/agent_trace_workbench/main.py +++ b/agent_trace_workbench/main.py @@ -24,6 +24,7 @@ day_runs_to_csv, report_to_csv, run_tools_to_csv, + status_trend_to_csv, trend_to_csv, ) from .handlers import ReplayPolicy, load_handler_config @@ -99,6 +100,9 @@ def dashboard( if selected_day else None ) + status_breakdown = app.state.store.status_trend( + days, agent_name=selected_agent or None + ) return render_template( request, "dashboard.html", @@ -106,6 +110,7 @@ def dashboard( "runs": runs, "stats": _stats(runs), "trend": chart, + "status_bars": _status_bars(status_breakdown), "query": q or "", "trend_agents": app.state.store.trend_agents(), "selected_agent": selected_agent, @@ -117,6 +122,7 @@ def dashboard( else None ), "trend_links": _trend_links(selected_agent, days), + "status_links": _status_links(selected_agent, days), "store": app.state.store.store_info(), "telemetry": _telemetry_info(), "scheduler": _scheduler_status(app), @@ -382,6 +388,23 @@ def api_trend( def api_trend_agents() -> list[str]: return app.state.store.trend_agents() + @app.get("/api/trend/statuses", response_model=None) + def api_trend_statuses( + days: int = Query(default=14, ge=1, le=90), + agent: str | None = Query(default=None, max_length=200), + export_format: str = Query(default="json", alias="format"), + ) -> Response | list[dict[str, Any]]: + buckets = app.state.store.status_trend(days, agent_name=agent) + if export_format == "csv": + return _download_response( + status_trend_to_csv(buckets, agent_name=agent or ""), + "status-trend.csv", + "text/csv; charset=utf-8", + ) + if export_format != "json": + raise HTTPException(status_code=400, detail="format must be 'json' or 'csv'") + return buckets + @app.get("/api/trend/{day}", response_model=None) def api_trend_day( day: str, @@ -819,6 +842,111 @@ def _trend_links(selected_agent: str, days: int) -> dict[str, str]: } +def _status_links(selected_agent: str, days: int) -> dict[str, str]: + """Return the JSON and CSV export links for the status breakdown panel. + + The links keep the active agent filter and any non-default window + size, so a download matches what the panel draws. Agent names are + URL-encoded because they may contain spaces or punctuation. + """ + + params: dict[str, Any] = {} + if days != 14: + params["days"] = days + if selected_agent: + params["agent"] = selected_agent + if not params: + return {"json": "/api/trend/statuses", "csv": "/api/trend/statuses?format=csv"} + prefix = urlencode(params) + return { + "json": f"/api/trend/statuses?{prefix}", + "csv": f"/api/trend/statuses?{prefix}&format=csv", + } + + +_STATUS_PRIORITY = {"ok": 0, "error": 1, "unset": 2} + + +def _status_order(statuses: list[str]) -> list[str]: + """Return status names in a stable visual order. + + Known statuses sort by their priority so the stacked bars always + draw ok at the bottom and error above it. Unknown statuses follow in + alphabetical order, because an imported trace may carry a name the + workbench does not recognise. + """ + + return sorted(statuses, key=lambda status: (_STATUS_PRIORITY.get(status, 3), status)) + + +def _status_class(status: str) -> str: + """Map a run status to a stable CSS color class.""" + + return status if status in _STATUS_PRIORITY else "other" + + +def _status_bars(trend: list[dict[str, Any]]) -> dict[str, Any]: + """Shape a status breakdown into SVG-ready stacked bars for the dashboard. + + The helper maps each day to a bar on a fixed view box. Each bar + stacks one segment per status, scaled by the busiest day in the + window. It returns the bar geometry and the legend totals so the + template can draw the panel without any charting dependency. + """ + + width = 680 + height = 110 + pad_x = 8 + pad_y = 10 + count = len(trend) + step = (width - 2 * pad_x) / max(count - 1, 1) + bar_width = min(max(round(step * 0.55, 2), 6.0), 40.0) + plot_height = height - 2 * pad_y + max_total = max((item["runs"] for item in trend), default=0) + bars = [] + for index, item in enumerate(trend): + x = round(pad_x + index * step, 2) + segments = [] + y_top = height - pad_y + for status in _status_order(list(item["statuses"])): + segment_count = item["statuses"][status] + if max_total == 0: + continue + segment_height = round(segment_count / max_total * plot_height, 2) + y0 = y_top + y1 = round(y_top - segment_height, 2) + y_top = y1 + segments.append( + { + "status": status, + "cls": _status_class(status), + "count": segment_count, + "y0": y0, + "y1": y1, + } + ) + bars.append({"x": x, "day": item["day"], "runs": item["runs"], "segments": segments}) + status_names = sorted( + {status for item in trend for status in item["statuses"]}, + key=lambda status: (_STATUS_PRIORITY.get(status, 3), status), + ) + totals = {status: 0 for status in status_names} + for item in trend: + for status, segment_count in item["statuses"].items(): + totals[status] += segment_count + legend = [ + {"status": status, "cls": _status_class(status), "total": totals[status]} + for status in status_names + ] + return { + "width": width, + "height": height, + "bar_width": bar_width, + "bars": bars, + "legend": legend, + } + + def _day_href(selected_agent: str, days: int, day: str) -> str: """Return the drill-down link for one day on the trend chart. diff --git a/agent_trace_workbench/storage.py b/agent_trace_workbench/storage.py index 66bbcb5..c39d858 100644 --- a/agent_trace_workbench/storage.py +++ b/agent_trace_workbench/storage.py @@ -772,6 +772,62 @@ def failure_trend( ) return buckets + def status_trend( + self, + days: int = 14, + *, + agent_name: str | None = None, + ) -> list[dict[str, Any]]: + """Return a daily run status breakdown over the last N days. + + The trend groups runs by the calendar day (UTC) when they + started. Each bucket reports the total runs and a per-status + count, so a reviewer sees the shape of the day beside the + failure line. Pass agent_name to restrict the breakdown to one + agent. Empty days stay in the window with an empty status map, + so the axis matches the failure trend. + """ + + if days < 1: + raise ValueError("days must be at least 1") + safe_days = min(days, 90) + start_day = (_now_utc() - timedelta(days=safe_days - 1)).date() + start_text = start_day.strftime("%Y-%m-%d") + query = """ + SELECT substr(started_at, 1, 10) AS day, + status, + COUNT(*) AS count + FROM runs + WHERE substr(started_at, 1, 10) >= ? + """ + params: list[Any] = [start_text] + if agent_name: + query += " AND agent_name = ?" + params.append(agent_name) + query += """ + GROUP BY day, status + ORDER BY day ASC, status ASC + """ + with traced_operation("storage.status_trend", {"trend.days": safe_days}): + with self._connect() as connection: + rows = connection.execute(query, params).fetchall() + per_day: dict[str, dict[str, int]] = {} + for row in rows: + counts = per_day.setdefault(row["day"], {}) + counts[row["status"]] = row["count"] + buckets: list[dict[str, Any]] = [] + for offset in range(safe_days): + day = (start_day + timedelta(days=offset)).strftime("%Y-%m-%d") + statuses = per_day.get(day, {}) + buckets.append( + { + "day": day, + "runs": sum(statuses.values()), + "statuses": statuses, + } + ) + return buckets + def runs_on_day( self, day: str, diff --git a/pyproject.toml b/pyproject.toml index 18a4412..bac403a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "agent-trace-workbench" -version = "1.6.0" +version = "1.7.0" description = "A local workbench for recording, replaying, comparing, and inspecting agent traces." readme = "README.md" requires-python = ">=3.11" diff --git a/static/styles.css b/static/styles.css index f43d5c2..e4444e9 100644 --- a/static/styles.css +++ b/static/styles.css @@ -219,6 +219,22 @@ input[type="checkbox"] { width: 16px; height: 16px; accent-color: var(--navy); c .trend-foot strong { color: var(--navy); font: 500 20px Georgia, serif; letter-spacing: -.02em; } .trend-foot .trend-failures strong { color: var(--coral); } .trend-foot .button { margin-left: auto; } +.trend-status { margin-top: 24px; padding-top: 20px; border-top: 1px solid var(--line); } +.trend-status-head { display: flex; align-items: baseline; justify-content: space-between; gap: 20px; margin-bottom: 16px; } +.trend-status-head h3 { margin-top: 5px; color: var(--navy); font: 500 19px Georgia, serif; } +.trend-status .trend-chart { margin-top: 14px; } +.status-svg { display: block; width: 100%; height: 110px; overflow: visible; } +.status-bar { transition: opacity .16s ease; } +.status-bar:hover { opacity: .82; } +.bar-ok { fill: var(--teal); } +.bar-error { fill: var(--coral); } +.bar-unset { fill: #b8c0c9; } +.bar-other { fill: var(--blue); } +.trend-legend { display: flex; align-items: center; flex-wrap: wrap; gap: 20px; margin-top: 18px; padding-top: 15px; border-top: 1px solid var(--line); color: var(--muted); font: 10px ui-monospace, SFMono-Regular, Consolas, monospace; letter-spacing: .06em; text-transform: uppercase; } +.legend-item { display: inline-flex; align-items: center; gap: 8px; } +.legend-swatch { width: 9px; height: 9px; } +.legend-item strong { color: var(--navy); font: 500 14px Georgia, serif; letter-spacing: -.01em; } +.trend-legend .button { margin-left: auto; } .day-actions { display: flex; align-items: center; gap: 10px; margin-top: 16px; } .day-actions .button { min-height: 42px; } .scheduler-section .section-heading { align-items: center; } @@ -229,4 +245,4 @@ input[type="checkbox"] { width: 16px; height: 16px; accent-color: var(--navy); c .scheduler-stats > div { padding: 16px 18px; border: 1px solid var(--line); background: #fff; } .scheduler-stats .field-label { display: block; margin-bottom: 8px; } .scheduler-stats .mono { color: var(--navy); font-size: 12px; } -@media (max-width: 800px) { .shell { width: min(100% - 32px, 620px); } .topbar { padding: 0 16px; } nav { gap: 15px; } .nav-api { display: none; } .hero { padding: 63px 0 54px; } .workspace-grid, .run-grid, .metric-grid { grid-template-columns: 1fr; } .metric-grid-four { grid-template-columns: 1fr 1fr; } .diff-detail-grid { grid-template-columns: 1fr; } .compare-form { grid-template-columns: 1fr; gap: 4px; } .compare-vs { padding: 10px 0 0; text-align: left; } .compare-form .button { margin-top: 13px; } .detail-hero { display: block; } .detail-actions { margin-top: 25px; } .tool-grid { grid-template-columns: 1fr; } .filter-bar { grid-template-columns: 1fr; } .footer { display: block; } .footer span { display: block; margin-top: 6px; } .scheduler-stats { grid-template-columns: 1fr 1fr; } .trend-foot .button { margin-left: 0; } } +@media (max-width: 800px) { .shell { width: min(100% - 32px, 620px); } .topbar { padding: 0 16px; } nav { gap: 15px; } .nav-api { display: none; } .hero { padding: 63px 0 54px; } .workspace-grid, .run-grid, .metric-grid { grid-template-columns: 1fr; } .metric-grid-four { grid-template-columns: 1fr 1fr; } .diff-detail-grid { grid-template-columns: 1fr; } .compare-form { grid-template-columns: 1fr; gap: 4px; } .compare-vs { padding: 10px 0 0; text-align: left; } .compare-form .button { margin-top: 13px; } .detail-hero { display: block; } .detail-actions { margin-top: 25px; } .tool-grid { grid-template-columns: 1fr; } .filter-bar { grid-template-columns: 1fr; } .footer { display: block; } .footer span { display: block; margin-top: 6px; } .scheduler-stats { grid-template-columns: 1fr 1fr; } .trend-foot .button, .trend-legend .button { margin-left: 0; } } diff --git a/templates/dashboard.html b/templates/dashboard.html index 9dc509b..7deccfc 100644 --- a/templates/dashboard.html +++ b/templates/dashboard.html @@ -2,7 +2,7 @@ {% block title %}Runs | Agent Trace Workbench{% endblock %} {% block content %}
-
LOCAL OBSERVABILITY / RELEASE 1.6
+
LOCAL OBSERVABILITY / RELEASE 1.7

See what your agent did.
Replay why it did it.

Ingest local JSON traces, import OTLP exports, watch a folder, inspect every tool boundary, replay recorded work, compare runs, label evidence for review, and read a folder-level library report without a hosted service.

@@ -87,6 +87,29 @@

See what your agent did.
Replay why it did it.

CSV
+
+
+
STATUS BREAKDOWN

What shapes each day?

+ run counts by status{% if selected_agent %} ยท {{ selected_agent }}{% endif %} +
+ +
+ {% for item in status_bars.legend %}{{ item.status }}{{ item.total }}{% endfor %} + JSON + CSV +
+
{% else %}
+

{% if selected_agent %}No runs for {{ selected_agent }} in this window{% else %}No runs in this window{% endif %}

Record a run and the dashboard draws its daily failure line.

{% endif %} diff --git a/tests/test_csv_export.py b/tests/test_csv_export.py index f2100b5..cfab2c6 100644 --- a/tests/test_csv_export.py +++ b/tests/test_csv_export.py @@ -14,6 +14,7 @@ day_runs_to_csv, report_to_csv, run_tools_to_csv, + status_trend_to_csv, trend_to_csv, ) from agent_trace_workbench.main import create_app @@ -318,6 +319,50 @@ def test_api_trend_csv_carries_agent_filter(tmp_path, baseline, support): assert all(row["agent_name"] == "support-assistant" for row in rows) +def test_status_trend_csv_lists_one_row_per_status(tmp_path, baseline, candidate): + store = TraceStore(tmp_path / "api.db") + store.ingest(baseline, "baseline.json") + store.ingest(candidate, "candidate.json") + _set_started(store, baseline.run_id, 2) + _set_started(store, candidate.run_id, 2) + + rows = _read_csv(status_trend_to_csv(store.status_trend(7))) + + assert list(rows[0])[0] == "day" + active = [row for row in rows if row["day"] == _day(2)] + assert {row["status"]: row["runs"] for row in active} == {"ok": "1", "error": "1"} + assert active[0]["agent_name"] == "" + + +def test_status_trend_csv_repeats_the_active_agent(tmp_path, baseline, candidate): + store = TraceStore(tmp_path / "api.db") + store.ingest(baseline, "baseline.json") + store.ingest(candidate, "candidate.json") + _set_started(store, baseline.run_id, 2) + _set_started(store, candidate.run_id, 2) + + rows = _read_csv( + status_trend_to_csv( + store.status_trend(7, agent_name="catalog-assistant"), "catalog-assistant" + ) + ) + + assert rows + assert all(row["agent_name"] == "catalog-assistant" for row in rows) + + +def test_status_trend_csv_omits_empty_days(tmp_path, baseline): + store = TraceStore(tmp_path / "api.db") + store.ingest(baseline, "baseline.json") + _set_started(store, baseline.run_id, 5) + + rows = _read_csv(status_trend_to_csv(store.status_trend(7))) + + assert len(rows) == 1 + assert rows[0]["status"] == "ok" + assert rows[0]["runs"] == "1" + + def test_cli_trend_prints_json_buckets(tmp_path, baseline, candidate, monkeypatch, capsys): store = TraceStore(tmp_path / "cli.db") store.ingest(baseline, "baseline.json") diff --git a/tests/test_trend.py b/tests/test_trend.py index d24eeb0..ba119a6 100644 --- a/tests/test_trend.py +++ b/tests/test_trend.py @@ -504,3 +504,236 @@ def test_cli_trend_day_rejects_bad_format(tmp_path, monkeypatch, capsys): ) with pytest.raises(SystemExit, match="YYYY-MM-DD"): main() + + +def test_status_trend_counts_runs_by_status(tmp_path, baseline, candidate): + store = TraceStore(tmp_path / "trend.db") + store.ingest(baseline, "baseline.json") + store.ingest(candidate, "candidate.json") + _set_started(store, baseline.run_id, 2) + _set_started(store, candidate.run_id, 2) + + breakdown = store.status_trend(7) + + assert len(breakdown) == 7 + bucket = _bucket(breakdown, 2) + assert bucket["runs"] == 2 + assert bucket["statuses"] == {"ok": 1, "error": 1} + + +def test_status_trend_keeps_empty_days_in_window(tmp_path, baseline): + store = TraceStore(tmp_path / "trend.db") + store.ingest(baseline, "baseline.json") + _set_started(store, baseline.run_id, 5) + + breakdown = store.status_trend(7) + + empty = _bucket(breakdown, 1) + assert empty["runs"] == 0 + assert empty["statuses"] == {} + + +def test_status_trend_spreads_across_days(tmp_path, baseline, candidate): + store = TraceStore(tmp_path / "trend.db") + store.ingest(baseline, "baseline.json") + store.ingest(candidate, "candidate.json") + _set_started(store, baseline.run_id, 3) + _set_started(store, candidate.run_id, 1) + + breakdown = store.status_trend(7) + + assert _bucket(breakdown, 3)["statuses"] == {"ok": 1} + assert _bucket(breakdown, 1)["statuses"] == {"error": 1} + assert _bucket(breakdown, 2)["runs"] == 0 + + +def test_status_trend_filters_by_agent(tmp_path, baseline, candidate, support): + store = TraceStore(tmp_path / "trend.db") + store.ingest(baseline, "baseline.json") + store.ingest(candidate, "candidate.json") + store.ingest(support, "support.json") + _set_started(store, baseline.run_id, 2) + _set_started(store, candidate.run_id, 2) + _set_started(store, support.run_id, 2) + + catalog = store.status_trend(7, agent_name="catalog-assistant") + support_trend = store.status_trend(7, agent_name="support-assistant") + + assert _bucket(catalog, 2)["statuses"] == {"ok": 1, "error": 1} + assert _bucket(support_trend, 2)["statuses"] == {"ok": 1} + + +def test_status_trend_rejects_zero_days(tmp_path, baseline): + store = TraceStore(tmp_path / "trend.db") + store.ingest(baseline, "baseline.json") + + with pytest.raises(ValueError, match="days"): + store.status_trend(0) + + +def test_status_trend_caps_large_windows(tmp_path, baseline): + store = TraceStore(tmp_path / "trend.db") + store.ingest(baseline, "baseline.json") + + breakdown = store.status_trend(500) + + assert len(breakdown) == 90 + + +def test_status_trend_unknown_agent_returns_empty_buckets(tmp_path, baseline): + store = TraceStore(tmp_path / "trend.db") + store.ingest(baseline, "baseline.json") + _set_started(store, baseline.run_id, 2) + + breakdown = store.status_trend(7, agent_name="ghost-agent") + + assert len(breakdown) == 7 + assert all(bucket["statuses"] == {} for bucket in breakdown) + + +def test_api_status_trend_returns_buckets(tmp_path, baseline, candidate): + client = TestClient(create_app(tmp_path / "api.db")) + client.post("/api/traces", json=baseline.as_jsonable()) + client.post("/api/traces", json=candidate.as_jsonable()) + store = TraceStore(tmp_path / "api.db") + _set_started(store, baseline.run_id, 2) + _set_started(store, candidate.run_id, 2) + + buckets = client.get("/api/trend/statuses", params={"days": 7}).json() + + assert len(buckets) == 7 + assert _bucket(buckets, 2)["statuses"] == {"ok": 1, "error": 1} + + +def test_api_status_trend_filters_by_agent(tmp_path, baseline, candidate, support): + client = TestClient(create_app(tmp_path / "api.db")) + client.post("/api/traces", json=baseline.as_jsonable()) + client.post("/api/traces", json=candidate.as_jsonable()) + client.post("/api/traces", json=support.as_jsonable()) + store = TraceStore(tmp_path / "api.db") + _set_started(store, baseline.run_id, 2) + _set_started(store, candidate.run_id, 2) + _set_started(store, support.run_id, 2) + + buckets = client.get( + "/api/trend/statuses", params={"days": 7, "agent": "support-assistant"} + ).json() + + assert _bucket(buckets, 2)["statuses"] == {"ok": 1} + + +def test_api_status_trend_csv_returns_attachment(tmp_path, baseline, candidate): + client = TestClient(create_app(tmp_path / "api.db")) + client.post("/api/traces", json=baseline.as_jsonable()) + client.post("/api/traces", json=candidate.as_jsonable()) + store = TraceStore(tmp_path / "api.db") + _set_started(store, baseline.run_id, 2) + _set_started(store, candidate.run_id, 2) + + response = client.get( + "/api/trend/statuses", params={"days": 7, "format": "csv"} + ) + + assert response.status_code == 200 + assert response.headers["content-type"].startswith("text/csv") + assert response.headers["content-disposition"] == ( + 'attachment; filename="status-trend.csv"' + ) + assert response.text.splitlines()[0] == "day,agent_name,status,runs" + + +def test_api_status_trend_rejects_unknown_format(tmp_path): + client = TestClient(create_app(tmp_path / "api.db")) + + assert ( + client.get("/api/trend/statuses", params={"format": "xml"}).status_code == 400 + ) + + +def test_cli_status_trend_prints_buckets(tmp_path, baseline, candidate, monkeypatch, capsys): + store = TraceStore(tmp_path / "cli.db") + store.ingest(baseline, "baseline.json") + store.ingest(candidate, "candidate.json") + _set_started(store, baseline.run_id, 2) + _set_started(store, candidate.run_id, 2) + + monkeypatch.setattr( + "sys.argv", + ["atw", "--db", str(tmp_path / "cli.db"), "trend", "--statuses", "--days", "7"], + ) + main() + + buckets = json.loads(capsys.readouterr().out) + assert _bucket(buckets, 2)["statuses"] == {"ok": 1, "error": 1} + + +def test_cli_status_trend_prints_csv(tmp_path, baseline, candidate, monkeypatch, capsys): + store = TraceStore(tmp_path / "cli.db") + store.ingest(baseline, "baseline.json") + store.ingest(candidate, "candidate.json") + _set_started(store, baseline.run_id, 2) + _set_started(store, candidate.run_id, 2) + + monkeypatch.setattr( + "sys.argv", + [ + "atw", + "--db", + str(tmp_path / "cli.db"), + "trend", + "--statuses", + "--days", + "7", + "--format", + "csv", + ], + ) + main() + + out = capsys.readouterr().out + assert out.splitlines()[0] == "day,agent_name,status,runs" + + +def test_dashboard_shows_status_breakdown(tmp_path, baseline, candidate): + client = TestClient(create_app(tmp_path / "api.db")) + client.post("/api/traces", json=baseline.as_jsonable()) + client.post("/api/traces", json=candidate.as_jsonable()) + store = TraceStore(tmp_path / "api.db") + _set_started(store, baseline.run_id, 2) + _set_started(store, candidate.run_id, 2) + + page = client.get("/").text + + assert "STATUS BREAKDOWN" in page + assert "What shapes each day?" in page + assert "status-svg" in page + assert "bar-error" in page + assert "legend-item" in page + assert "/api/trend/statuses?format=csv" in page + + +def test_dashboard_status_breakdown_respects_agent_filter( + tmp_path, baseline, candidate, support +): + client = TestClient(create_app(tmp_path / "api.db")) + client.post("/api/traces", json=baseline.as_jsonable()) + client.post("/api/traces", json=candidate.as_jsonable()) + client.post("/api/traces", json=support.as_jsonable()) + store = TraceStore(tmp_path / "api.db") + _set_started(store, baseline.run_id, 2) + _set_started(store, candidate.run_id, 2) + _set_started(store, support.run_id, 2) + + page = client.get("/", params={"agent": "support-assistant"}).text + + assert "STATUS BREAKDOWN" in page + assert "/api/trend/statuses?agent=support-assistant&format=csv" in page + + +def test_dashboard_status_breakdown_empty_state(tmp_path): + client = TestClient(create_app(tmp_path / "api.db")) + + page = client.get("/").text + + assert "No runs in this window" in page + assert "status-svg" not in page