Extend Agent Trace Workbench

CHANGELOG.md

@@ -4,6 +4,20 @@ All notable changes to Agent Trace Workbench appear in this file.
 
 The version format follows a release cycle. A release adds one coherent capability to the workbench.
 
+## 1.12.0 - 2026-08-04
+
+### Added
+
+- A failure-first review queue for unlabeled runs.
+- Status filters for the review page, API, and CLI.
+- Ordered failed-span summaries in review queue results.
+- Deterministic tests for priority ordering, filters, API output, CLI output, and page context.
+
+### Changed
+
+- Version numbers moved to 1.12.0.
+- The review table shows the first failed span before a reviewer opens the run.
+
 ## 1.11.0 - 2026-08-04
 
 ### Added

README.md

@@ -32,6 +32,8 @@ Release 1.10 adds a span detail panel to the error timeline. Click a marker or a
 
 Release 1.11 adds ordered failed-span summaries to each day drill-down card and day export.
 
+Release 1.12 adds a failure-first review queue. Filter unlabeled runs by status and read failed-span context before opening a run.
+
 ## Value
 
 Agent debugging needs evidence at tool boundaries.
@@ -87,7 +89,7 @@ SQLite runs in WAL mode with a busy timeout. Readers keep a committed snapshot.
 
 The OpenTelemetry integration stays local by default. Set `ATW_OTEL_CONSOLE=1` to print workbench spans. Set `ATW_OTEL_COLLECTOR_ENDPOINT` to export them to a local collector.
 
-Each stored run keeps a local label and note. They form the review context for long-lived evidence. The server can sweep old evidence on an interval. The scheduler stays off unless you set `ATW_CLEANUP_EVERY_SECONDS`.
+Each stored run keeps a local label and note. They form the review context for long-lived evidence. The review queue orders failures first and includes failed-span summaries. The server can sweep old evidence on an interval. The scheduler stays off unless you set `ATW_CLEANUP_EVERY_SECONDS`.
 
 ## Setup
 
@@ -207,7 +209,17 @@ The command prints matching run summaries.
     "run_id": "run-candidate-001",
     "agent_name": "catalog-assistant",
     "status": "error",
-    "tool_count": 3
+    "tool_count": 3,
+    "error_summary": [
+      {
+        "name": "agent.run",
+        "message": "agent.run ended with status error"
+      },
+      {
+        "name": "reserve_inventory",
+        "message": "reservation window expired"
+      }
+    ]
   }
 ]
 ```
@@ -1154,6 +1166,17 @@ curl.exe "http://127.0.0.1:8000/api/review"
 
 The review page links each run to its annotation form.
 
+The queue shows failed runs first. It sorts each group by start time. Each failed run includes its ordered failed-span summary.
+
+Filter the queue by status.
+
+```powershell
+python -m agent_trace_workbench.cli review --status error
+curl.exe "http://127.0.0.1:8000/api/review?status=error"
+```
+
+The API and CLI return the same run summary. The summary adds an `error_summary` list for failed spans.
+
 ## Bulk labeling
 
 Apply one label to several runs at once.
@@ -1497,7 +1520,7 @@ python scripts/check_requirements.py
 python -m compileall agent_trace_workbench tests
 ```
 
-Current verification passes 360 tests, Ruff lint, dependency checks, and Python compilation. CI installs from `requirements-lock.txt` and runs these checks on Python 3.11, 3.12, and 3.13 for every push and pull request.
+Current verification passes 365 tests, Ruff lint, dependency checks, and Python compilation. CI installs from `requirements-lock.txt` and runs these checks on Python 3.11, 3.12, and 3.13 for every push and pull request.
 
 ## Limitations
 
@@ -1513,6 +1536,8 @@ Labels and notes stay local to the workbench database. Portable export files do
 
 The review list shows runs with an empty label. A blank label counts as unreviewed.
 
+The review queue filters by recorded run status. It prioritizes failures, then older runs.
+
 Bulk labeling sets the label only. It leaves the notes on each run untouched.
 
 Retention counts from the last ingestion time. A re-ingest resets the clock.
@@ -1639,6 +1664,7 @@ The span exporter sends each workbench span as it ends. It does not batch spans.
 - Release 1.9 complete: add a run-level error timeline to the run detail page.
 - Release 1.10 complete: add a span detail panel to the run-level error timeline.
 - Release 1.11 complete: add ordered failed-span summaries to day cards, API output, CLI output, and CSV export.
+- Release 1.12 complete: add a failure-first review queue with status filters and failed-span context.
 - Next: choose the next bounded evidence-review slice.
 
 ## Repository map

agent_trace_workbench/__init__.py

@@ -1,3 +1,3 @@
 """Agent Trace Workbench package."""
 
-__version__ = "1.11.0"
+__version__ = "1.12.0"

agent_trace_workbench/cli.py

@@ -130,6 +130,12 @@ def build_parser() -> argparse.ArgumentParser:
         "review", help="List runs that still need a review label"
     )
     review.add_argument("--limit", type=int, default=20)
+    review.add_argument(
+        "--status",
+        choices=["all", "ok", "error"],
+        default="all",
+        help="Show only healthy or failed unlabeled runs",
+    )
     review.add_argument(
         "--label",
         default=None,
@@ -421,10 +427,12 @@ def main() -> None:
         else:
             print(json.dumps(store.list_comparisons(args.limit), indent=2))
     elif args.command == "review":
+        review_status = None if args.status == "all" else args.status
         if args.label is not None:
             _validate_annotation("label", args.label)
             run_ids = args.run_ids or [
-                run["run_id"] for run in store.unreviewed_runs(100)
+                run["run_id"]
+                for run in store.unreviewed_runs(100, status=review_status)
             ]
             if not run_ids:
                 raise SystemExit("No runs to label")
@@ -433,7 +441,11 @@ def main() -> None:
         elif args.run_ids:
             raise SystemExit("Provide --label together with --run-id")
         else:
-            print(json.dumps(store.unreviewed_runs(args.limit), indent=2))
+            print(
+                json.dumps(
+                    store.unreviewed_runs(args.limit, status=review_status), indent=2
+                )
+            )
     elif args.command == "report":
         if args.older_than_days < 1:
             raise SystemExit("--older-than must be at least 1 day")

agent_trace_workbench/main.py

@@ -8,7 +8,7 @@
 from datetime import datetime, timedelta, timezone
 from html import escape
 from pathlib import Path
-from typing import Any
+from typing import Any, Literal
 from urllib.parse import urlencode
 
 from fastapi import FastAPI, HTTPException, Query, Request
@@ -229,8 +229,9 @@ def compare_page(
     def review_page(
         request: Request,
         limit: int = Query(default=50, ge=1, le=200),
+        status: Literal["ok", "error"] | None = Query(default=None),
     ) -> Any:
-        runs = app.state.store.unreviewed_runs(limit)
+        runs = app.state.store.unreviewed_runs(limit, status=status)
         totals = app.state.store.library_report()["totals"]
         return render_template(
             request,
@@ -239,6 +240,8 @@ def review_page(
                 "runs": runs,
                 "totals": totals,
                 "limit": limit,
+                "status": status or "",
+                "failure_count": sum(run["status"] == "error" for run in runs),
                 "store": app.state.store.store_info(),
                 "telemetry": _telemetry_info(),
                 "scheduler": _scheduler_status(app),
@@ -304,8 +307,9 @@ def api_runs(
     @app.get("/api/review")
     def api_review(
         limit: int = Query(default=20, ge=1, le=100),
+        status: Literal["ok", "error"] | None = Query(default=None),
     ) -> list[dict[str, Any]]:
-        return app.state.store.unreviewed_runs(limit)
+        return app.state.store.unreviewed_runs(limit, status=status)
 
     @app.post("/api/review/labels")
     def api_bulk_label(payload: BulkLabelRequest) -> dict[str, Any]:

agent_trace_workbench/storage.py

@@ -543,23 +543,38 @@ def search_runs(self, query: str, limit: int = 20) -> list[dict[str, Any]]:
                 ).fetchall()
                 return _summarize_runs(connection, rows)
 
-    def unreviewed_runs(self, limit: int = 20) -> list[dict[str, Any]]:
-        """Return run summaries that have no review label.
+    def unreviewed_runs(
+        self,
+        limit: int = 20,
+        *,
+        status: str | None = None,
+    ) -> list[dict[str, Any]]:
+        """Return unlabeled runs with failure context for review.
 
-        A reviewer starts here. Each returned run has an empty label, so
-        it has not been triaged yet. The review page links each run to its
-        annotation form.
+        Failed runs come first. Within each status, older runs come first.
+        Pass status to show only ok or error runs.
         """
 
+        if status not in {None, "ok", "error"}:
+            raise ValueError("status must be 'ok', 'error', or None")
         safe_limit = max(1, min(limit, 100))
-        with traced_operation("storage.unreviewed_runs", {"review.limit": safe_limit}):
+        attributes: dict[str, Any] = {"review.limit": safe_limit}
+        if status is not None:
+            attributes["review.status"] = status
+        with traced_operation("storage.unreviewed_runs", attributes):
             with self._connect() as connection:
-                rows = connection.execute(
-                    "SELECT * FROM runs WHERE label = '' "
-                    "ORDER BY started_at DESC, run_id DESC LIMIT ?",
-                    (safe_limit,),
-                ).fetchall()
-                return _summarize_runs(connection, rows)
+                query = "SELECT * FROM runs WHERE label = ''"
+                params: list[Any] = []
+                if status is not None:
+                    query += " AND status = ?"
+                    params.append(status)
+                query += (
+                    " ORDER BY CASE WHEN status = 'error' THEN 0 ELSE 1 END, "
+                    "started_at ASC, run_id ASC LIMIT ?"
+                )
+                params.append(safe_limit)
+                rows = connection.execute(query, params).fetchall()
+                return _summarize_runs(connection, rows, include_error_summary=True)
 
     def library_report(self, *, older_than_days: int = 30) -> dict[str, Any]:
         """Return a folder-level summary of the local trace library.

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "agent-trace-workbench"
-version = "1.11.0"
+version = "1.12.0"
 description = "A local workbench for recording, replaying, comparing, and inspecting agent traces."
 readme = "README.md"
 requires-python = ">=3.11"

static/styles.css

@@ -185,6 +185,10 @@ tbody tr:last-child td { border-bottom: 0; }
 .mono { font: 11px ui-monospace, SFMono-Regular, Consolas, monospace; }
 .row-changed td { background: rgba(217,95,79,.035); }
 .row-alert { color: var(--coral); font-weight: 700; }
+.review-evidence { display: flex; min-width: 230px; flex-direction: column; gap: 3px; white-space: normal; }
+.review-evidence strong { color: var(--coral); font: 800 9px ui-monospace, SFMono-Regular, Consolas, monospace; letter-spacing: .08em; text-transform: uppercase; }
+.review-evidence span { overflow: hidden; max-width: 300px; color: var(--navy); font-size: 11px; text-overflow: ellipsis; white-space: nowrap; }
+.review-evidence small, .review-clear { color: var(--muted); font: 10px ui-monospace, SFMono-Regular, Consolas, monospace; }
 .review-section { padding-top: 48px; }
 .cleanup-section { padding-top: 48px; }
 .cleanup-section-last { padding-bottom: 88px; }

templates/dashboard.html

@@ -2,7 +2,7 @@
 {% block title %}Runs | Agent Trace Workbench{% endblock %}
 {% block content %}
 <section class="hero">
-    <div class="eyebrow">LOCAL OBSERVABILITY / RELEASE 1.9</div>
+    <div class="eyebrow">LOCAL OBSERVABILITY / RELEASE 1.12</div>
     <h1>See what your agent did.<br><em>Replay why it did it.</em></h1>
     <p class="hero-copy">Ingest local JSON traces, import OTLP exports, watch a folder, inspect every tool boundary, replay recorded work, compare runs, label evidence for review, and read a folder-level library report without a hosted service.</p>
     <div class="hero-actions">
@@ -42,7 +42,7 @@ <h1>See what your agent did.<br><em>Replay why it did it.</em></h1>
             <a href="#recent-runs"><span class="view-number">A</span><span><strong>Inspect</strong><small>Timeline, attributes, and failures</small></span><span class="view-arrow">&#8599;</span></a>
             <a href="#recent-runs"><span class="view-number">B</span><span><strong>Replay</strong><small>Stable handlers or recorded fallback</small></span><span class="view-arrow">&#8599;</span></a>
             <a href="#recent-runs"><span class="view-number">C</span><span><strong>Compare</strong><small>Calls, timing, results, and outcomes</small></span><span class="view-arrow">&#8599;</span></a>
-            <a href="/review"><span class="view-number">D</span><span><strong>Review</strong><small>Triage runs that still need a label</small></span><span class="view-arrow">&#8599;</span></a>
+            <a href="/review"><span class="view-number">D</span><span><strong>Review</strong><small>Failures first with failed-span context</small></span><span class="view-arrow">&#8599;</span></a>
             <a href="/report"><span class="view-number">E</span><span><strong>Report</strong><small>Folder-level summary of the library</small></span><span class="view-arrow">&#8599;</span></a>
         </div>
         <div class="architecture-line"><span class="dot"></span>FastAPI <span class="line"></span> SQLite <span class="line"></span> OpenTelemetry <span class="line"></span> Local collector</div>

templates/review.html

@@ -3,29 +3,34 @@
 {% block content %}
 <section class="detail-hero review-hero">
     <div>
-        <div class="eyebrow">REVIEW LIST / UNLABELED EVIDENCE</div>
-        <h1>Give every run a label.</h1>
-        <p class="hero-copy">Runs without a label have not been triaged yet. Open one, note what it showed, and it leaves this list.</p>
+        <div class="eyebrow">REVIEW QUEUE / FAILURE-FIRST</div>
+        <h1>Start with the runs that need answers.</h1>
+        <p class="hero-copy">Failed runs appear first. Each row shows its first failed span. Label a run after you inspect it.</p>
     </div>
     <div class="detail-actions">
-        <a class="button button-quiet" href="/api/review?limit={{ limit }}">JSON</a>
+        <a class="button button-quiet" href="/api/review?limit={{ limit }}{% if status %}&amp;status={{ status }}{% endif %}">JSON</a>
         <a class="button button-primary" href="/report">Library report <span aria-hidden="true">&#8594;</span></a>
     </div>
 </section>
 <section class="metric-grid detail-metrics">
     <div class="metric-card"><span class="metric-label">Needs review</span><strong>{{ runs|length }}<small> shown</small></strong><span class="metric-note">in this view</span></div>
+    <div class="metric-card {% if failure_count %}metric-alert{% endif %}"><span class="metric-label">Failures shown</span><strong>{{ failure_count }}</strong><span class="metric-note">priority evidence</span></div>
     <div class="metric-card"><span class="metric-label">Unlabeled total</span><strong>{{ totals.unlabeled_runs }}</strong><span class="metric-note">across the library</span></div>
-    <div class="metric-card {% if totals.failure_runs %}metric-alert{% endif %}"><span class="metric-label">Failure runs</span><strong>{{ totals.failure_runs }}</strong><span class="metric-note">unlabeled or not</span></div>
 </section>
 <section class="section-block review-section">
-    <div class="section-heading"><div><div class="eyebrow">REVIEW QUEUE</div><h2>Unlabeled runs</h2></div><span class="section-meta">{{ totals.labeled_runs }} labeled · {{ totals.unlabeled_runs }} left</span></div>
+    <div class="section-heading"><div><div class="eyebrow">REVIEW QUEUE</div><h2>{% if status == 'error' %}Failed{% elif status == 'ok' %}Healthy{% else %}Unlabeled{% endif %} runs</h2></div><span class="section-meta">{{ runs|length }} shown &middot; failures first</span></div>
+    <div class="state-filter" aria-label="Review status filter">
+        <a class="chip{% if not status %} chip-active{% endif %}" href="/review?limit={{ limit }}">All</a>
+        <a class="chip{% if status == 'error' %} chip-active{% endif %}" href="/review?limit={{ limit }}&amp;status=error">Failures</a>
+        <a class="chip{% if status == 'ok' %} chip-active{% endif %}" href="/review?limit={{ limit }}&amp;status=ok">Healthy</a>
+    </div>
     {% if runs %}
     <form id="bulk-label-form" class="bulk-label" aria-label="Apply one label to the selected runs">
         <input id="bulk-label-input" name="label" maxlength="80" placeholder="Label, e.g. triaged" aria-label="Label to apply">
         <button class="button button-primary" type="submit">Apply to selected <span aria-hidden="true">&#8594;</span></button>
         <span id="bulk-label-status" class="form-status" role="status"></span>
     </form>
-    <div class="table-wrap"><table><thead><tr><th><input type="checkbox" id="select-all" aria-label="Select all runs"></th><th>Status</th><th>Agent</th><th>Run</th><th>Tool calls</th><th>Duration</th><th>Source</th><th>Actions</th></tr></thead><tbody>{% for run in runs %}<tr><td><input type="checkbox" class="run-check" value="{{ run.run_id }}" aria-label="Select {{ run.run_id }}"></td><td><span class="badge badge-{{ run.status }}">{{ run.status }}</span></td><td>{{ run.agent_name }}</td><td class="mono">{{ run.run_id }}</td><td>{{ run.tool_count }}</td><td>{{ run.duration_ms|round(1) }} ms</td><td class="mono">{{ run.source_dir or 'api' }}/{{ run.source_name }}</td><td class="table-actions"><a class="button button-quiet" href="/runs/{{ run.run_id }}">Review</a></td></tr>{% endfor %}</tbody></table></div>
+    <div class="table-wrap"><table><thead><tr><th><input type="checkbox" id="select-all" aria-label="Select all runs"></th><th>Status</th><th>Agent</th><th>Run</th><th>Evidence</th><th>Tool calls</th><th>Duration</th><th>Source</th><th>Actions</th></tr></thead><tbody>{% for run in runs %}<tr><td><input type="checkbox" class="run-check" value="{{ run.run_id }}" aria-label="Select {{ run.run_id }}"></td><td><span class="badge badge-{{ run.status }}">{{ run.status }}</span></td><td>{{ run.agent_name }}</td><td class="mono">{{ run.run_id }}</td><td>{% if run.error_summary %}<div class="review-evidence"><strong>{{ run.error_summary|length }} failed span{% if run.error_summary|length != 1 %}s{% endif %}</strong><span>{{ run.error_summary[0].name }} &middot; {{ run.error_summary[0].message }}</span>{% if run.error_summary|length > 1 %}<small>+ {{ run.error_summary|length - 1 }} more</small>{% endif %}</div>{% else %}<span class="review-clear">No failed spans</span>{% endif %}</td><td>{{ run.tool_count }}</td><td>{{ run.duration_ms|round(1) }} ms</td><td class="mono">{{ run.source_dir or 'api' }}/{{ run.source_name }}</td><td class="table-actions"><a class="button button-quiet" href="/runs/{{ run.run_id }}">Review</a></td></tr>{% endfor %}</tbody></table></div>
     {% else %}
     <div class="empty-state"><span class="empty-mark">&#10003;</span><h3>Nothing left to review</h3><p>Every run in the library has a label. Add one to a run to clear its review flag.</p></div>
     {% endif %}