Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,34 @@ Every entry reports millisecond measurement error and a confusion matrix. See `d
reported.

### Added
- **`hotato console` + `/calls`: the daily surface over scored production
calls.** `hotato console --production-db DB` is serve + the production
evidence database + the score-on-arrival worker in one command, one process,
one browser tab, opening on the new `/calls` view (every serve flag and
behavior -- loopback bind, bearer-token auth, append-only audit -- is
identical, and every `hotato serve` flag keeps working). `/calls` is a
browsable feed of the derived score sidecar joined read-only with the
session metadata: evidence time and arrival stamp (each labeled with its
clock), evidence-derived duration, session state, score state, the worst
dimension with the measured failure-reason sentence, per-call hop-latency
p50/p95 with the declared authorities behind it, and evidence-lane
completeness. `SCORED`, `NOT_SCORABLE` (with its reason), and `ERROR` are
first-class rows -- a refusal or crash is shown, never hidden and never
rendered as OK. Filters (session state, scorability, since/until window)
are query params; pagination is keyset from the first page (no OFFSET); a
trends strip reports volume, the score-state split, the candidate-moment
share, and per-kind hop-latency p50/p95 over the filtered window (kinds are
never pooled into one number). `/calls/<id>` renders one call: per-dimension
observations (never blended), the ranked candidate moments with their
measured magnitudes and plain-English timing sentences, the timing
waterfall with derived-vs-event-reported values kept apart, every hop row
with its declared authority, scorer version + config hash, the session's
evidence lanes, and the local audio path exactly as the evidence recorded
it. Both views have `?format=json` mirrors built from the same model; the
feed's mirror carries an ETag, answers a matching `If-None-Match` with a
body-less 304, and the page's small inline script polls it every few
seconds to show "updated Ns ago" and re-render rows only on change -- no
external code, and the page is complete as served with JavaScript off.
- **`hotato serve --score-production`: score-on-arrival for the production
evidence plane.** A background worker in the serve process polls the
evidence database (read-only, `mode=ro`) for sessions reaching
Expand Down
61 changes: 57 additions & 4 deletions docs/WORKSPACE.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# The team workspace: `hotato serve`
# The team workspace: `hotato serve` and `hotato console`

A self-hosted, local web app for a team to read a voice agent's
conversation-QA state: release readiness, the scenario matrix, a
conversation inspector, failure clusters, and production health.
conversation-QA state: the live call feed, release readiness, the scenario
matrix, a conversation inspector, failure clusters, and production health.
Stdlib-only (`http.server` + `sqlite3`) -- no framework, no build step,
nothing that phones home. It serves the same fleet registry and evidence
store the CLI writes, reading that data directly instead of passing a
Expand Down Expand Up @@ -45,13 +45,30 @@ redirects to strip the token from the address bar (see [Auth](#auth)).
Exit codes: `0` clean shutdown (Ctrl-C); `2` usage error (unusable registry or
token, or the port was unavailable).

## The five views
## `hotato console`: the call console in one command

```
hotato console --production-db .hotato/production.sqlite3
```

One command, one process, one browser tab: `serve` with the production
evidence database wired (read-only, `mode=ro`), the score-on-arrival worker
on, and the printed URL landing on the live call feed at `/calls`. Every
serve behavior is identical -- loopback bind by default, bearer-token auth on
every request, the append-only audit log, a `?format=json` mirror on every
view -- and `--workspace`, `--host`, `--port`, `--registry`, `--token`,
`--token-file`, and `--no-open` work exactly as they do on `serve`. The same
exit codes apply.

## The views

Every view has a machine mirror at `?format=json` (same auth, same data) so
agents and scripts can drive the workspace without scraping HTML.

| View | URL | Shows |
|---|---|---|
| **Calls** | `/calls` | The console feed over scored production calls (see [The call feed](#the-call-feed-calls)). |
| **One call** | `/calls/<id>` | One call's derived score record: per-dimension observations, ranked candidate moments, the timing waterfall, evidence lanes, and the audio path as recorded. |
| **Release readiness** | `/` | Pre-ship home screen: per-release rollup of suites/runs/evaluations -- required-suite completion, scenario/run counts, **failures by dimension** (outcome / policy / conversation / speech / reliability), inconclusive count, real-vs-simulated split, and **new-vs-fixed since the previous release**. Small samples flagged (`low sample, N=3`), never smoothed. |
| **Scenario matrix** | `/scenarios` | Rows are scenarios, columns are the current and previous release, with a per-dimension status and **reliability** (`pass^k` where a scenario has repetitions). Filterable by `agent`, `release`, `suite`, `status`. |
| **Conversation inspector** | `/conversation/<id>` | One conversation: evidence manifest, transcript, trace spans, per-dimension evaluations with rationale and citations (deterministic checks and model-judged/advisory results in **separate lanes**), reviewer decisions. Every digest links to the raw evidence (`/evidence/<digest>`); redacted transcript segments and trace spans render `[redacted]`, in both HTML and JSON. |
Expand Down Expand Up @@ -124,6 +141,42 @@ content (the one wall-clock column, `created_at`, is excluded from the
canonical comparison). A sidecar written by a different schema version is
refused with that rebuild instruction.

### The call feed (`/calls`)

`/calls` renders the score sidecar joined read-only with the evidence
database's session metadata, newest arrival first. Each row shows the
evidence-clock timestamp beside the arrival stamp (each labeled with its
clock), the evidence-derived duration, the session state, the score state,
the worst dimension with the measured failure-reason sentence, per-call
hop-latency p50/p95 with the declared authorities behind it, and
evidence-lane completeness (missing required lanes named). `SCORED`,
`NOT_SCORABLE` (with its reason), and `ERROR` are all first-class rows: a
refused or crashed session is shown, never hidden and never rendered as OK.

Filters are query params -- `state` (session state), `scorability`
(`SCORED`/`NOT_SCORABLE`/`ERROR`), and a `since`/`until` window (epoch
seconds or RFC3339); a malformed filter or cursor is a 400, never a silently
dropped filter. Pagination is keyset (`cursor`, `limit`) -- every page is one
bounded query, and the trends strip above the table reports call volume, the
score-state split, the candidate-moment share, and per-kind hop-latency
p50/p95 (nearest-rank) over the whole filtered window; hop kinds keep their
own percentiles and state their authorities.

The feed's JSON mirror carries an `ETag` and answers a matching
`If-None-Match` with a body-less 304. The page's small inline script polls
that mirror every few seconds, shows an "updated Ns ago" indicator, and
re-renders the rows only when the ETag changes -- no external code, and with
JavaScript off the page is complete as served (reload for the latest).

`/calls/<id>` is one call: per-dimension observations (candidate counts and
worst measured magnitude per scan kind, each on its own), the ranked
candidate moments with their measured magnitudes and plain-English timing
sentences, the timing waterfall (turn spans with derived durations kept apart
from event-reported values; every hop row with its timestamp, latency, and
declared authority), scorer version + config hash, the session's evidence
lanes, a link to the production evidence plane, and the local audio path
exactly as the evidence recorded it -- the recording stays on this machine.

## Auth

Every request is authenticated against one shared **bearer token**:
Expand Down
61 changes: 57 additions & 4 deletions llms-full.txt

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

69 changes: 69 additions & 0 deletions src/hotato/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,11 @@ def _atomic_write_json(path: str, obj) -> None:
(2, "usage error: an unusable registry/token, or the port was "
"unavailable"),
),
"console": (
(0, "the console served and shut down cleanly (Ctrl-C)"),
(2, "usage error: an unusable production database, registry, or "
"token, or the port was unavailable"),
),
"report": (
(0, "report written, every scorable event passed"),
(1, "a scorable event failed"),
Expand Down Expand Up @@ -5294,6 +5299,21 @@ def _cmd_serve(args) -> int:
)


def _cmd_console(args) -> int:
# console = serve with the production evidence db wired and the score-on-
# arrival worker on, landing on the live call feed. One command, one
# process, one browser tab; every serve behavior (bind, auth, audit, JSON
# mirrors) is identical.
from . import serve as _serve

return _serve.run_serve(
workspace=args.workspace, host=args.host, port=args.port,
registry=args.registry, token=args.token, token_file=args.token_file,
open_browser=not args.no_open, production_db=args.production_db,
score_production=True, landing="/calls",
)


# The canonical core loop: one linear path, first touch to a CI gate. This is the
# single source of truth rendered identically everywhere a newcomer meets hotato:
# the GET STARTED block atop `hotato --help`, `hotato describe`'s core_loop, the
Expand Down Expand Up @@ -10350,6 +10370,55 @@ def _fleet_parser(parent, name, dotted, help_text):
"$HOTATO_NO_BROWSER is set)")
srv.set_defaults(func=_cmd_serve)

con = sub.add_parser(
"console",
help="the call console: serve + score-on-arrival over the production "
"evidence database, landing on the live call feed",
description=(
"One command, one process, one browser tab: run the workspace "
"server with the production evidence database wired (read-only, "
"mode=ro) and the score-on-arrival worker on, opening on the live "
"call feed at /calls. Completed sessions are scored one at a time "
"with the deterministic scorer into a console.sqlite3 sidecar "
"beside the evidence database; SCORED, NOT_SCORABLE (with its "
"reason), and ERROR are all first-class visible states. Every "
"serve behavior is identical: loopback bind by default, "
"bearer-token auth on every request, append-only audit, and a "
"?format=json mirror on every view."),
epilog=_exit_codes_epilog("console"),
formatter_class=argparse.RawDescriptionHelpFormatter,
)
con.add_argument(
"--production-db",
required=True,
metavar="PATH",
help=(
"the production evidence SQLite database to read (mode=ro) and "
"score; the console.sqlite3 sidecar of derived score records "
"lives beside it"
),
)
con.add_argument("--workspace", "-w", default="default", metavar="ID",
help="workspace id to serve (default 'default')")
con.add_argument("--host", default="127.0.0.1", metavar="HOST",
help="bind address (default 127.0.0.1; a non-loopback "
"host exposes the workspace to your network and "
"prints a warning)")
con.add_argument("--port", type=int, default=8321, metavar="PORT",
help="listen port (default 8321)")
con.add_argument("--registry", default=None, metavar="PATH",
help="registry home directory (default ~/.hotato/fleet)")
con.add_argument("--token", default=None, metavar="TOKEN",
help="bearer token (default: reuse the stored one, else "
"generate + store 0600 under the workspace state dir)")
con.add_argument("--token-file", default=None, metavar="PATH",
help="read the bearer token from PATH instead of --token")
con.add_argument("--no-open", action="store_true",
help="do not open a browser on start (auto-open is also "
"skipped when stdout is not a TTY, in CI, or when "
"$HOTATO_NO_BROWSER is set)")
con.set_defaults(func=_cmd_console)

# Operational families live in a standalone registrar so this integration
# remains a one-line semantic merge as the main CLI evolves.
from . import ops_cli as _ops_cli
Expand Down
Loading
Loading