Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,24 @@ Every entry reports millisecond measurement error and a confusion matrix. See `d
reported.

### Added
- **`hotato serve --score-production`: score-on-arrival for the production
evidence plane.** A background worker in the serve process polls the
evidence database (read-only, `mode=ro`) for sessions reaching
`COMPLETE`/`QUIESCENT` and scores each one -- one at a time -- with the
deterministic scorer over the session's recorded two-channel audio, writing
durable records to a `console.sqlite3` sidecar beside the evidence
database: per-dimension observations (never blended), ranked candidate
moments, one measured failure-reason sentence, and timing derived from
evidence event timestamps (per-hop latency keeps the reporting event's
declared authority; turn spans and end-to-end are labeled
`derived:event_timestamps`). Refusal is a first-class `NOT_SCORABLE` record
with its reason; a scorer crash or persist failure on one session becomes a
visible `ERROR` record -- never a silent skip, never a claimed score before
the sidecar write commits -- and the worker continues to the next session.
The sidecar is derived data with a versioned schema: `--rebuild-scores`
regenerates it entirely from the evidence database and exits, and the same
evidence database always rebuilds to identical content. Bind, auth, and the
server's read-only route surface are unchanged.
- **`--junit` on `hotato suite run` and `hotato prove`.** Both commands also
write a JUnit XML report any standard CI test widget ingests, mirroring
each command's existing grouping: one `<testsuite>` per dimension (suite
Expand Down
43 changes: 43 additions & 0 deletions docs/WORKSPACE.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,8 @@ redirects to strip the token from the address bar (see [Auth](#auth)).
| `--port` | `8321` | listen port |
| `--registry` | `~/.hotato/fleet` | registry home directory |
| `--production-db` | none | read session manifests and alerts from this separate Hotato production SQLite database in `/health` (mode=ro; see below) |
| `--score-production` | off | with `--production-db`: score completed sessions in the background into a `console.sqlite3` sidecar beside the evidence database (see below) |
| `--rebuild-scores` | off | with `--production-db`: deterministically regenerate the entire `console.sqlite3` sidecar from the evidence database, then exit |
| `--token` | none | supply the bearer token yourself |
| `--token-file` | none | read the bearer token from a file (first line) |

Expand Down Expand Up @@ -81,6 +83,47 @@ carry a fleet workspace id, so the UI states `workspace_scope =
not_encoded_by_production_schema` instead of silently assigning those sessions
to the workspace being served.

### Score-on-arrival (`--score-production`)

```bash
hotato serve --workspace default \
--production-db .hotato/production.sqlite3 --score-production
```

A background worker in the same process polls the evidence database (same
`mode=ro` read-only discipline) for sessions that reached
`COMPLETE`/`QUIESCENT` and scores each one with the deterministic scorer over
the session's recorded two-channel audio (the path named by the
`media.asset.available` event's `data.path`). Bind and auth are unchanged;
the server gains no new routes or write endpoints from this flag.

Each session becomes one durable record in `console.sqlite3` beside the
evidence database:

- **`SCORED`** -- per-dimension observations (candidate counts and worst
measured magnitude per scan kind, never blended), the ranked candidate
moments, and one plain-English failure-reason sentence built only from
measured numbers;
- **`NOT_SCORABLE`** -- the scorer's refusal with its reason (audio lane
unavailable, no recorded path, a one-channel or unreadable recording);
- **`ERROR`** -- a scorer crash or persist failure on that session, with its
reason; the worker records it and continues to the next session.

Every record carries the scorer version and a config hash, and every timing
figure derives from evidence event timestamps: per-hop latency rows keep the
reporting event's declared `authority`, turn spans and the end-to-end figure
are labeled `derived:event_timestamps`, and reported turn fields
(`yield_latency_ms`, `overlap_ms`, `duration_ms`) stay in a separate
`reported` block. Sessions are scored one at a time and a record is claimed
only after its sidecar write commits.

The sidecar is derived data -- the evidence database stays the only
authority. `--rebuild-scores` regenerates the whole sidecar from the evidence
database and exits; the same evidence database always rebuilds to identical
content (the one wall-clock column, `created_at`, is excluded from the
canonical comparison). A sidecar written by a different schema version is
refused with that rebuild instruction.

## Auth

Every request is authenticated against one shared **bearer token**:
Expand Down
43 changes: 43 additions & 0 deletions llms-full.txt

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

38 changes: 38 additions & 0 deletions src/hotato/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -5268,12 +5268,29 @@ def _cmd_start(args) -> int:


def _cmd_serve(args) -> int:
if args.rebuild_scores:
# One-shot: deterministically regenerate the console sidecar from the
# evidence database and exit; no server is started.
if not args.production_db:
raise ValueError(
"--rebuild-scores needs --production-db (the evidence database "
"the sidecar is derived from)"
)
from . import console_worker as _console

result = _console.run_rebuild(args.production_db)
print("hotato serve --rebuild-scores: regenerated %s" % result["sidecar"])
print(" scored: %d not scorable: %d errors: %d"
% (result["scored"], result["not_scorable"], result["errors"]))
return 0

from . import serve as _serve # lazy: the workspace server + its stdlib deps

return _serve.run_serve(
workspace=args.workspace, host=args.host, port=args.port,
registry=args.registry, token=args.token, token_file=args.token_file,
open_browser=not args.no_open, production_db=args.production_db,
score_production=args.score_production,
)


Expand Down Expand Up @@ -10301,6 +10318,27 @@ def _fleet_parser(parent, name, dotted, help_text):
"database into /health; opened read-only without payload access"
),
)
srv.add_argument(
"--score-production",
action="store_true",
help=(
"with --production-db: run the score-on-arrival worker alongside "
"the server -- completed sessions are scored one at a time with "
"the deterministic scorer and recorded (SCORED / NOT_SCORABLE "
"with reason / ERROR) in a console.sqlite3 sidecar beside the "
"evidence database"
),
)
srv.add_argument(
"--rebuild-scores",
action="store_true",
help=(
"with --production-db: deterministically regenerate the entire "
"console.sqlite3 sidecar from the evidence database, then exit "
"(the sidecar is derived data; the same evidence database always "
"rebuilds to identical content)"
),
)
srv.add_argument("--token", default=None, metavar="TOKEN",
help="bearer token (default: reuse the stored one, else "
"generate + store 0600 under the workspace state dir)")
Expand Down
Loading
Loading