hotato 1.14.0: the local-first AI engineering platform#33
Merged
Conversation
… (C2 crash, M1 OOM) QA found two adversarial inputs on the transcript path: a negative timestamp crashed the scorer with a leaked traceback (indexed before frame 0), and a mistyped huge end (50000s) built an unbounded per-hop timeline and exhausted memory. Both now refuse up front with a clean usage error (exit 2). Regression tests added.
hotato turn-taking eval8 of 8 scenarios pass. 0 fail. No regression.
RegressionsNone. Reproducible timing measured locally from call audio. Swap the bundled self-test step for your own captured recordings to gate on your agent. github.com/attenlabs/hotato |
… init prints a guarded command (C1/M2/M3) Three defects the adversarial QA found on primary paths: - C1 (CRITICAL): `run --stereo` fabricated a timing FAIL on byte-identical channels (comparing a channel to itself). It now routes through the same same-signal NOT SCORABLE refusal (exit 2) that trust/investigate use, before any number is emitted. Eligible recordings stay byte-identical. - M2 (MAJOR): `investigate label --expect yield` on a talk-over catch created a contract with no talk-over bound, so the golden path went GREEN, contradicting the docs' red gate. A talk-over over the 1.0s prompt-yield ceiling now auto-pins max_talk_over_sec, so the caught magnitude gates red (exit 1) as the docs promise; prompt yields stay green; an explicit --max-talk-over wins. - M3 (MAJOR): `init --auto` printed a verify next-command that errored on its own empty scaffold; it now prints the same shell-guarded no-op the generated CI job uses. Regression tests added; targeted suites green; engine byte-identical. Claude-Session: https://claude.ai/code/session_019fxCGvT5gqHCoevCiv3R4u
…44 lines) Mirrors the rebuilt homepage: pain-led hero (your voice agent is failing real calls, your tests say it's fine), a scannable four-plane capability table (Catch/Test/Gate/Observe), a tight quickstart and CI block, and condensed onramp/specs. Voice-first (dropped chat). Regenerated PyPI readme + llms-full.
Fixes the adversarial-QA defects on primary paths and ships an audible, honestly-labeled demo, with every derived surface reconciled to the measured values. QA (5): - C1: `run --stereo` no longer fabricates a FAIL on byte-identical channels; it routes through the same NOT SCORABLE refusal (exit 2) before any number. - C2/M1: the transcript scorer refuses a negative timestamp (was a crash) and a mistyped huge `end` (was an unbounded allocation) with a clean usage error. - M2: `investigate label --expect yield` on a talk-over catch auto-pins max_talk_over_sec so the contract gates red (exit 1) as the docs promise. - M3: `init --auto` prints a shell-guarded verify command that works on its own empty scaffold. Demo audio + honesty: - The fd-01 missed-interruption clip is now an audible, deterministically- synthesized barge-in (caller onset 2.0s, measured talk-over 2.65s, CLI display 2.66s), replacing an inaudible clip. The scenario is labeled a synthesized fixture; the demo report, SDK fixtures, neural-ONNX golden, and docs report the same measured numbers. - The mixed demo battery (synthesized fd-01 + recorded fd-02) is described honestly: dropped the "two REAL recorded calls" framing and the decorative "honest per-stack map". Copy: - README streamlined to the voice-first identity; PyPI long-description, llms.txt / llms-full.txt, atlas records, and TRUST-GALLERY regenerated. Version bumped 1.13.0 -> 1.13.1 across pyproject / __init__ / server.json / CITATION / README pin. Claude-Session: https://claude.ai/code/session_01BqJW5Dey1DzqBeBCAJXrH9
…m (WAVE 1) Repositions the top-level identity from "regression testing for voice agents" to "the local-first AI engineering platform" -- trace, evaluate, test, and gate any AI agent on your machine. The four planes map to capabilities SHIPPED in 1.13.0, so every claim is true of this wheel: - Observe: observe capture/cost/percentiles/report + trace ingest (OTel). - Evaluate: assert (deterministic assertions) + a separated local-judge lane. - Test: simulate + the gauntlet turn-taking stress suite. - Gate: content-addressed contracts (verify is the gate) + baseline + pr. The turn-taking wedge stays the on-ramp and leads the demo (the shareable contradiction: the text eval passed the call, the timing did not). Nothing is claimed that the package does not do; the deterministic core stays separate from the model-judge lane. Surfaces flipped coherently: README (new hero + four planes + local-first privacy plane), README.pypi + llms.txt + llms-full (regenerated), AGENTS.md, pyproject description (the PyPI Summary), server.json, CITATION.cff, the CLI top-level description, the bare-hotato first-run screen, and the package docstring. copy_lint clean; targeted suites green. Footer flavor corrected: "Catch it while it's hot" -> "Know when to pass it on" (the hot-potato metaphor is about knowing when to pass vs hold, not catching and holding), kept subtle. Claude-Session: https://claude.ai/code/session_01BqJW5Dey1DzqBeBCAJXrH9
The platform identity was right but the upside read defensively. Leads now with what you get and why you would switch off a hosted platform: - README hero: "Everything you reach for a hosted platform to do: trace, evaluate, test, and gate your LLM and voice agents, on your own machine. Free at any scale. Byte-reproducible. Nothing leaves it." - New "Why it is different" table: same four jobs; free vs metered per seat and per event; byte-for-byte reproducible vs varies run to run; local vs on their servers; CI-native vs needs their service. No competitor is named. - pyproject description (PyPI Summary) and llms.txt lead reframed to the same head-to-head. No em-dashes; no competitor names in product copy. Claude-Session: https://claude.ai/code/session_01BqJW5Dey1DzqBeBCAJXrH9
Six community-requested fixtures and checks, each built on the existing corpus and say-do infra, deterministic (seed = sha256(id)) and self-validating. - #11 leading-edge-onset: a dropped-leading-audio class where the caller onset is a short leading burst at the interruption boundary, so a pipeline that clips it is measurable against the ground-truth reference. - #12 structured-utterance: digit-string and email cadences where an intra-item pause must not be read as turn end (extends the mid-utterance-pause class). - #14 browser-telephony-parity: one scenario, two renders (a clean browser leg and a silence-gapped telephony leg), so "passes in the browser, fails on the phone" is a deterministic parity assertion. - #13 interrupted-tool-call: a say-do fixture bundle (clean-cancel, double-fire, zombie) where an interruption mid tool-call double-fires or discards a committed side effect; the say-do check catches the log-vs-evidence gap. - #15 dtmf_conformance: a pure-stdlib Goertzel detector that confirms claimed DTMF digits are audibly present in the delivered audio at the claimed time, extending the evidence-over-logs stance to DTMF. - #16 RFC-ROLEPLAY-FIXTURES.md: the share-safe role-play fixture recipe (script shape, consent/PII/attestation mapped to the schema, dual-channel capture, the validate.py PASS gate), linked from CONTRIBUTING and docs/SUBMITTING. Regenerated PyPI readme + llms-full. All new tests green; corpus audio renders deterministically from build_classes.py (gitignored, like the existing classes). Closes #11 Closes #12 Closes #13 Closes #14 Closes #15 Closes #16 Claude-Session: https://claude.ai/code/session_01BqJW5Dey1DzqBeBCAJXrH9
The six new corpus fixtures and the dtmf_conformance check add backwards-compatible functionality, so the aggregate of the QA fixes, the audible demo, the platform reposition, and the six issues ships as a minor bump (1.13.0 -> 1.14.0). Version bumped across pyproject / __init__ / server.json / CITATION / README pin / llms.txt / docs (CI, TRUST-GALLERY); atlas records, PyPI readme, and llms-full regenerated. CHANGELOG 1.14.0 entry consolidates the release. Claude-Session: https://claude.ai/code/session_01BqJW5Dey1DzqBeBCAJXrH9
Tightens the repo to tip-top shape alongside the reposition: - docs/README.md: a new index over all ~80 docs, grouped by the four planes (Observe / Evaluate / Test / Gate) plus Getting started, Connect your stack, Self-host, Reference, and Contributing. GitHub renders it as the docs/ landing page. Internal SAA docs are intentionally excluded. - .gitignore: the internal SAA artifacts (behavior card, fix pointer, and their script and test) are now excluded by the committed .gitignore, not only by a local .git/info/exclude, so a fresh clone or a forced add can never commit them to the public repo. - Sub-READMEs and top-level docs reframed to the platform identity where they still led with "conversation QA for voice agents"; copy-law cleanups (em-dashes removed, authenticity-protest and absence phrasing dropped) in CONTRIBUTING, AGENTS, METHODOLOGY, the TypeScript SDK, the control plane, the reference agent, the demo, and the vapi-defaults READMEs. No broken internal links (audited). No tracked cruft removed: the 11 MiB sample report and the root saydo/ fixture are both referenced, so they stay. Claude-Session: https://claude.ai/code/session_01BqJW5Dey1DzqBeBCAJXrH9
docs/README.md was untracked when the cleanup commit ran build_llms_full, so its FILE boundary header was missing. Now that it is committed, the regenerated llms-full.txt and README.pypi.md include it. Restores the doc-sync invariant. Claude-Session: https://claude.ai/code/session_01BqJW5Dey1DzqBeBCAJXrH9
quantumCF
changed the title
CI runs `ruff check src tests`; the new dtmf_conformance module imported an unused `Optional`, and its test imported an unused `check_dtmf_conformance`. Both removed; `ruff check src tests` is clean. Out-of-scope pre-existing lint in adapters/scripts is untouched (CI does not lint those paths). Claude-Session: https://claude.ai/code/session_01BqJW5Dey1DzqBeBCAJXrH9
Three of the seven corpus scenario classes were added this release; the module docstring, an inline comment, and the argparse description still said 'four'. Rendered output is byte-identical (docstring-only change). Claude-Session: https://claude.ai/code/session_01BqJW5Dey1DzqBeBCAJXrH9
The interrupted-tool-call fixture (#13) ships a voice_trace.jsonl per variant, but MANIFEST.in's examples include listed *.json *.py *.md *.sh *.yaml and not *.jsonl, so the traces were missing from the sdist and the say-do test failed when run from the extracted sdist tree (sdist-guard). Verified: the 3 traces are now packaged and all six #13 tests pass from the extracted sdist tree. Claude-Session: https://claude.ai/code/session_01BqJW5Dey1DzqBeBCAJXrH9
The broad `recursive-include docs *` swept the gitignored internal SAA files (SAA-BEHAVIOR-CARD.md, SAA-FIX-POINTER.md, saa_behavior_card.py, test_saa_card.py) into the sdist -- MANIFEST is independent of git, so .gitignore does not keep them out of a built sdist. That would have shipped crown-jewel files to PyPI and also broke the llms-full doc-boundary test in the extracted sdist tree (the filesystem glob found docs the git-generated llms-full has no header for). Explicit + defensive MANIFEST excludes now keep them out. Verified: no untracked file leaks into the sdist, and the full suite passes from the extracted tree. Claude-Session: https://claude.ai/code/session_01BqJW5Dey1DzqBeBCAJXrH9
Hardens the 2026-07-22 packaging-leak law into three enforced layers so a gitignored internal/crown-jewel file can never reach PyPI again: - scripts/check_dist_no_leak.py: scans a built sdist/wheel and REFUSES (exit 1) on any untracked member (not git-tracked, not build-generated) or any internal/secret pattern (SAA*, HANDOFF*, .env secrets, credentials, *.pem, .claude/, north-star, holocron). Stdlib-only, reads listings only. - Wired fail-closed into the sdist-guard CI job (scans the built sdist) and the publish workflow (scans the distribution before attest/upload; a hit aborts the whole publish). - tests/test_dist_no_leak.py pins the scanner logic, the MANIFEST SAA excludes, and the CI + publish wiring. Complements the MANIFEST excludes (c4aafb5) and the committed .gitignore. Claude-Session: https://claude.ai/code/session_01BqJW5Dey1DzqBeBCAJXrH9
test_scaffold_auto_gate_next_command_runs_clean_on_fresh_scaffold runs the printed CI-gate command via subprocess shell=True. That command is an `if ... fi` POSIX guard for the generated GitHub Actions bash step; on Windows shell=True uses cmd.exe, which cannot parse it (portability (windows-latest) failure). The command is never meant to run on cmd.exe, so skip the execution test there; its exit-0 behaviour stays covered on POSIX shells. Claude-Session: https://claude.ai/code/session_01BqJW5Dey1DzqBeBCAJXrH9
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
hotato 1.14.0 — the local-first AI engineering platform
Repositions hotato from "regression testing for voice agents" to the
local-first AI engineering platform (trace, evaluate, test, and gate any AI
agent on your machine), sharpens the value proposition into a direct head-to-head
with hosted platforms, ships the six community-requested fixtures/checks, and
tightens the repo to release shape. Everything is true of the shipped wheel; the
turn-taking wedge stays the on-ramp and leads the demo.
Reposition + upside
llms,AGENTS.md,the CLI description, the first-run screen, and metadata.
byte-reproducible enough to gate CI, and local. A "Why it is different"
head-to-head table (no competitor named; "hosted platforms" is the category).
fd-01demo is an audible, synthesized barge-in (talk-over 2.66sCLI / 2.65s verdict), labeled as a synthesized fixture.
QA fixes
run --stereorefuses byte-identical channels (NOT SCORABLE, exit 2).investigate label --expect yieldon a talk-over catch gates red.init --autoprints a verify command that works on its own scaffold.Six issues
Closes #11 Closes #12 Closes #13 Closes #14 Closes #15 Closes #16
leading-edge-onset,structured-utterance,browser-telephony-parity; a say-dointerrupted-tool-callbundle; apure-stdlib
dtmf_conformancecheck; and the role-play fixture RFC.Repo cleanup
docs/README.mdindex over all ~80 docs, grouped by the four planes..gitignore(were only in alocal exclude), so they can never leak to the public repo.
Full gate green (4600+ tests). Version bumped 1.13.0 -> 1.14.0 in lockstep
across every surface; derived docs regenerated.
https://claude.ai/code/session_01BqJW5Dey1DzqBeBCAJXrH9