From 8573f14ccb113dbe89d13afc3f2803371e3831fe Mon Sep 17 00:00:00 2001 From: "David J. Kim" Date: Wed, 22 Jul 2026 13:30:40 -0400 Subject: [PATCH 1/3] release: 1.15.1 -- calibrate the proof claim, correct three overclaims Acts on an external audit's claim-integrity findings against the 1.15.0 surfaces. No new capability; this makes the claims match the evidence. - hotato prove now states its CLAIM SCOPE and EVIDENCE AUTHORITY. A contracts-only run reads "Captured Evidence", a suite/gauntlet "Test Suite", and a before/after run reaches "Candidate Revision" (or "Deployed Revision") ONLY when the caller binds the candidate identity (--candidate-config-hash, --provider, --deployment-id). It can never claim more than the lanes support, so "release proof" is no longer applied to a bare contracts re-verification. proof.v1 gains claim_scope + evidence_authority; the renderer headlines the scope. Adversarially verified; the anti-over-elevation cases are pinned. - Package description drops "everything you use a hosted platform for" (untrue of the public package: no dataset/prompt management, hosted execution, or team collaboration). - The product one-liner is now a capability, not an automatic guarantee: "Turn production failures into portable tests, run candidate releases against them, and carry evidence with every release." - docs/COMPARE.md replaces the caricature of hosted platforms with a property-by-property table that is honest about where a managed platform (team collaboration, trace search, dataset/prompt management, hosted execution) is the stronger fit; the "vary run to run" claim is scoped to the model-judge lane. Version 1.15.1 in lockstep across every surface; derived regenerated. Claude-Session: https://claude.ai/code/session_01BqJW5Dey1DzqBeBCAJXrH9 --- CITATION.cff | 2 +- README.md | 27 ++- README.pypi.md | 27 ++- .../addressed-backchannel-yielded.json | 12 +- .../addressed-interruption-missed.json | 12 +- docs/CI.md | 10 +- docs/COMPARE.md | 48 +++-- docs/LIFECYCLE.md | 4 +- docs/TRUST-GALLERY.md | 2 +- llms-full.txt | 95 ++++++---- llms.txt | 2 +- pyproject.toml | 4 +- server.json | 4 +- src/hotato/__init__.py | 2 +- src/hotato/cli.py | 33 +++- src/hotato/prove.py | 176 +++++++++++++++++- tests/test_prove.py | 127 +++++++++++++ 17 files changed, 470 insertions(+), 117 deletions(-) diff --git a/CITATION.cff b/CITATION.cff index 03a0d11..3ac6b1f 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -15,7 +15,7 @@ type: software authors: - name: "Attention Labs, Inc." license: MIT -version: 1.15.0 +version: 1.15.1 date-released: "2026-07-21" repository-code: "https://github.com/attenlabs/hotato" url: "https://hotato.dev" diff --git a/README.md b/README.md index d34164b..a4c1fc7 100644 --- a/README.md +++ b/README.md @@ -32,7 +32,7 @@ That transcript passed every text eval. The timing did not. hotato pins the catc ## The loop -Every production failure becomes a portable test, every candidate runs against it, and every release carries evidence. Five steps, one command each, nothing leaves your machine. +Turn production failures into portable tests, run candidate releases against them, and carry evidence with every release. Five steps, one command each, nothing leaves your machine. | | | | | :-- | :-- | :-- | @@ -40,7 +40,7 @@ Every production failure becomes a portable test, every candidate runs against i | **Catch** | deterministic scoring finds what text evals miss: timing, say-do, policy | `hotato investigate call.wav` | | **Pin** | your label turns the catch into a portable, content-addressed contract | `hotato investigate label STATE#1 --expect yield` | | **Test** | simulate, stress, and drive candidates against everything you pinned | `hotato gauntlet` | -| **Prove** | every evidence lane composed into one fail-closed release proof, in CI | `hotato prove --contracts contracts/` | +| **Prove** | every evidence lane composed into one fail-closed, content-addressed proof, in CI | `hotato prove --contracts contracts/` | Deterministic. Byte-reproducible. Free, MIT. Agent-native over MCP. Every verdict carries its evidence across five dimensions (outcome, policy, conversation, speech, reliability), and production feeds the next loop: `hotato production export-regression` turns a live session back into a test. @@ -54,15 +54,15 @@ Same loop a hosted platform runs. Three things it cannot offer. | :-- | :-- | :-- | | Observe, catch, pin, test, prove | yes | yes | | Price at scale | free, MIT, any volume | metered per seat and per event | -| Verdicts | byte-for-byte reproducible, gate a build | vary run to run | +| Verdicts | byte-for-byte reproducible, gate a build | model-judge lanes vary run to run | | Your traces and prompts | stay on your machine | live on their servers | | Runs in CI, offline | yes | needs their service | Full comparison: [`docs/COMPARE.md`](docs/COMPARE.md) -## From a bad call to a release proof +## From a bad call to a proof -One recording in. The pinned failure becomes a gate that stays red until the agent stops failing that call, and every gate you have composes into one receipt: +One recording in. The pinned failure becomes a gate that stays red until the agent stops failing that call, and every gate you have composes into one receipt whose headline states exactly what it establishes: ```console $ hotato investigate ./call.wav @@ -73,14 +73,13 @@ $ hotato investigate label '.hotato/investigate-state.json#1' --expect yield created hotato contract: call-8s-yield $ hotato prove --contracts contracts/ -hotato prove: proof -- overall FAIL (exit 1) - lane verdict counts - contracts fail contracts=1 passed=0 failed=1 tampered=0 refused=0 -content_id: sha256:2959c905ee6c1030... -proof: .hotato/proofs/proof/proof.json +Captured Evidence: FAIL + A lane failed or regressed; see the table. + contracts fail contracts=1 passed=0 failed=1 +content_id: sha256:c907143d711a84ae... ``` -A contract re-measures the captured failure under the pinned policy on every CI run, the same discipline a snapshot test gives you. `hotato prove` composes every lane you have (contracts, suites, before/after batteries, the stress suite) into one fail-closed verdict with a content-addressed receipt: pass only when every lane passed, and "could not tell" is never green. +`hotato prove` composes every lane you have (contracts, suites, before/after batteries, the stress suite) into one fail-closed, content-addressed receipt: pass only when every lane passed, and "could not tell" is never green. The headline is the **claim scope** the evidence supports, never more: contracts alone re-measure stored evidence (**Captured Evidence**), a suite establishes a **Test Suite** ran, and a before/after run reaches **Candidate Revision** only when you bind the candidate identity (`--candidate-config-hash`, `--provider`). ## Quickstart @@ -91,7 +90,7 @@ uvx hotato start --demo hotato investigate ./call.wav # 3. pin the caught moment as a regression contract hotato investigate label '.hotato/investigate-state.json#1' --expect yield -# 4. compose every gate into one release proof, on every pull request +# 4. compose every gate into one proof, scoped to what it establishes hotato prove --contracts contracts/ ``` @@ -109,10 +108,10 @@ jobs: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - - uses: attenlabs/hotato@v1.15.0 + - uses: attenlabs/hotato@v1.15.1 with: contracts: contracts/ - hotato-version: 1.15.0 + hotato-version: 1.15.1 ``` Copy-paste workflow with a commit-SHA pin: [`docs/CI.md`](docs/CI.md). diff --git a/README.pypi.md b/README.pypi.md index 336627e..3245b8e 100644 --- a/README.pypi.md +++ b/README.pypi.md @@ -32,7 +32,7 @@ That transcript passed every text eval. The timing did not. hotato pins the catc ## The loop -Every production failure becomes a portable test, every candidate runs against it, and every release carries evidence. Five steps, one command each, nothing leaves your machine. +Turn production failures into portable tests, run candidate releases against them, and carry evidence with every release. Five steps, one command each, nothing leaves your machine. | | | | | :-- | :-- | :-- | @@ -40,7 +40,7 @@ Every production failure becomes a portable test, every candidate runs against i | **Catch** | deterministic scoring finds what text evals miss: timing, say-do, policy | `hotato investigate call.wav` | | **Pin** | your label turns the catch into a portable, content-addressed contract | `hotato investigate label STATE#1 --expect yield` | | **Test** | simulate, stress, and drive candidates against everything you pinned | `hotato gauntlet` | -| **Prove** | every evidence lane composed into one fail-closed release proof, in CI | `hotato prove --contracts contracts/` | +| **Prove** | every evidence lane composed into one fail-closed, content-addressed proof, in CI | `hotato prove --contracts contracts/` | Deterministic. Byte-reproducible. Free, MIT. Agent-native over MCP. Every verdict carries its evidence across five dimensions (outcome, policy, conversation, speech, reliability), and production feeds the next loop: `hotato production export-regression` turns a live session back into a test. @@ -54,15 +54,15 @@ Same loop a hosted platform runs. Three things it cannot offer. | :-- | :-- | :-- | | Observe, catch, pin, test, prove | yes | yes | | Price at scale | free, MIT, any volume | metered per seat and per event | -| Verdicts | byte-for-byte reproducible, gate a build | vary run to run | +| Verdicts | byte-for-byte reproducible, gate a build | model-judge lanes vary run to run | | Your traces and prompts | stay on your machine | live on their servers | | Runs in CI, offline | yes | needs their service | Full comparison: [`docs/COMPARE.md`](https://github.com/attenlabs/hotato/blob/main/docs/COMPARE.md) -## From a bad call to a release proof +## From a bad call to a proof -One recording in. The pinned failure becomes a gate that stays red until the agent stops failing that call, and every gate you have composes into one receipt: +One recording in. The pinned failure becomes a gate that stays red until the agent stops failing that call, and every gate you have composes into one receipt whose headline states exactly what it establishes: ```console $ hotato investigate ./call.wav @@ -73,14 +73,13 @@ $ hotato investigate label '.hotato/investigate-state.json#1' --expect yield created hotato contract: call-8s-yield $ hotato prove --contracts contracts/ -hotato prove: proof -- overall FAIL (exit 1) - lane verdict counts - contracts fail contracts=1 passed=0 failed=1 tampered=0 refused=0 -content_id: sha256:2959c905ee6c1030... -proof: .hotato/proofs/proof/proof.json +Captured Evidence: FAIL + A lane failed or regressed; see the table. + contracts fail contracts=1 passed=0 failed=1 +content_id: sha256:c907143d711a84ae... ``` -A contract re-measures the captured failure under the pinned policy on every CI run, the same discipline a snapshot test gives you. `hotato prove` composes every lane you have (contracts, suites, before/after batteries, the stress suite) into one fail-closed verdict with a content-addressed receipt: pass only when every lane passed, and "could not tell" is never green. +`hotato prove` composes every lane you have (contracts, suites, before/after batteries, the stress suite) into one fail-closed, content-addressed receipt: pass only when every lane passed, and "could not tell" is never green. The headline is the **claim scope** the evidence supports, never more: contracts alone re-measure stored evidence (**Captured Evidence**), a suite establishes a **Test Suite** ran, and a before/after run reaches **Candidate Revision** only when you bind the candidate identity (`--candidate-config-hash`, `--provider`). ## Quickstart @@ -91,7 +90,7 @@ uvx hotato start --demo hotato investigate ./call.wav # 3. pin the caught moment as a regression contract hotato investigate label '.hotato/investigate-state.json#1' --expect yield -# 4. compose every gate into one release proof, on every pull request +# 4. compose every gate into one proof, scoped to what it establishes hotato prove --contracts contracts/ ``` @@ -109,10 +108,10 @@ jobs: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - - uses: attenlabs/hotato@v1.15.0 + - uses: attenlabs/hotato@v1.15.1 with: contracts: contracts/ - hotato-version: 1.15.0 + hotato-version: 1.15.1 ``` Copy-paste workflow with a commit-SHA pin: [`docs/CI.md`](https://github.com/attenlabs/hotato/blob/main/docs/CI.md). diff --git a/atlas/records/addressed-backchannel-yielded.json b/atlas/records/addressed-backchannel-yielded.json index 4a5cc57..de3084e 100644 --- a/atlas/records/addressed-backchannel-yielded.json +++ b/atlas/records/addressed-backchannel-yielded.json @@ -164,7 +164,7 @@ "provenance": { "hotato": { "name": "hotato", - "version": "1.15.0" + "version": "1.15.1" }, "inputs": [ { @@ -196,7 +196,7 @@ ], "source_result_digest": "sha256:7a691d5993389b47f2b43b7afbe7316bbb0d662e703fa0f8a82e39b93b052c08" }, - "record_id": "sha256:dd3c63fcaeff31c7af4cecce0612aa524e554f7ae40d0c362f2d5ddef8491472", + "record_id": "sha256:70f0fc141627f293d1ca134f307a0560f41103f43d5ef967b7d27f40c060cb4f", "reproduction": { "argv": [ "hotato", @@ -222,7 +222,7 @@ "version": "1.0" }, "reproduction_metadata": { - "hotato_version": "1.15.0", + "hotato_version": "1.15.1", "reviewer_principal": "hotato-examples", "working_directory": ".", "bundle": { @@ -240,7 +240,7 @@ "verify_output": "verify.json", "selector": "fd-02-backchannel-yielded", "record_out": "record", - "expected_record_id": "sha256:dd3c63fcaeff31c7af4cecce0612aa524e554f7ae40d0c362f2d5ddef8491472" + "expected_record_id": "sha256:70f0fc141627f293d1ca134f307a0560f41103f43d5ef967b7d27f40c060cb4f" }, "supersedes": { "record_id": "sha256:6c5e0e6de73004d8683c0987812d849c46fa9f9f8f204eb4da71e600b636e3a0", @@ -268,8 +268,8 @@ }, { "command": "hotato record render verify.json#fd-02-backchannel-yielded --out record", - "output": "wrote failure record (FAIL, fd-02-backchannel-yielded) to record/: failure-record.html, failure-record.json, failure-record.md, failure-record.svg\nrecord_id: sha256:dd3c63fcaeff31c7af4cecce0612aa524e554f7ae40d0c362f2d5ddef8491472\n" + "output": "wrote failure record (FAIL, fd-02-backchannel-yielded) to record/: failure-record.html, failure-record.json, failure-record.md, failure-record.svg\nrecord_id: sha256:70f0fc141627f293d1ca134f307a0560f41103f43d5ef967b7d27f40c060cb4f\n" } ], - "content_digest": "sha256:dc05c98c555099225b575ebc5de7f8f95f9c94bcfd6e01345cf3f02f92c258ec" + "content_digest": "sha256:6efe74abf4d49ceb0d51f8673e174510eff437a89958cbdf90901049e3b36980" } diff --git a/atlas/records/addressed-interruption-missed.json b/atlas/records/addressed-interruption-missed.json index a7e5fd1..9f96964 100644 --- a/atlas/records/addressed-interruption-missed.json +++ b/atlas/records/addressed-interruption-missed.json @@ -164,7 +164,7 @@ "provenance": { "hotato": { "name": "hotato", - "version": "1.15.0" + "version": "1.15.1" }, "inputs": [ { @@ -196,7 +196,7 @@ ], "source_result_digest": "sha256:d3ec5aa5bbbdac05ef1ee7ad6d2612afd3916504e313e16df56ea3c334cb1346" }, - "record_id": "sha256:ad9c62d49f76f62c11c25184a13cd63669e5e6fee9f1dfc5f8093fb35dbe6eaa", + "record_id": "sha256:25f01ea209620cbd36b6ec662dfce124eb989f53f303be808c65f2f6d90ed674", "reproduction": { "argv": [ "hotato", @@ -222,7 +222,7 @@ "version": "1.0" }, "reproduction_metadata": { - "hotato_version": "1.15.0", + "hotato_version": "1.15.1", "reviewer_principal": "hotato-examples", "working_directory": ".", "bundle": { @@ -242,7 +242,7 @@ "verify_output": "verify.json", "selector": "fd-01-missed-interruption", "record_out": "record", - "expected_record_id": "sha256:ad9c62d49f76f62c11c25184a13cd63669e5e6fee9f1dfc5f8093fb35dbe6eaa" + "expected_record_id": "sha256:25f01ea209620cbd36b6ec662dfce124eb989f53f303be808c65f2f6d90ed674" }, "supersedes": { "record_id": "sha256:772fbb2a13addeb5cfbfb82d3c1652b15cb547d004e2db3db4341779c99c4336", @@ -270,8 +270,8 @@ }, { "command": "hotato record render verify.json#fd-01-missed-interruption --out record", - "output": "wrote failure record (FAIL, fd-01-missed-interruption) to record/: failure-record.html, failure-record.json, failure-record.md, failure-record.svg\nrecord_id: sha256:ad9c62d49f76f62c11c25184a13cd63669e5e6fee9f1dfc5f8093fb35dbe6eaa\n" + "output": "wrote failure record (FAIL, fd-01-missed-interruption) to record/: failure-record.html, failure-record.json, failure-record.md, failure-record.svg\nrecord_id: sha256:25f01ea209620cbd36b6ec662dfce124eb989f53f303be808c65f2f6d90ed674\n" } ], - "content_digest": "sha256:c3d8d1ed551fc070610b61304a316e9166296a84128b23b963a533ad610f371e" + "content_digest": "sha256:d38d9900d7e3265b62b96fb05bdcb8189717b239ce5796d35a1cd06bee246967" } diff --git a/docs/CI.md b/docs/CI.md index c983a3a..e2559f1 100644 --- a/docs/CI.md +++ b/docs/CI.md @@ -19,7 +19,7 @@ Conversation failed: Agent did not yield; measured talk-over was 2.66 s. Share in a PR: hotato-failure-record/failure-record.md Share as an image: hotato-failure-record/failure-record.svg - Verify the record: uvx --from hotato==1.15.0 hotato record verify hotato-failure-record/failure-record.json + Verify the record: uvx --from hotato==1.15.1 hotato record verify hotato-failure-record/failure-record.json ``` Preview it locally, then scaffold the durable gate into your own repository @@ -43,11 +43,11 @@ offline -- it runs the pinned Action revision itself off PYTHONPATH (no pip, no package index), installs no model, no ASR, no Node tool, calls no external judge, and reads no secret. -Composite Action since v1.4.0. Adopt the current release (v1.15.0), pinned +Composite Action since v1.4.0. Adopt the current release (v1.15.1), pinned by its full commit SHA; resolve the tag to its SHA first: ```bash -git ls-remote https://github.com/attenlabs/hotato refs/tags/v1.15.0 +git ls-remote https://github.com/attenlabs/hotato refs/tags/v1.15.1 ``` Then commit this workflow (replace the `attenlabs/hotato` pin with the SHA @@ -69,7 +69,7 @@ jobs: - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 - id: hotato # Pin by full commit SHA (immutable); the comment names the release. - uses: attenlabs/hotato@ # v1.15.0 + uses: attenlabs/hotato@ # v1.15.1 with: suite: tests/voice/qa.suite.json agent: support-agent @@ -107,7 +107,7 @@ Everything else is optional: | Value | Effect | |---|---| | `action` (default) | Installs the pinned Action revision itself, `--no-deps`, no package-index egress -- exactly the revision your workflow pinned | -| an exact version, e.g. `1.15.0` | `pip install --no-deps hotato==1.15.0` | +| an exact version, e.g. `1.15.1` | `pip install --no-deps hotato==1.15.1` | | `preinstalled` | Skips installation (hotato is already on the runner) | A range or `latest` is refused, so the pin always names one exact diff --git a/docs/COMPARE.md b/docs/COMPARE.md index ba7254f..398d190 100644 --- a/docs/COMPARE.md +++ b/docs/COMPARE.md @@ -4,31 +4,45 @@ Local-first testing and observability for AI agents, next to the hosted platforms that run the same jobs as a service. The short version: the same lifecycle, three structural differences. -## The lifecycle, side by side +## Property by property -Every step below runs on your machine, from the CLI, with an exit code CI can -gate on. The right column is what the equivalent step costs on a hosted -platform, structurally, whoever the vendor is. +The hosted platforms (LangSmith, Langfuse, Phoenix, LangWatch, and the +voice-specific tools) are capable products: many support deterministic code +evaluators, datasets, experiments, human review, and self-hosting. This table +compares default properties, not a caricature, and it is honest about where a +managed platform is the stronger fit. -| Lifecycle step | hotato | Hosted platforms | +| Property | hotato default | Typical managed deployment | |---|---|---| -| **Observe** traces, cost, latency | `hotato observe` on the OTel spans you already emit, locally | your traces live on their servers | -| **Catch** the failures evals miss | `hotato investigate` / `sweep`: deterministic timing and say-do scoring | model-scored, so the number drifts run to run | -| **Pin** a failure as a test | `hotato investigate label`: a portable, content-addressed contract | a dataset row inside their account | -| **Test** candidates against it | `hotato simulate` / `drive` / `gauntlet`, seeded and byte-reproducible | simulation credits, metered per run | -| **Prove** a release | `hotato prove`: every lane composed into one fail-closed receipt | a dashboard score you cannot re-derive | -| **Confirm** in production | `hotato production` alerts, exported back into the loop as tests | alerts inside their service | +| Raw evidence location | your local filesystem or VPC | the vendor's service | +| Price at scale | free and MIT, any volume | metered per seat, run, or event | +| Deterministic timing lane (turn-taking, say-do) | built in | varies by platform | +| Content-addressed contracts + release proofs | built in | platform-specific | +| Offline verification, no service dependency | built in | often service-dependent | +| Model-judge lane | separate and advisory by default | commonly integrated into the score | +| Hosted parallel execution at scale | you operate it | managed by the vendor | +| Persistent trace search, datasets, prompt management | limited | commonly built in | +| Team collaboration, roles, review queues | local workspace | mature hosted collaboration | -## The three structural differences +## Where hotato is the clear pick -- **Price at scale.** Free and MIT at any volume. There is no per-seat, per-run, - or per-event meter anywhere in the loop. -- **Verdicts you can gate on.** The same input scores the same way on every - machine, byte for byte, so an exit code can block a merge. A judged score - that varies run to run cannot. - **Your data stays yours.** Recordings, traces, prompts, and backend state never leave your machine, and a contract or proof stays verifiable if you stop using hotato, if the vendor changes, or if the service is down. +- **Verdicts you can gate on.** The deterministic lanes score the same input + the same way on every machine, byte for byte, so an exit code can block a + merge. A model-judged score that varies run to run cannot. +- **Free at any volume.** No per-seat, per-run, or per-event meter anywhere. + +## Where a managed platform may fit better + +A hosted platform is often the faster path if you need mature multi-user +collaboration, a persistent searchable trace explorer with saved views, +built-in dataset and prompt management, or vendor-operated execution at high +concurrency. hotato is deliberately local and CLI-first; those surfaces are +limited today. The two compose: point a hosted platform's OTel export at +`hotato trace ingest` and keep the deterministic test and release-evidence +layer on your own machine. ## The layer hotato is not diff --git a/docs/LIFECYCLE.md b/docs/LIFECYCLE.md index e9a4923..48b081c 100644 --- a/docs/LIFECYCLE.md +++ b/docs/LIFECYCLE.md @@ -1,7 +1,7 @@ # The hotato lifecycle -Every production failure becomes a portable test, every candidate runs against -it, and every release carries evidence. That sentence is the whole product; this +Turn production failures into portable tests, run candidate releases against +them, and carry evidence with every release. That is the whole product; this page maps every command onto it. The loop has five steps you drive, and one that feeds itself: diff --git a/docs/TRUST-GALLERY.md b/docs/TRUST-GALLERY.md index cb28b97..8c6c6d5 100644 --- a/docs/TRUST-GALLERY.md +++ b/docs/TRUST-GALLERY.md @@ -1,7 +1,7 @@ # Trust gallery: eight recordings, eight verdicts Eight named input conditions, each with the `hotato` output it produces. -Every block below is verbatim CLI output from hotato 1.15.0, offline. The rows +Every block below is verbatim CLI output from hotato 1.15.1, offline. The rows correspond to the [trust matrix](TRUST-MATRIX.md). The clean and echo cases use the bundled examples diff --git a/llms-full.txt b/llms-full.txt index 240354b..d2f4cf6 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -36,7 +36,7 @@ That transcript passed every text eval. The timing did not. hotato pins the catc ## The loop -Every production failure becomes a portable test, every candidate runs against it, and every release carries evidence. Five steps, one command each, nothing leaves your machine. +Turn production failures into portable tests, run candidate releases against them, and carry evidence with every release. Five steps, one command each, nothing leaves your machine. | | | | | :-- | :-- | :-- | @@ -44,7 +44,7 @@ Every production failure becomes a portable test, every candidate runs against i | **Catch** | deterministic scoring finds what text evals miss: timing, say-do, policy | `hotato investigate call.wav` | | **Pin** | your label turns the catch into a portable, content-addressed contract | `hotato investigate label STATE#1 --expect yield` | | **Test** | simulate, stress, and drive candidates against everything you pinned | `hotato gauntlet` | -| **Prove** | every evidence lane composed into one fail-closed release proof, in CI | `hotato prove --contracts contracts/` | +| **Prove** | every evidence lane composed into one fail-closed, content-addressed proof, in CI | `hotato prove --contracts contracts/` | Deterministic. Byte-reproducible. Free, MIT. Agent-native over MCP. Every verdict carries its evidence across five dimensions (outcome, policy, conversation, speech, reliability), and production feeds the next loop: `hotato production export-regression` turns a live session back into a test. @@ -58,15 +58,15 @@ Same loop a hosted platform runs. Three things it cannot offer. | :-- | :-- | :-- | | Observe, catch, pin, test, prove | yes | yes | | Price at scale | free, MIT, any volume | metered per seat and per event | -| Verdicts | byte-for-byte reproducible, gate a build | vary run to run | +| Verdicts | byte-for-byte reproducible, gate a build | model-judge lanes vary run to run | | Your traces and prompts | stay on your machine | live on their servers | | Runs in CI, offline | yes | needs their service | Full comparison: [`docs/COMPARE.md`](docs/COMPARE.md) -## From a bad call to a release proof +## From a bad call to a proof -One recording in. The pinned failure becomes a gate that stays red until the agent stops failing that call, and every gate you have composes into one receipt: +One recording in. The pinned failure becomes a gate that stays red until the agent stops failing that call, and every gate you have composes into one receipt whose headline states exactly what it establishes: ```console $ hotato investigate ./call.wav @@ -77,14 +77,13 @@ $ hotato investigate label '.hotato/investigate-state.json#1' --expect yield created hotato contract: call-8s-yield $ hotato prove --contracts contracts/ -hotato prove: proof -- overall FAIL (exit 1) - lane verdict counts - contracts fail contracts=1 passed=0 failed=1 tampered=0 refused=0 -content_id: sha256:2959c905ee6c1030... -proof: .hotato/proofs/proof/proof.json +Captured Evidence: FAIL + A lane failed or regressed; see the table. + contracts fail contracts=1 passed=0 failed=1 +content_id: sha256:c907143d711a84ae... ``` -A contract re-measures the captured failure under the pinned policy on every CI run, the same discipline a snapshot test gives you. `hotato prove` composes every lane you have (contracts, suites, before/after batteries, the stress suite) into one fail-closed verdict with a content-addressed receipt: pass only when every lane passed, and "could not tell" is never green. +`hotato prove` composes every lane you have (contracts, suites, before/after batteries, the stress suite) into one fail-closed, content-addressed receipt: pass only when every lane passed, and "could not tell" is never green. The headline is the **claim scope** the evidence supports, never more: contracts alone re-measure stored evidence (**Captured Evidence**), a suite establishes a **Test Suite** ran, and a before/after run reaches **Candidate Revision** only when you bind the candidate identity (`--candidate-config-hash`, `--provider`). ## Quickstart @@ -95,7 +94,7 @@ uvx hotato start --demo hotato investigate ./call.wav # 3. pin the caught moment as a regression contract hotato investigate label '.hotato/investigate-state.json#1' --expect yield -# 4. compose every gate into one release proof, on every pull request +# 4. compose every gate into one proof, scoped to what it establishes hotato prove --contracts contracts/ ``` @@ -113,10 +112,10 @@ jobs: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - - uses: attenlabs/hotato@v1.15.0 + - uses: attenlabs/hotato@v1.15.1 with: contracts: contracts/ - hotato-version: 1.15.0 + hotato-version: 1.15.1 ``` Copy-paste workflow with a commit-SHA pin: [`docs/CI.md`](docs/CI.md). @@ -6282,7 +6281,7 @@ Conversation failed: Agent did not yield; measured talk-over was 2.66 s. Share in a PR: hotato-failure-record/failure-record.md Share as an image: hotato-failure-record/failure-record.svg - Verify the record: uvx --from hotato==1.15.0 hotato record verify hotato-failure-record/failure-record.json + Verify the record: uvx --from hotato==1.15.1 hotato record verify hotato-failure-record/failure-record.json ``` Preview it locally, then scaffold the durable gate into your own repository @@ -6306,11 +6305,11 @@ offline -- it runs the pinned Action revision itself off PYTHONPATH (no pip, no package index), installs no model, no ASR, no Node tool, calls no external judge, and reads no secret. -Composite Action since v1.4.0. Adopt the current release (v1.15.0), pinned +Composite Action since v1.4.0. Adopt the current release (v1.15.1), pinned by its full commit SHA; resolve the tag to its SHA first: ```bash -git ls-remote https://github.com/attenlabs/hotato refs/tags/v1.15.0 +git ls-remote https://github.com/attenlabs/hotato refs/tags/v1.15.1 ``` Then commit this workflow (replace the `attenlabs/hotato` pin with the SHA @@ -6332,7 +6331,7 @@ jobs: - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 - id: hotato # Pin by full commit SHA (immutable); the comment names the release. - uses: attenlabs/hotato@ # v1.15.0 + uses: attenlabs/hotato@ # v1.15.1 with: suite: tests/voice/qa.suite.json agent: support-agent @@ -6370,7 +6369,7 @@ Everything else is optional: | Value | Effect | |---|---| | `action` (default) | Installs the pinned Action revision itself, `--no-deps`, no package-index egress -- exactly the revision your workflow pinned | -| an exact version, e.g. `1.15.0` | `pip install --no-deps hotato==1.15.0` | +| an exact version, e.g. `1.15.1` | `pip install --no-deps hotato==1.15.1` | | `preinstalled` | Skips installation (hotato is already on the runner) | A range or `latest` is refused, so the pin always names one exact @@ -8633,31 +8632,45 @@ Local-first testing and observability for AI agents, next to the hosted platforms that run the same jobs as a service. The short version: the same lifecycle, three structural differences. -## The lifecycle, side by side +## Property by property -Every step below runs on your machine, from the CLI, with an exit code CI can -gate on. The right column is what the equivalent step costs on a hosted -platform, structurally, whoever the vendor is. +The hosted platforms (LangSmith, Langfuse, Phoenix, LangWatch, and the +voice-specific tools) are capable products: many support deterministic code +evaluators, datasets, experiments, human review, and self-hosting. This table +compares default properties, not a caricature, and it is honest about where a +managed platform is the stronger fit. -| Lifecycle step | hotato | Hosted platforms | +| Property | hotato default | Typical managed deployment | |---|---|---| -| **Observe** traces, cost, latency | `hotato observe` on the OTel spans you already emit, locally | your traces live on their servers | -| **Catch** the failures evals miss | `hotato investigate` / `sweep`: deterministic timing and say-do scoring | model-scored, so the number drifts run to run | -| **Pin** a failure as a test | `hotato investigate label`: a portable, content-addressed contract | a dataset row inside their account | -| **Test** candidates against it | `hotato simulate` / `drive` / `gauntlet`, seeded and byte-reproducible | simulation credits, metered per run | -| **Prove** a release | `hotato prove`: every lane composed into one fail-closed receipt | a dashboard score you cannot re-derive | -| **Confirm** in production | `hotato production` alerts, exported back into the loop as tests | alerts inside their service | - -## The three structural differences - -- **Price at scale.** Free and MIT at any volume. There is no per-seat, per-run, - or per-event meter anywhere in the loop. -- **Verdicts you can gate on.** The same input scores the same way on every - machine, byte for byte, so an exit code can block a merge. A judged score - that varies run to run cannot. +| Raw evidence location | your local filesystem or VPC | the vendor's service | +| Price at scale | free and MIT, any volume | metered per seat, run, or event | +| Deterministic timing lane (turn-taking, say-do) | built in | varies by platform | +| Content-addressed contracts + release proofs | built in | platform-specific | +| Offline verification, no service dependency | built in | often service-dependent | +| Model-judge lane | separate and advisory by default | commonly integrated into the score | +| Hosted parallel execution at scale | you operate it | managed by the vendor | +| Persistent trace search, datasets, prompt management | limited | commonly built in | +| Team collaboration, roles, review queues | local workspace | mature hosted collaboration | + +## Where hotato is the clear pick + - **Your data stays yours.** Recordings, traces, prompts, and backend state never leave your machine, and a contract or proof stays verifiable if you stop using hotato, if the vendor changes, or if the service is down. +- **Verdicts you can gate on.** The deterministic lanes score the same input + the same way on every machine, byte for byte, so an exit code can block a + merge. A model-judged score that varies run to run cannot. +- **Free at any volume.** No per-seat, per-run, or per-event meter anywhere. + +## Where a managed platform may fit better + +A hosted platform is often the faster path if you need mature multi-user +collaboration, a persistent searchable trace explorer with saved views, +built-in dataset and prompt management, or vendor-operated execution at high +concurrency. hotato is deliberately local and CLI-first; those surfaces are +limited today. The two compose: point a hosted platform's OTel export at +`hotato trace ingest` and keep the deterministic test and release-evidence +layer on your own machine. ## The layer hotato is not @@ -9787,8 +9800,8 @@ FILE: docs/LIFECYCLE.md # The hotato lifecycle -Every production failure becomes a portable test, every candidate runs against -it, and every release carries evidence. That sentence is the whole product; this +Turn production failures into portable tests, run candidate releases against +them, and carry evidence with every release. That is the whole product; this page maps every command onto it. The loop has five steps you drive, and one that feeds itself: @@ -13130,7 +13143,7 @@ FILE: docs/TRUST-GALLERY.md # Trust gallery: eight recordings, eight verdicts Eight named input conditions, each with the `hotato` output it produces. -Every block below is verbatim CLI output from hotato 1.15.0, offline. The rows +Every block below is verbatim CLI output from hotato 1.15.1, offline. The rows correspond to the [trust matrix](TRUST-MATRIX.md). The clean and echo cases use the bundled examples diff --git a/llms.txt b/llms.txt index dbffa08..798a407 100644 --- a/llms.txt +++ b/llms.txt @@ -36,7 +36,7 @@ > --format json` regenerates the same command data straight from the CLI's > argparse, so it never drifts from the real flags. > -> Version 1.15.0 +> Version 1.15.1 ## Start here diff --git a/pyproject.toml b/pyproject.toml index a937d57..204007e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -5,8 +5,8 @@ build-backend = "setuptools.build_meta" [project] name = "hotato" -version = "1.15.0" -description = "Local-first testing and observability for AI agents: trace, evaluate, test, and gate your LLM and voice agents on your own machine. Everything you use a hosted platform for, free and MIT at any scale, byte-reproducible, and CI-native. Nothing leaves your machine." +version = "1.15.1" +description = "Local-first testing and observability for AI agents. Trace, evaluate, simulate, and gate agents on your own infrastructure: deterministic, byte-reproducible, free, MIT, CI-native. Nothing leaves your machine." readme = "README.pypi.md" # Support policy: requires-python is lower-bounded only. There is no upper cap # on purpose -- an upper pin (e.g. <3.14) locks installs out of new interpreters diff --git a/server.json b/server.json index ef8fdde..13cd2f5 100644 --- a/server.json +++ b/server.json @@ -2,7 +2,7 @@ "$schema": "https://static.modelcontextprotocol.io/schemas/2025-12-11/server.schema.json", "name": "io.github.attenlabs/hotato", "description": "Local-first testing and observability for AI agents. Trace, evaluate, test, and gate. MIT.", - "version": "1.15.0", + "version": "1.15.1", "repository": { "url": "https://github.com/attenlabs/hotato", "source": "github", @@ -14,7 +14,7 @@ "registryType": "pypi", "registryBaseUrl": "https://pypi.org", "identifier": "hotato", - "version": "1.15.0", + "version": "1.15.1", "runtimeHint": "uvx", "transport": { "type": "stdio" diff --git a/src/hotato/__init__.py b/src/hotato/__init__.py index b43f07e..bd92605 100644 --- a/src/hotato/__init__.py +++ b/src/hotato/__init__.py @@ -64,6 +64,6 @@ # absent or can describe a DIFFERENT installed copy than the code executing. # tests/test_version_lockstep.py enforces the match; 0.4.0 shipped # self-reporting 0.3.1 because nothing did. -__version__ = "1.15.0" +__version__ = "1.15.1" __all__ = ["run_single", "run_suite", "LIMITS", "SUITE_ID", "__version__"] diff --git a/src/hotato/cli.py b/src/hotato/cli.py index 261c6c7..fae6ca4 100644 --- a/src/hotato/cli.py +++ b/src/hotato/cli.py @@ -2132,6 +2132,8 @@ def _cmd_prove(args) -> int: contracts=args.contracts, suite=args.suite, agent=args.agent, before=args.before, after=args.after, min_n=args.min_n, gauntlet=args.gauntlet, name=args.name, + candidate_config_hash=args.candidate_config_hash, + provider=args.provider, deployment_id=args.deployment_id, ) out_dir = args.out or os.path.join(".hotato", "proofs", proof["name"]) os.makedirs(out_dir, exist_ok=True) @@ -8978,10 +8980,19 @@ def build_parser() -> argparse.ArgumentParser: "prove", help="compose every evidence lane you have (contracts, suite, " "before/after, gauntlet) into one fail-closed, content-addressed " - "release proof; exit 0 only when every lane passed", + "proof whose CLAIM SCOPE reflects the evidence given; contracts " + "alone is Captured Evidence, not a release proof; exit 0 only " + "when every lane passed", description=( "hotato prove composes the evidence lanes you already ran into " - "ONE portable release proof. Each flag activates one lane: " + "ONE portable proof, headlined by the CLAIM SCOPE the evidence " + "actually supports -- contracts alone re-measures stored evidence " + "and reads as 'Captured Evidence', a suite/gauntlet run as 'Test " + "Suite', and a before/after lane reaches 'Candidate Revision' " + "(or 'Deployed Revision') only when you also bind the candidate " + "identity via --candidate-config-hash and --provider (plus " + "--deployment-id). It never claims more than the lanes support. " + "Each flag activates one lane: " "--contracts re-scores a directory of hotato contracts through " "contract verify's own logic; --suite runs a suite.v1 of " "conversation-tests through the suite runner; --before/--after " @@ -9040,6 +9051,24 @@ def build_parser() -> argparse.ArgumentParser: pv.add_argument("--gauntlet", action="store_true", help="gauntlet lane: run the bundled deterministic " "turn-taking stress suite") + pv.add_argument("--candidate-config-hash", dest="candidate_config_hash", + default=None, metavar="HASH", + help="bind the candidate's config identity (an opaque " + "digest, not a path): with --provider it raises a " + "before/after proof from Test Suite to Candidate " + "Revision; recorded as evidence but never raises the " + "scope without the before/after lane") + pv.add_argument("--provider", default=None, metavar="NAME", + help="the candidate's provider/runner name; with " + "--candidate-config-hash it grounds the candidate " + "identity a before/after proof needs to claim " + "Candidate Revision") + pv.add_argument("--deployment-id", dest="deployment_id", + default=None, metavar="ID", + help="a deployment identifier: only on top of a bound " + "candidate (--candidate-config-hash + --provider) " + "does it raise a before/after proof to Deployed " + "Revision; alone it is recorded but never elevates") pv.add_argument("--name", default=None, metavar="NAME", help="name the proof (letters, digits, dot, underscore, " "hyphen); also the default output directory name " diff --git a/src/hotato/prove.py b/src/hotato/prove.py index 7d043bb..367e5a9 100644 --- a/src/hotato/prove.py +++ b/src/hotato/prove.py @@ -60,6 +60,119 @@ VERDICT_INCONCLUSIVE = "inconclusive" VERDICT_REFUSED = "refused" +# ========================================================================= +# claim scope + evidence authority: exactly what the proof establishes, and +# on what strength of evidence. Both fail DOWN -- the proof may never claim +# more than the activated lanes actually support. +# ========================================================================= + +# claim_scope, weakest -> strongest. The proof's scope is the STRONGEST scope +# any activated lane supports, where each lane's own scope is already capped +# by the evidence it holds (the before/after lane cannot reach a candidate or +# deployed revision without the candidate identity bound alongside it). +SCOPE_CAPTURED_EVIDENCE = "captured_evidence" # re-measures stored evidence +SCOPE_TEST_SUITE = "test_suite" # a suite/battery executed +SCOPE_CANDIDATE_REVISION = "candidate_revision" # bound candidate identity +SCOPE_DEPLOYED_REVISION = "deployed_revision" # bound + a deployment id +CLAIM_SCOPES = ( + SCOPE_CAPTURED_EVIDENCE, + SCOPE_TEST_SUITE, + SCOPE_CANDIDATE_REVISION, + SCOPE_DEPLOYED_REVISION, +) + +# The base scope each lane can support on its own evidence. The before/after +# (``verify``) lane is elevated above ``test_suite`` only when the caller also +# grounds the candidate identity (see :func:`_lane_claim_scope`). +_LANE_BASE_SCOPE = { + "contracts": SCOPE_CAPTURED_EVIDENCE, + "suite": SCOPE_TEST_SUITE, + "verify": SCOPE_TEST_SUITE, + "gauntlet": SCOPE_TEST_SUITE, +} + +# evidence_authority, weakest -> strongest. This release ships only the two +# lower rungs: every current lane scores deterministically, so it is +# ``measured``; a future asserted-only lane would be ``asserted``. The upper +# rungs (paired / runner_authenticated / organization_signed / +# independently_attested) are NOT claimable yet and are never emitted. +AUTHORITY_ASSERTED = "asserted" +AUTHORITY_MEASURED = "measured" +AUTHORITY_PAIRED = "paired" +AUTHORITY_RUNNER_AUTHENTICATED = "runner_authenticated" +AUTHORITY_ORGANIZATION_SIGNED = "organization_signed" +AUTHORITY_INDEPENDENTLY_ATTESTED = "independently_attested" +EVIDENCE_AUTHORITIES = ( + AUTHORITY_ASSERTED, + AUTHORITY_MEASURED, + AUTHORITY_PAIRED, + AUTHORITY_RUNNER_AUTHENTICATED, + AUTHORITY_ORGANIZATION_SIGNED, + AUTHORITY_INDEPENDENTLY_ATTESTED, +) + +# Every lane in this release scores deterministically -- it MEASURES its +# verdict rather than asserting it. +_MEASURING_LANES = frozenset(LANES) + +# Human-facing claim-scope titles for the rendered headline. +SCOPE_TITLE = { + SCOPE_CAPTURED_EVIDENCE: "Captured Evidence", + SCOPE_TEST_SUITE: "Test Suite", + SCOPE_CANDIDATE_REVISION: "Candidate Revision", + SCOPE_DEPLOYED_REVISION: "Deployed Revision", +} + +# The one-line description of what a PASS at each scope establishes. +_SCOPE_PASS_SUBLINE = { + SCOPE_CAPTURED_EVIDENCE: "All stored contracts re-measured successfully " + "under their pinned policies.", + SCOPE_TEST_SUITE: "A test suite executed and every activated lane passed.", + SCOPE_CANDIDATE_REVISION: "The bound candidate revision passed every " + "activated lane.", + SCOPE_DEPLOYED_REVISION: "The deployed revision passed every activated " + "lane.", +} + + +def _lane_claim_scope(lane_name: str, *, candidate_bound: bool, + deployment_bound: bool) -> str: + """The scope a single activated lane supports, given whether the caller + grounded a candidate revision (config hash + provider) and, on top of + that, a deployment id. Only the before/after lane can be elevated; every + other lane keeps its base scope.""" + if lane_name == "verify" and candidate_bound: + return SCOPE_DEPLOYED_REVISION if deployment_bound \ + else SCOPE_CANDIDATE_REVISION + return _LANE_BASE_SCOPE[lane_name] + + +def _derive_claim_scope(lanes: List[Dict[str, Any]], *, + candidate_bound: bool, + deployment_bound: bool) -> str: + """The proof's overall claim scope: the STRONGEST scope any activated lane + supports on its own capped evidence. Contracts-only stays + ``captured_evidence`` (adding nothing cannot elevate it); a suite or + gauntlet lane establishes that a ``test_suite`` ran; the before/after lane + reaches a candidate or deployed revision only with the identity bound.""" + scopes = [ + _lane_claim_scope(entry["lane"], candidate_bound=candidate_bound, + deployment_bound=deployment_bound) + for entry in lanes + ] + return max(scopes, key=CLAIM_SCOPES.index) + + +def _derive_evidence_authority(lanes: List[Dict[str, Any]]) -> str: + """The strength of the evidence under the claim. Every lane in this + release measures deterministically, so any activated lane yields + ``measured``; the paired / signed / attested rungs are not claimable yet + and a candidate binding does NOT raise this to ``runner_authenticated`` + (the runner is not authenticated in this release).""" + if any(entry["lane"] in _MEASURING_LANES for entry in lanes): + return AUTHORITY_MEASURED + return AUTHORITY_ASSERTED + EXIT_PASS = 0 EXIT_FAIL = 1 EXIT_INCONCLUSIVE = 2 @@ -295,6 +408,9 @@ def run_prove( min_n: int = 3, gauntlet: bool = False, name: Optional[str] = None, + candidate_config_hash: Optional[str] = None, + provider: Optional[str] = None, + deployment_id: Optional[str] = None, ) -> Dict[str, Any]: """Run every activated evidence lane through its existing module function and compose the results into one ``hotato.proof.v1`` dict (including @@ -338,6 +454,20 @@ def run_prove( overall = _overall(lanes) exit_code = {VERDICT_PASS: EXIT_PASS, VERDICT_FAIL: EXIT_FAIL, VERDICT_INCONCLUSIVE: EXIT_INCONCLUSIVE}[overall] + + # The candidate identity binds only when BOTH the config hash and the + # provider are supplied; a deployment id elevates only on top of that + # binding. Any of the three supplied WITHOUT the before/after lane are + # recorded as evidence below but never raise the scope (only the verify + # lane reads them). + candidate_bound = bool(candidate_config_hash and provider) + deployment_bound = candidate_bound and bool(deployment_id) + claim_scope = _derive_claim_scope( + lanes, candidate_bound=candidate_bound, + deployment_bound=deployment_bound, + ) + evidence_authority = _derive_evidence_authority(lanes) + proof = { "tool": "hotato", "schema_version": SCHEMA_VERSION, @@ -345,9 +475,25 @@ def run_prove( "hotato_version": __version__, "created_at": _deterministic_created_at(), "lanes": lanes, + "claim_scope": claim_scope, + "evidence_authority": evidence_authority, "overall": overall, "exit_code": exit_code, } + # Share-safe, digest-stable record of any supplied candidate binding. These + # are opaque identifiers (a config digest, a provider name, a deployment + # id), never paths; they are stored verbatim so the content address covers + # them, and only present when at least one was given. + binding: Dict[str, Any] = {} + if candidate_config_hash is not None: + binding["candidate_config_hash"] = candidate_config_hash + if provider is not None: + binding["provider"] = provider + if deployment_id is not None: + binding["deployment_id"] = deployment_id + if binding: + proof["evidence"] = binding + proof["content_id"] = compute_content_id(proof) return proof @@ -373,13 +519,32 @@ def _table_rows(proof: Dict[str, Any]) -> List[Dict[str, str]]: return rows +def _scope_subline(proof: Dict[str, Any]) -> str: + """The one line under the headline: what a PASS at this claim scope + establishes, or why a non-pass did not establish it.""" + overall = proof["overall"] + if overall == VERDICT_FAIL: + return "A lane failed or regressed; see the table." + if overall == VERDICT_INCONCLUSIVE: + return "A lane refused or came back inconclusive; see the table." + return _SCOPE_PASS_SUBLINE[proof["claim_scope"]] + + def render_text(proof: Dict[str, Any], proof_path: Optional[str] = None) -> str: - """The tight per-lane table + overall verdict + content address (+ where - the proof landed, when the caller wrote it).""" + """The claim-scope headline + one-line subhead, then the tight per-lane + table + overall verdict + content address (+ where the proof landed, when + the caller wrote it). The headline states exactly what the proof + establishes -- e.g. ``Captured Evidence: PASS`` for a contracts-only run, + never a bare 'release proof'.""" + title = SCOPE_TITLE[proof["claim_scope"]] lines = [ + f"{title}: {proof['overall'].upper()}", + f" {_scope_subline(proof)}", f"hotato prove: {proof['name']} -- overall " f"{proof['overall'].upper()} (exit {proof['exit_code']})", + f" claim_scope: {proof['claim_scope']} " + f"evidence_authority: {proof['evidence_authority']}", ] rows = _table_rows(proof) lane_w = max(len("lane"), *(len(r["lane"]) for r in rows)) @@ -399,9 +564,14 @@ def render_text(proof: Dict[str, Any], def render_md(proof: Dict[str, Any]) -> str: """proof.md: the same per-lane table, overall verdict, and content address as the text rendering, in markdown.""" + title = SCOPE_TITLE[proof["claim_scope"]] lines = [ f"# hotato proof: {proof['name']}", "", + f"## {title}: {proof['overall'].upper()}", + "", + _scope_subline(proof), + "", f"Overall: **{proof['overall'].upper()}** " f"(exit {proof['exit_code']}). Pass requires every activated lane " "to pass; an inconclusive or refused lane exits non-zero.", @@ -413,6 +583,8 @@ def render_md(proof: Dict[str, Any]) -> str: lines.append(f"| {r['lane']} | {r['verdict']} | {r['counts']} |") lines += [ "", + f"- claim_scope: {proof['claim_scope']}", + f"- evidence_authority: {proof['evidence_authority']}", f"- hotato_version: {proof['hotato_version']}", f"- created_at: {proof['created_at']}", f"- content_id: `{proof['content_id']}`", diff --git a/tests/test_prove.py b/tests/test_prove.py index 6922480..65252c0 100644 --- a/tests/test_prove.py +++ b/tests/test_prove.py @@ -268,3 +268,130 @@ def test_gauntlet_lane_passes_and_counts_the_bundled_suite(tmp_path, capsys): assert lane["verdict"] == "pass" assert lane["counts"]["total"] == 10 assert lane["counts"]["passed"] == 10 + + +# --- claim scope + evidence authority: the artifact states exactly what it +# establishes, and never claims more than its evidence supports ---------- + +def _empty_dir(tmp_path, name): + d = tmp_path / name + d.mkdir() + return d + + +def test_contracts_only_is_captured_evidence_not_a_release_proof( + tmp_path, capsys): + # A contracts-only run re-measures stored evidence; it binds no candidate + # and no deployment, so the strongest thing it can say is "Captured + # Evidence" -- never a bare "release proof". + cdir = _passing_contracts_dir(tmp_path) + out = tmp_path / "proofout" + rc = cli.main(["prove", "--contracts", str(cdir), "--out", str(out)]) + assert rc == 0 + proof = _read_proof(out) + assert proof["claim_scope"] == "captured_evidence" + assert proof["evidence_authority"] == "measured" + + text = capsys.readouterr().out + assert "Captured Evidence" in text + assert "release proof" not in text.lower() + # the JSON "headline" is the claim_scope field itself + assert proof["claim_scope"] == "captured_evidence" + assert "release proof" not in (out / "proof.md").read_text( + encoding="utf-8").lower() + + +def test_gauntlet_lane_is_test_suite_scope(tmp_path, capsys): + out = tmp_path / "proofout" + assert cli.main(["prove", "--gauntlet", "--out", str(out)]) == 0 + capsys.readouterr() + proof = _read_proof(out) + assert proof["claim_scope"] == "test_suite" + assert proof["evidence_authority"] == "measured" + + +def test_contracts_plus_gauntlet_is_test_suite_scope(tmp_path, capsys): + # contracts (captured_evidence) + gauntlet (test_suite): the whole + # evidence set establishes that a test suite ran AND stored evidence held, + # so the proof reads as the stronger "Test Suite". + cdir = _passing_contracts_dir(tmp_path) + out = tmp_path / "proofout" + assert cli.main(["prove", "--contracts", str(cdir), "--gauntlet", + "--out", str(out)]) == 0 + capsys.readouterr() + proof = _read_proof(out) + assert {lane["lane"] for lane in proof["lanes"]} == {"contracts", + "gauntlet"} + assert proof["claim_scope"] == "test_suite" + + +def test_before_after_without_binding_stays_test_suite(tmp_path): + # The anti-over-elevation case: a before/after lane on its own re-runs a + # battery -- it does NOT identify a candidate revision, so it may not be + # elevated above "Test Suite". + before = _empty_dir(tmp_path, "before") + after = _empty_dir(tmp_path, "after") + out = tmp_path / "proofout" + cli.main(["prove", "--before", str(before), "--after", str(after), + "--out", str(out)]) + proof = _read_proof(out) + assert any(lane["lane"] == "verify" for lane in proof["lanes"]) + assert proof["claim_scope"] == "test_suite" + assert proof["claim_scope"] != "candidate_revision" + + +def test_before_after_with_candidate_binding_is_candidate_revision(tmp_path): + before = _empty_dir(tmp_path, "before") + after = _empty_dir(tmp_path, "after") + out = tmp_path / "proofout" + cli.main(["prove", "--before", str(before), "--after", str(after), + "--candidate-config-hash", "sha256:cfg-abc123", + "--provider", "acme-runner", "--out", str(out)]) + proof = _read_proof(out) + assert proof["claim_scope"] == "candidate_revision" + # the binding is recorded, share-safely, in the proof's evidence block + assert proof["evidence"]["candidate_config_hash"] == "sha256:cfg-abc123" + assert proof["evidence"]["provider"] == "acme-runner" + # this release does not authenticate the runner, so authority stays measured + assert proof["evidence_authority"] == "measured" + + +def test_deployment_id_without_candidate_binding_never_deployed_revision( + tmp_path): + # A deployment id alone cannot reach "Deployed Revision": without the + # candidate identity (config hash + provider) bound, there is nothing to + # attach the deployment to. + before = _empty_dir(tmp_path, "before") + after = _empty_dir(tmp_path, "after") + out = tmp_path / "proofout" + cli.main(["prove", "--before", str(before), "--after", str(after), + "--deployment-id", "deploy-77", "--out", str(out)]) + proof = _read_proof(out) + assert proof["claim_scope"] != "deployed_revision" + assert proof["claim_scope"] == "test_suite" + # supplied without raising the scope, it is still recorded as evidence + assert proof["evidence"]["deployment_id"] == "deploy-77" + + +def test_new_flags_stay_deterministic_and_leak_no_absolute_path( + tmp_path, monkeypatch): + monkeypatch.setenv("SOURCE_DATE_EPOCH", "1700000000") + before = _empty_dir(tmp_path, "before") + after = _empty_dir(tmp_path, "after") + args = [ + "prove", "--before", str(before), "--after", str(after), + "--candidate-config-hash", "sha256:cfg-abc123", + "--provider", "acme-runner", "--deployment-id", "deploy-77", + ] + out_a = tmp_path / "proof-a" + out_b = tmp_path / "proof-b" + cli.main(args + ["--out", str(out_a)]) + cli.main(args + ["--out", str(out_b)]) + a = (out_a / "proof.json").read_bytes() + b = (out_b / "proof.json").read_bytes() + assert a == b + # the content address covers the new fields, and no caller path leaks + raw = a.decode("utf-8") + assert str(tmp_path) not in raw + assert _prove.compute_content_id(json.loads(a)) == json.loads(a)[ + "content_id"] From 98b9fda1abbf760ccced1ebe7ace04e38d66e9af Mon Sep 17 00:00:00 2001 From: "David J. Kim" Date: Wed, 22 Jul 2026 13:31:08 -0400 Subject: [PATCH 2/3] changelog: 1.15.1 claim-integrity entry Claude-Session: https://claude.ai/code/session_01BqJW5Dey1DzqBeBCAJXrH9 --- CHANGELOG.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1133453..46ef2c3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,25 @@ Every entry reports millisecond measurement error and a confusion matrix. See `d ## [Unreleased] +## [1.15.1] - 2026-07-22 + +### Changed +- **`hotato prove` states its claim scope.** The proof envelope gains + `claim_scope` (captured_evidence, test_suite, candidate_revision, + deployed_revision) and `evidence_authority`, and the renderer headlines the + scope. A contracts-only run reads **Captured Evidence**, not a "release + proof"; a before/after run reaches **Candidate Revision** only when the caller + binds the candidate identity with `--candidate-config-hash` and `--provider` + (and `--deployment-id` for a deployed revision). The proof never claims more + than its lanes support. No new scoring engine; this is a calibration. +- **Corrected three overclaims on the public surfaces.** The package + description drops "everything you use a hosted platform for" (the public + package has no dataset/prompt management, hosted execution, or team + collaboration). The product one-liner is a capability, not an automatic + guarantee. `docs/COMPARE.md` replaces the caricature of hosted platforms with + an honest property-by-property table that names where a managed platform is + the stronger fit. + ## [1.15.0] - 2026-07-22 ### Added From 1cfa75bd88ae8e3d215b9f36a83bad434a7a3e65 Mon Sep 17 00:00:00 2001 From: "David J. Kim" Date: Wed, 22 Jul 2026 13:33:33 -0400 Subject: [PATCH 3/3] docs: prove docstring says scoped proof, not release proof Claude-Session: https://claude.ai/code/session_01BqJW5Dey1DzqBeBCAJXrH9 --- src/hotato/prove.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/hotato/prove.py b/src/hotato/prove.py index 367e5a9..4e00125 100644 --- a/src/hotato/prove.py +++ b/src/hotato/prove.py @@ -1,5 +1,5 @@ """``hotato prove``: compose the evidence lanes you already ran into ONE -portable, content-addressed release proof. +portable, content-addressed proof whose CLAIM SCOPE states exactly what it establishes. It adds no new scoring engine: every number here is one existing command already measures -- ``hotato contract verify`` (the contracts lane, via