Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,36 @@ Every entry reports millisecond measurement error and a confusion matrix. See `d

## [Unreleased]

## [1.15.0] - 2026-07-22

### Added
- **`hotato prove`: one command, one release proof.** Composes the evidence
lanes you already ran into a single fail-closed verdict with a portable
receipt: `--contracts` re-scores contracts through contract verify's own
logic, `--suite` runs a conversation-test suite, `--before/--after` scores
the battery rollup through `hotato verify` (with its min-n refusal), and
`--gauntlet` runs the bundled stress suite. It adds no new scoring engine.
Overall is pass only when every activated lane passed; any failure fails the
proof, and a lane that could not support its claim is inconclusive and exits
non-zero, so CI never reads "could not tell" as green. Writes
`proof.json` (schema `hotato.proof.v1`) plus `proof.md`: verdicts, counts,
relative names, and sha256 digests only, with a content address that
regenerates identically under a pinned `SOURCE_DATE_EPOCH`.
- **`docs/LIFECYCLE.md`: the whole loop on one page.** Observe, catch, pin,
test, prove, and the production edge that feeds the next loop, with every
command mapped to its step.

### Changed
- **The story is the loop.** README, the docs index, and `docs/COMPARE.md` are
restructured around one sentence: every production failure becomes a portable
test, every candidate runs against it, and every release carries evidence.
The comparison page now walks the lifecycle side by side with hosted
platforms and states the three structural differences (free at any scale,
verdicts reproducible enough to gate a build, data that stays yours).
- **Concrete positioning.** Every surface leads with "Local-first testing and
observability for AI agents"; the positioning-lockstep test enforces the new
line and retires the vague platform noun.

## [1.14.0] - 2026-07-21

### Changed
Expand Down
4 changes: 2 additions & 2 deletions CITATION.cff
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
cff-version: 1.2.0
message: If you use hotato, please cite it as below.
title: "hotato: the local-first AI engineering platform"
title: "hotato: local-first testing and observability for AI agents"
abstract: >-
Trace, evaluate, test, and gate any AI agent on your machine, with nothing
leaving it. Four planes over one install: observe (traces, tokens, cost, and
Expand All @@ -15,7 +15,7 @@ type: software
authors:
- name: "Attention Labs, Inc."
license: MIT
version: 1.14.0
version: 1.15.0
date-released: "2026-07-21"
repository-code: "https://github.com/attenlabs/hotato"
url: "https://hotato.dev"
Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Contributing to hotato

Thanks for being here. hotato is the local-first AI engineering platform: it simulates,
Thanks for being here. hotato is local-first testing and observability for AI agents: it simulates,
evaluates, reviews, and tracks calls across five dimensions (outcome, policy,
conversation, speech, reliability), with the evidence behind every result.
Deterministic checks stay separate from the model-judged rubric, and no output is
Expand Down
53 changes: 30 additions & 23 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,56 +10,59 @@

# hotato

**The local-first AI engineering platform.**
**Local-first testing and observability for AI agents.**

Everything you reach for a hosted platform to do: trace, evaluate, test, and gate your LLM and voice agents, on your own machine. Free at any scale. Byte-reproducible. Nothing leaves it.
Your evals are green. Your agent still ships bugs they can't see: talk-over, dead air, a tool it swore it ran. hotato catches them on your machine and gates CI so they stay fixed.

**[hotato.dev](https://hotato.dev)**

</div>

Hosted observability and eval platforms meter your traffic, keep your traces and prompts on their servers, and score your evals with a model, so the number drifts and cannot gate a build. hotato runs the same four jobs (tracing, evals, tests, and CI gates) on your own machine: free and MIT at any scale, byte-for-byte reproducible, and offline by default.
Free, open-source, and deterministic: the same call scores the same way every run, so you can gate a build on it. Your traces and prompts never leave your machine, and there's no per-seat or per-event bill as you scale.

**Catch your first failure in seconds. One command, no account:**
**Your first catch in seconds. One command, no account:**

```console
$ uvx hotato start --demo
Conversation failed: Agent did not yield; measured talk-over was 2.66 s.
talk-over 2.66s the agent kept talking while the caller held the floor
```

Your text eval read the words on that call and passed it. The timing failed. hotato scores what the transcript can't see, pins the catch as a CI contract, and reproduces the verdict byte for byte on every machine. It measures timing and say-do, not intent.
That transcript passed every text eval. The timing did not. hotato pins the catch as a CI contract that reproduces byte for byte, so a fixed bug stays fixed. It measures timing and say-do, not intent.

## What it does
## The loop

Four planes, one install, nothing leaves your machine.
Every production failure becomes a portable test, every candidate runs against it, and every release carries evidence. Five steps, one command each, nothing leaves your machine.

| | | |
| :-- | :-- | :-- |
| **Observe** | traces, tokens, cost, and latency, from the OTel spans you already emit | `hotato observe report traces/` |
| **Evaluate** | deterministic assertions plus a separated local-judge lane, no blended score | `hotato assert run` |
| **Test** | simulate calls, stress-test turn-taking, pin any failure as a fixture | `hotato gauntlet` |
| **Gate** | content-addressed contracts fail the build on a regression, in CI | `hotato contract verify` |
| **Catch** | deterministic scoring finds what text evals miss: timing, say-do, policy | `hotato investigate call.wav` |
| **Pin** | your label turns the catch into a portable, content-addressed contract | `hotato investigate label STATE#1 --expect yield` |
| **Test** | simulate, stress, and drive candidates against everything you pinned | `hotato gauntlet` |
| **Prove** | every evidence lane composed into one fail-closed release proof, in CI | `hotato prove --contracts contracts/` |

Deterministic. Byte-reproducible. Free, MIT. Agent-native over MCP.
Deterministic. Byte-reproducible. Free, MIT. Agent-native over MCP. Every verdict carries its evidence across five dimensions (outcome, policy, conversation, speech, reliability), and production feeds the next loop: `hotato production export-regression` turns a live session back into a test.

Every verdict carries its evidence, scored across five dimensions: outcome, policy, conversation, speech, and reliability.
The whole loop, command by command: [`docs/LIFECYCLE.md`](docs/LIFECYCLE.md).

## Why it is different

Same four jobs a hosted platform runs. Three things it cannot offer.
Same loop a hosted platform runs. Three things it cannot offer.

| | hotato | Hosted platforms |
| :-- | :-- | :-- |
| Trace, evaluate, test, gate | yes | yes |
| Observe, catch, pin, test, prove | yes | yes |
| Price at scale | free, MIT, any volume | metered per seat and per event |
| Verdicts | byte-for-byte reproducible, gate a build | vary run to run |
| Your traces and prompts | stay on your machine | live on their servers |
| Runs in CI, offline | yes | needs their service |

## From a bad call to a CI gate
Full comparison: [`docs/COMPARE.md`](docs/COMPARE.md)

One recording in. The pinned failure becomes a gate that stays red until the agent stops failing that call:
## From a bad call to a release proof

One recording in. The pinned failure becomes a gate that stays red until the agent stops failing that call, and every gate you have composes into one receipt:

```console
$ hotato investigate ./call.wav
Expand All @@ -69,11 +72,15 @@ $ hotato investigate ./call.wav
$ hotato investigate label '.hotato/investigate-state.json#1' --expect yield
created hotato contract: call-8s-yield

$ hotato contract verify contracts/
[FAIL] call-8s-yield 0/1 contracts pass; exit_code=1
$ hotato prove --contracts contracts/
hotato prove: proof -- overall FAIL (exit 1)
lane verdict counts
contracts fail contracts=1 passed=0 failed=1 tampered=0 refused=0
content_id: sha256:2959c905ee6c1030...
proof: .hotato/proofs/proof/proof.json
```

A contract re-measures the captured failure under the pinned policy on every CI run, the same discipline a snapshot test gives you. Same input, same verdict, byte for byte, on every machine.
A contract re-measures the captured failure under the pinned policy on every CI run, the same discipline a snapshot test gives you. `hotato prove` composes every lane you have (contracts, suites, before/after batteries, the stress suite) into one fail-closed verdict with a content-addressed receipt: pass only when every lane passed, and "could not tell" is never green.

## Quickstart

Expand All @@ -84,8 +91,8 @@ uvx hotato start --demo
hotato investigate ./call.wav
# 3. pin the caught moment as a regression contract
hotato investigate label '.hotato/investigate-state.json#1' --expect yield
# 4. gate every pull request on it
hotato contract verify contracts/
# 4. compose every gate into one release proof, on every pull request
hotato prove --contracts contracts/
```

Keep it with `pipx install hotato`, drive it over MCP with `uvx --from "hotato[mcp]" hotato-mcp`, or walk the path in [`docs/GETTING-STARTED.md`](docs/GETTING-STARTED.md).
Expand All @@ -102,10 +109,10 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: attenlabs/hotato@v1.14.0
- uses: attenlabs/hotato@v1.15.0
with:
contracts: contracts/
hotato-version: 1.14.0
hotato-version: 1.15.0
```

Copy-paste workflow with a commit-SHA pin: [`docs/CI.md`](docs/CI.md).
Expand Down
53 changes: 30 additions & 23 deletions README.pypi.md

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

12 changes: 6 additions & 6 deletions atlas/records/addressed-backchannel-yielded.json
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,7 @@
"provenance": {
"hotato": {
"name": "hotato",
"version": "1.14.0"
"version": "1.15.0"
},
"inputs": [
{
Expand Down Expand Up @@ -196,7 +196,7 @@
],
"source_result_digest": "sha256:7a691d5993389b47f2b43b7afbe7316bbb0d662e703fa0f8a82e39b93b052c08"
},
"record_id": "sha256:1203b251213d691d03f9ce42219e47f6ac1eea508b5299f87371fdef4ba54949",
"record_id": "sha256:dd3c63fcaeff31c7af4cecce0612aa524e554f7ae40d0c362f2d5ddef8491472",
"reproduction": {
"argv": [
"hotato",
Expand All @@ -222,7 +222,7 @@
"version": "1.0"
},
"reproduction_metadata": {
"hotato_version": "1.14.0",
"hotato_version": "1.15.0",
"reviewer_principal": "hotato-examples",
"working_directory": ".",
"bundle": {
Expand All @@ -240,7 +240,7 @@
"verify_output": "verify.json",
"selector": "fd-02-backchannel-yielded",
"record_out": "record",
"expected_record_id": "sha256:1203b251213d691d03f9ce42219e47f6ac1eea508b5299f87371fdef4ba54949"
"expected_record_id": "sha256:dd3c63fcaeff31c7af4cecce0612aa524e554f7ae40d0c362f2d5ddef8491472"
},
"supersedes": {
"record_id": "sha256:6c5e0e6de73004d8683c0987812d849c46fa9f9f8f204eb4da71e600b636e3a0",
Expand Down Expand Up @@ -268,8 +268,8 @@
},
{
"command": "hotato record render verify.json#fd-02-backchannel-yielded --out record",
"output": "wrote failure record (FAIL, fd-02-backchannel-yielded) to record/: failure-record.html, failure-record.json, failure-record.md, failure-record.svg\nrecord_id: sha256:1203b251213d691d03f9ce42219e47f6ac1eea508b5299f87371fdef4ba54949\n"
"output": "wrote failure record (FAIL, fd-02-backchannel-yielded) to record/: failure-record.html, failure-record.json, failure-record.md, failure-record.svg\nrecord_id: sha256:dd3c63fcaeff31c7af4cecce0612aa524e554f7ae40d0c362f2d5ddef8491472\n"
}
],
"content_digest": "sha256:316481e041ab644927ebcd0512e0628772b4d6d5afe4295075a546d4003abd4a"
"content_digest": "sha256:dc05c98c555099225b575ebc5de7f8f95f9c94bcfd6e01345cf3f02f92c258ec"
}
Loading
Loading