Whisperer

▶ Live demo

A drop-in, model-agnostic memory layer for long voice-agent calls.

A cheap model distills the live conversation into a typed, evidence-backed state ledger
and re-grounds the agent every turn — so it never loses the thread, and the audio context isn't re-paid each turn.

_{Built at HackRome · 13 Jun 2026 — Giovanni Di Fonzo · Daniele Giovanardi · Gabriele Loreti}

Watch the demo →  ·  Devpost  ·  HackRome

---

Long voice calls drift and get expensive. OpenAI itself notes that "instruction adherence can drift" as context grows, and that "turns later in the session will be more expensive." Managed platforms ship hard limits to cope — Retell caps context around 32k, OpenAI documents blind truncation — so by minute 10 the agent has quietly forgotten what the caller said at minute 1.

Today the fix is artisanal: every team rewrites its own context handling. Whisperer packages it as an add-on you bolt onto any voice stack (OpenAI Realtime, ElevenLabs Agents, Vapi, Retell).

The number — measured, not claimed

Same agent, same prompt, same call. A binary judge asks "does it recall the minute-1 fact?" — yes/no, with the turn citation. Batch of N=10 per side.

Side	Recall (N=10)	How
base (no layer)	0 / 10	item-capped window → the minute-1 fact falls out by recall time
suggeritore (Whisperer on)	10 / 10	answers from the compact ledger; every verdict carries a [t41] citation (the recall turn shown in the replay; server/evidence/ documents the real batch)

Plus: a split-screen demo, a live HUD that writes the ledger to screen by itself, and the recall moment — the base flounders, Whisperer answers exactly, with proof.

What we built — the precise truth

caller speaks  →  distiller writes the ledger (every 4 turns)  →  injector sends [ledger + question]  →  agent answers from memory  →  judge scores recall (N=10)

Whisperer is a memory layer that sits next to the voice agent. Three real pieces, all in the code:

1. The distiller — gpt-4o-mini, structured output. Every 4 turns it reads the live transcript and writes a typed, append-only state ledger: identity · objective · facts · commitments. Every entry cites the transcript turn that proves it. A deterministic _reconcile step drops any fact that doesn't point to a real caller turn → no hallucinated memory. This is the IP. → sdk/whisperer/state_store.py · spec/SPEC.md §1–2

2. Compact re-grounding — compact_input, the real session-rotation of SPEC §3. Each turn, instead of resending the whole conversation, Whisperer sends [compact ledger + current question]. The agent answers from the distilled state — light while the base re-pays its growing window. This is the product, and it's measured. → sdk/whisperer/injector.py

3. The judge + harness — a binary judge reads a recording and returns remembers: yes/no with the turn citation. In batch (N=10/side) it produces the number. It runs against the real workflow, so the number isn't gamed at the measurement layer. → harness/judge.py · harness/runner.py

Honesty notes — the limits are the credibility

We under-promise by a hair, on purpose. State these proactively; keep them ready in Q&A.

• Cost — the important one. The on-screen counter shows real, measured numbers from server/evidence/fullcontext-qa (28-turn batch, N=10): base $0.30 vs Whisperer $0.23, 1.30× — the base re-pays its growing context every turn while Whisperer sends a compact ledger and stays flat. That's the direction, measured, labeled "misurato". The dramatic magnitude (e.g. 7.6×) is a projection for a full 10–20 min call — stated verbally, never passed off as the measured on-screen number. On a short 28-turn demo forgetting and cost-magnitude are coupled, so the live gap is a modest 1.3×; on a real call it compounds as the base keeps climbing and Whisperer stays flat. The mock cost.json is gone.
• Why the base forgets. The base is capped to reproduce the real 32k hard-cap managed platforms ship (Retell) and the blind truncation OpenAI documents. The number measures recall under that real condition — not an invented handicap.
• Watchdog (if asked). Implemented (SPEC §4) and on by default (SUGGERITORE_WATCHDOG, sdk/whisperer/watchdog.py): after each reply a cheap check asks whether it contradicts a known fact and, on drift, re-injects that single fact and lets the agent answer again. Recall in the measured scenarios is the same with it on or off (the layer already hits full recall); the watchdog hardens the contradiction edge that periodic re-grounding alone doesn't cover. The published cost number was measured with the watchdog off (periodic re-grounding + compact state only) — reproduce that baseline with SUGGERITORE_WATCHDOG=off.
• Distiller cost (if asked). Yes, the layer runs a cheap text model; its overhead is small next to the audio context the base re-pays every turn.

Backup evidence for Q&A lives in server/evidence/fullcontext-qa/: a full-context base experiment showing 5/10 even with uncapped context (context rot), and a 1.30× real, measured cost divergence in the right direction — "we measured the direction; we project the magnitude."

Repo layout

Path	What lives here
sdk/	whisperer-sdk — the layer extracted as an installable package: distiller, injector, watchdog, cost meter, connectors
server/	FastAPI voice agent (from openai-voice-agent-sdk-sample) that consumes whisperer-sdk + demo agent/connectors
harness/	Binary judge + batch runner (N=10/side) → the number · token/cost meter
web/	Next.js dashboard: split-screen base vs suggeritore + live memory HUD + cost counter
spec/	SPEC.md (design + JSON contracts) · PROMPTS.md (Codex kickoffs) · fixtures/ (mock data + scenarios)

The folders couple only through the JSON contracts in spec/SPEC.md §7. Each builds against spec/fixtures/, never against the others — which is what lets web/ run with no backend (mock-first).

Run it

Prereqs: Node 18+, uv, and a .env in the repo root (copy .env.example) with OPENAI_API_KEY (+ ELEVENLABS_API_KEY for voice).

1. The dashboard (the demo) — runs on fixtures, no backend needed

cd web
npm install
npm run dev          # http://localhost:3000 — split-screen, live HUD, cost meter

2. The harness (the number)

# validate the judge against the bundled fixture
python harness/runner.py --mode fixture

# score N real recordings per side, and aggregate per-run cost
python harness/runner.py --mode live --scenario nonna \
  --base recordings/nonna_base_run*.jsonl \
  --sug recordings/nonna_sug_run*.jsonl \
  --cost-dir recordings/

3. The live server (engine)

cd server && make sync                        # npm install + uv sync
cd server/server && uv run python server.py   # FastAPI + /ws on :8000

# generate a fresh batch of recordings (N=10 per side; --scenario default: nonna)
uv run python batch_run.py --mode suggeritore --n 10 --scenario nonna
uv run python batch_run.py --mode base        --n 10 --scenario nonna

# other scenarios live in spec/fixtures/scenarios/ (reso, cambio-consegna, long-call);
# long-call sweeps call length: add --turns N to cap to the first N caller turns
uv run python batch_run.py --mode base --scenario long-call --turns 20

Recordings are reproducible. The commands above (re)generate recordings/<scenario>_*.jsonl on demand, so per-scenario batches are not all committed. The repo keeps only the long-call evidence (recordings/long-call_* capped + recordings/uncapped/ full-context — the cost proof for #3); regenerate any other scenario with --scenario <id> whenever you need it.

4. The recall×cost benchmark (the measured 2×2) — one command, end to end

make bench        # repo root — 3 modes (base_full / base_cap / suggeritore) × N=5 real runs,
                  # multi-probe recall judge, per-turn cost with modelled prompt caching.
                  # Regenerates spec/fixtures/studierai-oral.json (+ the web copy, page /studierai)
                  # and research/recall-cost-benchmark.md. ARGS="--skip-batch" reuses recordings.

The design and honesty rules live in research/recall-cost-benchmark-SPEC.md; the measured result and method are in research/recall-cost-benchmark.md.

The layer is toggled by env vars (read by the SDK in sdk/whisperer/): SUGGERITORE_MODE (on/off), SUGGERITORE_STATE_PATH, SUGGERITORE_COST_PATH, SUGGERITORE_INJECT_EVERY, SUGGERITORE_DISTILL_EVERY, SUGGERITORE_BASE_CAP, SUGGERITORE_WATCHDOG (SPEC §4 drift guard, opt-in). The agent's tools call a per-client connector selected by WHISPERER_API_CONNECTOR (default api_shopdemo) — swap in a real connector without touching the agent (see server/server/app/api_template.py.example).

Naming: suggeritore is the internal identifier for the layer-on side (the project's original name). The product is Whisperer.

The contracts — spec/SPEC.md §7

Four JSON shapes are the only coupling between folders; mocks live in spec/fixtures/:

Shape	What it is
transcript turn	one line of the call — [t{n}], role, text
state.json	the append-only ledger — identity · objective · facts · commitments, each with a citing turn
cost_event	per-turn { agent, turn, tokens_in, tokens_out, usd_cumulative } (SPEC §5)
verdict	the judge output — { remembers, citation, … } (SPEC §6)

Demo audio

Real recordings of a grandmother (Neapolitan, with consent) live in web/public/audio/, named by transcript turn and matching spec/fixtures/transcript.jsonl word for word: the gift is a watch for grandson Luca, it must arrive before the 20th (graduation), order 4471, delivered to neighbor sig.ra Pina, interno 3. At minute 10 the base asks the caller to repeat everything; Whisperer confirms the deadline and delivery from memory.

Built with — an agentic coding stack

Codex was the primary builder, and the Codex-signed commit trail is the proof of build: the repo's history shows the layer, the judge, and the dashboard built today, in the open. Every feature started from one closed prompt — "read spec/SPEC.md, propose the integration plan into the sample" — never "build everything". One task at a time, a fresh session per task, gated against hallucination.

Tool	Role in the build
OpenAI Codex	Primary builder — server, harness, dashboard · the signed commit trail
Claude Code (Opus)	Orchestration — scaffold, SPEC.md, all fixtures, audio pipeline, verdict view, docs
Context7 MCP	Fresh library docs at build time — zero hallucinated APIs
shadcn	Official CLI + MCP for real UI components — never hand-rolled
Magic (21st.dev) MCP	On-demand component generation

Discipline that kept it honest: library APIs via Context7, UI via the shadcn CLI, one closed task per session, mock-first against spec/fixtures/ so every folder builds in parallel without touching the others.

Sponsors used

OpenAI Agents SDK · structured outputs (gpt-4o-mini distiller + binary judge) · ElevenLabs (voice) · Codex (the builder, and the commit trail).

Team

Giovanni Di Fonzo	LinkedIn
Daniele Giovanardi	LinkedIn
Gabriele Loreti	LinkedIn