Skip to content

M5 W4: contained hosted voice on current ElevenLabs APIs #10

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
owieschon merged 1 commit into from
Jul 11, 2026
Merged

M5 W4: contained hosted voice on current ElevenLabs APIs #10

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@ state/*.db
.DS_Store
_r*.txt

# Local credentials and environment overrides
.env.local

# Internal process docs — do not publish
_handoff/

Expand Down
49 changes: 49 additions & 0 deletions docs/DECISION_LOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1528,3 +1528,52 @@ supports a safe single-line part-only amendment. Order response provenance now
carries line-scoped SKU, requested quantity, and disclosed unit price so the
agent edge cannot crash or launder a priced order line. **Status:** locked for
the W1 risk slice; durable event/conversation storage remains outstanding.

## 2026-07-11 — D19: hosted platform history is authoritative; model tool arguments are not

**Decision.** The ElevenLabs `system__conversation_history` stream is the only
caller-text authority at the hosted voice edge. The gateway validates an
append-only prefix against its durable cursor and processes exactly one next user
entry. Its idempotency identity combines conversation ID, user-entry ordinal,
and content hash. Model-authored tool arguments are empty and cannot supply or
replace caller text, caller identity, quantity, account, agent binding, or quote
facts. Forks, regressions, gaps, and multiple unprocessed user entries fail
closed before order mutation.

The agent may close or transfer only when the gateway's narrow control envelope
explicitly permits it. Binding responses are exact gateway `say` values. Hosted
voice uses `Channel.VOICE`; requested quantity remains line-scoped provenance,
while ERP on-hand quantity is never exposed to the hosted model.

**Why.** A model is useful for conversational routing but is not a trustworthy
source of binding caller facts. Making platform history authoritative preserves
natural hosted conversation without allowing a tool call to turn “ten” into
“twenty” or to bypass durable replay and assent gates.

**Outcome.** `/agent/turn`, `gateway.platform_history`, the custom-LLM router,
and the hosted prompt share one fail-closed contract. Retry, restart, concurrent
caller, planted argument, close/transfer, exact-say, and history-fork tests pass.
**Status:** implemented and verified offline; credentialed hosted-call evidence
remains external.

## 2026-07-11 — D20: authenticated durable call completion releases post-call work

**Decision.** A raw post-call webhook is not itself permission to deliver.
Qualified ElevenLabs completion signals are HMAC-verified, environment-bound,
idempotently persisted with source/effective/received times, and reconciled
against the conversation status API. A missing or permanently uncertain event
holds delivery and raises one manual alert; it never blind-sends. Transcript
content is not retained—only bounded metadata and a digest.

Agent migration updates an existing standalone tool and staged agent branch via
current APIs. It never creates a parallel agent, preserves the assigned phone or
SIP resource and intentional latency/audio/persona settings, fetches back a
secret-redacted normalized receipt, and produces a compensating rollback plan.
Current hosted tests use `run-tests` and bounded invocation polling rather than
the deprecated simulation endpoint.

**Outcome.** Signature/replay/extra-field/wrong-binding/delay/loss and v1 schema
migration tests pass offline. Production remains blocked on existing staging
agent/tool/branch IDs, credentials, real hosted test IDs, an inbound call,
credentialed webhook latency/loss evidence, and one exercised rollback receipt.
**Status:** implementation locked; external staging proof outstanding.
35 changes: 35 additions & 0 deletions docs/M5_BUILD_JOURNAL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -700,3 +700,38 @@ Durable `ConversationStore`/processed-turn replay, append-only line/readback/
assent events, restart recovery, pending-clause continuation through part
disambiguation, configured quantity/line limits, and full attribute/cancel
reference coverage remain required before `owieschon-ix3.2` can close.

## 2026-07-11 — Session 10: W4 hosted voice and authenticated post-call release

Moved the existing hosted agent contract to current ElevenLabs surfaces without
creating a second agent. The migration code manages one standalone
`parts_gateway_turn` tool through `tool_ids`, current built-ins, environment-
scoped custom-LLM configuration, current interruption mode, normalized
secret-redacted capture/fetch-back receipts, and a compensating rollback plan.
The current Agent Testing runner uses `run-tests` and invocation polling and
keeps raw results in ignored state.

Platform conversation history is now authoritative. The durable prefix cursor
rejects forks, regressions, gaps, and turn batching; model-authored binding input
cannot alter caller text or quantity. The custom-LLM path substitutes exact
gateway speech, enforces close/transfer authorization, uses voice disclosure
semantics, and no longer exposes on-hand counts. Issuance speech names the
store-issued quote, selected channel, masked contact, post-call timing, and the
quote-not-sales-order boundary without claiming delivery already happened.

Added authenticated post-call persistence and reconciliation. Qualified events
are HMAC-verified and environment-bound, v1 stores migrate non-destructively,
replay is idempotent, missing events remain held, and permanent uncertainty
alerts once. Transcript text is discarded after hashing.

**Verification:** full pytest exits zero at 902 tests collected (the existing
expected skips remain); ruff is clean; the configured correctness-critical mypy
gate reports no issues across 86 source files. Focused runtime typing across all
new W4 modules is also clean. P14/P15 capability probes correctly remain
production-blocked rather than manufacturing credentialed evidence.

**External W4 proof still required:** existing staging agent/tool/branch/version/
environment/phone-or-SIP IDs and ElevenLabs credentials; a fetch-back receipt;
an inbound staging call; actual hosted Agent Test IDs and raw results; a
credentialed post-call latency/loss receipt; and one exercised rollback. No
production or Vercel deployment was attempted.
261 changes: 107 additions & 154 deletions docs/VOICE_AGENT.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,158 +1,111 @@
# Voice agent: hosted speech shell over the deterministic gateway

## The decision

Two ways to run a phone agent over this gateway:

1. **Self-hosted media pipeline** — Twilio Media Streams → streaming STT →
gateway → TTS, all wired in `runtime/app.py` (`/voice-stream`). We built and
live-tested this. It works, but it makes us responsible for turn-taking,
barge-in, small talk, greeting cadence, and voice quality — the parts that a
hosted voice-agent platform already solves well, and that a live test call
showed are hard to get right by hand (robotic cadence, jumping the caller).

2. **Hosted speech shell + gateway-as-a-tool** (this document) — an ElevenLabs
Agent owns speech, turn-taking, small talk, and the natural voice; for **any**
part/availability/price question it calls **one** server tool, `resolve_part`,
which is our deterministic gateway (`POST /agent/turn`). The agent never
produces a SKU or a price itself — it can only relay what the tool returns.

We chose (2) for the conversational surface and kept (1) in the tree as the
self-hosted fallback. The division of labor:

| Concern | Owner |
|---|---|
| Natural voice, turn-taking, barge-in, small talk, greeting | ElevenLabs Agent |
| Speech-to-text | ElevenLabs Agent (hosted STT) |
| **Resolving the part** (grammar + retrieval + never-invent) | **gateway tool** |
| **Availability / lead time** | **gateway tool** |
| **Pricing behind account verification** | **gateway tool** |
| **Escalation to a human** | **gateway tool** |
| Self-improvement data capture | gateway (`observe_agent_turn`) |

The thesis of the whole repo holds unchanged: **rules own every binding fact;
the model only handles speech and phrasing.** Moving speech to ElevenLabs does
not move the guarantee — it stays in code, behind the tool.

## What the agent can and cannot do

The agent's *only* freedom is to be a friendly human voice. Every part fact is
gated by the tool, enforced two ways:

- **In code (primary).** `/agent/turn` runs the same `gateway.turn` as every
other channel: never-invent (every RESOLVED answer references a real catalog
row), pricing refused until the session is verified, escalation on repeated
failure. The agent cannot reach a SKU, a price, stock, or a ship date except
as the `say` string the tool hands back. No prompt can change that.
- **In the prompt + platform guardrails (defense in depth).** The system prompt
(`voice_agent/SYSTEM_PROMPT.md`) and the ElevenLabs Guardrails control layer
tell the model the same thing, so it does not *try* to free-lance and then get
blocked. Belt and suspenders.

## Grounding (authoritative ElevenLabs sources)

The system prompt and guardrail config follow ElevenLabs' own guidance
(fetched 2026-06-07):

- **Six-block prompt** — Personality, Environment, Tone, Goal, Guardrails,
Tools. Models pay special attention to the `# Guardrails` heading; the most
important rules are stated twice and tagged "This step is important."
<https://elevenlabs.io/docs/eleven-agents/best-practices/prompting-guide>
- **Guardrails** — four types (Focus, Manipulation, Content, Custom) across
three layers (system-prompt hardening, user-input validation, independent
agent-response validation). The binding guarantee lives in the prompt's
`# Guardrails` section (always sent) and in code at `/agent/turn`; the
ElevenLabs *platform* Guardrails 2.0 layer is a versioned object we enable in
the dashboard (our intended policy — Focus + Manipulation + Content + a custom
`parts_only_from_tool` rule — is version-controlled in
`runtime/voice_agent.py:guardrail_config()`). Voice runs in streaming mode; on
a hard violation the call ends rather than risk an unmandated value.
<https://elevenlabs.io/docs/eleven-agents/best-practices/guardrails>
- **Hallucination prevention** — ground the agent in tool/knowledge data and
instruct "never guess or make up information"; we go further and give it *no*
part knowledge of its own — the tool is the only source.
<https://elevenlabs.io/docs/eleven-agents/best-practices/prompting-guide>
- **TTS-friendly phrasing** — the gateway already returns speech-shaped text
(`gateway/spoken.py`: "5 by 24 inch", not `5"X24`), so the prompt tells the
agent to read `say` verbatim and not re-render the numbers.
- **Server (webhook) tool + create-agent API** — the tool and payload shapes in
`runtime/voice_agent.py`.
<https://elevenlabs.io/docs/eleven-agents/customization/tools/server-tools> ·
`POST https://api.elevenlabs.io/v1/convai/agents/create` (header `xi-api-key`)

## Data-capture parity

The self-hosted path feeds the always-on self-improvement loop (flagging
uncertain moments for periodic human review, see `gateway/shadow.py`). The
hosted path keeps that parity: `/agent/turn` calls
`ContinuousImprovement.observe_agent_turn(text)` on every turn (active only when
`SKU_IMPROVEMENT` is configured), so moving speech off-box does not lose the
"improve the service" signal.

## Persona is one source of truth

The hosted agent's greeting and voice come from the **same** operator-configurable
`VoicePersona` the local runtime uses — `build_agent_payload(persona=…)` reads
`first_message` from `persona.opening()` and `voice_id` from
`persona.resolved_voice_id()`. So `SKU_VOICE_NAME`, `SKU_VOICE_ACCENT`,
`SKU_VOICE_GREETING`, and `SKU_VOICE_ID[_<ACCENT>]` configure the ElevenLabs
agent exactly as they configure the phone stack. No second place to edit.

## Deploy

The build and validation are pure (no network, no key) — CI exercises them in
`tests/test_voice_agent_config.py`. Only `--apply` touches the API.
# Voice agent: hosted speech shell over the contained gateway

## Production architecture

ElevenLabs owns telephony audio, ASR, TTS, turn-taking, barge-in, and brief small
talk. The gateway owns every consequential business transition and binding fact.
The hosted agent has one standalone webhook tool, `parts_gateway_turn`, pointing
to `POST /agent/turn`.

The tool accepts no model-authored business argument. ElevenLabs populates these
three fields from system variables:

- `system__conversation_id`
- `system__conversation_history`
- `system__agent_turns`

The gateway validates the history as an append-only prefix of its durable cursor,
processes exactly the next user entry, and derives the idempotency key from the
conversation ID, user ordinal, and user-content hash. Model text, tool arguments,
assistant history, forks, regressions, and gaps cannot mutate binding state.

The custom LLM endpoint at `POST /v1/chat/completions` is the containment seam.
When the last message is a gateway tool result, it returns the tool's `say`
exactly and never calls a model. Free turns still pass the fabrication filter.
`end_call` and `transfer_to_number` are current built-in tools, but the router
permits them only when the latest gateway `control` envelope authorizes them.

## Current ElevenLabs interfaces

- Standalone tool: `POST/PATCH /v1/convai/tools[/{tool_id}]`
- Agent tool references: `conversation_config.agent.prompt.tool_ids`
- Built-ins: `conversation_config.agent.prompt.built_in_tools`
- Custom LLM: environment-scoped `conversation_config.agent.llm.custom_llm`
- Agent update: `PATCH /v1/convai/agents/{agent_id}?branch_id={branch_id}`
- Tests: `POST /v1/convai/agents/{agent_id}/run-tests`, then bounded polling of
`GET /v1/convai/test-invocations/{invocation_id}`
- Completion: HMAC-authenticated `post_call_transcription`, reconciled against
`GET /v1/convai/conversations/{conversation_id}` when the callback is missing

Removed inline `prompt.tools`, model-authored utterance parameters, deprecated
interruption fields, and the legacy simulation runner are forbidden by tests.

## Interruption policy

The gateway tool uses `interruption_mode=disable_during_tool`. A caller cannot
corrupt an in-flight lookup, but can interrupt the spoken readback after the tool
returns. That interruption appears as the next authoritative user entry and is
handled as an edit or clarification. Readbacks are chunked and persist their
position; an edit invalidates completion and restarts the final gate.

## Completion and delivery boundary

Quote issuance says the store-issued quote number, chosen delivery channel,
masked verified destination, and that sending happens after the call. It states
that the artifact is a quote—not a sales order—and never claims “sent” or
“delivered” before a provider callback.

`POST /webhooks/elevenlabs/post-call` validates `ElevenLabs-Signature` over the
raw body, checks timestamp tolerance and exact agent/branch/version/environment,
and stores only completion metadata plus a payload digest. Transcript and audio
are not retained there. Missing callbacks are reconciled through the conversation
status API; unresolved calls remain held and produce one manual-review alert.

## Offline validation

```bash
uv run --extra dev python scripts/elevenlabs_agent.py --validate
uv run --extra dev python scripts/elevenlabs_agent.py --dry-run
uv run --extra dev pytest -q tests/test_voice_agent_config.py \
tests/test_runtime_conversation_durability.py tests/test_custom_llm_route.py \
tests/test_post_call.py tests/test_elevenlabs_migration.py
```

## Credentialed staging migration

The migration command never creates an agent. It requires explicit IDs for the
existing agent, staging branch, immutable rollback version, and standalone tool.
It captures the phone/SIP assignment and latency/audio/persona settings before
mutation, updates the existing tool and branch, fetches both back, compares every
intentional source field, and writes a secret-redacted receipt under `state/`.

```bash
# 1. Confirm the prompt still carries every non-negotiable guardrail:
python scripts/elevenlabs_agent.py --validate

# 2. Review the exact create payload (no network):
AGENT_TOOL_BASE_URL=https://<public-host> \
python scripts/elevenlabs_agent.py --dry-run | less

# 3. Stand up a public URL the ElevenLabs agent can reach for /agent/turn
# (cloudflared quick tunnel or your ingress), then create the agent:
AGENT_TOOL_BASE_URL=https://<public-host> \
ELEVENLABS_API_KEY=... \
python scripts/elevenlabs_agent.py --apply
# update later with: --apply --agent-id <id>
export ELEVENLABS_API_KEY=...
export ELEVENLABS_AGENT_ID=...
export ELEVENLABS_STAGING_BRANCH_ID=...
export ELEVENLABS_TOOL_ID=...
export AGENT_TOOL_BASE_URL=https://staging-gateway.example

uv run --extra dev python scripts/elevenlabs_agent.py --capture-receipt
uv run --extra dev python scripts/elevenlabs_agent.py --apply
uv run --extra dev python scripts/agent_grid.py
```

`create_or_update_agent()` re-runs `validate_system_prompt()` and **refuses to
deploy** a prompt that has lost a guardrail — the same check the tests assert.

Schema notes (live-verified against the create API, 2026-06-07): a POST webhook
tool uses `api_schema.request_body_schema` (not `body_params_schema`);
`caller_id` binds to the `system__conversation_id` system dynamic variable;
English agents must use `eleven_turbo_v2` or `eleven_flash_v2` TTS (we use flash
v2 for the lowest phone latency). A successful `--apply` returns an `agent_id`;
verify the tool fires end-to-end without a phone via
`POST /v1/convai/agents/{id}/simulate-conversation` (supply
`simulation_specification.dynamic_variables.system__conversation_id`, which real
calls inject automatically).

### First-deploy checks (do these in the ElevenLabs dashboard)

- **`caller_id` binding.** The tool sends `caller_id` so the gateway session —
and thus account verification — persists across the call. We bind it to the
system dynamic variable `{{system__conversation_id}}`; confirm the agent is
actually injecting it (not letting the model fill it), so concurrent callers
do not share a session.
- **LLM id.** `SKU_AGENT_LLM` (default `claude-opus-4-7`) must match a model
ElevenLabs currently serves (`GET /v1/convai/llm`). EU data-residency
workspaces restrict some models.
- **Attach a phone number** to the agent (or point your Twilio number at the
ElevenLabs SIP/inbound integration) and place a test call.

## What we give up vs. the self-hosted path

- **STT vendor choice / keyterm biasing.** ElevenLabs uses its own hosted STT;
we lose the AssemblyAI keyterm-prompt tuning. Mitigation: the gateway's
deterministic grammar already absorbs garbled alphanumerics, and the readback
states the decoded part back for confirmation.
- **Full transcript control.** The hosted agent owns the media; our per-turn
`assistant` events on `/voice-stream` are richer. Mitigation: `/agent/turn`
still journals every turn through the gateway and feeds the improvement loop.
- **A second dependency** (ElevenLabs Agents, not just TTS). Accepted: it buys
the conversational quality a live call showed we should not hand-roll.
After an inbound staging call proves the pinned phone reaches the new version,
exercise the compensating rollback once:

```bash
uv run --extra dev python scripts/elevenlabs_agent.py --rollback
```

The apply path restores the prior tool and commits the immutable prior agent
configuration as the new staging-branch tip automatically if update or fetch-back
validation fails. No traffic is deployed to production by these commands.

## External blockers

Without ElevenLabs workspace credentials and the tenant's existing agent, branch,
version, tool, phone/SIP, environment variables, and hosted test IDs, the repo can
prove the migration machinery offline but cannot honestly claim the inbound-call,
hosted-test, latency/loss, or rollback staging receipts. Those are explicit P14/
P15 production blockers, not reasons to weaken the local controls.
Loading
Loading