telnyx-offleash

A complete Telnyx voice agent. Every layer through Telnyx. No console.

A complete real-time voice phone agent where every layer (transport, speech-to-text, text-to-speech, and LLM inference) runs through Telnyx, in about 1,600 lines of Python. One control plane, one API key, end to end. The speech and inference engines reachable through that one Telnyx API include both Telnyx-native and third-party models (see Pipeline options); "through Telnyx" means one integration surface and one bill, not that every model is Telnyx-built.

Full write-up and analysis: Voice agents are not pipelines. They are competing loops.

The line count is honest and checkable. 1,608 lines of code (non-blank, non-comment), 2,188 with blanks and comments, agent only. Run it yourself:

find offleash -name '*.py' -exec grep -vE '^\s*(#|$)' {} + | wc -l

The gap it fills

Telnyx ships every primitive a self-hosted voice agent needs, but no open reference assembles them into a controllable conversational loop with real tool calling. The managed Telnyx AI Assistant is configured in a console, lacking control in core areas. This repo is the same primitives with the leash off:

• Real tool calling: functions you define, dispatched and looped in your own code (see the restaurant demo).
• Model and prompt tuning: the system prompt is rebuilt every turn from live state, and the model is a value you set.
• Custom loop behavior: barge-in, flow nodes, spend caps, and teardown are all yours to change.

It is built in the open, minimal and bulletproof. It does not depend on the reference framework it was lifted from; the code is lifted in, not imported.

Architecture

This is pure Telnyx Call Control orchestration. Speech-to-text and text-to-speech are issued as call commands and their results arrive as webhooks. There is no media WebSocket to manage.

        PSTN
   caller  ────────────►  Telnyx Programmable Voice
     ▲                       │  transport + STT (transcription_start)
     │  speak audio          │         + TTS (speak)
     │                       │
     │   webhooks            │   Call Control commands
     │   call.initiated      │   answer
     │   call.answered       ▼   transcription_start
     │   call.transcription      speak
     │   call.speak.ended        playback_stop   (barge-in)
     │   call.hangup             hangup
     │       │                       ▲
     │       ▼                       │
   offleash server (FastAPI)  ───────┘
   Ed25519-verified webhook
           │
           ▼
   VoiceAgent  ── one event queue per call ──►  turn manager ─► LLM tool loop
           │                                                         │
           └──────────  chat/completions  ──────────────────────────┘
                        Telnyx Inference (Kimi-K2.5, OpenAI-compatible, with tools)

Per call, the loop is:

1. The caller speaks. Telnyx transcribes and posts call.transcription webhooks (interim and final).
2. The server turns those into events on the agent's queue. The turn manager assembles a complete user turn.
3. The agent calls Telnyx Inference, runs the tool loop (executing functions and feeding results back) until the model returns text, then speaks the reply with one speak command.
4. Barge-in: a transcript arriving while the agent is speaking issues playback_stop, cancels the in-flight response, and returns to listening.
5. On a terminal flow node or a caller hangup, the call tears down cleanly.

The files:

File	Role
offleash/agent.py	The conversational loop and state machine: tool loop, flow nodes, barge-in, interrupted-turn rollback, spend caps.
offleash/telnyx.py	Thin typed wrappers over Call Control, the LLM client, and Ed25519 webhook verification. No abstraction.
offleash/server.py	FastAPI webhook receiver and per-call lifecycle dispatcher.
offleash/tools.py	The tool registry and the three restaurant demo tools.
offleash/prompts.py	The system-prompt composer and the Golden Fork agent config.
offleash/settings.py	Flat, env-driven settings.
offleash/{conversation,turn_manager,barge_in,limits,retry,logging,types}.py	The orchestration core, lifted from voice-agent-lite.

A note on barge-in (a finding, not an omission)

This is what a voice agent built only on Telnyx's public primitives actually looks like, and the primitive surface sets the responsiveness floor.

Telnyx speech-to-text (transcription_start) and text-to-speech (speak) are leg-bound Call Control commands, not stream-attachable: transcripts are delivered as webhooks and speech plays directly onto the call leg. There is no way to point Telnyx STT at a media stream. So a purely single-vendor agent interrupts via a transcript-triggered playback_stop, which sets a barge-in floor that is transcript-bound rather than frame-level. Measured over real PSTN calls, in this configuration (Google interim transcription triggering playback_stop), that floor is a median of 1.3 s, about 82% of which is the wait for the first STT interim; offleash's own contribution (decide to stop, issue playback_stop) is about 1 ms. The number is specific to this engine and configuration, not a universal Telnyx figure. The full method, conditions, N, and per-component median/p95/p99 are in BENCHMARK.md. To keep the floor as low as the primitives allow, this build uses the Google transcription engine for interim results and fires the interrupt on the first interim transcript while the agent is speaking, not on the final.

Frame-level barge-in requires running Telnyx Media Streaming for the audio path (bidirectional RTP) and detecting speech onset locally with a VAD, so the agent reacts to the sound of speech rather than waiting for a transcript. That is a different architecture, and it is its own repo: telnyx-onset, the media-stream sibling to this call-command build. offleash is the smaller, call-command implementation; onset is the media-stream implementation with frame-level interruption. Pick offleash for turn-based agents where the caller does not talk over the system, and onset when you need natural, interruption-heavy conversation.

Benchmark: barge-in latency

Barge-in responsiveness is measured, not asserted. BENCHMARK.md documents the method (a controlled-caller harness, "Option 1"), the conditions, N, and median/p95/p99 per component, with the bottleneck decomposed. The harness, the raw per-event data, and the analysis live in bench/ on main, kept separate from the runtime. It runs the server in bench mode, dials the agent from a second number on the same connection, bridges the legs, and injects a barge-in stimulus at a known time. Run it:

# Stop any normal server on the port first; the harness must own it.
python -m offleash bench --from +1XXXXXXXXXX --events 25 --label run1 --out bench/data/run1.jsonl
python -m bench.analyze bench/data/run1.jsonl

The restaurant demo

The default agent is Ava, the reservations assistant for "The Golden Fork", recreated faithfully from the voice-agent-lite demo so the two are a direct comparison. It runs a three-node flow (booking, then confirm, then a terminal farewell) over three tools: check_availability, make_reservation, and get_menu. The reservation confirmation number is deterministic (a crc32 of the name) so a given caller always gets the same number.

A happy path: the agent greets, collects date, time, and party size, checks availability, asks for a name, makes the reservation, reads back the confirmation, and hangs up.

Setup

You need a Telnyx account. One V2 API key authenticates Call Control, transcription, speak, and inference.

1. Create an API key. Portal > Account Settings > API Keys > Create API Token. Copy it once.
2. Get the webhook public key. Portal > Account Settings > Keys & Credentials > Public Key. This is the Ed25519 key used to verify webhooks.
3. Create a Voice API (Call Control) Application. Portal > Voice > Programmable Voice > Call Control Applications. Set its webhook URL to {your public https url}/webhook. Attach an Outbound Voice Profile if you want to place outbound calls. The Application id is your TELNYX_CONNECTION_ID.
4. Buy a phone number and assign it to that application (for inbound).
5. Expose your local server publicly. For local development, run a tunnel such as ngrok http 8000 and set the application's webhook URL (in the Telnyx portal) to {that https url}/webhook. The webhook URL lives in the portal, not in the app's environment.

Then configure the environment. Copy .env.example to .env and fill it in:

TELNYX_API_KEY=          # one key for Call Control and Inference
TELNYX_PUBLIC_KEY=       # Ed25519 public key for webhook verification
TELNYX_PHONE_NUMBER=     # E.164, the "from" number for outbound
TELNYX_CONNECTION_ID=    # Voice API Application id

Everything else (model, voice, STT engine, limits) has sensible defaults in .env.example.

Run

With uv:

uv run --extra dev offleash

Or with a plain virtualenv:

python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
python -m offleash

The server listens on PORT (default 8000) and serves the signed webhook at /webhook and a health check at /health. Point your Telnyx application's webhook URL at {your public https url}/webhook and call your number.

To place an outbound call (the server must be running to handle the resulting webhooks):

python -m offleash call +1XXXXXXXXXX

Tests

uv run --extra dev pytest

The suite covers the tools, Ed25519 webhook verification, the full reservation flow, barge-in and recovery, speak-generation correlation, the token budget, the tool-round cap, LLM-failure fallback, response supersession, capacity rejection, and clean teardown, all with fakes so no Telnyx account or network is needed.

Telnyx gotchas (things the docs don't flag)

These are the non-obvious Telnyx requirements this agent ran into in practice. Each one cost real time and none are called out in the docs:

• Use a Call Control (Voice API) application, not a TeXML application. TeXML posts form-encoded webhooks and expects XML back; this agent needs JSON event webhooks and drives the call with REST commands. If a form-encoded webhook arrives, the server logs webhook.texml_application_detected.
• The account must be verified to at least Level 2. Trial accounts prepend an abuse notice onto all Telnyx machine-generated speech, so the agent's voice is replaced by that notice. Add a payment method and complete verification.
• The transcription language is a short code such as en. The transcription API rejects en-US (which the speak command, confusingly, does accept), so the speak and transcription languages are separate settings.

Pipeline options

Each stage is env-swappable. The defaults below are chosen for this architecture (Option A: pure Call Control, transcript-triggered barge-in), not just for raw quality. The current Telnyx catalog is larger than what ships as default; the trade-offs that actually bite this design are called out per stage.

STT - transcription_start (default: Google / en)

Barge-in here fires on the first interim transcript, so it depends on the STT engine forwarding interim results through Call Control transcription webhooks. Google reliably forwarded interims in our tests and is the engine used in the published benchmark. Other engines showed inconsistent interim-forwarding across our tests: nova-3 delivered zero interims in one test and twelve in another, and Soniox and Deepgram Flux were single-test observations that remain unverified. We are not publishing an engine comparison until this is characterized properly with a real sample size. If you select a non-Google engine, verify interim delivery on your own account before relying on transcript-triggered barge-in.

Select an engine with STT_ENGINE and a specific model with STT_MODEL (blank uses the engine's default). The pairing is validated against a known matrix at startup (settings.py), so a bad combination fails fast instead of 422-ing on the first call. The agent only requests interim results from engines the matrix marks as interim-capable; when interims do not arrive, barge-in falls back to final transcripts. Deepgram Flux exposes native turn detection tunable with STT_EOT_THRESHOLD / STT_EOT_TIMEOUT_MS (range-validated, applied only for Deepgram+flux). Note that in our single test Flux did not forward interims through this surface, and interim behavior for non-Google engines is under re-test; the agent still derives end-of-turn from the is_final flag, so consuming Flux's native EndOfTurn event would mean teaching turn_manager.py to key off Flux's turn signal, a separate follow-up.

LLM - Telnyx Inference, OpenAI-compatible (default: moonshotai/Kimi-K2.5)

Kimi-K2.5 (non-reasoning; enable_thinking is disabled in code for latency) is the benchmarked project default. Telnyx currently recommends Kimi-K2.6 for voice AI. Levers, via LLM_MODEL:

• moonshotai/Kimi-K2.6 - Telnyx's current recommendation for highest-intelligence voice AI; enable_thinking remains disabled for latency.
• MiniMaxAI/MiniMax-M3-MXFP8 - Telnyx's lowest-cost option while maintaining high intelligence; pick this to optimize spend.
• zai-org/GLM-5.2 - optimized for coding and reasoning with a 1M-token context window; Telnyx especially recommends it for function calling.

Model IDs are the ones the account's GET /v2/ai/models actually returns; verify availability there if you pin a different one.

TTS - speak (default: Telnyx.NaturalHD.astra)

Voice (TTS_VOICE)	Tier	Notes
Telnyx.NaturalHD.astra (default)	Telnyx-native HD	Higher fidelity, multilingual, handles "um/uh" and laughter; low-latency.
Telnyx.Natural.abbie	Telnyx-native	Lowest latency and cost, English-only.
Telnyx.Ultra.<uuid>	Telnyx-native	Most expressive; voices are UUID-identified.
ElevenLabs.* / AWS.Polly.*-Neural / Azure.*Neural	third-party	Premium / neural passthrough voices.

All TTS options are pure env swaps - no code change. speak accepts the full locale (TTS_LANGUAGE=en-US), unlike transcription which wants the short code.

Related

voice-agent-lite is the vendor-neutral, swappable framework this agent was lifted from: multiple STT, TTS, and LLM providers behind clean interfaces, with Twilio and Telnyx transports. If you want to compare providers or keep your options open, start there. This repo is the opposite bet: one vendor, no abstraction, the smallest honest implementation of a Telnyx-only voice agent.