vLLM Chatterbox Stream — OpenAI-compatible TTS with real streaming

Multilingual text-to-speech (Chatterbox) served on vLLM, exposed through the OpenAI /v1/audio/speech API with real audio streaming and low time-to-first-byte. 23 languages, voice cloning.

• Two-stage TTS: an autoregressive T3 model (a Llama backbone) emits speech tokens — served by vLLM's AsyncLLMEngine with continuous batching — and S3Gen (CosyVoice2 flow-matching decoder + HiFi-GAN vocoder) turns them into a 24 kHz waveform.
• Real streaming: with stream: true the server emits raw PCM chunks as they are generated, so playback starts in ~1 second instead of waiting for the whole clip.

More detail: docs/serving.md (full API & tuning) · docs/maintainers-runbook.md (ops) · docs/technical-primer.md (how it works).

---

Install

Linux + NVIDIA GPU. Requires git and Python 3.10+ (and uv for the fast path).

git clone https://github.com/wuxuedaifu/vllm-chatterbox-stream.git
cd vllm-chatterbox-stream
uv venv && source .venv/bin/activate
uv pip install -e .            # or: pip install -e .

Model weights download automatically from the Hugging Face Hub on first run.

Start the server

chatterbox-serve --variant multilingual --host 0.0.0.0 --port 8000

Recommended low-latency flag (≈12% faster TTFB, identical audio):

CHATTERBOX_T3_CUDA_GRAPHS=1 chatterbox-serve --variant multilingual --port 8000

Flag	Default	Meaning
--variant	multilingual	multilingual (23 langs) or english
--host / --port	0.0.0.0 / 8000	bind address
--voices	voices/voices.json	voice-preset file
--max-batch-size	16	vLLM max_num_seqs + scheduler concurrency budget
--max-model-len	1000	max speech tokens per request

---

Run with Docker

The image bundles the server and runs chatterbox-serve on port 8000. An NVIDIA GPU and the NVIDIA Container Toolkit are required at runtime.

Build:

docker build -t vllm-chatterbox-stream .

Run (mount a Hugging Face cache so weights download only once):

docker run --gpus all -p 8000:8000 \
  -v $HOME/.cache/huggingface:/root/.cache/huggingface \
  vllm-chatterbox-stream

Enable the low-latency CUDA-graphs path:

docker run --gpus all -p 8000:8000 \
  -e CHATTERBOX_T3_CUDA_GRAPHS=1 \
  -v $HOME/.cache/huggingface:/root/.cache/huggingface \
  vllm-chatterbox-stream

The server is then reachable at http://localhost:8000/v1/audio/speech.

---

Real streaming & TTFB

With stream: true the response is a chunked audio/pcm body: T3 speech tokens are consumed as deltas and an S3GenStreamer re-decodes the growing prefix into ~1 s audio chunks (HiFi-GAN cache_source + overlap-fade), so the first audio arrives long before generation finishes.

Measured on one A100 (vLLM 0.10.0):

Metric	Streaming	Non-streaming
Time-to-first-byte (Arabic, ~17 s clip)	≈ 1.1 s	≈ 7.6 s (~7× slower)
Single-request TTFB p50 (short EN, CUDA graphs on)	≈ 0.68 s	—
Inter-chunk gap	≈ 0.6 s	n/a
Real-time factor (RTF)	≈ 0.5	≈ 0.5

TTFB tuning (streaming)

Setting	Default	Effect (measured)
Startup warmup	on	Throwaway generation at startup so the first real request skips cold-start (~1.5 s → ~0.81 s). Disable with CHATTERBOX_SKIP_WARMUP=1.
CHATTERBOX_T3_CUDA_GRAPHS=1	off	vLLM CUDA graphs for T3. TTFB 809 → 714 ms (−12%), audio identical (corr 0.9999). Recommended on.
CHATTERBOX_STREAM_FIRST_HOP	12	Smaller first hop emits the first chunk sooner (tune to taste).
CHATTERBOX_STREAM_NTIMESTEPS	—	Fewer CFM decode steps roughly halve decode time (some quality cost).

Note: PCM is currently the only streaming format (mp3/opus containers are non-streaming for now). Streaming re-decodes the prefix each chunk (O(N²) total compute — fine for sentence-length inputs; incremental caching is a future optimization).

---

Concurrency

chatterbox-serve runs on vLLM's AsyncLLMEngine with continuous batching. The scheduler admits up to --max-batch-size (default 16) requests so they batch together; a separate, smaller semaphore bounds concurrent S3Gen decodes (CHATTERBOX_S3GEN_CONC, default 2) so they don't thrash VRAM.

Closed-loop HTTP load test (A100, T3 CUDA graphs on, 79-char EN input) — at each concurrency level N, N virtual users each stream a request, await completion, and repeat for a fixed window. Metrics over successful requests:

Concurrency (N)	TTFB p50	TTFB p95	Throughput (req/s)	Errors
1	0.68 s	0.71 s	0.45	0%
2	1.07 s	1.19 s	0.33	0%
4	1.98 s	—	0.47	0%
8	3.72 s	—	0.43	0%
16	4.47 s	—	0.37	0%
32	18.6 s	—	0.45	0%

TTFB stays roughly flat up to the batch capacity (16); beyond it, requests queue and TTFB climbs — raise --max-batch-size (more VRAM) or add GPU replicas. Throughput holds at ~0.4 req/s regardless of N: that is the single-GPU compute ceiling (≈2 concurrent requests already saturate the GPU), not a bug. Scale total throughput with faster per-request compute (CUDA graphs, fewer CFM steps) or more replicas.

Reproduce with test_scripts/concurrency_test.py (closed-loop tester; writes a CSV + Markdown table).

---

API endpoints

Method & path	Purpose
POST /v1/audio/speech	synthesize speech (non-streaming or streaming)
GET /v1/models	list the served model (chatterbox)
GET /health	readiness probe → {"status":"ok"}

Usage

curl — non-streaming (returns the whole clip):

curl -s -X POST http://localhost:8000/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{"model":"chatterbox","input":"Hello from Chatterbox.","voice":"uae_man","language":"en","response_format":"mp3"}' \
  -o out.mp3

curl — streaming (low TTFB; raw PCM chunks piped straight to a player):

curl -N -X POST http://localhost:8000/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{"model":"chatterbox","input":"مرحبا بالعالم","voice":"uae_man","language":"ar","stream":true,"response_format":"pcm"}' \
  --output - | aplay -r 24000 -f S16_LE -c 1

OpenAI Python client (Chatterbox extras go through extra_body):

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

# Non-streaming
resp = client.audio.speech.create(
    model="chatterbox",
    voice="uae_man",
    input="Hello from Chatterbox.",
    response_format="mp3",
    extra_body={"language": "en", "exaggeration": 0.5},
)
resp.stream_to_file("out.mp3")

# Streaming: write PCM chunks as they arrive
with client.audio.speech.with_streaming_response.create(
    model="chatterbox",
    voice="uae_man",
    input="Hello from Chatterbox.",
    response_format="pcm",
    extra_body={"language": "en", "stream": True},
) as resp:
    with open("out.pcm", "wb") as f:
        for chunk in resp.iter_bytes():
            f.write(chunk)   # raw int16, 24 kHz, mono

Request — POST /v1/audio/speech (JSON)

Field	Type	Default	Meaning
model	string	chatterbox	model id
input	string	required	text to synthesize
voice	string	default	preset name from voices/voices.json
reference_audio	string	null	base64 or http(s) URL clip to clone (overrides voice)
response_format	string	mp3	mp3 | wav | opus | flac | pcm | aac
stream	bool	false	stream audio chunks as generated (use pcm)
speed	float	1.0	playback speed, 0 < speed ≤ 4
language	string	en	one of 23 codes (see below)
exaggeration	float	0.5	emotion intensity, 0–1
vad	bool	true	drop hallucinated non-speech (Silero VAD; non-streaming)
temperature	float	0.8	sampling
top_p / min_p / repetition_penalty	float	0.8 / 0.1 / 2.0	sampling

Response

• Non-streaming: HTTP body = the audio file, Content-Type: audio/<response_format>.
• Streaming (stream: true): chunked audio/pcm body emitted as generated — raw little-endian int16, 24 kHz, mono.
• Errors: JSON {"error": "..."} — 400 (bad voice/language), 503 (engine not ready), 500 (other).

Voices: edit voices/voices.json (name → reference clip path, or null for the built-in default). Keep reference clips ~6–10 s of clean speech.

---

Language samples

23 languages, all generated by this server. Codes: ar da de el en es fi fr he hi it ja ko ms nl no pl pt ru sv sw tr zh.

Language	Code	Sample	Language	Code	Sample
Arabic	ar	▶	Japanese	ja	▶
Danish	da	▶	Korean	ko	▶
German	de	▶	Malay	ms	▶
Greek	el	▶	Dutch	nl	▶
English	en	▶	Norwegian	no	▶
Spanish	es	▶	Polish	pl	▶
Finnish	fi	▶	Portuguese	pt	▶
French	fr	▶	Russian	ru	▶
Hebrew	he	▶	Swedish	sv	▶
Hindi	hi	▶	Swahili	sw	▶
Italian	it	▶	Turkish	tr	▶
			Chinese	zh	▶

Regenerate the set with test_scripts/multilang_test.py.

---

Known limitation — audio hallucinations

Chatterbox itself has a known defect: generated speech can occasionally contain hallucinated, blurry, or garbled non-speech artifacts (buzzing, breaths, or meaningful-sounding rambling past the end of the text). The non-streaming path mitigates this with a Silero VAD post-filter (enabled by default; disable per request with "vad": false) that keeps only speech regions, plus a text-proportional max_tokens cap to bound rambling. These reduce the problem substantially but do not eliminate it entirely — it is an upstream model limitation, not a server bug. VAD is not yet applied to the streaming path (it needs a causal hangover buffer — planned).

---

License

MIT. Chatterbox © 2025 Resemble AI; vLLM port © 2025 David Jia Wei Li. See LICENSE.