You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
feat(voice): add VoxCPM2 (OpenBMB) as a fifth TTS backend candidate [spec 02] (#10)
Adds VoxCPM2 (mlx-community/VoxCPM2-8bit) to the MLX backend registry as a
post-L0 blind-A/B candidate. The thin MlxAudioBackend + profile seam means the
whole model is one PROFILES row — smoke-verified end-to-end through the standard
SynthesisRequest -> wav path: loads (mlx-audio #649 does not block), 48 kHz
native, measured RTF ~=1.6 on M3 Pro/8bit (slower than real-time -> a
quality-reference / pre-generated candidate, not the real-time default).
Also fixes a registry-drift bug the review caught: the CLI --voice choices were
hardcoded and rejected 'voxcpm2' even though build_voice() accepted it. Now
derived from PROFILES so future rows are CLI-selectable automatically, guarded
by a new regression test.
- src: PROFILES row; --voice choices derived from the registry
- spec 02 §3.3/§4/§5/§6 + DESIGN shortlist + README/pyproject aligned
- tests (test-first): registry/integration params + CLI-accepts-every-voice guard
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: specs/DESIGN.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -108,7 +108,7 @@ Each item records the **why**, to avoid re-litigating later.
108
108
109
109
### 3.5 Output / TTS: hot-swappable, human-ness is the soul
110
110
-**`VoiceProvider` abstraction**: TTS is a hot-swappable backend, not hard-coded. Each model is its own adapter, switchable by one config line; you can even **mount different models per scenario** (a fast one for live replies, a warm/rich one for proactive broadcasts).
111
-
-**Candidate pool** (decide the primary after a blind A/B): Qwen3-TTS, CosyVoice2, Chatterbox Multilingual V3, OpenAudio S1-mini. *(spec 02 wires the MLX-runnable experiment shortlist — **Spark**[primary], Qwen3-TTS, Chatterbox, Dia — as local experiment-phase voices per §3.7.)*
111
+
-**Candidate pool** (decide the primary after a blind A/B): Qwen3-TTS, CosyVoice2, Chatterbox Multilingual V3, OpenAudio S1-mini. *(spec 02 wires the MLX-runnable experiment shortlist — **Spark**[primary], Qwen3-TTS, Chatterbox, Dia, and VoxCPM2 — as local experiment-phase voices per §3.7.)*
112
112
-**TTS runs as an always-on warm sidecar process.***Rationale*: models load slowly (seconds, several GB), so keep them warm rather than loading on every utterance; crash isolation — a TTS crash must not take down the radio brain; cross-process is also the cleanest seam for hot-swapping.
113
113
-*Selection notes (from mid-2026 research)*:
114
114
-**Licensing is deferred to the two-phase strategy (§3.7)**: during local experimentation, non-commercially-licensed models (Spark/CosyVoice2/Fish/IndexTTS2, etc.) are fine to try; the *distributable* voice is a paid/licensed choice made at distribution time. So the experiment pool is unconstrained by license — an experiment pick is not a commitment to ship it.
@@ -281,7 +281,7 @@ v1 ships as **a sequence of sub-specs**, ordered so that **every step runs and a
281
281
| # | sub-spec (`specs/specNN/…`) | Part it delivers | Milestone | Depends on |
282
282
|---|---|---|---|---|
283
283
|**01 ✅**|`core-loop`| Single-process spine: CLI Host + Program Director (talk-only policy) + Brain (Claude SDK, subscription auth) + static persona load + typed talk-back + session-only history + AudioPlayer (basic, sole audio authority, manual stop) + segment cadence. **Declares the outbound interface contracts** (VoiceProvider / MusicProvider / Memory seams). |**L0**| — |
|**03-01**|`brain-harness`| The general **brain-harness** seam — turn the isolated brain into a tool/skill-using agent (§3.2), preserving local-env isolation, with a fast content-agnostic context-insertion mechanism. **First capability: Claude-driven music discovery** — the harnessed brain searches, judges, and **pulls** a track (`MusicProvider` impl; yt-dlp default, musicdl optional), returning an `AudioClip`. **Find + pull only — no playback/scheduling/announce** (those are 03-02). |**L1** (radio feel) | 01 |
286
286
|**03-02**|`ducking`| Source-agnostic mixing audio **engine** (replaces the afplay `AudioPlayer`): ffmpeg→PCM + numpy mix + gain-envelope **ducking**; a typed interjection **ducks** music instead of hard-stopping it. **Owns music playback + Director talk↔music scheduling + optional DJ "up next" announce** — the tracks 03-01 pulls are scheduled and played here. |**L1** (radio feel) | 01, 03-01 |
287
287
|**03-03**|`guide-harness`| A second capability on the harness (03-01): shape the **native Claude Code agent** (built-in tools, step-by-step `default` confirmation — never `bypassPermissions`) to diagnose and, with user consent, **fix why the music dependency (yt-dlp) isn't working in the user's environment** (e.g. a corporate proxy's untrusted CA). Deterministic preflight triggers it; confirmations flow through the CLI Host (no TUI). Makes 03's music **actually usable** on constrained machines. |**L1** (music works everywhere) | 01, 03-01 |
> -**steps 1–2 (done)**: standardized adapter boundary (`SynthesisRequest` §3.5), JSON-lines-over-stdio sidecar (`python -m murmur.voice.sidecar`) with a `TtsBackend` interface + no-model `FakeBackend`, and the supervising `SidecarVoiceProvider` (spawn/wait-for-ready, restart-on-death + retry, synth-timeout kills the proc to avoid pipe desync). `build_voice`: `stub` / `qwen3` / `sidecar-fake`; `--voice` flag. Verified by 63 unit tests + a real two-process end-to-end run on `sidecar-fake`. Acceptance §3 (kill→recover) and §4 (hot-swap) are covered by the `sidecar-fake` path.
5
-
> -**step 3 (code in place, not yet verified)**: a **thin generic `MlxAudioBackend`** over `mlx-audio` (optional `tts-mlx` extra; lazy-imported) + a **profile registry** wiring four backends — `spark` (primary), `qwen3`, `chatterbox`, `dia`(§3.3). The deterministic layer (request→`generate` kwarg mapping, profile merge, backend selection) is unit-tested with a fake model; real model load/synth is a parametrized tagged integration test (`pytest -m integration`). Acceptance §5.1 ("sounds clearly human") / §5.2 ("warm") + the blind A/B among the four gate on a hands-on Mac run (install the extra, download models, judge by ear) — the agent cannot self-verify a voice.
5
+
> -**step 3 (code in place, not yet verified)**: a **thin generic `MlxAudioBackend`** over `mlx-audio` (optional `tts-mlx` extra; lazy-imported) + a **profile registry** wiring the backends — `spark` (primary), `qwen3`, `chatterbox`, `dia`, plus the post-L0 `voxcpm2` candidate (§3.3). The deterministic layer (request→`generate` kwarg mapping, profile merge, backend selection) is unit-tested with a fake model; real model load/synth is a parametrized tagged integration test (`pytest -m integration`). Acceptance §5.1 ("sounds clearly human") / §5.2 ("warm") + the blind A/B among the candidates gate on a hands-on Mac run (install the extra, download models, judge by ear) — the agent cannot self-verify a voice.
6
6
> **Part**: The `VoiceProvider` implementation + the warm TTS sidecar + the first adapter. See master [`../DESIGN.md`](../DESIGN.md) §3.5 (TTS = soul, pluggable, warm sidecar), §4 (architecture).
7
7
> **Milestone**: L0 (01+02 = the first audible version).
8
8
> **Conventions**: English; written for a coding agent. Design-level — mechanism and contracts, not final code.
@@ -60,9 +60,10 @@ contract:
60
60
the payload tiny (text in, file path out) — do not stream audio bytes over the
61
61
IPCin L0.
62
62
63
-
### 3.3 Backends — a thin MLX layer over four models (Spark primary)
63
+
### 3.3 Backends — a thin MLX layer over the candidate models (Spark primary)
64
64
L0 wires **four**TTS backends, all on Apple Silicon via **`mlx-audio`** (the Mac
65
-
hub for these models, master §3.5). Because all four share one runtime and one
65
+
hub for these models, master §3.5); **VoxCPM2** (OpenBMB) was added post-L0 as a
66
+
fifth blind-A/B candidate (§5). Because all of them share one runtime and one
66
67
`load_model(repo) → model.generate(text, …)`API, they are served by a **single
67
68
generic `MlxAudioBackend`** parameterized by a per-model **profile** — not one
68
69
class per model. This is the "thin middle layer": the model-specific differences
@@ -74,6 +75,7 @@ collapse into a config profile + the `params` escape hatch (§3.5).
74
75
|`qwen3`|`Qwen3-TTS-12Hz-0.6B-Base-bf16`|0.6B| multi (zh) |24 kHz; voice presets / voice-design. |
75
76
|`chatterbox`|`chatterbox-fp16`|~0.5B| multi (en-strong) | expressive, emotion-exaggeration control. |
76
77
|`dia`|`Dia-1.6B-fp16`|1.6B| en | ultra-real dialogue/emotion (English wildcard). |
78
+
|`voxcpm2`|`VoxCPM2-8bit`|2B| multi (zh) | tokenizer-free, 48 kHz native (confirmed); strong naturalness +long-form continuation + native streaming (`generate_streaming`). Heaviest — **measured RTF ≈ 1.6 on M3 Pro /8bit** (whole-clip, warm), i.e. slower than real-time → a **quality-reference / pre-generated** candidate, **not** the real-time default. 8bit for the A/B; drop to `VoxCPM2-4bit`ifRTF must improve. |
77
79
78
80
- Repo ids are the L0 defaults and**confirmed on first hands-on run** (they can shift; §6). All are open-weight and local; licensing is per the **two-phase model strategy** (master §3.7) — any good open model is fair game during local experimentation, and the *distributable* voice is a paid/licensed choice made at distribution time (so e.g. Spark's CC-BY-NC is fine to experiment with now, not a commitment to ship).
79
81
- The sidecar loads the selected model at `start()`and warms it with one throwaway synth so the first real `synthesize`is fast.
@@ -110,7 +112,7 @@ class SynthesisRequest:
110
112
fields it supports and ignores the rest.
111
113
112
114
**The thin middle layer — one generic backend + a profile registry.** Because
113
-
the four L0 models (§3.3) all run on `mlx-audio` through the same
115
+
the L0 models plus the VoxCPM2 candidate (§3.3) all run on `mlx-audio` through the same
114
116
`load_model(repo) → model.generate(text, …)`API, they are **not** four classes
115
117
— they are **one `MlxAudioBackend`** whose only per-model state is a **profile**:
116
118
```python
@@ -123,29 +125,29 @@ class MlxProfile: # one row per model; adding a model = adding a row
123
125
```
124
126
`MlxAudioBackend.synthesize` merges the profile defaults with the incoming
125
127
`SynthesisRequest` (request wins), maps them to `generate()` kwargs, renders the
126
-
whole clip (§3.4), and writes the wav. The registry (`spark`/`qwen3`/`chatterbox`/`dia`)
128
+
whole clip (§3.4), and writes the wav. The registry (`spark`/`qwen3`/`chatterbox`/`dia`/`voxcpm2`)
127
129
lives in one place; `build_backend(name)` looks up the profile and constructs the
128
130
one backend. A future non-MLX model (e.g. a PyTorch CosyVoice2/Fish for
129
131
pre-generation) would be a *separate*`TtsBackend`in its own process/env, but the
130
132
core contract andIPC are still unchanged.
131
133
132
134
-**Zero-shot voice cloning** (`reference_audio`/`reference_text`) stays a *designed-for axis*, **not wired in L0** — L0 uses preset voices. The slot exists so a cloning backend needs no contract change.
133
-
-`config` (01 §3.1) selects the backend by name; the profile supplies the per-model defaults used (with the core's `synthesize(text, scenario=…)` call) to build the `SynthesisRequest`. The core contract and IPC are unchanged when swapping backends — proven by hot-swapping among the four.
135
+
-`config` (01 §3.1) selects the backend by name; the profile supplies the per-model defaults used (with the core's `synthesize(text, scenario=…)` call) to build the `SynthesisRequest`. The core contract and IPC are unchanged when swapping backends — proven by hot-swapping among the candidates.
134
136
135
137
---
136
138
137
139
## 4. Dependencies
138
140
- [`01-core-loop.md`](../spec01/01-core-loop.md) — owns the `VoiceProvider`/`AudioClip` contract this implements.
139
-
- External (Mac): MLX+`mlx-audio` (one optional extra `tts-mlx` covers allfour backends) + each model's weights (downloaded once from HF/ModelScope). No network **at inference time** (local models) — consistent with master §3.1 ("only network hops: inference + music"; TTS is local). The weight download is one-time setup, not a runtime hop.
141
+
- External (Mac): MLX+`mlx-audio` (one optional extra `tts-mlx` covers allthe MLX backends) + each model's weights (downloaded once from HF/ModelScope). No network **at inference time** (local models) — consistent with master §3.1 ("only network hops: inference + music"; TTS is local). The weight download is one-time setup, not a runtime hop.
140
142
141
143
---
142
144
143
145
## 5. Acceptance criteria
144
146
1. With the sidecar started, `synthesize("…")` returns an `AudioClip` the core can play, and the speech sounds **clearly human** (not robotic `say`-tier) — the L0 bar for"soul."
145
147
2. The model is**warm**: the second and later `synthesize` calls do **not**reload the model; per-call latency is small enough that the talk loop feels live on the target Mac.
146
148
3. Killing the sidecar process does **not** crash the core; the core reports the failure and recovers (restart) on the next call.
147
-
4. Switching the configured backend name (`spark`/`qwen3`/`chatterbox`/`dia`) changes the voice **without any change to spec-01 code** (proves the hot-swap seam) — the thin `MlxAudioBackend`+ profile registry serve all four.
148
-
5. **Blind A/B among the four** (master §10.3eval track): render the same Chinese line through each and pick the primary by ear. `spark` leads going in; watch its 16 kHz output against `qwen3`'s 24 kHz.
149
+
4. Switching the configured backend name (`spark`/`qwen3`/`chatterbox`/`dia`/`voxcpm2`) changes the voice **without any change to spec-01 code** (proves the hot-swap seam) — the thin `MlxAudioBackend`+ profile registry serve them all.
150
+
5. **Blind A/B among the candidates** (master §10.3eval track): render the same Chinese line through each and pick the primary by ear. `spark` leads going in; watch its 16 kHz output against `qwen3`'s 24 kHz and `voxcpm2`'s 48 kHz (VoxCPM2 is the quality contender but ~3× slower — judge whether its naturalness earns the latency, or only for pre-generated segments).
149
151
150
152
---
151
153
@@ -154,3 +156,5 @@ core contract and IPC are still unchanged.
154
156
- Audio handoff: file-path (chosen for L0) vs shared-memory/streamed PCM (lower latency, needed if/when streaming TTS lands in spec 04).
155
157
- Exact mlx-community repo ids + per-model `generate` kwargs (voice presets, Dia's `[S1]/[S2]` speaker tags, Chatterbox `exaggeration`/`cfg`, Spark gender/pitch): the §3.3 repos are best-effort defaults, confirmed/tuned on the first hands-on run. Each maps onto `SynthesisRequest` fields + `params`; the profile registry absorbs the per-model differences.
156
158
- Exact Qwen3-TTS voice/preset selection andany Chinese/English voice mapping — deferred to first hands-on run.
159
+
-**VoxCPM2 quant**: candidate wired at `mlx-community/VoxCPM2-8bit` (~3.2GB disk, ~4–5GB resident on the target 18GB Mac). 8bit is deliberate — the blind A/B should hear the model near full quality; `VoxCPM2-4bit` (~2.3GB) is the fallback only if real-time RTF on M3 Pro forces it. Resolved for now; revisit after the hands-on RTF check.
160
+
-~~**VoxCPM2 mlx-audio load risk** ([Blaizzy/mlx-audio#649](https://github.com/Blaizzy/mlx-audio/issues/649))~~ **Resolved by smoke**: the installed `mlx-audio` loads `voxcpm2` fine (only a benign `transformers` model_type warning) and synthesizes a real 48 kHz wav through the standard seam — verified via a throwaway `scratch/` smoke, and covered on-demand by `test_mlx_backend_renders_a_real_nonempty_wav[voxcpm2]` (`pytest -m integration`). Re-check only if the pinned `mlx-audio` version changes.
0 commit comments