Add Echo-TTS (community model)

[5uck1ess]

Ready for review. The DiT denoiser passes its parity gate against PyTorch; the remaining gaps are listed honestly under What's still missing, and the two open questions are structural ones only you can answer — see Scope.

Note on ENGINE_BUILD_TESTS: a full build currently fails on three tests/moss_tts_local/*_parity.cpp files. That is pre-existing on main, not from this branch — those files and every moss source here are byte-identical to upstream, and I reproduced the identical error in a clean upstream/main worktree configured from scratch. Flagging it so it isn't attributed to this PR.

Adds Echo-TTS as a community model: an English zero-shot voice-cloning TTS model. A 2.8B diffusion transformer generates 80-D latents in PCA space, decoded to 44.1 kHz by the Fish S1-DAC autoencoder. Cloning needs a reference wav, no transcript. Text is byte-level UTF-8 — no phonemiser, no G2P, no pronunciation dependency.

Echo-TTS is on the candidate list in #34 (struck through, "contributions are welcome").

Credit

The implementation is @dignome's. They built it from this PR's design notes, published it at dignome/audio.cpp-echo-tts, and offered it here rather than opening a competing PR. I've integrated it, reviewed it, cut the parts that don't belong upstream, and fixed what the reviews found. Their commits are preserved as Co-authored-by.

Evidence

Numerical parity against PyTorch

tests/echo_tts/echo_tts_dit_parity.cpp against a dump from tools/community_models/echo_tts_reference.py. Gates are cosine over the flattened tensors and max-absolute-error — cosine alone cannot see a uniform scale error, and the host Philox stream matches CUDA to ~2 ULP rather than bit-exactly.

The reference defaults to bfloat16; this GGUF is F16. Both dumps are shown because the difference between them turns out to be the single largest term:

Check	vs bf16 ref	vs f16 ref	Gate	Verdict
Denoiser, one conditional forward at t = 0.7	0.999977	0.999999188, max-abs 0.010	≥ 0.999	PASS
40-step sampler, reference initial noise injected	0.913082	0.988459, max-abs 1.07	≥ 0.999	below gate
40-step sampler, our own seeded draw	0.905481	0.976972, max-abs 1.87	≥ 0.999	below gate

The denoiser probe is the number that carries this port. It settles the four details that fail silently — half-head RoPE, the interleaved (not NEOX) rotary pairing, the speaker patchify reshape, and the adaLN shift/scale/gate order.

It does not isolate the DiT blocks alone: the reference text ids and speaker latents are injected, but prepare_conditioning() then runs this port's own text encoder, speaker encoder and KV projections. Enough to catch a wrong block, not enough to localise one. Per-block dumps exist in the packer (--blocks) and have not been run.

The 40-step trajectory is below the gate and is reported as a FAIL of the check as written. What is established:

• Not the RNG — injecting the reference's own initial noise scores no better than our seeded draw (0.988 vs 0.977).
• Dominated by dtype — re-dumping the reference at float16 moves the trajectory 0.905 → 0.977 and the denoiser 0.999977 → 0.999999.
• Compounds with step count — 0.9965 at 4 steps, 0.9055 at 40 (bf16 ref). Monotonic degradation is the signature of accumulating per-step rounding amplified by dual CFG at 3.0/8.0, not a structural defect.
• A line-by-line review of the sampler against inference.py found no defect — schedule, inclusive CFG bounds, single application of truncation_factor, three-lane CFG combination, the Euler update, and the speaker-KV boundary all agree.

That is an explanation, not a proof. Treat the trajectory as unverified.

End to end

Executed on an RTX 3090 (sm_86), CUDA, F16 GGUF converted locally from jordand/echo-tts-base + jordand/fish-s1-dac-min:

Check	Result
Conversion	manifest OK — 1117 DiT tensors, 219 blockwise tensors dropped, 495 codec tensors
GGUF verifier	pass — 1614 tensors, F16 1043 / F32 571 (norm weights held at F32)
latent_scale	0.0555555559694767 (= 1/18), matching the reference
Generation	exit 0, 44 100 Hz mono, no NaNs, peak 0.80
ASR round-trip, 32 words	WER 0.0 %, 0 edits (faster-whisper-large-v3-turbo)
Throughput	9.195 s audio in 7.89 s wall — RTF 0.86 cold, including the 5.5 GB load

WER on 32 words is a small sample and scores intelligibility only. It shows the pipeline runs end to end and produces the right words; it does not score speaker identity, so a wrong patchify reshape could yield fluent, correctly-worded speech in the wrong voice and still read 0 %. The denoiser cosine above is what covers that.

Generation cost is essentially constant between a 15-word and a 32-word run (7.75 s vs 7.89 s) because the window is fixed at 640 frames, so longer text inside a chunk is close to free.

Independent reviews

Two rounds, Codex (GPT-5.6) and Grok 4.6 each time — first on the contributed tree before adopting it, then on my own verification commits. Codex checked the ggml port line-by-line against the reference PyTorch; Grok judged adoption and upstream fitness.

Codex cleared the four items the original author flagged as possibly-silently-wrong: half-head RoPE rotates the first heads/2 on the head axis (model.py:199,217), GGML_ROPE_TYPE_NORMAL is the correct interleaved convention against model.py:21's reshape(...,-1,2) pairing, the patchify reshape gives frame-then-channel ordering per model.py:458, and adaLN order is shift/scale/gate per model.py:64. Its one CANNOT-TELL — tensor names — was settled by running the converter against the real checkpoint.

Both then independently flagged the same feature, and Codex supplied the mechanism:

The adaptive generation window is now off by default. It shrank the 640-frame window to a text-length estimate, justified in-comment as costing "time, never fidelity" because a missing flattening point retries at full length on the same Philox seed. The noise claim is true; the fidelity conclusion doesn't follow, because the seed isn't what changes. Echo's generated self-attention is fully non-causal — self_mask = torch.ones((batch_size, seq_len)) at model.py:249 — so every latent position attends across the whole window. Shrinking 640 to 128 changes the computation at every retained position, not just how many survive. And the retry only fires when no flattening point is found, so a short window that yields a plausible flat tail is never corrected. The optimisation stays, behind AUDIOCPP_ECHO_TTS_ADAPTIVE_WINDOW=1.

Scope

Cut from the contributed branch before landing: the root README.md dump, echotts-server.json (hardcoded machine paths), webui dist/ and package-lock.json build artifacts, and validate_model_spec.py (a generic schema checker that isn't Echo-specific and deserves its own PR).

Two things here are not additive and I'd rather flag them than have you find them:

• src/models/fish_audio/codec.cpp. Echo reuses the in-tree Fish codec, which needs continuous z_q access. Two separate things, and I had originally described them as one:

  The decode side is a pure refactor. build_decode_quantizer was split into build_zq_from_codes + build_decode_from_zq and reassembled as a composition of the two. Same ops, same order, same graph. I earlier called this "restructured"; that overstated it.

  The encode side did change behaviour, and I've since fixed it. &z_q was passed unconditionally, so every fish_audio request built an extra ggml_sub node and marked it ggml_set_output even though fish_audio never reads it. The arithmetic was unaffected, but ggml_set_output pins that buffer and keeps both x and the quantiser residual live to the end of the graph, where otherwise the residual is free to be reused — a real allocation change for a family that gains nothing from it. EncodeGraph now takes want_z_q; with it false no node is built, no output is set, nothing is expanded, so the construction sequence is identical to upstream's. encode_reference passes false, only encode_zq passes true, and matches() carries the flag so the two graphs can't be confused.

  That shrinks the blast radius to zero for existing fish_audio users, which I think is a better answer than a test proving I didn't break it. Still happy to split this into a preceding PR if you'd rather review it on its own.

• src/framework/audio/wav_reader.cpp (+176). PCM8/PCM32/float64/A-law/mu-law/WAVEFORMATEXTENSIBLE support. This is a general WAV improvement that Echo doesn't strictly need. Say the word and I'll pull it into a separate PR.

Long-form and the fixed window

Echo generates at most 640 latents (640 × 2048 ÷ 44100 = 29.7215 s). Upstream's blockwise sampler subdivides that window rather than extending it (sum(block_sizes) + continuation_length < 640), and upstream notes it "hasn't been thoroughly tested."

So long text goes through the framework chunker, as the rest of the repo does — runtime::chunk_text_request at 300 codepoints, speaker conditioning cached once per session, append_audio_buffer to concatenate. chatterbox is the closest analogue. Dropping the blockwise path also drops latent_encoder, wk_latent and wv_latent: the 219 tensors the converter reports discarding, ~294 M parameters.

long_form is deliberately not claimed in capabilities. It appears nowhere in the C++, and only 5 of 22 TTS/clone families declare it — the 17 that don't include chatterbox, fish_audio, higgs_audio_tts, index_tts2, qwen3_tts, voxcpm2 and pocket_tts. Happy to add it if you'd rather it be declared.

What's still missing

• No fish_audio regression test, and no fish_audio checkpoint on my machine to build one against. The z_q output is now opt-in (see Scope), so fish_audio's own graph is identical to upstream's and there is nothing left for such a test to catch — but that is an argument from construction, not from execution, and I'd rather say so than imply I ran it.
• No per-block DiT activation dump, so the passing denoiser cosine proves correctness without localising where a future regression would live.
• The 40-step trajectory residual is explained but not closed.
• No A/B of the flash-attention path against AUDIOCPP_ECHO_TTS_NO_FLASH=1 on a fixed seed.
• No listening comparison of F16 against Q8_0. Q8 works but hasn't been auditioned, so the docs don't recommend it.
• Warm RTF and VRAM-stability-across-requests numbers.
• combine_cfg_lanes and euler_timestep_schedule have no registered coverage. They were checked by hand (6.6e-07 and 0.0 diff against a numpy transcription of inference.py); that hand run is not what CI holds, and the docs now say so.

Registered tests: tests/echo_tts/echo_tts_host_units.cpp via add_engine_unittest/add_test, CPU-only, no checkpoint needed — WhisperD normalisation, full byte-token id vectors, truncation and padding, PCA projection and inversion pinned independently against hand-computed values on a rectangular non-symmetric basis, and the flattening crop including both its thresholds. Mutation-checked against the four defects the second review said the earlier fixtures couldn't catch (transposed basis, mean dropped on both legs, scale dropped on both legs, zero-window search instead of thresholds); all four now fail the suite.

Licence

Echo-TTS is CC-BY-NC-SA-4.0, and the restriction covers generated audio, not just the weights (inherited from the Fish S1-DAC dependency). Flagging it rather than leaving it to be inferred. There's in-tree precedent — fish_audio carries the identical restriction from the identical dependency — and audio.cpp's Apache 2.0 licence is unaffected, since weights are a separate download. Documented in the model doc so nobody ships product on non-commercial output.

One question

Anything you'd want structured differently — file layout, option naming, whether the fish_audio and wav_reader changes should land as their own PRs first, or whether this belongs in community_models at all.

Why this model

Picked by comparing every model tracked in tts-bench — 62 local TTS models benchmarked across speed, objective scores, and blind human preference — against the existing support table.

Measure	Echo-TTS	Field
Blind cloning Elo	1162	#3 of 40 (35 games, 738 cloning votes)
Speaker similarity	0.836	2nd of 41
UTMOS / WER	4.21 / 7.45 %	—
Frozen pairwise study	21-1-6	near-tied 1st of 28

Caveats worth stating: the cloning arena averages ~30 games per model, so gaps under ~100 Elo are noise, and the ranking uses a single reference clip. Echo is top-3 on votes and 2nd on objective SIM, which are independent measurements.

[dignome]

I had a go at it. Can use anything you want from here if it helps.

https://github.com/dignome/audio.cpp-echo-tts

[5uck1ess]

@dignome appreciate it. i was all out of tokens building a SaaS. will take a look and implement.

[5uck1ess]

@dignome this was extremely useful — thank you. I've integrated it and it's now the body of this PR, with your commits preserved as Co-authored-by and credit in the model doc and models.md.

It works. Converted an F16 GGUF locally from jordand/echo-tts-base + jordand/fish-s1-dac-min and ran it on a 3090:

• manifest OK — 1117 DiT tensors, 219 blockwise dropped, 495 codec
• GGUF verifier passes, 1614 tensors, norm weights held at F32
• ASR round-trip through faster-whisper-large-v3-turbo: 0.0% WER, 0 edits on 32 words
• 9.195 s of audio in 7.89 s wall — RTF 0.86 cold, including the 5.5 GB load

Sounds good too, not just measurable-good.

I ran it past Codex and Grok before adopting. Codex checked your five "most likely to be wrong" items against the reference PyTorch line by line and cleared four of them — half-head RoPE on the head axis (model.py:199,217), GGML_ROPE_TYPE_NORMAL as the correct interleaved pairing against model.py:21, the patchify reshape ordering (model.py:458), and adaLN shift/scale/gate (model.py:64). Your fifth, the tensor names, got settled by the converter itself printing manifest OK against the real checkpoint. Your instinct about which parts were dangerous was right, and the parts you were worried about turned out fine.

One real find, and both reviewers hit it independently. The adaptive generation window defaulted on, with the reasoning that an under-estimate "costs time, never fidelity" because the retry reuses the Philox stream. The noise argument is correct but the conclusion doesn't follow, because the seed isn't what changes — model.py:249 is self_mask = torch.ones((batch_size, seq_len)), so generated self-attention is fully non-causal and every latent position attends across the whole window. Going 640 → 128 changes the computation at every retained position, not just how many survive. And the retry only fires when no flattening point is found, so a short window that yields a plausible flat tail never gets corrected. I've kept the implementation and moved it behind AUDIOCPP_ECHO_TTS_ADAPTIVE_WINDOW=1 — the cost saving is real, it just can't be the default until it's been A/B'd on a fixed seed.

Trimmed for upstream scope: the root README section, echotts-server.json, the webui dist/ and lockfile artifacts, and validate_model_spec.py (genuinely useful, but generic — worth its own PR). Folded the perf writeup into docs/community_models/echo_tts_performance.md.

Still draft. Missing a real parity pass against PyTorch, and a fish_audio regression test — your z_q seam restructured build_decode_quantizer rather than just extending it, so that core family's decode path changed and needs its own coverage before this can merge.

Genuinely appreciate you publishing it and offering it up instead of racing a competing PR.

hats off to you sir @dignome

[0xShug0]

@5uck1ess @dignome Sorry, I didn’t realize this PR was ready for review... Could you elaborate on why you need to change the Fish Audio and framework libraries?

[dignome]

We definitely do not need the framework change - I was just trying to avoid converting all my wav files as the cli/server path has weaker support for audio formats than the web ui (in my fork).

I'll have to look why Claude decided to rework the fish audio codec but it most likely has something to do with Echo using the fish audio S1 codec (audiocpp currently has an implementation of the S2 audio codec). Would make more sense if we do modify existing fish audio codec to rename the zq stuff as s1 just to clarify what it is doing.

[5uck1ess]

@0xShug0

Reverted wav_reader.cpp to upstream.

The reader rejects WAVEFORMATEXTENSIBLE files saying it needs PCM16, when PCM16 is what's inside, just tagged 0xFFFE which line 187 never unwraps. Been there a while, nothing to do with this PR. Can send it separately if you want it.

Fish Audio I do need. Echo's DiT works in continuous latents, not codebook indices, so it needs z_q in and out of the autoencoder, and that's the same Fish DAC already in the tree. Other option was vendoring a second copy of a codec you already ship.

Two changes there and I'd run them together in the PR body:

• decode: build_decode_quantizer split in two and called one after the other. Same ops, same order. I wrote "restructured" which oversold it.
• encode: this one did change. I was building the z_q output unconditionally, so every fish encode carried an extra node marked as a graph output that fish never reads. No math change, but it pins a buffer that'd otherwise get reused. Now behind want_z_q, and with it off the graph matches yours.

3 files, +248/-10, and those 10 lines are the only deletions in the PR. Happy to split it out first if you'd rather review it on its own.

On z_q -> s1, @dignome, I don't think so? The codec constants (1024 dim, 4096 semantic, 9x1024 residual, 44.1k, 2048 frame) are both the S1-DAC-min and the S2 Pro config, since S2 Pro kept S1's RVQ codec. One graph, two weight sets. And S1 is already Echo's speaker tag. If the name's unclear, encode_quantized_latents works whatever weights you load.

Still open:

• sampler's at 0.977 against the 0.999 gate I set. one-step denoiser passes at 0.999999. it's dtype (reference runs bf16, my gguf is F16, matching them takes it 0.905 -> 0.977) and it grows with step count, so rounding rather than a bug. went through the sampler line by line and found no defect. still calling it below gate.
• no fish_audio regression test. graph's identical to yours with z_q off, but I don't have a fish_audio checkpoint here to actually run one.

Rebased on main, 31 files, out of draft.

[dignome]

This is the best interpretation I can find for why Claude chose to modify the existing fish audio codec. I'm not a professional coder so most of the below was generated via an AI to help me understand why these changes were made.

=============

The proposed Echo implantation reuses the existing Fish S2-DAC codec, but unlike Fish S1-mini/S2, it does not generate discrete codebook IDs. Echo generates continuous latents that are converted directly into the codec’s internal z_q representation.

z_q is the existing fish codec’s quantized latent audio representation.

The name means:

z: a hidden or latent representation
q: quantized

For each approximately 46 ms audio frame, z_q is a vector of 1,024 floating-point values describing the sound.

The Fish codec already computes and consumes z_q internally, but its public API only accepted or returned codebook IDs. The changes expose this existing boundary:

encode_latents() returns z_q for Echo’s reference-audio conditioning.
decode_latents() accepts Echo’s generated z_q and runs the existing decoder tail.

This reuses the existing codec implementation without duplicating it or adding a lossy z_q → codebook IDs → z_q conversion. Normal Fish Audio decoding remains codebook IDs → z_q → audio.

[5uck1ess]

@dignome that's right, and I've gone with encode_latents / decode_latents — pushed. Clearer than zq and it doesn't pin a shared graph to one model generation.

One correction on the lossy bit, and it's the half that matters.

Encode is not lossy through codes. z_q is defined as the sum of the dequantised codebook entries, which is literally what build_zq_from_codes computes, so audio → codes → z_q gives you the same z_q. encode_latents is just convenience.

Decode is where it breaks. Echo's DiT emits latents that aren't on the RVQ lattice at all — session goes pca_unproject → decoder, never quantising — so pushing that back through codes means re-quantising to the nearest entries and dropping everything in between. That's the reason decode_latents has to exist.

Put that in the header comment since it's easy to get backwards.

And yeah, your numbers check out: 2048 samples at 44.1k is 46.4 ms, 1024 dim.

[5uck1ess]

The two red checks look like runner trouble on your side rather than this branch — mac came back cancelled with no steps recorded and Windows was startup_failure, both before anything compiled. Linux and Nix weren't scheduled at all, and the pushes since haven't triggered any new runs.

The commit before these passed all four. I ran the same three workflows on my fork at the same SHA to check: Linux (cpu + vulkan), macOS and Windows all green. Also built the CI config locally (Debug, gcc-13, CUDA and Vulkan off, same three targets) — clean, and check_loader_catalog_sync.py passes.

I can't rerun your jobs without admin, so flagging rather than fixing.

[dignome]

Can find some premade gguf files for f16 and q8_0 here:

https://huggingface.co/dignome/Echo-TTS/tree/main

Updated to reflect change to model spec (e8b886e)

[0xShug0]

@5uck1ess @dignome Thanks for the updates! Since fish audio and echo tts share codec components in some way, I promoted it as a framework/runtime and refactored fish audio. This should make it easier to extend and maintain while reducing across-model dependencies. Could you try to use the new runtime? I think the new runtime should be able to support the echo tts use case. If not feel free to extend the codec runtime additively.

Could you also normalize the spec option names? You can find examples in #128 and the specs of the v1 core models under model_specs/

I'll do more testing after wrapping up release 0.7.

[0xShug0]

The reader rejects WAVEFORMATEXTENSIBLE files saying it needs PCM16, when PCM16 is what's inside, just tagged 0xFFFE which line 187 never unwraps. Been there a while, nothing to do with this PR. Can send it separately if you want it.

Yes, a follow-up PR would be appreciated!

[0xShug0]

We definitely do not need the framework change - I was just trying to avoid converting all my wav files as the cli/server path has weaker support for audio formats than the web ui

A follow-up PR would be appreciated!

[5uck1ess]

Both done, rebased on main.

Codec runtime. No extension needed — the promoted runtime already had both seams. Echo builds a FishDacCodecComponent from its own codec weights and calls encode_latents/decode_latents. The fish_audio diff is gone entirely, so the "no regression test for fish_audio" caveat in my docs goes with it.

Option names. irodori_tts was the closest match structurally (dual CFG + RF steps), so I used its spelling:

num_steps             -> num_inference_steps
cfg_scale_text        -> text_guidance_scale
cfg_scale_speaker     -> speaker_guidance_scale
cfg_interval          -> guidance_interval
reference_max_seconds -> reference_duration_sec
sequence_length       -> max_duration_sec

The last one also changed units. sequence_length was latent frames, which is an internal number. It's seconds now, quantised down to a 46.44 ms frame so the value is a real ceiling — ask for 2.0 s and you get 1.997 s, never more. Unreleased model, so no legacy aliases.

Kept truncation_factor and speaker_kv_scale — neither concept has a normalized spelling in-tree and both are domain terms rather than Python leaks. Happy to rename if you'd rather.

Worth noting the contract comes from the spec embedded in the GGUF, not model_specs/, so I re-embedded it to test: every new name is accepted, every old one is rejected. Full ctest green and an end-to-end CUDA clone still round-trips through ASR at 0 edits.

Still open from before: the 40-step sampler trajectory sits at 0.977 against the 0.999 gate. Dtype accounts for most of it — matching the bf16 reference to the F16 GGUF moved it 0.905 -> 0.977 — and a line-by-line sampler review found no defect, but the residual isn't closed. Single-step denoiser is 0.999999.

Wav reader follow-up is ready on my side, will open it separately.

[dignome]

GATES="--denoiser-gate 0 --sampler-gate 0 --denoiser-max-abs 1e9 --sampler-max-abs 1e9"
=== gguf=echo-tts-f16.gguf ref=echo_ref_f32.bin ===
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 32141 MiB):
Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes, VRAM: 32141 MiB
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
text_length=140 speaker_frames=300
denoiser probe at t=0.7000
denoiser cosine=0.999999948 max_abs=0.005709 rms=0.000425 gate=0.000/1000000000.000 PASS
sampler, reference initial noise injected
injected cosine=0.999464125 max_abs=0.305232 rms=0.006684 gate=0.000/1000000000.000 PASS
sampler steps=40 sequence_length=640 seed=0
sampler cosine=0.999674003 max_abs=0.264109 rms=0.005214 gate=0.000/1000000000.000 PASS
echo_tts_dit_parity: ok

Just needed a higher precision reference against echo-tts-base to pass the parity test.

Generate with the attached
echo_tts_reference.py

/mnt/work/echo-tts# python echo_tts_reference.py --speaker /mnt/work/audio.cpp-pr180/assets/resources/sample.wav --force-dtype float32 -o echo_ref_f32.npz

The gguf files I linked earlier are updated.

Only thing I can see that could be fixed now or later is to add echo_tts model to the web ui catalog.

webui/configs/model_params.json:

"echo_tts": [ {"name": "num_inference_steps", "type": "slider", "label": "Sampling steps", "default": 20, "minimum": 8, "maximum": 40, "step": 1, "precision": 0, "info": "Number of Euler sampling steps."}, {"name": "guidance_interval", "type": "slider", "label": "Interval", "default": 1, "minimum": 1, "maximum": 3, "step": 1, "precision": 0, "info": "Guidance refresh interval. Higher values improve speed and work best with more generation steps; 1 gives the highest fidelity."} ],

webui/configs/models_catalog.json:

{ "id": "echotts", "display_name": "EchoTTS (voice clone)", "family": "echo_tts", "path": "models/Echo-TTS-GGUF", "task": "clon", "mode": "offline", "download_id": "echotts", "min_vram_gb": 10 },

There may be other steps to update the webui not 100% on that.