Promote reusable codec hooks for Fish Audio and CosyVoice - #310
Merged
Conversation
5uck1ess
added a commit
to 5uck1ess/audio.cpp-fork
that referenced
this pull request
Aug 27, 2026
0xShug0#310 promoted the Fish S1-DAC into engine::codecs::FishDacCodecRuntime, and it already exposes the two seams Echo needed -- encode_latents and decode_latents. Echo now builds a FishDacCodecComponent from its own codec weights and drops every reference to the fish_audio model. The diff against fish_audio is gone with it: this PR no longer touches that family at all, so the "no regression test for fish_audio" caveat no longer applies and is removed. Verified: echo_tts_host_units and the full ctest suite pass, and an end-to-end CUDA clone (F16 GGUF, RTX 3090) transcribes back at 0 edits.
0xShug0
pushed a commit
that referenced
this pull request
Aug 27, 2026
* docs: add Echo-TTS port design spec Design for porting Echo-TTS (jordand/echo-tts-base, 2.8B DiT + Fish S1-DAC) into audio.cpp as a community model. Architecture verified against upstream source and safetensors headers, not inferred. Key findings: - EchoDiT: 24 blocks, d=2048, joint attention, adaLN, byte-level text - Fixed 640-latent / 29.72s generation window - Blockwise path subdivides that window, does not extend it - Decode and encode need near-disjoint Fish submodules - 303.6M of the Fish checkpoint is regenerable buffers, not weights Staged M0-M4 with per-milestone gates and a hard Definition of Ready before the PR leaves draft. * docs: spec self-review fixes - Resolve RoPE theta open question (10000.0, complex-valued, model.py:9) - Add timestep embedding formula - Warn RTF vs RTFx are inverses (tts-bench vs audio.cpp conventions) - Cite the actual schema validator for the M0 gate - Define 'cosine' precisely (flattened 1-D, with max-abs-error) - Add decomposition note: M0+M1 in one plan, M2/M3/M4 separate * docs: add M0+M1 implementation plan for Echo-TTS port 13 tasks, each gated on executed evidence: - M0 (T1-2): spec v1 registration + draft PR - M1 (T3-13): converter, parity dumps, GGUF, assets, tokenizer, text/speaker encoders, 24-block DiT, dual-CFG Euler sampler, PCA inverse + Fish decode, crop, warm bench Every stage gates on cosine >= 0.999 vs PyTorch before the next begins. Task 12 requires a human ear check - tensor parity cannot catch a wrong flattening-point crop. PR stays draft through M1; cloning still needs an injected .npy until M2 lands native speaker encoding. * docs: correct build invocation and CUDA arch in M0+M1 plan - Repo has no CMakePresets.json; --preset would fail. Use scripts/build_linux.sh or cmake --build build/linux-cuda-release. - Existing build tree pins CMAKE_CUDA_ARCHITECTURES=75 (Turing) on an sm_86 card. Task 13 now reconfigures to 86 before measuring RTF, otherwise the number is invalid. - Note AUDIOCPP_MODEL_SET=full so the family compiles in automatically. * docs: fix Task 1 gate to use --list-loaders (no --list-families flag) Baseline on this branch is registered_loaders=42, verified. Also note that a 'requires a schema v1 model contract' failure means a stale binary, not a broken tree. * feat(echo_tts): register family with spec v1 and silence stub Plan Task 1. Spec-backed loader (no loader.cpp), schema_version 1, capabilities.clone deliberately omits long_form until M3 earns it. Verified by execution: registered_loaders 42 -> 43, echo_tts appears as 'clon (offline)', spec parses. Fix over Codex's draft: guard used VoiceTaskKind::Tts, but the family registers as a clone task, so every real invocation would have thrown. Corrected to VoiceCloning, matching confucius4_tts:185. The registration gate could not catch this - --list-loaders enumerates loaders without constructing a session. * docs(echo_tts): add community model doc Documents the fixed 29.72s window, why blockwise does not extend it, the CC-BY-NC-SA output restriction (with the fish_audio precedent), benchmark provenance, options, and the WhisperD text format. Also drops an unrelated .gitignore change that was accidentally swept into an earlier docs commit, so the PR diff stays scoped. * chore: keep internal planning docs out of the upstream PR The design spec and implementation plan are our working process, not content for audio.cpp. Preserved on the local echo-tts-planning branch and still on disk; just untracked here so the PR diff stays scoped to the actual contribution. * feat(echo_tts): full implementation — DiT, sampler, codec seam, converter Replaces the M0 silence stub with a complete port, contributed by @dignome and offered for use in this PR (see PR #180 discussion). DiT trunk, 24 blocks, joint attention with flash-attn path Byte tokenizer + WhisperD normalisation Euler dual-CFG sampler with independent text/speaker guidance PCA inverse + flattening-point crop Fish S1-DAC z_q seam, reusing the in-tree fish_audio codec GGUF converter (F16 and Q8_0) plus a manifest and verifier Long-form via the framework text chunker at 300 codepoints Scope trimmed from the source branch before landing: the root README dump, echotts-server.json (hardcoded machine paths), webui build artifacts, and a generic validate_model_spec.py that is not Echo-specific and belongs in its own PR. Two corrections on top of the contributed tree: resolve_reference_max_samples' comment claimed it falls back to the trained maximum; it returns kDefaultReferenceMaxSamples (15 s). The code is intentional and the spec publishes 15.0 in both scopes, so the comment was the error, not the behaviour. The status table still carried this PR's original milestone list, which said M2 was not started and that cloning needed a pre-computed speaker latent. session.cpp calls codec_->encode_zq directly, so both claims were false. Rewritten to separate what is implemented from what is numerically verified, because nothing in the ggml graph has been checked against PyTorch yet. Co-authored-by: dignome <dignome@gmail.com> * fix(echo_tts): make the adaptive generation window opt-in Independent reviews by Codex and Grok both landed on this feature, from different angles. Codex supplied the mechanism. The code defaulted the generation window to a text-length estimate, justified in-comment as "an under-estimate costs time, never fidelity", because a missing flattening point retries at full length and generate_torch_cuda_randn is a sequential Philox stream, so the retry draws bit-identical noise. The noise claim is true. The fidelity conclusion does not follow, because the seed is not what changes. Echo's generated self-attention is fully non-causal -- model.py:249 is self_mask = torch.ones((batch_size, seq_len), dtype=torch.bool, ...) so every latent position attends across the whole window. Shrinking 640 to 128 changes the computation at every retained position, not merely how many positions survive. The reference defaults to 640 (inference.py:353). The retry also only fires when no flattening point is found, so a short window that happens to produce a plausible flat tail is never corrected and silently ships different audio. Inverted to AUDIOCPP_ECHO_TTS_ADAPTIVE_WINDOW=1. The cost saving is real and the implementation stays; only the default changes, until it has been A/B'd against the full window on a fixed seed. Codex also cleared the four highest-risk items the original author flagged as possibly-silently-wrong, against the reference: half-head RoPE rotates the first heads/2 on the head axis (model.py:199,217), GGML_ROPE_TYPE_NORMAL is the correct interleaved convention against model.py:21's reshape(...,-1,2) pairing, the speaker patchify reshape produces frame-then-channel ordering matching model.py:458, and the adaLN chunk order is shift/scale/gate per model.py:64. Tensor names remain unproven pending a real checkpoint. * docs(echo_tts): list the family in the community models index Adds the models.md row, with attribution to @dignome for the implementation. Flagged by review as the one doc that was never updated. * docs(echo_tts): record the end-to-end run and what it does not prove Executed on an RTX 3090 (sm_86) against a locally converted F16 GGUF. Conversion reports manifest OK, the verifier passes, and generation round-trips through faster-whisper at 0.0% WER on 32 words. Throughput is RTF 0.86 cold, including the 5.5 GB load. A 0% WER rules out the silent-wrong failure modes -- half-head RoPE, rotary pairing, patchify layout and adaLN chunk order would each produce fluent but incorrect speech. It is still not per-tensor parity, so the missing evidence is listed explicitly rather than implied: no cosine gate against PyTorch, no flash-attn A/B, no fish_audio regression test for the restructured build_decode_quantizer, no F16-vs-Q8 listen, and no registered C++ tests. * test(echo_tts): add host-side unit tests for tokenizer, PCA and crop The contributed port's host-side units were verified by hand and never committed as tests. This registers them as a CPU test target that needs neither a GPU nor the 5.5 GB checkpoint, so CI can hold them. Coverage: WhisperD normalisation, including the asymmetric double-quote rewrite upstream applies to U+201D but not U+201C. That asymmetry looks like a bug and is load-bearing -- "fixing" it silently desyncs the token stream from the reference, so the test pins it. Byte tokenisation: exact token counts and prefixes, [S1] tagging, bracket/paren suppression, truncation at 768 including the BOS. PCA forward/inverse round trip on an orthonormal basis, plus a separate assertion that latent_scale is applied, so dropping it on either leg fails rather than cancelling out. find_flattening_point on three cases: a mid-sequence flattening, a latent that never flattens, and one flat from frame zero. Every expected value was produced by executing the reference implementation (inference.py tokenizer_encode / find_flattening_point), not by reasoning about what it should return. Verified by mutation rather than by the tests merely passing: removing the colon rewrite fails the normalisation case, and dropping the inverse PCA scale fails the round trip at element 0. Both reverted. * test(echo_tts): add a DiT parity harness and pass the denoiser gate Adds the numerical parity evidence the PR was missing. A 0% WER shows the pipeline is right end to end; it does not show the DiT graph matches PyTorch, which is what the silent-wrong failure modes would break. tests/echo_tts/echo_tts_dit_parity.cpp Loads the GGUF, injects the reference's own text ids, mask and speaker latent, and scores by cosine plus max-absolute-error. Cosine alone hides a uniform scale error and max-abs alone is dominated by one outlier, so both are reported. Not registered with add_test -- it needs a 5.5 GB GGUF and a PyTorch dump, so it is hand-driven like dots_tts_vocoder_parity. tools/community_models/echo_tts_pack_reference.py Packs echo_ref.npz into a flat binary so the harness needs no npz parser in C++. EchoDitRuntime::denoise_once(x, t, lanes) A testing seam. sample() alone cannot isolate a wrong block from a wrong integration step; feeding the reference's own x and t makes any difference attributable to the graph. Result on an RTX 3090 against the F16 GGUF, reference dumped from upstream at a fixed seed and a fixed timestep t=0.7: denoiser cosine=0.999976711 max_abs=0.086061 rms=0.008939 PASS That clears the 0.999 gate and settles the four items flagged as possibly-silently-wrong: half-head RoPE, the rotary pairing convention, the speaker patchify reshape and the adaLN chunk order all now have a number behind them rather than a code reading. The full 40-step trajectory from our own seeded noise scores cosine 0.905, below the gate. Not yet reported as pass or fail: the harness now also runs the sampler from the reference's OWN initial noise, which discriminates between RNG divergence and a real integration defect. That run is queued behind an unrelated tts-bench job holding the GPU. * fix(echo_tts): denoise_once divided the input by lane count The seam recomputed sequence_length as x.size() / (latent_size * lanes), but x is always a SINGLE lane: sampler.cpp calls denoise(x_t, t, 3) with an x_t of exactly `elements` and expects elements * 3 back. `lanes` selects the width of the OUTPUT, not the input. The three-lane path therefore threw on a non-divisible size instead of running. Caught by executing the parity harness, not by reading it. * test(echo_tts): close the tautologies two reviews found, publish both cosines Codex and Grok reviewed the verification commits (not dignome's port) and converged on the same four things. All are addressed here. Parity harness -------------- max-absolute error was computed, printed, and never gated -- `actual = 1000 * expected` scored cosine 1.0 and PASSed. Both checks now require cosine AND max-abs, with `--denoiser-max-abs` / `--sampler-max-abs`. Verified by running the denoiser probe at `--denoiser-max-abs 0.001`: cosine still 0.999999, verdict FAIL. The header claimed the probe attributes any difference "to the graph alone". It does not: `prepare_conditioning()` runs this port's own text encoder, speaker encoder and KV projections, so the number covers the combined conditioning-plus-denoiser path. Corrected in the header and the model doc. The bundle reader took signed name/element lengths straight from the file into `resize()` and treated every dtype tag other than 1 as float. Bounded and validated; the little-endian assumption is now stated rather than implied. Host unit tests --------------- The PCA fixture was a square identity, which is its own transpose, so a transposed basis read passed, and a mean or scale dropped on *both* legs cancelled in the round trip. Replaced with a rectangular, non-symmetric orthonormal basis (2 components over 4 features), with projection and inversion each pinned against independently computed values -- confirmed against numpy. The round trip is now exact to 1e-6 on an in-subspace vector rather than 1e-3 on a bijection. Token tests checked a length and a 12-id prefix; a length-preserving rewrite (signed-char sign extension on multibyte UTF-8) passed. Full id vectors are now pinned for all five cases, generated by executing `tokenizer_encode`. The flattening fixtures were all extreme active-or-zero, so an implementation that merely searched for an all-zero window passed without evaluating either threshold. Added a quiet-but-non-zero tail (0.02 -> 30) and a flat-but-loud tail (0.5 -> 60), both confirmed against `find_flattening_point`. `require_close` compares `fabs(a - b) > tolerance`, which is false for NaN, so NaN passed every float assertion. Wrapped locally with an explicit finite check rather than changing shared test code. Also added `pad_to_max` coverage and truncation *content* -- previously only its length was asserted. Mutation-checked, each reverted after: transposed basis, mean dropped on both legs, scale dropped on both legs, and zero-window search instead of the thresholds. All four now fail the suite; all four passed it before. The sampler residual -------------------- Re-dumping the reference at float16 to match the GGUF (it defaults to bfloat16) moves the denoiser probe from cosine 0.999977 to 0.999999188 and the 40-step trajectory from 0.905481 to 0.976972. Combined with the earlier step-count sweep -- 0.9965 at 4 steps, 0.9055 at 40 -- and Codex finding no defect in a line-by-line read of the sampler against inference.py, the residual is accumulating per-step rounding amplified by dual CFG at 3.0/8.0, not a structural defect. That is an explanation, not a proof: the trajectory is still reported as below the gate. Documentation ------------- docs/community_models/echo_tts.md claimed "No cosine >= 0.999 comparison" and "No C++ unit tests are registered". Both were false at HEAD. It also used a 0 % WER on 32 words to rule out four silent-failure modes, including the speaker patchify reshape -- but WER scores words, not speaker identity, so that one could produce fluent correct text in the wrong voice and still read 0 %. Rewritten: both cosines published in one table, the trajectory marked below gate, WER demoted to what it actually shows, and the hand-run numbers separated from what CI holds. Dropped the internal planning docs again -- a third `git add -A` re-added them in 85ee6d8. Now in .git/info/exclude so it cannot recur. * revert(framework): drop the wav_reader format expansion from this PR @0xShug0 asked why this PR touches the framework, and @dignome answered it better than I could: it was never needed here. It rode in with the contributed tree at fa8206c, where it existed to spare their fork's CLI and server path from converting wav files the webui already accepted. Echo needs none of it. Reverted to upstream's reader verbatim; nothing in the echo_tts or fish_audio changes referenced the added helpers. Verified by execution after the revert: wav_reader_test passes, echo_tts_host_units passes, and an Echo-TTS clone from a stereo 24 kHz PCM16 reference still runs clean on CUDA -- RC 0, 4.55 s of 44.1 kHz mono. The dropped support (PCM8, PCM32, float64, A-law, mu-law and WAVEFORMATEXTENSIBLE) is genuinely useful and worth its own PR on its own merits. It just isn't Echo's to carry. * docs(echo_tts): follow the encode_latents rename The accessor was renamed from encode_zq in the previous commit; the model doc still named the old one. The remaining z_q mentions in the docs refer to the upstream PyTorch DAC.encode_zq / DAC.decode_zq, which are correct as written and should not be renamed with our C++ API. * refactor(echo_tts): move onto the shared Fish DAC codec runtime #310 promoted the Fish S1-DAC into engine::codecs::FishDacCodecRuntime, and it already exposes the two seams Echo needed -- encode_latents and decode_latents. Echo now builds a FishDacCodecComponent from its own codec weights and drops every reference to the fish_audio model. The diff against fish_audio is gone with it: this PR no longer touches that family at all, so the "no regression test for fish_audio" caveat no longer applies and is removed. Verified: echo_tts_host_units and the full ctest suite pass, and an end-to-end CUDA clone (F16 GGUF, RTX 3090) transcribes back at 0 edits. * spec(echo_tts): normalize option names to the v1 conventions Per #128 and the names already in use across model_specs_v1. irodori_tts is the closest structural match -- dual CFG plus RF steps -- and Echo now uses the same spelling: num_steps -> num_inference_steps cfg_scale_text -> text_guidance_scale cfg_scale_speaker -> speaker_guidance_scale cfg_interval -> guidance_interval reference_max_seconds -> reference_duration_sec (request and session) sequence_length -> max_duration_sec The last one also changes units. sequence_length was latent frames -- an internal number leaking into the public surface -- so it becomes seconds, in line with the *_sec convention. It is quantised *down* to a whole 46.44 ms frame and clamped to the trained window, so the value is a real ceiling: asking for 2.0 s yields 1.997 s, never more. truncation_factor and speaker_kv_scale keep their names: neither concept has an existing normalized spelling in-tree, and both are domain terms rather than Python internals. The model is unreleased, so no legacy aliases are kept. Internal struct fields are untouched; only the parse boundary in session.cpp moves. Verified against a GGUF re-embedded with the new spec: every normalized name is accepted, every old name is rejected by the contract, max_duration_sec truncates (2.0 -> 1.997 s, 1.0 -> 0.975 s) and clamps at both ends, and the clone still transcribes back at 0 edits. * test(echo_tts): close the sampler parity gap — it was the reference's dtype The 40-step trajectory sat at 0.977 against a 0.999 gate and this document called it unverified. It was never this port: ggml accumulates in F32 whatever the stored weight type is, so the like-for-like comparison against an F16 GGUF is a float32 reference, not the bfloat16 one upstream loads by default. The rounding being measured was PyTorch's. Adds --force-dtype to the reference dumper (contributed by @dignome), which also disables TF32 for the float32 path -- TF32 has a 10-bit mantissa, no better than F16, and Ampere would otherwise use it for matmuls and defeat the point. Against a float32 reference, with the gates enforced rather than disabled, on an RTX 3090: denoiser cosine=0.999999899 max_abs=0.012564 gate=0.999/0.250 PASS injected cosine=0.999515574 max_abs=0.318055 gate=0.999/4.000 PASS sampler cosine=0.999541572 max_abs=0.318892 gate=0.999/4.000 PASS @dignome found this and reported the same three PASSes on a 5090. The run above is an independent reproduction, not a restatement of theirs. M1 is now numerically verified end to end, and the status table says so. * feat(webui): add Echo-TTS to the model catalog and parameter surface Suggested by @dignome. Bounds and defaults are copied from model_specs/echo_tts.json so the two cannot drift: 40 steps, text/speaker guidance at 3.0/8.0, truncation 0.8, guidance_interval 1, reference trim 15 s. download_id points at the echo_tts_orig package, whose download is still declared unsupported in the spec -- there is no official GGUF to point it at yet, so the entry makes the model selectable once weights are on disk rather than downloadable from the UI. * spec(echo_tts): point downloads at the published GGUFs, drop target_voice Two of @0xShug0's three review points. Downloads: the spec declared packaging unsupported, the only such package in the tree. It now names @dignome's published GGUFs -- Q8_0 default, F16 alongside -- pinned to a revision so a later push upstream cannot change what a given audio.cpp checkout resolves to. The webui entry follows the default package and its VRAM floor drops from 10 GB to 8 with Q8_0. target_voice: removed from the spec and the docs option table. Nothing read it. The speaker reference arrives through --voice-ref, which the CLI and server hand to the session directly rather than as a request option, so advertising target_voice described a path that did not exist. The warm-bench case moves to voice_ref, matching every other clone family and what model_perf.py expects. * refactor(codecs): drop the fish_audio.codec.* tensor-sample traces The third of @0xShug0's review points. The Fish DAC encode graph logged four tensor dumps -- after_downsample, after_pre_module, encoder_latent and reference_codes -- and reference_codes was not even behind trace_log_enabled(). With logging on they fire per request, which Echo made obvious because it encodes a speaker reference on every uncached clone. Removing them also unpins memory. Each traced tensor got a ggml_set_output(), and ggml_gallocr_free_node() returns early for output tensors, so their buffers stayed live for the whole encode graph whether or not anything was reading them. Verified: full ctest green, and the Echo DiT parity harness returns bit-identical cosines to the run before this change (denoiser 0.999999899, injected 0.999515574, sampler 0.999541572), so the encode graph's arithmetic is untouched. No fish_audio GGUF was available locally to exercise that family end to end; its code path through encode_codes is unchanged and covered by the suite. --------- Co-authored-by: dignome <dignome@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validation
Notes