Add Echo-TTS (community model)

CMakeLists.txt

@@ -710,6 +710,24 @@ audiocpp_add_model(neutts
         engine::models::neutts::make_neutts_loader
 )
 
+audiocpp_add_model(echo_tts
+    SOURCES
+        src/community_models/echo_tts/session.cpp
+        src/community_models/echo_tts/tokenizer.cpp
+        src/community_models/echo_tts/latent_post.cpp
+        src/community_models/echo_tts/sampler.cpp
+        src/community_models/echo_tts/dit.cpp
+    INCLUDES
+        engine/community_models/echo_tts/session.h
+        engine/community_models/echo_tts/config.h
+        engine/community_models/echo_tts/tokenizer.h
+        engine/community_models/echo_tts/latent_post.h
+        engine/community_models/echo_tts/sampler.h
+        engine/community_models/echo_tts/dit.h
+    LOADERS
+        engine::models::echo_tts::make_echo_tts_loader
+)
+
 if (MSVC)
     set_source_files_properties(
         src/community_models/inflect_v2/frontend.cpp
@@ -2031,6 +2049,17 @@ if (ENGINE_BUILD_TESTS)
 
     add_engine_unittest(dots_tts_vocoder_parity tests/dots_tts/dots_tts_vocoder_parity.cpp)
 
+    # Needs the GGUF and a PyTorch reference dump, so it is driven by hand
+    # rather than registered with add_test -- same as dots_tts_vocoder_parity.
+    add_engine_unittest(echo_tts_dit_parity tests/echo_tts/echo_tts_dit_parity.cpp)
+
+    add_engine_unittest(echo_tts_host_units tests/echo_tts/echo_tts_host_units.cpp)
+
+    add_test(
+        NAME echo_tts_host_units
+        COMMAND echo_tts_host_units
+    )
+
     add_engine_unittest(midi_file_test tests/unittests/test_midi_file.cpp)
 
     add_test(

docs/community_models/echo_tts_autoencoder_reuse.md

@@ -0,0 +1,138 @@
+# Echo-TTS: autoencoder reuse
+
+Status: verified against checkpoint sizes and upstream source. No weights were
+downloaded to reach these conclusions; every number below is reproducible from
+`autoencoder.py` plus the file sizes Hugging Face reports.
+
+> The "Integration seam" section below was written when the codec still lived
+> inside the `fish_audio` model. Upstream has since promoted it to a framework
+> runtime (`engine::codecs::FishDacCodecRuntime`, #310), which exposes the two
+> seams this document argued for -- `encode_latents` and `decode_latents` --
+> directly. Echo now calls that runtime and touches no `fish_audio` code at all.
+> The analysis is kept because the reasoning about *which* weights Echo needs
+> still holds.
+
+## Summary
+
+Echo-TTS depends on the Fish S1-DAC autoencoder, and **audio.cpp already
+implements that exact autoencoder** -- today in
+`src/framework/codecs/fish_dac_codec_runtime.cpp`, at the time of writing inside
+the `fish_audio` model. The Echo port does not need a new decoder,
+encoder, quantiser, or window-limited transformer. It needs a `z_q` seam on the
+existing one.
+
+This changes the cost of milestones M1 and M2 substantially relative to the
+original PR plan, which scoped "Fish decode" and "native speaker encoding
+(Fish encoder + RVQ)" as separate pieces of work.
+
+## Evidence
+
+### Configuration
+
+`jordand/fish-s1-dac-min/config.json` reports:
+
+    sample_rate 44100, encoder_dim 64, encoder_rates [2,4,8,8], latent_dim 1024,
+    decoder_dim 1536, decoder_rates [8,8,4,2], n_codebooks 9, codebook_size 1024,
+    codebook_dim 8, semantic_codebook_size 4096, causal true
+
+Every one of these matches the constants already compiled into the shared
+codec: `kCodecDim` 1024, semantic codebook 4096, nine residual
+quantisers of 1024, codebook dim 8, a final decoder snake at 96 channels
+(= 1536 / 2^4), and causal convolutions throughout.
+
+### Parameter budget
+
+Deriving the parameter count from `autoencoder.py` and comparing against the
+1.87 GB Hugging Face reports for `pytorch_model.safetensors`:
+
+| Component | Parameters |
+| --- | ---: |
+| Encoder | 76,851,328 |
+| Decoder | 54,102,722 |
+| Quantiser (incl. pre/post transformers) | 260,475,040 |
+| **Total** | **391,429,090** |
+
+At F32 that is 1.566 GB. The `Transformer` base class registers two buffers per
+instance — a `freqs_cis` table and a `block_size^2` boolean `causal_mask` — which
+for the three surviving transformer instances (encoder block 3 at block_size
+16384, quantiser pre/post at 4096) comes to 305 MB. Together: **1.871 GB**,
+against the 1.87 GB reported. This also reproduces the 303.6 MB
+"regenerable buffers" figure noted on the PR.
+
+The match only holds once the decoder is counted **without** a transformer, which
+leads to the next point.
+
+### The decoder has no transformer
+
+`build_ae` passes `decoder_transformer_layers=[4, 0, 0, 0]`, which reads as though
+decoder block 0 carries a 4-layer transformer. It does not. `DecoderBlock.__init__`
+constructs `transformer_module` into a local variable and then builds
+`self.block = nn.Sequential(Snake1d, conv_trans, ResidualUnit x3)` without it.
+The module is never assigned to `self`, so it is not a submodule, has no
+parameters, and is absent from the checkpoint. `EncoderBlock`, by contrast, does
+include `transformer_module` in its `Sequential`.
+
+Two independent checks agree:
+
+1. The 1.87 GB file size only reconciles when the decoder transformer is excluded
+   (including it predicts 2.05 GB, and adds a second 16384x16384 mask buffer that
+   would break the 303.6 MB figure).
+2. The codec implementation already loads the encoder transformer conditionally
+   at `block_index == 3` and loads no transformer anywhere in the decoder path.
+
+The C++ was evidently written against the real checkpoint, and it agrees with
+the source reading. Worth knowing before anyone "fixes" the apparent omission.
+
+## Integration seam
+
+Echo needs continuous `z_q` where `fish_audio` uses discrete codes. Both seams
+sit at existing boundaries in the codec, and the framework runtime now exposes
+them as `decode_latents` and `encode_latents`:
+
+**Decode.** `DAC.decode_zq` is `post_module -> upsample -> decoder`.
+`build_decode_quantizer` already performs exactly that chain; it just derives its
+input by looking up codebook entries first:
+
+    latent = build_quantizer_out(semantic) + sum(build_quantizer_out(residual_i))
+    latent = build_window_transformer(..., post_module, 128)   <- Echo enters here
+    for stage in upsample: ...
+
+Echo supplies `latent` directly from the PCA inverse and runs from the
+`post_module` line onward -- which is what `decode_latents` does.
+
+**Encode.** `DAC.encode_zq` quantises and then sums the dequantised results:
+`z_q = z_q_semantic + z_q_residual`. The encode graph already computes each
+`quantized` term internally on the way to emitting code indices; `encode_latents`
+returns that sum instead of discarding it. That gives native speaker encoding
+without new model code, which is most of milestone M2.
+
+## Consequences for packaging
+
+The *implementation* is shared; the *weights* are not.
+
+`fish_audio` ships Fish Audio S2 Pro. Echo is trained against the Fish S1 DAC
+(`jordand/fish-s1-dac-min`, a mirror of `fishaudio/openaudio-s1-mini`), and its
+PCA basis is fitted to that codec's latent space. The S2 technical report says S2
+retains S1's RVQ codec, and the shapes line up (10 codebooks, ~21 Hz), but
+"retains the codec" in a report can mean the architecture rather than identical
+weights -- and a retrained-but-isomorphic codec would yield wrong audio with no
+error raised anywhere. That is not a risk worth taking to save a download.
+
+`convert_echo_tts.py` therefore packages the S1 codec into Echo's GGUF under the
+`codec_weights` prefix, folding weight normalisation and dropping the 305 MB of
+regenerable buffers. Echo hands that tensor source to the framework runtime with
+a `FishDacCodecConfig` that differs from the defaults in only four fields
+(`sample_rate`, `frame_length`, `total_codebooks`, `quantizer_codebooks`) -- the
+rest already describe S1-DAC.
+
+Verified against the real checkpoint manifest: the folded output supplies all 220
+tensor names the codec loads, and the 541 stored tensors resolve to 455 after
+folding and buffer removal.
+
+## Caveat
+
+Everything above is derived from source reading plus file-size arithmetic. The
+parameter total agreeing with the reported size to three significant figures is
+strong evidence, but it is not the same as having loaded the tensors. The
+tensor-name check in `convert_echo_tts.py` and a parity run against
+`echo_tts_reference.py` remain the gates before any of this is claimed as done.

docs/community_models/echo_tts_dit_status.md

@@ -0,0 +1,133 @@
+# Echo-TTS DiT: implementation status
+
+## What exists
+
+| Component | State | Verification |
+| --- | --- | --- |
+| Byte tokenizer + WhisperD normalisation | complete | executed, output checked by hand |
+| PCA forward / inverse | complete | executed, cross-checked against numpy |
+| Flattening-point crop | complete | executed |
+| Euler dual-CFG sampler | complete | executed, matches a numpy transcription of `inference.py` to 6.6e-07 |
+| Timestep embedding | complete | matches `model.py` exactly (0.0 diff) |
+| Attention mask construction | complete | layout checked against upstream `cat()` semantics and ggml constraints |
+| DiT graph (encoders, joint attention, adaLN, blocks) | written | compiles against real headers; **never executed** |
+| Weight loading (1,117 tensors) | written | compiles; **tensor names unconfirmed against a real checkpoint** |
+| Conditioning / denoiser graph execution | written | compiles; **never executed** |
+| Fish codec `z_q` seam | not started | — |
+| Session integration | not started | — |
+
+The distinction in that last column is the important one. Everything above the
+line was run and compared against a reference. Everything below it has only been
+type-checked. A clean compile here means the framework APIs are used correctly;
+it says nothing about whether the numbers are right.
+
+## Design decisions worth reviewing
+
+### Flash attention in the DiT joint attention
+
+`joint_attention` uses `ggml_flash_attn_ext`, which never materialises the
+`(lanes, heads, seq, keys)` scores tensor. That tensor was the largest
+per-request allocation in the model:
+
+| Case | Keys | Scores tensor removed |
+| --- | ---: | ---: |
+| Typical (64 text bytes, 10 s reference) | 793 | 97 MB per attention |
+| Long text, 30 s reference | 1569 | 193 MB per attention |
+| Worst case (768 text, 5 min reference) | 3008 | 370 MB per attention |
+
+Live across 24 blocks with `ggml_gallocr` reuse, the practical saving is a few
+hundred MB to over a gigabyte, and flash attention is also faster.
+
+This was initially written with the explicit lowering on the belief that the
+speaker-unconditional CFG lane produces fully masked rows, which would make
+`-inf` softmax to NaN. That was wrong: `make_denoiser_mask` leaves the self block
+of every row unmasked, so a query always attends to at least its own 640
+positions and no row can be fully masked.
+
+Two details the flash path requires. The mask must be F16, so the masked value
+is `-65000` rather than `-1e9`; the latter converts to `-inf` in F16, which would
+reintroduce exactly the NaN hazard the explicit path was chosen to avoid. And
+`q->ne[2] % mask->ne[2]` and `q->ne[3] % mask->ne[3]` must both be zero, which
+holds because the mask carries a singleton head axis and matches the lane count.
+
+Set `AUDIOCPP_ECHO_TTS_NO_FLASH=1` to fall back to the explicit lowering and F32
+mask, for A/B comparison without a rebuild.
+
+The two encoders still use the explicit lowering. Their sequences are short (a
+few hundred tokens at most) so the scores tensors are small, and the speaker
+encoder is causal with no explicit mask, which the flash path rejects.
+
+### Speaker references are encoded in chunks
+
+`encode_speaker` splits the reference into ~29.7 s chunks (640 latents x 2048
+samples), zero-pads the last one, and concatenates the per-chunk latents,
+following `inference.py::get_speaker_latent_and_mask`. Upstream's comment calls
+that the longest chunk seen in training, so this is a fidelity matter as much as
+a memory one -- encoding several minutes in a single pass is a different
+computation from what the model saw.
+
+The memory difference is large, because the Fish encoder's first stages run at
+the full 44.1 kHz rate. A single 64-channel activation is 0.34 GB for one chunk
+against 3.04 GB for a 4m29s reference encoded in one pass, and several such
+tensors are live at once. Fixed-size chunks also mean one encode graph is built
+and reused across all chunks.
+
+After chunking, the dominant per-request allocation at long reference lengths is
+the persistent KV cache: 0.59 GB at 4m29s and 0.65 GB at the 297 s maximum,
+stored F32. Halving it to F16 is the obvious next step if that ever matters.
+
+### KV cache as a separate backend buffer
+
+The conditioning encoders and the denoiser are separate graphs so the encoders
+run once per request rather than once per sampler step. They share the cached
+projections through tensors allocated in their own `ggml_context` and backend
+buffer, referenced as leaves by both graphs. `ggml_gallocr` leaves
+already-allocated tensors alone, so the conditioning graph writes into them with
+`ggml_cpy` and the denoiser graph reads them directly.
+
+Consequence: changing text length or speaker length invalidates the cache and
+every graph built against it. `prepare_conditioning` tears all of it down and
+rebuilds, which is correct but means a request with new conditioning pays full
+graph construction. Acceptable given that a 40-step sample dominates.
+
+### Speaker KV scaling round-trips through the host
+
+`scale_speaker_kv` reads the cached tensors back, scales, and re-uploads, because
+the cache has no graph attached. This runs at most twice per request (once to
+apply, once to undo at the threshold) and touches at most 24 layers x 2 tensors.
+It is not on the per-step path. If it ever shows up in a profile, the fix is a
+tiny scaling graph rather than a host round trip.
+
+## Things most likely to be wrong
+
+Listed in rough order of how much damage they would do and how hard they would
+be to spot without a parity run:
+
+1. **Tensor names.** Derived from `model.py`'s module structure, corroborated by
+   a parameter count matching the published file size to ten digits, but never
+   resolved against an actual checkpoint. `convert_echo_tts.py --model-dir ...`
+   settles this in seconds and prints exactly what is wrong if anything is.
+2. **Half-head RoPE.** Heads 0-7 rotate, 8-15 do not. Implemented as
+   slice/rope/concat on the head axis. Wrong here means plausible-sounding but
+   incorrect audio, with no shape error.
+3. **Rotary pairing convention.** `GGML_ROPE_TYPE_NORMAL` (interleaved), matching
+   upstream's complex view of adjacent pairs. The in-tree `rf_dit.cpp` uses NEOX,
+   so copying from it would be wrong.
+4. **Speaker patchify reshape.** Folding `patch_size` frames into the feature
+   axis assumes row-major frame-then-channel ordering. A transposed reading would
+   still produce correct shapes.
+5. **adaLN chunk order.** `shift, scale, gate` from `cond_embed.chunk(3, -1)`.
+   A permutation here is silent.
+
+Items 2-5 are all caught by the per-block parity dumps from
+`tools/community_models/echo_tts_reference.py`, which is why that script dumps
+per-block activations at a fixed timestep rather than only the final output.
+
+## Next steps
+
+1. Run `convert_echo_tts.py` against the real checkpoint to confirm item 1.
+2. Build on a machine with a GPU and run the parity comparison per block.
+3. Split `fish_audio/codec.cpp`'s `build_decode_quantizer` at the `post_module`
+   boundary and expose the summed `quantized` term from
+   `build_encode_quantizer`, giving Echo decode and native speaker encoding.
+4. Wire the session: tokenize, encode speaker, sample, PCA inverse, decode, crop.