QVAC-19261: tts-cpp: add Parler-TTS engine (mini-v1 / large-v1, CPU)#92
Open
pratiknarola-t wants to merge 14 commits into
Open
QVAC-19261: tts-cpp: add Parler-TTS engine (mini-v1 / large-v1, CPU)#92pratiknarola-t wants to merge 14 commits into
pratiknarola-t wants to merge 14 commits into
Conversation
Review StatusCurrent Status: ❌ PENDING Pending reviews: Needs 1 Management or Team Lead, and 1 more from Management, Team Lead, or Member. |
New self-contained engine tts_cpp::parler: Flan-T5 description encoder (RMSNorm, relative-position bias, no attention scaling, gated-GELU) with per-description cross-KV precompute, MusicGen-style delay-pattern decoder LM (9 summed codebook embeddings, 9 LM heads, HF-faithful logits processor / min-new-tokens / stopping, exact-erf GELU, sinusoidal positions, token-major KV slab), and a DAC 44.1 kHz codec decoder (RVQ from_codes, snake activation with the +1e-9 guard, unpadded conv-transpose + view trim, F32 im2col convs). Single GGUF per model (parler.* metadata drives mini/large; no code branching) produced by scripts/convert-parler-to-gguf.py with two-sided tensor-name completeness checks and weight-norm folding; the T5 unigram tokenizer (precompiled charsmap + Metaspace + Viterbi) ships in the GGUF. Verified vs the HF PyTorch reference (scripts/dump-parler-reference.py fixtures): tokenizer ids exact (12-case corpus incl. unicode), T5 max_abs 1.3e-4, decoder prefill + teacher-forced steps max_abs <= 5.8e-4 with per-codebook argmax equality, delay/logits-processor decisions exact vs a trace of the real HF classes, DAC 121-124 dB SNR, and the full greedy e2e token trace matches HF exactly (429/429 steps; wav 121.5 dB SNR). Direct-vs-sched dispatch is bit-identical, ASAN is clean, and the full existing ctest suite stays green (83/83 runnable). The only shared-code touch is tts-cli model-family detection (parler.arch sniffed before the chatterbox tokenizer fallback) plus the --description flag; a standalone parler-cli mirrors supertonic-cli. F16 conversion keeps the T5 encoder in F32 (Flan-T5 activations overflow the f16 range).
- Repair weight-norm parametrizations from the safetensors ground truth after loading: transformers' sharded low-mem load leaves parametrizations.weight.original0/1 at init values for large-v1's DAC (mini-v1 single-file loads are unaffected; both v1 checkpoints ship byte-identical DAC weights), so without this the large fixtures encode a corrupted codec. - Prepend a cloning no-op logits processor so output_logits stays raw: the ParlerTTS EOS gate mutates scores in place, and for large-v1 (no min_new_tokens processor cloning before it) the -inf EOS mask was stamped into the dumped "raw" logits.
pratiknarola-t
force-pushed
the
qvac-19261-parler-tts
branch
from
July 15, 2026 14:00
dfaebde to
2618a23
Compare
- reset cross-KV state before the fallible re-encode work in parler_encode_description and invalidate the engine's description cache when encoding fails, so a failed re-encode can never leave a stale-but-plausible cross-attention state behind - validate max_frames in the engine: values <= n_codebooks cannot yield a single audio frame and now throw up front (with a regression test) - honor --output-sample-rate on the parler tts-cli path via the existing sinc resampler - wrong-arch engine test: skip honestly when the fixture is not staged and assert the arch-check message when it is - drop the unused n_gpu_layers engine option; record delayed_len from the reconstructed token sequence in the fixture dump metadata
- converter: per-tensor recipe tiers — bulk decoder matmuls at the target type, embedding tables + LM heads kept at q6_K/q8_0, T5 matmuls always q8_0 (quantized dots re-quantize activations per block, so the f16 activation-overflow trap does not apply); norms/biases/alphas/positions/ DAC stay f32 as before; per-tensor dequantize-roundtrip self-check - k-quants (Q4_K/Q6_K) are dequantize-only in gguf-py, so q4_k_m encodes through the built ggml library via ctypes (ggml_quantize_chunk — the same encoder inference runs against); sanitizer build dirs are skipped when auto-locating the library - PARLER_TEST_REPORT_ONLY=1 makes test-parler-t5/decoder print stage metrics and argmax-agreement without enforcing the f32 tolerance bars (non-finite output still fails); default strict behavior is unchanged - mini-v1 sizes: q8_0 1.01 GiB, q4_k_m 0.80 GiB, q4_0 0.82 GiB
Tier-decomposition on large-v1 isolated q4_k_m's audio collapse (argmax agreement 44%->17%, sampled generation running to max_length without EOS) to the 9 LM heads at q6_K alone: heads feed the logits directly, and q6_K's 2.0% per-row error (vs 0.6% at q8_0, measured) corrupts every step's sampling/EOS decision, drifting the trajectory off-manifold. Embedding tables at q6_K measure indistinguishable from the q8_0 baseline and keep that tier. Recipe split from 3 to 4 tiers accordingly; sizes are unchanged (heads are ~15M params). Agreement after the fix: mini 57->67%, large 14->40% with natural EOS on both.
…nization
parler-v1 has no text front-end and voices raw digits badly ('12' comes
out garbled while 'twelve' is clean, A/B-verified by ear). Normalize the
prompt (never the description) at the engine layer so every consumer
gets it: English cardinals incl. thousands separators, decimals,
ordinals; leading-zero and >15-digit runs are read digit-by-digit.
Default ON via EngineOptions::normalize_numbers; opt out with
--no-normalize-numbers. Normalized output for the digit fixture prompt
is bit-identical to synthesizing the hand-worded sentence.
The teacher-forced trace previously ran case0 only; factor it into a helper and aggregate argmax agreement across case0+case1. Trace length follows the fixture (dump --max-step-logits), now 200 steps per case: f32 passes strict parity 3600/3600, f16 reports 99.61% agreement.
Recipes chosen by a 200-step teacher-forced argmax grid over per-tier types (scripts/parler-quant-grid.py): f16 LM heads are the dominant quality lever (+3.4pt at 8-bit bulk for +10 MB; +20pt on large where head noise dominated), embedding tables matter least. Shipped recipes (mini agree vs f32, f16 ceiling 99.61%): q8_0 = Q8_0 bulk + F16 tables/heads 98.06% 1.16 GB q6_k = Q6_K bulk/tables + F16 heads 94.94% 0.98 GB q5_0 = Q5_0 bulk 90.36% 0.91 GB q4_k_m = unchanged 83.14% 0.86 GB q4_0 is dropped (q4_k_m beats it at equal size). Large-v1: 96.64 / 92.03 / 70.00 / 65.92, f16 ceiling 99.42. Also: --recipe per-tier override, optional importance-matrix weighting (scripts/compute-parler-imatrix.py + --imatrix, encoded via ggml_quantize_chunk), and a q8_0 byte-parity cross-check between the gguf-py and ggml encoders.
Sub-q6 tiers measure well below the quality floor (mini q5_0 90.4% / q4_k_m 83.1% argmax vs 94.9% for q6_k; the gap widens on large-v1: 70.0% / 65.9% vs 92.0%), so the shipped recipe set is q6_k and q8_0 only. Dropped combinations stay reproducible for research through --recipe overrides on a q6_k base; the grid driver's dormant arms are rewritten accordingly.
Indic-class checkpoints differ from mini/large in three storage details, all handled without model-specific branching: - The repo tokenizer is a SentencePiece-BPE PROMPT tokenizer (90714 vocab, 152116 merges, byte fallback, Metaspace prepend-first); descriptions keep the Flan-T5 unigram tokenizer, which the converter now fetches from the text encoder's own repo and embeds as before. New parler.prompt_tokenizer.* GGUF keys carry the BPE payload; the engine routes prompts through a new HF-faithful merge-rank BPE encoder when present. mini/large GGUFs re-convert byte-identically. - LM heads are stored fused ([9*vocab, d_model]); the converter splits them into the nine per-head tensors (rows k*V:(k+1)*V = head k, the exact inverse of the HF fused-view reshape). - The DAC is stored under transformers-DacModel names with weight-norm pre-folded; a second mapping branch moves them directly (the weights are numerically identical to mini's dac_44khz codec). The e2e test now reads its case texts from the fixture dir's meta.json so one binary serves every model's fixtures; indic fixture instances of the tokenizer/t5/decoder/dac/e2e/sched tests are registered alongside a new prompt-tokenizer parity test (20-case multilingual corpus, exact ids vs the HF fast tokenizer). Verified: full ctest green including the indic instances; teacher-forced argmax 3600/3600 at f32; greedy e2e trace 429/429 exact; DAC parity 124 dB SNR.
…mpts The indic checkpoint only voices numerals it saw in training: native Devanagari-class digits work, ASCII digits garble everywhere, and English word-injection reads wrong inside Indic text (by-ear findings). AI4Bharat ships no text front-end at all — their training pipeline verbalized digits, so raw digits are out-of-distribution. New normalize_numbers_indic(), routed for models with a prompt tokenizer: ASCII digit runs adopt the nearest letter context — an Indic script wins and the run becomes that script's native digits (positional decimal, digit-by-digit; all 13 digit-bearing scripts of the 21 languages), a Latin letter wins and the run falls through to the existing English-words pass. Native numerals always pass through. Best effort by design: scripts whose numerals were rare in training (e.g. Gujarati) remain unvoiceable — a model limitation; per-language number-words belong upstream where the prompt language is known. Review findings fixed: the shared danda (encoded in the Devanagari block) no longer resolves script context for neighbouring scripts; multiplication/division signs are not Latin letters; malformed UTF-8 bytes pass through verbatim but carry no context.
…cripts/parler/ Pure reorganization, no behavior change: the parler family (incl. the indic variant) now lives in its own folders like lavasr, with the redundant parler_ filename prefix dropped (the folder scopes the names). Includes, CMake source lists, script-relative paths (ggml-lib discovery moved one level deeper) and README paths updated; test target names and the public include/tts-cpp/parler/engine.h are unchanged. Verified: full rebuild + 16/16 parler ctest green; scripts py_compile clean and the converter's ggml-lib auto-discovery re-verified from the new location.
Add build_description() (include/tts-cpp/parler/description.h): renders a voice description from structured fields (voice, emotion, pitch, pace, expressivity, noise, reverb, quality) in the models' training-caption phrasing, so conditioning stays in-distribution. Emotion is a closed, case-insensitive set of the 12 trained speaking styles and renders a tone clause plus the trailing 'The intended style is <x>.' anchor the training captions used (RASMALAI, arXiv 2505.18609, Table 1). The all-default spec renders the models' recommended fallback caption verbatim, so everything works with no configuration. parler-cli grows the matching template flags; --description becomes optional and is mutually exclusive with them (presence-based check, empty value rejected). Exact-string unit tests pin the renderer; validation errors list the valid values. ASCII-only lowering keeps validation locale-independent, and DescriptionSpec stays header-only so shared-lib consumers can link it under hidden visibility.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new self-contained
tts_cpp::parlerengine: description-conditioned TTS forparler-tts/parler-tts-mini-v1andparler-tts/parler-tts-large-v1on CPU. mini/large differences are pure GGUF metadata (parler.*keys) — no code branching.Pipeline: Flan-T5 encoder (description → cross-attention K/V, precomputed once and cached per description; RMSNorm, relative-position bias, no attention scaling, gated-GELU) → delay-pattern decoder LM (9 summed codebook embeddings, 9 LM heads, MusicGen-style stagger with HF-faithful EOS gating / min-new-tokens / stopping, exact-erf GELU, sinusoidal positions, token-major KV slab) → DAC 44.1 kHz codec (RVQ
from_codes, snake activation incl. the+1e-9guard, unpaddedconv_transpose_1d+ view trim, F32-im2col convs) → mono PCM.Additive everywhere: the only shared-code touch is
tts-clifamily detection (parler.archsniffed before the chatterboxtokenizer.ggml.tokensfallback) plus a--descriptionflag; a standaloneparler-climirrorssupertonic-cli. Graph dispatch uses the sharedsched_dispatchdual path (fresh graph per pass, per the single-use contract).Conversion:
scripts/convert-parler-to-gguf.pyemits ONE GGUF per model (T5 + decoder + DAC + T5 unigram tokenizer with precompiled charsmap), with two-sided tensor-name completeness checks and weight-norm folding.--dtype f16keeps the entire T5 encoder F32 (Flan-T5 activations overflow the f16 range; ggml's f16 mul_mat converts activation rows to f16 for the dot product → NaN), plus norms/biases/alphas/positional table/DAC.Verification (fixtures from
scripts/dump-parler-reference.py, HF PyTorch reference)TTS_CPP_FORCE_SCHED=1Sampled decoding (model default: temp 1.0, top-k 50) terminates via natural EOS — the delay-pattern stopping that is known-broken in the TTS.cpp reference implementation (their issue #50).
Upstream quirks found (and worked around in the dump script — not engine code)
parametrizations.weight.original0/1at init values for large-v1's DAC (single-file mini is unaffected; both v1 checkpoints ship byte-identical DAC weights). Stock HF parler-large actually runs with that corrupted codec on modern stacks; the GGUF ships the true weights, and the reference dump repairs the parametrizations before dumping.ParlerTTSLogitsProcessormutates scores in place, contaminatingoutput_logitswhen nothing clones before it (large has no min-new-tokens processor) — the dump prepends a cloning no-op.Out of scope / follow-ups
scripts/setup-ggml.shpins a bundled-ggml SHA that predates the lavasr custom ops, so the bundled dev build of master is currently broken independent of this PR (worked around locally by checking out speech HEAD).