Skip to content

glm_tts_validation.md

Latest commit

 

History

History
428 lines (349 loc) · 16.2 KB

glm_tts_validation.md

File metadata and controls

428 lines (349 loc) · 16.2 KB
Raw

GLM-TTS validation

This report covers the community GLM-TTS implementation, its prepared safetensors package, and a standalone mixed Q8/F16 GGUF. Measurements were captured per request, with deterministic Python parity completed before the normal sampled path was measured.

Implementation scope

  • Adds the community glm_tts loader and offline tts / clon session.
  • Runs the ChatGLM tokenizer, Llama speech-token generator, Whisper-VQ reference encoder, CAMPPlus speaker encoder, Flow/DiT mel generator, and HiFT vocoder natively through the shared framework.
  • Adds the glm_tts model-manager package, package spec, CLI/server registry entry, user documentation, converter, and validation probes.
  • Supports prepared safetensors packages and standalone multi-component GGUF files with embedded configs, tokenizer files, and package metadata.
  • Keeps reusable component runtimes resident, caches prepared references, reuses shape-compatible graphs, and restricts the Llama output projection to GLM-TTS audio-token rows.
  • Extends the shared OuteTTS Llama runtime and Whisper embedding surface. GLM-TTS's CAMPPlus partial-segment normalization is additive and explicitly enabled only by the GLM-TTS session; the established shared default remains unchanged for Chatterbox, IndexTTS2, and Seed-VC.

Test environment

Tests were run on 2026-07-24 with:

  • Windows 11 and MSVC 2022 14.43
  • NVIDIA RTX 3090 24 GB
  • CUDA toolkit 12.4.131
  • PyTorch 2.5.1+cu121 and Transformers 5.13.0
  • GLM-TTS branch based on audio.cpp main f8fb0c1

The official Python reference was checked out at:

..\_reference\GLM-TTS

The prepared package id and paths were:

model-manager package: glm_tts
safetensors package:   ..\models\GLM-TTS
standalone GGUF:       ..\models\GLM-TTS-Q8\GLM-TTS_Q8.gguf
reference WAV:         ..\SAMPLES\EN_2.wav
reference transcript:  If you actually care about security

Install the package with:

python tools\model_manager_deprecated.py install glm_tts --models-dir ..\models

The installer downloads zai-org/GLM-TTS, converts Flow and HiFT from the official PyTorch checkpoints to safetensors, exports the tokenizer resources, and installs matching CAMPPlus safetensors weights.

Exact build commands

CPU:

.\scripts\build_windows.ps1 `
  -Preset windows-cpu-release -ConfigureOnly

cmake -S . -B build\windows-cpu-release `
  -DENGINE_BUILD_WARMBENCH=ON

cmake --build build\windows-cpu-release --parallel 8 --target `
  audiocpp_cli audiocpp_server `
  campplus_shared_default_probe `
  glm_tts_warm_bench glm_tts_llama_probe `
  glm_tts_frontend_probe glm_tts_campplus_probe `
  glm_tts_flow_conditioned_probe

CUDA:

$cuda = "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4"
$env:CUDA_PATH = $cuda
$env:CUDAToolkit_ROOT = $cuda
$env:CUDA_TOOLKIT_ROOT_DIR = $cuda
$env:CUDACXX = Join-Path $cuda "bin\nvcc.exe"
$env:CMAKE_CUDA_COMPILER = $env:CUDACXX
$env:NVCC_PREPEND_FLAGS = "-allow-unsupported-compiler"

.\scripts\build_windows.ps1 `
  -Preset windows-cuda-release -ConfigureOnly

cmake -S . -B build\windows-cuda-release `
  -DENGINE_BUILD_WARMBENCH=ON `
  -DCMAKE_CUDA_COMPILER="$env:CUDACXX"

cmake --build build\windows-cuda-release --parallel 8 --target `
  audiocpp_cli audiocpp_server audiocpp_gguf `
  campplus_shared_default_probe `
  glm_tts_warm_bench glm_tts_llama_probe `
  glm_tts_frontend_probe glm_tts_campplus_probe `
  glm_tts_flow_conditioned_probe

Both clean builds completed. The loader catalog advertises glm_tts with offline tts and clon tasks and the /v1/audio/speech endpoint.

Standalone GGUF

The quality-tested package keeps the autoregressive Llama group in F16 and stores the speech tokenizer, Flow, HiFT, and CAMPPlus groups as Q8_0:

.\build\windows-cuda-release\bin\audiocpp_gguf.exe `
  --input llama_weights=..\models\GLM-TTS\llm\model.safetensors.index.json `
  --input speech_tokenizer_weights=..\models\GLM-TTS\speech_tokenizer\model.safetensors `
  --input flow_weights=..\models\GLM-TTS\flow\model.safetensors `
  --input hift_weights=..\models\GLM-TTS\hift\model.safetensors `
  --input campplus_weights=..\models\GLM-TTS\frontend\campplus.safetensors `
  --root ..\models\GLM-TTS `
  --family glm_tts `
  --model-spec .\model_specs\glm_tts.json `
  --type q8_0 `
  --keep-type "llama_weights/*=f16" `
  --overwrite `
  --output ..\models\GLM-TTS-Q8\GLM-TTS_Q8.gguf

Inspection:

.\build\windows-cuda-release\bin\audiocpp_gguf.exe `
  --inspect ..\models\GLM-TTS-Q8\GLM-TTS_Q8.gguf
tensors=2076
rank0_scalars=122
embedded_sidecars=true
embedded_model_spec=true
model_spec_family=glm_tts
namespace=campplus_weights
namespace=flow_weights
namespace=hift_weights
namespace=llama_weights
namespace=speech_tokenizer_weights

The file is 5,143,813,728 bytes and has SHA-256 28E2B66333FD5F051466520F6A8A5F26CBA7F3B768891006168A4560972107CB. It needs no external model spec, tokenizer, config, or weight file.

Exact run commands

Standalone CLI:

.\build\windows-cuda-release\bin\audiocpp_cli.exe `
  --task clon --family glm_tts `
  --model ..\models\GLM-TTS-Q8\GLM-TTS_Q8.gguf `
  --backend cuda `
  --voice-ref ..\SAMPLES\EN_2.wav `
  --reference-text "If you actually care about security" `
  --text "Hello from G L M T T S. This voice was generated locally." `
  --top-k 25 --top-p 0.8 --temperature 1.0 `
  --seed 0 --max-tokens 256 `
  --out ..\outputs\glm_tts_pr_validation\standalone_q8_cli.wav `
  --log

The one-file path was materialized to audio.cpp's temporary GGUF resource directory, produced 131 speech tokens, 262 mel frames, and 5.24 seconds of valid 24 kHz mono PCM. ffmpeg -v error -i ... -f null - completed without an error. The output SHA-256 is DF8ACFDCD308EA10FDBEF37D03581DAA4E2222922E2DC54A3DD417EEF0263450.

Server configuration:

{
  "host": "127.0.0.1",
  "port": 18083,
  "backend": "cuda",
  "device": 0,
  "threads": 8,
  "lazy_load": false,
  "models": [
    {
      "id": "glm-tts-q8",
      "family": "glm_tts",
      "path": "E:/models/GLM-TTS-Q8/GLM-TTS_Q8.gguf",
      "task": "clon",
      "mode": "offline",
      "default_voice_preset": {
        "voice_ref": "E:/samples/EN_2.wav",
        "reference_text": "If you actually care about security"
      }
    }
  ]
}
.\build\windows-cuda-release\bin\audiocpp_server.exe `
  --config ..\outputs\glm_tts_pr_validation\server.cuda.json `
  --log

curl.exe http://127.0.0.1:18083/v1/audio/speech `
  -H "Content-Type: application/json" `
  -o ..\outputs\glm_tts_pr_validation\server_q8.wav `
  -d '{\"model\":\"glm-tts-q8\",\"input\":\"Hello from the GLM TTS server.\",\"max_tokens\":256,\"seed\":0,\"temperature\":1.0,\"top_k\":25,\"top_p\":0.8}'

GET /health returned {"status":"ok","backend":"cuda","models":1}. The speech request produced a valid WAV with SHA-256 9C818AB9804FF92A04F0EC5AB22B06B45F6E9DA60704F1A851C1367787F5169C.

Deterministic Python parity

The official model uses RAS sampling with temperature 1.0, top-k 25, and top-p 0.8. Those defaults remain unchanged for normal synthesis. For logic parity only, the upstream RAS selection and audio.cpp sampling were changed to argmax. Native FP32 weights were used on both sides. This isolates prompt, token, duration, Flow, and vocoder logic from stochastic sampling drift.

Python:

python tests\glm_tts\reference_warm_bench.py `
  ..\_reference\GLM-TTS `
  --request-file tests\glm_tts\parity_greedy_requests.json `
  --voice-ref ..\SAMPLES\EN_2.wav `
  --reference-text "If you actually care about security" `
  --out-dir ..\outputs\glm_tts_pr_validation\python_f32_greedy `
  --greedy

audio.cpp:

.\build\windows-cuda-release\bin\glm_tts_warm_bench.exe `
  --model ..\models\GLM-TTS `
  --backend cuda --threads 8 `
  --voice-ref ..\SAMPLES\EN_2.wav `
  --reference-text "If you actually care about security" `
  --request-file tests\glm_tts\parity_greedy_requests.json `
  --iterations 1 `
  --session-option glm_tts.weight_type=f32 `
  --audio-out-dir ..\outputs\glm_tts_pr_validation\cpp_f32_greedy

Per-request results:

Request Speech tokens C++ / Python WAV frames C++ / Python WAV cosine Log-mel cosine C++ wall Python wall
short_cold 75 / 75, exact 72,000 / 72,000 0.150483 0.992423 2,684.2 ms 2,687.7 ms
short_repeat 75 / 75, exact 72,000 / 72,000 0.150483 0.992423 887.8 ms 1,929.9 ms
technical 139 / 139, exact 133,440 / 133,440 0.144861 0.991100 1,442.5 ms 3,208.2 ms
longform 305 / 305, exact 292,800 / 292,800 -0.012583 0.991946 3,161.7 ms 6,505.0 ms

The low waveform cosine is phase-sensitive and reflects independent HiFT excitation. Frame counts and token ids are exact, while log-mel similarity is 0.9911 to 0.9924. No parity-only trace dumping is enabled during normal requests.

Component boundaries

The component probes use fixed inputs and fixed Flow noise:

Component Compared boundary Result
Text tokenizer and prompt normalized strings and full prompt ids exact
Whisper-VQ reference speech-token ids 63 / 63 exact
Llama greedy generated speech-token ids exact for 75, 139, and 305-token requests
CAMPPlus 192-value speaker embedding cosine 0.999999999977; max abs 2.52e-05; RMSE 8.86e-06
CAMPPlus shared default deterministic 250 x 80 input against unmodified main byte-for-byte exact; max abs 0
Flow, 10 steps 114 x 80 mel cosine 0.999999999231; max abs 0.001515; RMSE 0.000279

CampplusEncoderConfig::normalize_partial_segment_by_full_length defaults to false. Only GLM-TTS sets it to true to match the published GLM-TTS CAMPPlus ONNX graph. Chatterbox, IndexTTS2, Seed-VC, and other shared callers continue through the previous pooling graph. The regression probe campplus_shared_default_probe was compiled once against unmodified main and once against this branch; both emitted the same 192 serialized values.

The request options flow_noise_path, hift_source_random_path, and hift_prior_noise_count expose stochastic boundaries for targeted parity tests without adding normal-request log spam.

Normal sampled-path parity

The normal official settings were then restored: RAS, temperature 1.0, top-k 25, top-p 0.8, and seed 0. Native FP32 and the mixed Q8/F16 GGUF were compared against the official FP32 output per request:

Request Native FP32 log-mel / frames Mixed GGUF log-mel / frames
short_cold 0.902897; 77,760 / 89,280 0.902589; 77,760 / 89,280
short_repeat 0.902897; 77,760 / 89,280 0.902589; 77,760 / 89,280
technical 0.929266; 150,720 / 147,840 0.857288; 135,840 / 147,840
longform 0.862186; 313,920 / 306,240 0.861948; 313,920 / 306,240

For the short sampled request, token ids matched Python through token 37 and then diverged stochastically. The deterministic tests above demonstrate that the model logic and frame accounting match when sampling variation is removed.

Per-request performance

The normal mixed Q8/F16 matrix was run in three fresh processes. Each request was executed twice inside one long-lived session. Iteration 1 can build a new shape-specific graph; iteration 2 measures reuse of that request shape.

.\build\windows-cuda-release\bin\glm_tts_warm_bench.exe `
  --model ..\models\GLM-TTS-Q8\GLM-TTS_Q8.gguf `
  --backend cuda --threads 8 `
  --voice-ref ..\SAMPLES\EN_2.wav `
  --reference-text "If you actually care about security" `
  --request-file tests\glm_tts\warm_bench_requests.json `
  --audio-out-dir ..\outputs\glm_tts_pr_validation\q8_run1\audio `
  --log-file ..\outputs\glm_tts_pr_validation\q8_run1\trace.log

Mean and sample standard deviation across the three processes:

Request Iteration Audio Wall mean ± SD RTF mean ± SD
short_cold 1, reference miss 3.24 s 2,469.5 ± 637.1 ms 0.7622 ± 0.1966
short_cold 2 3.24 s 626.2 ± 2.4 ms 0.1933 ± 0.0008
short_repeat 1 3.24 s 617.1 ± 2.4 ms 0.1905 ± 0.0007
short_repeat 2 3.24 s 618.1 ± 2.2 ms 0.1908 ± 0.0007
technical 1 5.66 s 1,023.2 ± 2.6 ms 0.1808 ± 0.0005
technical 2 5.66 s 1,002.8 ± 2.6 ms 0.1772 ± 0.0005
longform 1 13.08 s 2,231.0 ± 3.6 ms 0.1706 ± 0.0003
longform 2 13.08 s 2,189.1 ± 6.1 ms 0.1674 ± 0.0005

The four Python sampled requests took 3,314.2, 2,429.6, 3,808.6, and 7,151.9 ms respectively. Using the comparable reference-cached audio.cpp requests, the mixed GGUF path was approximately 3.2x to 3.9x faster than the official Python reference.

All 24 WAVs from the three mixed-GGUF runs reduced to three hashes: one shared by the two identical short cases, one for technical, and one for longform. The repeated outputs are byte-identical across processes.

Memory and mem-saver tradeoff

Memory was sampled every 200 ms with nvidia-smi:

Path Peak GPU memory Warm repeated short request
Native FP32 audio.cpp 8,846 MiB 887.8 ms
Official Python FP32 8,628 to 8,787 MiB 1,929.9 ms
Mixed GGUF, resident 5,320 MiB 608.6 ms
Mixed GGUF, balanced glm_tts.mem_saver=true 3,878 MiB 622.3 ms
Mixed GGUF, glm_tts.aggressive_mem_saver=true 3,372 MiB 3,761.7 ms

Balanced mem-saver releases the approximately 1.58 GiB reference-only Whisper-VQ and CAMPPlus path after its output is cached, but keeps Llama, Flow, and HiFT warm. Across three warm repeats it saved 1,442 MiB (27.1%) versus resident mode while adding 13.7 ms (2.3%) mean latency. A reference-cache miss releases the generation path before reconstructing the reference encoders so the groups do not overlap in VRAM.

Aggressive mem-saver preserves the former unload-every-stage policy. It saved another 506 MiB, but rebuilding Llama, Flow, and HiFT made warm requests about 6.0x slower than balanced mode. All balanced, aggressive, and resident outputs reduced to the same SHA-256 hash: 72672D49755B1B3987318EC264FBEC0DBB0AA0175461818AAC56BD1404F56E88.

Windows reported peak working set around 25.0 GiB and private bytes between 33.8 and 44.5 GiB while loading this model. These counters include mapped GGUF pages, converted host tensors, and backend virtual reservations; private bytes must not be interpreted as committed physical RAM.

CPU path

A full short request was executed with the CPU build:

.\build\windows-cpu-release\bin\glm_tts_warm_bench.exe `
  --model ..\models\GLM-TTS-Q8\GLM-TTS_Q8.gguf `
  --backend cpu --threads 16 `
  --voice-ref ..\SAMPLES\EN_2.wav `
  --reference-text "If you actually care about security" `
  --text "Hello." --iterations 1 `
  --audio-out-dir ..\outputs\glm_tts_pr_validation\cpu_short\audio

It produced 0.64 seconds of valid audio in 4,963.3 ms, RTF 7.755. CPU is functional, but CUDA is the practical tested backend for interactive use.

Generated artifacts

..\outputs\glm_tts_pr_validation\standalone_q8_cli.wav
..\outputs\glm_tts_pr_validation\server_q8.wav
..\outputs\glm_tts_pr_validation\python_f32
..\outputs\glm_tts_pr_validation\python_f32_greedy
..\outputs\glm_tts_pr_validation\cpp_f32
..\outputs\glm_tts_pr_validation\cpp_f32_greedy
..\outputs\glm_tts_pr_validation\q8_run1
..\outputs\glm_tts_pr_validation\q8_run2
..\outputs\glm_tts_pr_validation\q8_run3
..\outputs\glm_tts_pr_validation\components
..\outputs\glm_tts_pr_validation\cpu_short
..\models\GLM-TTS-Q8\GLM-TTS_Q8.gguf

Known limitations

  • GLM-TTS supports Chinese and English.
  • Both advertised task names use the same zero-shot reference-conditioned path. A clean reference WAV and its exact transcript are required.
  • Normal sampled runs can diverge from Python because small numeric differences alter autoregressive multinomial choices. Greedy FP32 token and frame parity is exact.
  • Independent HiFT excitation makes waveform cosine phase-sensitive even when token ids and frame counts match; log-mel similarity is the useful metric.
  • The mixed GGUF keeps Llama in F16 because a fully Q8 autoregressive head had a larger quality and token-selection penalty.
  • CUDA and CPU were tested. Vulkan and Metal were not tested.
  • The upstream WeText normalizer was unavailable in the Windows Python environment. Parity uses English text that the identity fallback does not alter.