Document Voxtral Q4 GGUF results

Author: 0xShug0

README.md

@@ -50,7 +50,7 @@ audio.cpp would not be moving this quickly without generous contributors bringin
 
 Task tags: `TTS` text to speech, `Clone` voice cloning, `VC` voice conversion, `ASR` speech recognition, `Align` forced alignment, `VAD` voice activity detection, `Diar` speaker diarization, `Codec` audio codec, `Sep` source separation, `Music` music/song generation, `SFX` sound effects, `Edit` audio/music editing, `Design` voice design, `Dialogue` multi-speaker dialogue TTS, `Ctrl` TTS/clone voice control such as emotion, style, instruction, caption, or non-verbal tag control.
 
-Runtime tags: safetensors is the default model loading path. `GGUF 16/Q8` means both 16-bit and `q8_0` GGUF paths are tested; `GGUF Q8` means only `q8_0` is tested; `GGUF F32` means the original-F32 GGUF path is tested. See [docs/gguf.md](docs/gguf.md) for precision/status details. `Bundled` means the tiny runtime asset ships under `assets/framework/models` and needs no separate model download. `Stream` means the family exposes a streaming server/session path.
+Runtime tags: safetensors is the default model loading path. `GGUF 16/Q8/Q4` means those GGUF precision or quantization paths are tested; `GGUF Q8` means only `q8_0` is tested; `GGUF F32` means the original-F32 GGUF path is tested. See [docs/gguf.md](docs/gguf.md) for precision/status details. `Bundled` means the tiny runtime asset ships under `assets/framework/models` and needs no separate model download. `Stream` means the family exposes a streaming server/session path.
 
 | Family | Task | Lang | Variants | Runtime |
 |---|---|---|---|---|
@@ -80,7 +80,7 @@ Runtime tags: safetensors is the default model loading path. `GGUF 16/Q8` means
 | **vevo2** | TTS, Music, VC, Edit | en, zh | Vevo2 with Qwen2.5-0.5B AR model | GGUF 16 |
 | **vibevoice** | TTS, Dialogue | en, zh | VibeVoice-1.5B, VibeVoice-7B | GGUF 16/Q8 |
 | **vibevoice_asr** | ASR | auto | VibeVoice ASR | GGUF 16/Q8 |
-| **voxtral_realtime** | ASR | auto | Voxtral-Mini-4B-Realtime-2602 | GGUF 16/Q8, Stream |
+| **voxtral_realtime** | ASR | auto | Voxtral-Mini-4B-Realtime-2602 | GGUF 16/Q8/Q4, Stream |
 | **voxcpm2** | TTS, Clone, Design, Ctrl | ar, da, de, el, en, es, fi, fr, he, hi, id, it, ja, km, ko, lo, ms, my, nl, no, pl, pt, ru, sv, sw, th, tl, tr, vi, zh | VoxCPM2-2B, 48 kHz | GGUF 16/Q8, Stream |
 | **index_tts2** | TTS, Clone, Ctrl | zh, en | IndexTTS-2 | GGUF 16/Q8 |
 | **irodori_tts** | TTS, Clone, Design, Ctrl | ja | Irodori-TTS-500M-v3, Irodori-TTS-600M-v3-VoiceDesign | GGUF 16/Q8 |

docs/asr.md

@@ -255,12 +255,13 @@ audiocpp_cli --task asr --family vibevoice_asr --model models/VibeVoice-ASR --ba
 
 ## Voxtral Realtime
 
-Voxtral Realtime is a Mistral realtime ASR model with offline and streaming sessions. The model manager installs the Q8_0 standalone GGUF package by default; native Hugging Face directories and other standalone GGUF variants can also be used when provided directly.
+Voxtral Realtime is a Mistral realtime ASR model with offline and streaming sessions. The model manager installs the Q8_0 standalone GGUF package by default; native Hugging Face directories and other standalone GGUF variants can also be used when provided directly. A Q4_K GGUF package is also available for lower memory use and faster CUDA runs; in a quick path check its transcripts matched Q8_0 except for one capitalization-only difference.
 
 | Field | Value |
 |---|---|
 | Family | `voxtral_realtime` |
 | Model path | `models/Voxtral-Mini-4B-Realtime-2602-GGUF/voxtral-mini-4b-realtime-2602-q8_0.gguf` when installed through the model manager |
+| GGUF variants | `bf16`, `q8_0`, `q4_k` |
 | Task | `asr` |
 | Modes | `offline`, `streaming` |
 | Output | Transcription text |

docs/gguf.md

@@ -94,6 +94,12 @@ Status labels:
 | `voxcpm2` | Done | Pass | Pass | Pass (ASR match, drift) | Pass (ASR match, drift) |
 | `voxtral_realtime` | Done | Pass | --- | Pass | Pass |
 
+Additional lower-bit checks:
+
+| Family | Format | Tested |
+|---|---|---|
+| `voxtral_realtime` | `q4_k` | Pass (quick CUDA check; transcripts match Q8 except one capitalization-only difference) |
+
 Q8 packaging notes:
 
 - `chatterbox` Q8 is intentionally mixed type. Graph-sensitive scalar, norm,
@@ -109,6 +115,9 @@ Q8 packaging notes:
   projection or linear weights, while leaving the speaker encoder, lookup, and
   codebook-sensitive tensors unquantized. Quantizing those speaker-side tensors
   can produce long-form quality problems such as large silence.
+- `voxtral_realtime` also has a tested `q4_k` package. In a quick CUDA path
+  check it was smaller and faster than Q8_0, while transcript output matched
+  Q8_0 except for one capitalization-only difference.
 
 ## Build The Converter
 

docs/reports/gguf_q8_performance.md

@@ -22,6 +22,21 @@ Q8 gives the clearest end-to-end wins on the larger AR-style models:
 
 Smaller or already memory-light models still load and run with Q8, but the speed gain can be modest. Treat Q8 as a measured route choice, not a guaranteed win for every model.
 
+## Voxtral Q4_K Quick Check
+
+Voxtral also has a `q4_k` GGUF package. In this quick CUDA path-test check,
+Q4_K was smaller and faster than Q8_0 while producing effectively the same
+transcripts. Four requests matched exactly; one differed only by capitalization
+(`Mother Nature` vs `mother nature`).
+
+| Route | Q8 RTF | Q4_K RTF | Q4_K vs Q8 |
+|---|---:|---:|---:|
+| Offline short | 0.0862 | 0.0629 | 1.37x |
+| Offline medium | 0.0643 | 0.0476 | 1.35x |
+| Offline longer | 0.0576 | 0.0439 | 1.31x |
+| Offline sampled | 0.0630 | 0.0500 | 1.26x |
+| Streaming path | 0.1036 | 0.0904 | 1.15x |
+
 ## TTS Offline Long-Lived Session
 
 | Model | 16-bit speed vs real time | Q8 speed vs real time | Q8 vs 16-bit | 16-bit peak VRAM | Q8 peak VRAM |