feat(models): add IBM Granite Speech 5.0 470M TurboCTC ASR engine

[ampersandru]

Summary

Adds support for IBM Granite Speech 5.0 470M TurboCTC, an ultra-fast, compact 470M parameter English Automatic Speech Recognition (ASR) model based on a Conformer CTC architecture.

This integration includes:

• C++ Core Engine: HTK mel filterbank frontend, 16-layer Conformer with folded depthwise batch-norm, zero-copy Shaw relative positional embeddings, layer 8 self-conditioning, and greedy CTC collapse.
• Quantized GGUF Support: Standalone 8-bit (Q8_0) and float (F16/F32) packaging with embedded tokenizer JSON and model specs.
• ByteLevel BPE Decoding: Full GPT-2 byte-fallback decoding for clean punctuation and whitespace handling.
• Offline & Streaming Modes: Support for standard offline file transcription, VAD/fixed chunking, and live PCM audio streaming.
• Server Integration: Registered in model catalog and server.json for OpenAI-compatible /v1/audio/transcriptions and /v1/audio/transcriptions/live endpoints.

---

Technical Details

1. Frontend:

   • 16 kHz sample rate, 80-bin HTK mel spectrogram (0–8000 Hz) with 80.0 dB log-magnitude flooring.
   • First-order deltas ($win=3$), $2\times$ frame stacking + stride-2 subsampling (yielding 160-dim feature vectors at 25 Hz / 40 ms per frame).

2. Conformer Encoder:

   • 16 Conformer blocks ($d_{model}=1024, d_{ff}=4096, H=8, d_{head}=128$).
   • Subsampling depthwise convolutions at layers 0 and 1 downsampling the time axis to 12.5 Hz (80 ms per frame).
   • Pre-folded batch normalization in depthwise convolutions for zero runtime overhead.
   • Pre-indexed Shaw relative positional bias table ($128 \times 128 \times 128$) sliced dynamically using ggml_view_3d with batched matrix multiplications for $O(1)$ lookup without dynamic graph allocation.
   • Mid-layer CTC self-conditioning at layer 8 projected into layer 9.

3. Tokenizer & CTC Decoding:

   • 40,961 vocabulary with GPT-2 ByteLevel BPE tokenizer fallback in HuggingFaceTokenizerJson.
   • Greedy CTC blank (<|blank|>, ID 0) collapse.

---

Performance & Verification

Tested on LibriSpeech librispeech_test_clean_6930-75918-0000.wav (3.0s audio clip):

Mode	Platform	Latency	Real-Time Factor (RTF)	Parity Result
Q8_0 GGUF (481 MB)	CPU (AVX512 / OpenMP)	~73.8 ms	0.024 (~40x realtime)	100% exact match
Q8_0 GGUF (481 MB)	NVIDIA CUDA (RTX 5060 Ti)	~18.5 ms	0.006 (~160x realtime)	100% exact match

Golden Parity Test Output

Raw transcript: concord returned to its place amidst the tents Normalized actual: concord returned to its place amidst the tents Normalized expected: concord returned to its place amidst the tents PASS: Granite 5 ASR golden transcription verified successfully.

---

server.json

{
  "models": [
    {
      "id": "granite5asr",
      "family": "granite5asr",
      "path": "/models/granite5asr-q8_0.gguf",
      "task": "asr",
      "mode": "offline"
    }
  ]
}

CLI & Server Usage

CLI

# Offline transcription (safetensors or standalone Q8_0 GGUF)
audiocpp_cli --task asr --family granite5asr --model models/granite5asr-q8_0/granite5asr-q8_0.gguf --audio sample.wav

# Streaming mode
audiocpp_cli --task asr --family granite5asr --mode streaming --model models/granite5asr-q8_0/granite5asr-q8_0.gguf --audio sample.wav

python tools/community_models/convert_granite5asr.py \
    --checkpoint granite5asr \
    --converter build/bin/audiocpp_gguf \
    --type q8_0 \
    --output models/granite5asr-q8_0/granite5asr-q8_0.gguf

AI Disclaimer: Yes, I used AI to help me integrate Granite into audio.cpp. Tested and verified to work

[0xShug0]

@ampersandru Thanks for the PR! I just spoke with the model’s author on LinkedIn about integrating it into audio.cpp. Perfect timing!

Could you move the model under the community_models/ directory? Models under models/ go through more extensive testing, including Vulkan and Metal runtime validation. I’d like to make this model available early.

[ampersandru]

@ampersandru Thanks for the PR! I just spoke with the model’s author on LinkedIn about integrating it into audio.cpp. Perfect timing!

Could you move the model under the community_models/ directory? Models under models/ go through more extensive testing, including Vulkan and Metal runtime validation. I’d like to make this model available early.

Done! I've moved the Granite 5.0 ASR model implementation under community_models/ (include/engine/community_models/granite5asr/ and src/community_models/granite5asr/), registered the loader under engine::community_models::granite5asr, and verified all offline & streaming tests pass (although, this model is so fast, I just use offline)

[0xShug0]

@ampersandru Thanks for the update!

Could you remove the runtime.cpp changes from this PR?

That change is not required for the model integration and should be handled in a follow-up PR. It also invokes the external ffmpeg command from the server path, which may not be portable across environments and would require CI changes if we want this feature to actually work.

[ampersandru]

one bug I cant seem to fix, I tried adding this as a STT in Open WebUI and it errors out:
Error transcribing chunk: External: 500, message='Internal Server Error', url='https://audio.xxxxx.net/v1/audio/transcriptions'

Works in Home Assistant after linking it to wyoming_openai docker

[0xShug0]

one bug I cant seem to fix, I tried adding this as a STT in Open WebUI and it errors out: Error transcribing chunk: External: 500, message='Internal Server Error', url='https://audio.xxxxx.net/v1/audio/transcriptions'

Works in Home Assistant after linking it to wyoming_openai docker

Let’s keep it separate from this PR. I’ve already completed end-to-end inference testing, so the model port can be merged safely once the runtime.cpp changes are removed.

[ampersandru]

one bug I cant seem to fix, I tried adding this as a STT in Open WebUI and it errors out: Error transcribing chunk: External: 500, message='Internal Server Error', url='https://audio.xxxxx.net/v1/audio/transcriptions'
Works in Home Assistant after linking it to wyoming_openai docker

Let’s keep it separate from this PR. I’ve already completed end-to-end inference testing, so the model port can be merged safely once the runtime.cpp changes are removed.

I've removed the runtime.cpp changes from this PR and kept the PR strictly focused on the community_models/granite5asr integration, tests, spec, and GGUF converter. All changes are rebased cleanly on top of main

I will test another ASR model in openwebui, this might just be a specific issue related to OWU

[0xShug0]

one bug I cant seem to fix, I tried adding this as a STT in Open WebUI and it errors out: Error transcribing chunk: External: 500, message='Internal Server Error', url='https://audio.xxxxx.net/v1/audio/transcriptions'
Works in Home Assistant after linking it to wyoming_openai docker

Let’s keep it separate from this PR. I’ve already completed end-to-end inference testing, so the model port can be merged safely once the runtime.cpp changes are removed.

I've removed the runtime.cpp changes from this PR and kept the PR strictly focused on the community_models/granite5asr integration, tests, spec, and GGUF converter. All changes are rebased cleanly on top of main

I will test another ASR model in openwebui, this might just be a specific issue related to OWU

Looks like a similar issue as #312.

[ampersandru]

one bug I cant seem to fix, I tried adding this as a STT in Open WebUI and it errors out: Error transcribing chunk: External: 500, message='Internal Server Error', url='https://audio.xxxxx.net/v1/audio/transcriptions'
Works in Home Assistant after linking it to wyoming_openai docker

Let’s keep it separate from this PR. I’ve already completed end-to-end inference testing, so the model port can be merged safely once the runtime.cpp changes are removed.

I've removed the runtime.cpp changes from this PR and kept the PR strictly focused on the community_models/granite5asr integration, tests, spec, and GGUF converter. All changes are rebased cleanly on top of main
I will test another ASR model in openwebui, this might just be a specific issue related to OWU

Looks like a similar issue as #312.

might be a bit different, I posted the request/bug: #313

[0xShug0]

@ampersandru PR merged! Thanks!

[ampersandru]

ASR Benchmark: NVIDIA Parakeet-TDT 0.6B v3 vs IBM Granite Speech 5.0 470M (audio.cpp)

Hardware: NVIDIA GeForce RTX 5060 Ti (CUDA Backend)
Engine: audio.cpp Server Arena
Precision: Quantized Q8_0 (GGUF)

---

1. Benchmark & Performance Summary

Metric	NVIDIA Parakeet-TDT 0.6B v3	IBM Granite Speech 5.0 470M
Parameters	600M	470M
Q8_0 GGUF Size	~874 MB	~535 MB
Acoustic Architecture	FastConformer (8x Subsampling)	Conformer (4x Subsampling)
Decoder Mechanism	2-Layer LSTM + TDT Joint Network	Single-Layer Linear CTC Projection
Punctuation & Casing	True-Cased & Punctuated (Commas, Periods, Proper Nouns)	Normalized Lowercase (No punctuation by design)
Languages	25 European Languages	English
10.1s Audio Latency	31.98 ms (RTF: 0.00316 / 315x real-time)	55.46 ms (RTF: 0.00549 / 181x real-time)
11.8s Audio Latency	46.76 ms (RTF: 0.00397 / 251x real-time)	47.65 ms (RTF: 0.00404 / 246x real-time)
19.5s Audio Latency	65.34 ms (RTF: 0.00335 / 298x real-time)	69.47 ms (RTF: 0.00356 / 280x real-time)
20.2s Audio Latency	64.76 ms (RTF: 0.00319 / 312x real-time)	56.32 ms (RTF: 0.00278 / 359x real-time)

---

2. Side-by-Side Transcription Outputs

Test 1 (10.09s Audio)

• Parakeet-TDT 0.6B (31.98 ms):

  "I'm mostly just testing out the system to see how well it captures my voice. If you can hear this clearly, then the setup is probably working exactly as intended."

• Granite Speech 5.0 (55.46 ms):

  "i am mostly just testing out the system to see how well it captures my voice if you can hear this clearly then the setup is probably working exactly as intended"

---

Test 2 (19.49s Audio)

• Parakeet-TDT 0.6B (65.34 ms):

  "Um the iPhone is obviously one of the most important inventions, probably of all time. Um Steve Jobs came out with it in 2007. I started Facebook in 2004. So he was working on the iPhone while I was getting started with Facebook. So I I basically"

• Granite Speech 5.0 (69.47 ms):

  "the iphone is obviously one of the most important inventions probably of all time you know steve jobs came out with it in 2007 i started facebook in 2004 so he was working on the iphone while i was getting started with facebook so i basically"

---

Test 4 (11.77s Audio)

• Parakeet-TDT 0.6B (46.76 ms):

  "Oh my earliest memory I think I rem I remember I have a twin brother and my earliest memory is of my twin brother. I remember being with my twin brother like in a"

• Granite Speech 5.0 (47.65 ms):

  "0 my earliest memory i think i remember i have a twin brother and my earliest memory is of my twin brother i remember being with my twin brother like in a"

---

3. Architectural Differences & Why Granite Lacks Punctuation

1. Training Normalization: IBM trained Granite Speech 5.0 on standardized acoustic speech corpora (LibriSpeech, GigaSpeech, Common Voice) where target transcripts are pre-normalized to lowercased text without punctuation marks.
2. CTC Token Vocabulary: Granite’s CTC dictionary contains only lowercase characters (a-z, 0-9, ', space, and special tokens). It has no tokens for uppercase letters or punctuation marks (., ,, ?, !).
3. Parakeet's Direct Restoration: NVIDIA trained Parakeet-TDT with an 8,192-token SentencePiece vocabulary that includes punctuation and true-casing natively in the joint acoustic-language decoder.

---

4. Recommended Use Cases

• Home Assistant & Smart Home Voice Pipelines (NLU / Intent Parsing) ➔ IBM Granite 5.0:
  • Clean, unpunctuated, lowercased output avoids regex/intent parser failures caused by unexpected punctuation.
  • Extremely compact (~535 MB in Q8_0) and fast single-pass CTC inference.
• Chatbots, Open WebUI, Meeting Notes & Dictation ➔ NVIDIA Parakeet-TDT 0.6B:
  • Ready-to-read sentences with capitalization, commas, and periods out of the box without requiring a secondary punctuation restoration model.
  • 25 European language support with automatic language detection.