Skip to content

speech addons: enable per-arch ggml CPU variants on desktop linux-arm64 (whispercpp, bci, tts-ggml)#3272

Merged
ogad-tether merged 2 commits into
mainfrom
feat/speech-linux-arm64-cpu-variants
Jul 16, 2026
Merged

speech addons: enable per-arch ggml CPU variants on desktop linux-arm64 (whispercpp, bci, tts-ggml)#3272
ogad-tether merged 2 commits into
mainfrom
feat/speech-linux-arm64-cpu-variants

Conversation

@ogad-tether

@ogad-tether ogad-tether commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Extends the transcription-parakeet linux-arm64 CPU enablement (#3263) to bci-whispercpp, transcription-whispercpp and tts-ggml.

Problem

Desktop linux-arm64 prebuilds compiled ggml at the plain armv8-a baseline — the ARM dotprod/fp16/i8mm kernels were compiled out, leaving f16 and quantized models on the slowest generic paths (Android already builds GGML_CPU_ALL_VARIANTS).

Changes

The build-flags change lives in the ggml-speech vcpkg port (2026-07-14, qvac-registry-vcpkg#249) — per review guidance, a change for every consumer of the port belongs in the port, not in triplets. The port builds per-arch runtime-scored CPU MODULE variants (SIGILL-safe), statically links libc++ into the modules, stages them flat in lib/ (where the addons' existing staging glob already looks), and excludes the SVE tiers (2–4× slower than NEON on VL=128 cores — A/B linked in the port PR).

This PR:

  1. Bumps the -cpp constraintswhisper-cpp override 1.9.1#1 → 1.9.1#3 (transcription-whispercpp, bci-whispercpp) and tts-cpp >= 2026-07-13#2 (tts-ggml) — whose ports now require ggml-speech >= 2026-07-14 (per review: no direct ggml-speech dependency in the packages).
  2. transcription-whispercpp + bci-whispercpp: the backend-loading gate (ensureBackendsLoadedAndroidensureBackendsLoaded) widens from __ANDROID__ to __ANDROID__ || (__linux__ && __aarch64__) — with GGML_BACKEND_DL=ON no backend is statically registered, and the addon must call ggml_backend_load_all_from_path(backendsDir/BACKENDS_SUBDIR) before whisper_init. Gated to linux-arm64 only per review (linux-x64 stays on static backends). tts-ggml needs no addon change — it already threads backends_dir unconditionally through tts-cpp EngineOptions.
  3. CHANGELOG entries under [Unreleased] for all three packages.

Verification (ubuntu-24.04-arm, mean RTF, lower is better)

End-to-end benchmark runs with this change set (port content via overlay):

  • whisper (main baselinebranch): base f16 0.3316 → 0.0970 (3.4×), base q8_0 0.1270 → 0.0629 (2.0×), base q5_1 0.1774 → 0.1303, small f16 1.3450 → 0.3426 (3.9×), small q8_0 0.4561 → 0.2047 (2.2×)
  • tts-ggml (main baselinebranch): chatterbox q4 2.70 → 2.28 (−15%), supertonic ~flat
  • bci-whispercpp (branch): linux-arm64 leg green, CPU modules load and benchmark completes

All other platforms and Android unchanged/green in the same runs.

Merge order: qvac-registry-vcpkg#249 first (version resolution needs ggml-speech 2026-07-14 in the registry). Independent of #3263.

🤖 Generated with Claude Code

@ogad-tether ogad-tether requested review from a team as code owners July 14, 2026 13:04
@github-actions

github-actions Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Review Status

Current Status: ✅ APPROVED
Approvals so far: Team Lead: 1, Member: 1

@ogad-tether ogad-tether self-assigned this Jul 14, 2026

@GustavoA1604 GustavoA1604 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also put concise entry in CHANGELOG.md under [unreleased] for the packages

Comment thread packages/bci-whispercpp/addon/src/model-interface/bci/BCIModel.cpp Outdated
@ogad-tether

Copy link
Copy Markdown
Contributor Author

Reworked per review guidance:

  • Build flags moved out of the package triplets into the ggml-speech port (ggml-speech 2026-07-14: per-arch CPU variants on desktop aarch64 Linux qvac-registry-vcpkg#249) — the triplet approach was also setting the GGML flags for every vcpkg dependency in the manifests; the port is the right scope. The three packages now just consume ggml-speech 2026-07-14.
  • @GustavoA1604 the whisper/bci backend-loading gate is now __ANDROID__ || (__linux__ && __aarch64__) — linux-x64 stays on static backends.
  • CHANGELOG entries added under [Unreleased] for all three packages.

@ogad-tether ogad-tether requested a review from GustavoA1604 July 15, 2026 00:39
@ogad-tether ogad-tether force-pushed the feat/speech-linux-arm64-cpu-variants branch from 3a8b3ee to 26d35e4 Compare July 15, 2026 00:40
…speech 2026-07-14)

Desktop linux-arm64 prebuilds compiled ggml at the plain armv8-a
baseline: the ARM dotprod/fp16/i8mm kernels were compiled out and both
f16 and quantized models ran on the slowest generic paths (Android
already builds GGML_CPU_ALL_VARIANTS).

The build-flags change lives in the ggml-speech vcpkg port (2026-07-14,
qvac-registry-vcpkg PR #249): per-arch runtime-scored CPU MODULE
variants (SIGILL-safe on any aarch64 host), static libc++ in the
modules, flat .so staging in lib/, SVE tiers excluded (2-4x slower than
NEON on the VL=128 cores in the desktop fleet). This PR consumes the
new port version in bci-whispercpp, transcription-whispercpp and
tts-ggml, and makes the two whisper addons load the dynamically-loaded
backends on linux-arm64:

- WhisperModel/BCIModel: the ensureBackendsLoadedAndroid gate (now
  ensureBackendsLoaded) widens from __ANDROID__ to __ANDROID__ ||
  (__linux__ && __aarch64__) -- with GGML_BACKEND_DL=ON no backend is
  statically registered and the addon must call
  ggml_backend_load_all_from_path(backendsDir/BACKENDS_SUBDIR) before
  whisper_init. Kept off linux-x64, which stays on static backends.
  tts-ggml already threads backends_dir unconditionally through
  tts-cpp EngineOptions.

Verified end-to-end via overlay-port benchmark runs (ubuntu-24.04-arm,
mean RTF): whisper base f16 0.3316 -> 0.0970 / q8_0 0.1270 -> 0.0629,
small f16 1.3450 -> 0.3426 (run 29333221534); chatterbox q4 2.70 ->
2.28 (run 29333223845); bci green (run 29333225816). All other
platforms and Android unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@ogad-tether ogad-tether force-pushed the feat/speech-linux-arm64-cpu-variants branch from 26d35e4 to 728e30c Compare July 15, 2026 15:13
@ogad-tether

Copy link
Copy Markdown
Contributor Author

Per review: consumption now flows through the -cpp ports — tetherto/qvac-registry-vcpkg#249 bumps parakeet-cpp/whisper-cpp/tts-cpp to require ggml-speech 2026-07-14, and this PR just bumps the whisper-cpp override (1.9.1#3) / tts-cpp constraint (2026-07-13#2). Direct ggml-speech deps removed. Also rebased onto main (CHANGELOG conflict with #3265 resolved).

@github-actions

Copy link
Copy Markdown
Contributor

QVAC E2E — android — running

View run
Config: suite=smoke · filter=(none) · exclude=(none)

@ogad-tether ogad-tether merged commit 8581892 into main Jul 16, 2026
63 of 92 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

verified Authorize secrets / label-gate in PR workflows

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants