webgpu-onnxruntime-fix.md

WebGPU on q4 — why it was disabled, and why it no longer is

Status: re-enabled on 2026-08-04

WebGPU is now attempted for every dtype except q8. This document used to say the opposite, and to explain the exclusion with a root cause that turned out to be wrong. It is kept at this path, corrections and all, because it is linked from the upstream discussion in microsoft/onnxruntime#29599.

What this document previously claimed

That onnxruntime-web accumulated MatMul/MatMulNBits dot products in f16, that those sums overflowed 65504 on Whisper q4 and propagated NaN, that this was the cause of the [Music] hallucination, and that PR #29599 would fix it.

Two separate things were wrong with that.

Whisper q4 has no f16 to overflow. Reading the model file instead of assuming: onnx-community/whisper-small at dtype q4 contains zero fp16 tensors — input_features, last_hidden_state, every MatMulNBits activation and every scale is FLOAT, in the encoder and in the merged decoder. The accumulator was already f32 on every path, so #29599 is a no-op on this model. It cannot have been the fix, and it is not needed for this extension.

The July A/B did not measure what it thought it measured. It compared the published 1.22.0-dev.20250409 package against a from-source build of the PR branch. Those two differ by a full release cycle of unrelated changes, not by the accumulator patch, so any difference between them was unattributable.

What was actually measured, 2026-08-04

A bench that replicates src/offscreen/whisper.ts line for line, including the audio decode from src/content/index.ts (16 kHz AudioContext, the same peak normalisation), running three arms that differ only in which onnxruntime-web ends up in the bundle. Three real WhatsApp voice messages (25.7 s / 32.6 s / 39.2 s, Italian, language forced), cache cleared between runs.

arm	stack
A	@huggingface/transformers 3.8.1 + onnxruntime-web 1.22.0-dev.20250409 — what this repo pins today
B	3.8.1 + a from-source build of PR #29599
C	@huggingface/transformers 4.2.0 + onnxruntime-web 1.26.0-dev

On all three arms, including arm A, WebGPU output is character-identical to WASM. The failure does not reproduce on the currently pinned package. WebGPU is genuinely in use: with ORT profiling on, the executed kernels are listed one per graph node (BlockwiseMatMulNBits32, Conv2DMatMul, ReduceMeanShared).

Timings, arm A, both pipelines loaded in one page and runs alternated:

clip	WebGPU	WASM	ratio
25.7 s	15.9 s	24.5 s	1.54x
32.6 s	17.3 s	26.0 s	1.50x
39.2 s	23.4 s	34.7 s	1.48x

on an Intel Iris Xe (gen-12lp). A discrete RTX 3050 Ti is faster still, though the gap there is inflated by the WASM side thermally throttling.

Alternating the two backends in a single page matters: measured sequentially on a laptop, the same WASM run gives 37 s cold and 75 s hot, which is enough to invent or erase a 2x difference.

Do not bump @huggingface/transformers. Arm C produces the same text but is ~3x slower on WASM and ~1.6x slower on WebGPU than the pinned 3.8.1. The fastest configuration measured is the stack already pinned here, with WebGPU enabled.

All three models, not just the default

The earlier round only covered whisper-small, which is not enough to justify lifting an exclusion that applies to all of them. On the 25.7 s clip, arm A:

model	WebGPU	WASM	output
whisper-tiny	2.4 s	1.7 s	identical
whisper-small	11.9 s	17.6 s	identical
whisper-large-v3-turbo	19.4 s	54.4 s	identical

Correctness holds everywhere. Speed does not: tiny is faster on WASM, on an Intel Iris Xe and on a discrete GPU alike, and it stays that way at steady state over repeated runs, so it is dispatch overhead rather than shader warm-up. At that model size there is not enough compute per kernel to pay for going to the GPU.

So the choice is now per model, via the webgpu flag in MODEL_CONFIGS: off for tiny, on for small and large-v3-turbo. It is a speed decision, not a correctness one.

So what was the original [Music]?

Not identified, and worth being straight about rather than inventing a story. The exclusion has been in the code since the first commit, so the failure predates this repository's history and nothing recorded the conditions. What has been ruled out by re-running the same bench against each variable in turn:

• Not the ONNX Runtime version. Four stacks tested, spanning ~22 months of releases: onnxruntime-web 1.21.0-dev.20241024 (via transformers 3.0.2, the oldest thing ^3.0.0 resolves to), 1.22.0-dev.20250409 (pinned here), the #29599 source build, and 1.26.0-dev (via transformers 4.2.0). All correct.
• Not the browser. Chrome for Testing 135, 147 and installed 150 — April 2025 through August 2026 — same bundle, same audio. All correct.
• Not the packages this repo pins. package-lock.json has never changed since the first commit. The bytes that produced [Music] are the bytes that now produce correct output.
• Not the model. onnx-community/whisper-small has not been touched on the Hub since 2025-06-19, well before the exclusion was written.
• Not the model size. tiny and large-v3-turbo behave like small.

What is left is the GPU driver, or something about the original run that was never written down. The re-enable rests on the measurements above, not on a theory of what the old failure was — and the webgpu_crash_flag and WASM fallback are still there if it ever comes back.

What is still excluded

q8, on the same "never retested" footing the q4 exclusion turned out to have. Its reported failure was a Can't perform where op crash rather than wrong output, and no model configured here uses q8, so there is nothing to measure against.

Correction, 2026-08-05. The sentence above stopped being true the day per-module dtypes landed. whisper-tiny and whisper-base went to q8 on both modules, and whisper-small to a q8 encoder with a q4 decoder — while the WebGPU guard was simultaneously narrowed to check only the decoder, on the theory that the exclusion was about the Can't perform where op crash and that crash is decoder-shaped. Everything above measured whisper-small flat at q4; the q8-encoder configuration was never benchmarked, on any backend.

It reached users, and it does not crash — it returns gibberish. On the 2026-07-20 test clip, Italian forced, whisper-small on WebGPU produced T T T... X t I ( NT/ P," D Se..., Ton-. and, on a second run, failed with token_ids must be a non-empty array of integers. The encoder returns meaningless hidden states; the decoder then wanders the vocabulary or emits nothing at all.

The guard now excludes q8 in any module, which is what "q8 is excluded" always meant. Consequence: whisper-small runs on WASM, and the 1.5x WebGPU speed-up in the table above is currently unreachable for it. Getting it back means measuring a q4 encoder for accuracy, not relaxing the guard again.

Worth noting how this stayed invisible: the same release carried a chrome.runtime.getManifest() call into the offscreen document, where that function does not exist. It threw before any WebGPU load could start, so the narrowed guard never actually selected the GPU until the TypeError was fixed. Two bugs in one commit, the first hiding the second.

The existing safety net is unchanged: webgpu_crash_flag in localStorage skips WebGPU for a model that crashed on the previous attempt, and the catch around pipeline creation falls back to WASM.

Scope of the verification

Verified at the pipeline level — same code path, same audio decode, same inference options — not by driving the extension end-to-end through its offscreen document, which is not navigable as a tab and whose MV3 service worker is not awake at startup. Worth one manual pass on WhatsApp Web before tagging a release.

Unrelated, but found on the way

The q4f16 export of the same Whisper model does have a real fp16 overflow, in the decomposed LayerNorm's Pow, and it is now filed upstream as microsoft/onnxruntime#31626. This extension does not use q4f16, so it is not affected.