🧭 chore: Gemma 4 E2B QAT Mobile の Gate 1 再検証（Metal TQ2_0 カーネル待ち）

[tyabu12]

⚠️ Current scope (re-scoped 2026-08-13)

The retry this issue was originally filed for has run. google/gemma-4-E2B-it-qat-q4_0-gguf
was measured on a bumped pin inside the #1415 spike and scored a clean NO-GO (103 vs that
session's 113 incumbent, and the file is +7.8 % larger). Its sibling
unsloth/gemma-4-E2B-it-qat-GGUF → UD-Q4_K_XL was measured too and passed the Mac filter.
Both are now recorded in docs/models/eval-log.md (entries §2026-08-12 and §2026-08-13).

What this issue now tracks is the third QAT file, which the original body never named:
unsloth/gemma-4-E2B-it-qat-mobile-GGUF → gemma-4-E2B-it-qat-UD-Q2_K_XL.gguf (2.19 GB,
−29.6 % vs the incumbent). Its Gate 1 verdict is BLOCKED, and not on #1415 — the b10327
pin does clear the shared-KV loader gate, but the build then SIGSEGVs on its first inference
because llama.cpp's Metal backend ships no TQ-family kernels. Still absent at b10375.

New unblock condition: llama.cpp's Metal backend gains TQ2_0 kernels — at minimum
kernel_mul_mm_tq2_0_f32 — and llama.swift ships a release at that build. Cheap re-check
before downloading anything: grep the release xcframework binary for tq2_0 kernel names,
keeping a known-present kernel (q4_0 / q8_0 / q4_K) in the same grep as a positive control.

Already done, do not re-derive: ADR-011 P1 and P2 both PASS for the mobile
re-export (<|turn> id 105 / <turn|> id 106 are both token_type=3 CONTROL).

Result detail: this comment.
Ledger: docs/models/eval-log.md (#1447).

The original body follows unchanged, for the record. Read its "Candidate under test",
"Blocked by #1415" and "Already pre-cleared" sections as historical — they describe the
Q4_0 candidate, not the one above.

---

Summary

Resume the /model-eval Gate 1 battery for the Gemma 4 E2B QAT candidate
once the llama.cpp pin carries shared-KV tail-layer support. Today's run
(2026-08-08) was blocked at model load — zero inferences, so no quality
judgment exists yet for QAT.

Blocked by #1415. Do not start until a pin bump has landed (or #1415 lands a
spike result that makes this measurable).

Why this is a retry, not a fresh evaluation

The 2026-08-08 Gate 1 run failed 6/6 cells in ~0.9 s each, all with the identical
error before any inference:

llama_model_load: error loading model: missing tensor 'blk.15.attn_k.weight'

Cause: Gemma 4 E2B QAT GGUFs use a shared-KV tail-layer layout (541 tensors;
attn_k / attn_v / attn_k_norm omitted for layers 15–34) that pinned llama.cpp
b8694 cannot load. Confirmed not vendor-specific — unsloth's own QAT re-export
has the byte-identical 541-tensor layout, so the split is QAT-vs-non-QAT, not
Google-vs-unsloth. Detail in #1415.

The verdict was fail in the mechanical sense only (not all 6 cells ok); it is
a blocked outcome, not a quality rejection — the same distinction the Sarashina
2026-07-08 run drew before its clean 2026-07-23 re-eval.

Already pre-cleared (do not re-derive)

Measured on 2026-08-08 against the candidate file; these carry over:

Check	Result
ADR-011 P1 non-gated source	✅ HF gated: false, license: apache-2.0, anonymous resolve 302 → CDN 200
ADR-011 P2 CONTROL token flags	✅ <|turn> id=105 and <turn|> id=106 both token_type=3 (CONTROL)
EOG termination	✅ EOG set {1, 50, 106, 212} — a superset of the incumbent's {50, 106, 212}; <turn|> terminates in both
Prompt-format fields	✅ equivalent to the incumbent: systemPromptSuffix nil, assistantPrefix nil (chat template is read from the GGUF by llama.cpp)
File integrity	✅ 3,349,516,256 bytes / sha256 fa401b55b07ee70a54c6dae3903c783a6e65064312529ea57175cb5f8dec6634, both matching HF X-Linked-Size / X-Linked-ETag

Still open: P3 on-device GBNF PoC, P4 no catalog regression, P5 no raw
prompt-token sampler accept — all Gate 2, and all after Gate 1 passes.

Candidate under test

• google/gemma-4-E2B-it-qat-q4_0-gguf @ 675cff42a74c774d6cb76f76d8eacb49b48c9b93
  → gemma-4-E2B_q4_0-it.gguf (3.35 GB, QAT Q4_0)
• Worth evaluating alongside unsloth/gemma-4-E2B-it-qat-GGUF →
  gemma-4-E2B-it-qat-UD-Q4_K_XL.gguf (2.62 GB, sha256
  e531007218dfab990486a5de7676a6932d6ea8dea233d1f698d7c21cf8a16889): same shared-KV
  layout, but ~0.49 GB smaller than the shipped Q4_K_M (2.62 vs 3.11 GB), so if
  quality holds it is
  the more interesting of the two on download size and RAM headroom.

Profile note (carry forward)

The 2026-08-08 run reused the incumbent profile id gemma-4-e2b-q4-k-m — legitimate
per docs/models/onboarding.md (Stage 0 is required for a new family only, and
gemma is registered) after verifying the three prompt-format fields are equivalent.
Consequence: the harness stamps modelIdentifier as "Gemma 4 E2B (Q4_K_M)", so raw
JSONL mislabels the candidate. If this candidate advances, add a proper
ModelProfile in the registration PR. Alternatively add a candidate-only profile
first (the sarashina223B precedent) to keep the retry's logs self-describing.

Acceptance criteria

• All 6 battery cells reach status: ok (or a genuine non-load failure is
  recorded with its cause)
• Rubric scored for every ok cell; differentiation written against both
  incumbents
• Gate verdict recorded, with the explicit note that a Gate 1 pass only
  advances to the ADR-011 real-device PoC and is never an adoption
• Verdict appended to docs/models/eval-log.md (committed ledger) — replacing
  or superseding the blocked 2026-08-08 entry, not duplicating it
• Comparison against the b8694-era incumbent baseline explicitly flagged as
  cross-pin if the pin changed (see 🔧 chore: llama.cpp pin 上げの検討（b8694 → 現行）— QAT shared-KV 対応の可否を spike で判断 #1415 § eval-baseline re-basing)

Not agent-ready

Harness-gated stochastic judgment loop — the /model-eval rubric scoring is
in-session human/model judgment, and the asymmetric borderline→re-sample rule needs
an operator. Run interactively.

References

• 🔧 chore: llama.cpp pin 上げの検討（b8694 → 現行）— QAT shared-KV 対応の可否を spike で判断 #1415 — llama.cpp pin bump investigation (blocker)
• 🧭 chore: model candidates — rolling intake for the validation pipeline #979 — model candidate rolling intake
• docs/models/onboarding.md — two-gate procedure
• ADR-011 § Decision item 2 — the five hard prerequisites

[tyabu12]

Retry scope note from the #1415 spike (see
the result and its
scope correction):

The candidate this issue names — google/gemma-4-E2B-it-qat-q4_0-gguf — has now been measured
and is a NO-GO on quality. On a bumped pin (llama.swift 2.10327.0 / llama.cpp b10327) it loads
and completes all 6 battery cells, but scores 103 vs the incumbent's 113 on the same pin, and
its file is larger than the incumbent (3.35 vs 3.11 GB). So the Gate 1 retry this issue tracks
can be resolved as evaluated and rejected rather than left retry-gated.

But do not close this issue as "QAT rejected". Google's
QAT Mobile line was never part of
the evaluation. Its GGUF conversion unsloth/gemma-4-E2B-it-qat-mobile-GGUF →
gemma-4-E2B-it-qat-UD-Q2_K_XL.gguf is 2.19 GB (−30 % vs the incumbent) and carries the same
541-tensor shared-KV layout, so it is gated on the same pin bump and is a genuinely different
quantisation scheme (wNa8o8, 2-bit decoding layers). It is unmeasured in every respect —
including whether it loads on b10327 — and ADR-011 P1/P2 are pre-cleared only for the Google file,
not this re-export.

Suggested reframing: retitle/refocus this retry onto the mobile variant, or close it against the
Q4_0 result and open a fresh intake under #979 for the mobile GGUF.

[tyabu12]

QAT Mobile Gate 1 result — BLOCKED (Metal has no TQ2_0 kernels), and an unplanned PASS for the QAT Q4_K_XL sibling

Ran the reframed retry this issue's comment thread called for: the mobile variant as a fresh
candidate through the normal pipeline (P1/P2 sweep → Gate 1 battery on a bumped pin). Throwaway
worktree, main's pin untouched, no commit.

Headline: the mobile GGUF is not measurable on the shipping backend — a second runtime
gap sits behind the shared-KV one, and it is not fixable by choosing a newer pin. But the third
QAT file, unsloth/gemma-4-E2B-it-qat-GGUF → UD-Q4_K_XL, passes the Mac filter at −15.7 %
file size with no measured quality regression — which is the payoff #1415's spike looked for and
could not find.

---

unsloth/gemma-4-E2B-it-qat-mobile-GGUF → UD-Q2_K_XL — BLOCKED

Check	Result
ADR-011 P1 non-gated source	✅ HF gated: false, license: apache-2.0, anonymous resolve 302 → CDN 200
ADR-011 P2 CONTROL token flags	✅ <|turn> id 105 and <turn|> id 106 both token_type=3 (CONTROL) — identical to the incumbent, so the unsloth #5070 NORMAL-flag trap is not present in this re-export
File integrity	✅ 2,186,186,784 B / sha256 0a5bbc20…da28, both matching HF X-Linked-Size / X-Linked-ETag
shared-KV tail layer (the 2026-08-08 blocker)	✅ resolved — loads cleanly on b10327
Inference	❌ SIGSEGV on the first inference

So the P1/P2 sweep the scope-correction comment asked for is done and need not be re-derived on
a future retry.

Root cause, from lldb (the model had already loaded — run_start emitted, assign phase
completed, so this is compute, not the loader):

ggml_metal_library_compile_pipeline: failed to compile pipeline:
  base = 'kernel_mul_mm_tq2_0_f32'
  Error Domain=MTLLibraryErrorDomain Code=5
  "Function kernel_mul_mm_tq2_0_f32 was not found in the library"
→ thread #8, queue = 'ggml-metal', EXC_BAD_ACCESS (code=1, address=0x0)
   frame #0: llama`ggml_metal_encoder_set_pipeline + 8

llama.cpp's Metal backend ships no kernels at all for the TQ (ternary) types. Verified by
grepping the shipped framework binary with a positive control in the same grep —
kernel_mul_mm_q4_0 / q8_0 / q4_K all present, every tq1_0 / tq2_0 kernel absent (tq2_0
survives only as a block-struct / type name). Still absent at b10375, ~48 builds newer than the
pin under test, so this is not a "pick a newer tag" problem.

Secondary observation worth an upstream report: llama.cpp does not appear to reflect the missing
kernel in its op-support check, so it dereferences a NULL pipeline instead of falling back to CPU
for those nodes.

What wNa8o8 actually is, read from the GGUF header (probe positive-controlled against the two
local files first, reproducing their known 601 / 541 tensor counts):

file	tensor quant spread
incumbent Q4_K_M	F32=353, Q4_K=212, Q6_K=34, BF16=1, Q5_K=1
Google QAT Q4_0	Q4_0=275, F32=263, Q6_K=2, F16=1
QAT Mobile Q2_K_XL	F32=263, Q4_0=146, Q8_0=70, TQ2_0=61, F16=1

61 tensors at TQ2_0 are the model card's "targeted 2-bit decoding layers"; the 70 Q8_0 are the
8-bit activations side.

• Unblocked by: llama.cpp's Metal backend gaining TQ2_0 kernels (at minimum
  kernel_mul_mm_tq2_0_f32) and a llama.swift wrapper release at that build. Cheap re-check:
  grep the release xcframework binary for tq2_0 kernel names before downloading anything.
• Disposition: not rejected. At 2.19 GB (−29.6 % vs the incumbent) it is still the largest
  download saving in the family, and its quality remains genuinely untested.

---

unsloth/gemma-4-E2B-it-qat-GGUF → UD-Q4_K_XL — Gate 1 PASS (Mac filter only)

The file this issue's body already flagged as "worth evaluating alongside". Despite the name, its
tensor spread is Q4_0=278 / F32=263 (general.name: "smart Q4_0, QAT-lossless") — every
kernel it needs exists, so it runs. P1/P2 clear on the same sweep; 2,620,370,976 B / sha256
e5310072…6889, matching the HF headers.

Battery run on llama.swift 2.10327.0 (b10327) with a same-session incumbent control arm,
because #1415 measured judge drift at up to −17 points across sessions — the recorded 2026-07-08
and #1415 numbers are not comparable to these.

cell	incumbent control	QAT Q4_K_XL
word_wolf ja	17	22
word_wolf en	12	16
prisoners_dilemma ja	11 → re-sample 18	23
prisoners_dilemma en	18	22
bokete ja	21	18
bokete en	21	20
total	107 (100 with the first pd-ja sample)	121

Both arms mechanically clean: 6/6 ok, attempts_mean 1.00, language_mismatch_total 0, no crash.

The difference is more legible as mechanism than as points. The control's prisoners_dilemma
ja collapsed to all-cooperate — a five-way 12-point tie, with Ryuji declaring 裏切り twice and then
cooperating 4/4, so the scenario's premise never fired; its word_wolf ja had the wolf say
「柑橘類の皮の香り」 twice with no agent engaging it. The candidate's runs engage: Akira defects in
R2 citing Kenta's observed R1 pattern (cross-round adaptation), Kai lies in a whisper and then
betrays the partner he lied to, and the word_wolf majority picks up the wolf's kotatsu-mikan tell
and votes correctly. It is worse than the incumbent on bokete in both languages (two self-votes
in bokete ja) — this is not a clean sweep.

One small robustness signal: the control emitted stray JSON residue (」}) into inner_thought in
both its prisoners_dilemma ja runs; the candidate emitted none anywhere.

Method note — the pd-ja re-sample. The control's first prisoners_dilemma ja sample was the
outlier driving most of the headline gap, so that cell was re-run and both samples are recorded
rather than silently replacing the weaker one. The candidate arms were not re-sampled, so this
re-sample can only move the comparison in the incumbent's favour.

Caveats that bound the result: n=1 per cell, one judge, one session; the gap leans on
prisoners_dilemma ja. Treat +14 as a screening signal, not a measured delta.

A Gate 1 pass only advances the candidate to the ADR-011 real-device GBNF PoC (P3–P5). It is not
an adoption. Device-only risk is entirely untouched — Metal behaviour, the ~8 GB RAM headroom
estimate (ADR-002 §4), and app binary size are all Mac-harness-invisible.

Two costs that are not in the −0.49 GB headline, for whoever scopes adoption:

1. Per ModelRegistry's supersede convention, updating a model means a new id and new
   fileName with the old entry removed — so existing users re-download 2.62 GB, and the old
   3.11 GB file becomes an orphan surfaced in Settings → Models (never auto-deleted, ADR-015 / ✨ feat: model-update convention + storage-management UX #548).
   The saving accrues to new installs.
2. The removed id must move into RETIRED_MODEL_IDS in scripts/gallery_highlight_validate.py in
   the same PR (ADR-029) — it is currently an empty frozenset — or every shipped gallery
   highlight naming that model reddens.

---

Artifacts

Raw scorecards → data/models/eval-digest.md §2026-08-13, three sections
(gemma-4-e2b-qat-mobile-q2-k-xl, gemma-4-e2b-qat-q4-k-xl, gemma-4-e2b-q4-k-m = the control
arm), gitignored and local. Run logs → data/models/eval-runs/2026-08-13/. Each candidate got its
own throwaway ModelProfile so the logs self-describe and the digest's (date, profile_id)
section key does not collide across the three arms — which is what stopped #1415 from recording its
own numbers.