[Bug]: glm_tiny TF oracle is 30/32, not the documented 32/32 — forward-pass near-tie at 2 positions

[woolcoxm]

How I found this

I was starting work on #442 (SIMD-ifying the MLA-absorb f32 reductions in attention_rows). The plan from that issue was to use the glm_tiny token-exactness oracle as the correctness gate — "a fixed-order SIMD reduction can stay bit-identical and self-validates against the oracle." Before touching any code I wanted to confirm the gate was green, so I ran:

SNAP=./glm_tiny TF=1 NGEN=0 ./colibri

I expected 32/32 (the engine self-documents this at colibri.c:6177: "Engine self-test: SNAP=./glm_tiny TF=1 ./glm 64 16 16 (expect 32/32)"). It returned 30/32, deterministically, on current dev (c98e5f8, post-#298). That stopped the #442 work cold — I could not use a 30/32 oracle as a "did my SIMD change break anything" gate without first understanding the 2 mismatches.

This issue tracks those 2 mismatches. They are not caused by the #442 SIMD change (I verified — see "Correctness gate" below).

The mismatch

Deterministic across 5 runs and across a fresh oracle regeneration. Same 2 positions every time:

PREFILL (teacher-forcing) C vs oracle: 30/32 positions
[ORACLE] mismatch pos=5  expected=34  got=197
[ORACLE] mismatch pos=25 expected=103 got=136

Environment

• dev @ c98e5f8 (post the cuda+engine: full fmt=4 (grouped int4 gs=64) support + diagnostic harness #298 fmt=4 dense/attention fix)
• GCC 15.2.0, -march=x86-64-v3 (-O3 -fopenmp, no -ffast-math)
• Win32 / Git Bash
• transformers==5.13.1 (well above the 5.11 interleaved-RoPE floor from make_glm_oracle.py on transformers 5.5.0 produces an oracle the engine scores 25/32 against (expected 32/32) #281)
• glm_tiny: 5 layers, hidden=128, kv_lora=32, vocab=256

What it actually is (not an argmax bug)

I spent several hours convinced this was an argmax or buffer-aliasing bug, because the DEBUG_LOGITS dump I wrote to investigate it appeared to show the expected token winning while pred held a different one. That was a bug in my own diagnostic (the selection-sort seeded bv=0 with no NaN guard, producing misleading top-K output). Once I fixed the diagnostic, the picture inverted completely: the engine is correct, the logits genuinely disagree with the oracle at these 2 positions by a near-tie margin.

With a corrected top-5 dump, both mismatches are clean near-ties — the engine's argmax picks the true top-1 of its own logits, and the oracle's expected token is a close #2:

pos 5 (input token 58):

1. tok=197  logit=1.754010
2. tok=34   logit=1.721285   <- oracle expected (Δ=0.033)
3. tok=244  logit=1.404302

pos 25 (input token 103, inside the long 103-run at pos 14-25):

1. tok=136  logit=1.620979
2. tok=103  logit=1.597835   <- oracle expected (Δ=0.023)
3. tok=119  logit=1.496210

So the C engine's forward pass produces a logit distribution that is very close to the Python transformers reference at every position, but at 2 positions the top-1/top-2 ordering flips. This is a small forward-pass numeric discrepancy, not a logic error.

What I ruled out

• Oracle drift / stale weights. Regenerated glm_tiny + ref_glm.json from tools/make_glm_oracle.py — got a byte-identical reference (the generator is deterministic), and the fresh engine still scores 30/32.
• transformers < 5.11 (make_glm_oracle.py on transformers 5.5.0 produces an oracle the engine scores 25/32 against (expected 32/32) #281 RoPE bug). Running 5.13.1. make_glm_oracle.py hard-fails below 5.11 anyway.
• GLM [gMASK]<sop> prefix. Neither the Python forward (make_glm_oracle.py:147, model(torch.tensor([full]))) nor the C forward_all prepends it; full_ids starts directly with the prompt ([3, 14, 159, ...]), no prefix tokens. Inspected GlmMoeDsaForCausalLM.forward — it takes input_ids as-is.
• Non-determinism / OMP race. 5 identical runs, same result every time. Also reproduced at OMP_NUM_THREADS=1 (still 30/32, same positions).
• NaN-pinning argmax (my leading wrong hypothesis — the test_logit_nan.c regression class). The mismatched logits are finite; isfinite(lo[best]) was 1 at both positions. Swapping the raw argmax loop for the NaN-safe argmax_v (sample.h) made no difference — as expected, since there were no non-finites to skip.
• Buffer aliasing / mutation between argmax and memcpy. Probed lo[best] immediately after argmax and again immediately before the memcpy three lines later — identical values (Δ=0.0) at both positions. No mutation.

What I did not pin down

The actual source of the numeric divergence. The candidates, in rough order of likelihood, all live in forward_all → layers_forward → the MLA-absorb attention path or the RMSNorm/router upstream:

1. The idot path on lm_head. With g_idot on (banner: idot: avx2) and lm_head at fmt=1, matmul_qt quantizes x to int8 via qrow_i8 and computes dot_i8i8 * scale * sx — a different numeric computation than transformers's f32 lm_head. At a near-tie this could easily flip top-1/top-2. Would need to A/B with g_idot off (or lm_head at fmt=0) to confirm.
2. Reduction-order differences in the MLA-absorb attention path (the very reductions [Performance]: hot-path float reductions are scalar and don't auto-vectorize (router matmul + MLA-absorb attention) #442 is about — but pre-existing, before any SIMD).
3. RMSNorm epsilon or the router matmul feeding subtly different hidden states into the final projection.

I stopped here because this is a forward-pass numeric-resolution investigation that needs careful per-stage A/B isolation, and it was pulling me away from the #442 work that surfaced it. The 30/32 floor is stable and well-characterized, so it can serve as a regression gate in the meantime (a correctness change should not move it).

Proposal: implement the advertised DEBUG_LOGITS dump

While investigating this, I discovered that colibri.c:6202 advertises a feature that does not exist:

"[ORACLE] %d/%d mismatches — run: TF=1 DEBUG_LOGITS=1 for top-5 logit dump\n"

Nothing in the engine reads DEBUG_LOGITS. Anyone who hit an oracle mismatch, read that hint, and set the env var got nothing — which is exactly the trap I fell into when I had to write my own dump to see these near-ties.

I have a 32-line implementation on a local branch (fix/tf-oracle-argmax-mismatch, commit a9d1ba5): forward_all optionally stashes the full [S, vocab] logit matrix; the TF block prints a NaN-safe top-K at every mismatch position (K via DEBUG_LOGITS_K, default 5). It is what produced the exact numbers above.

Happy to PR it if useful — it is a developer-facing debug tool, not a behavior change to the engine, so it may be too small to merit its own PR. Leaving the call to the maintainers; the diff is available if anyone wants it.

Correctness gate (for #442)

The #442 SIMD reductions PR (#481) was gated against this 30/32 floor: the SIMD change preserves the exact same 2 mismatches at the exact same positions with the exact same wrong tokens. So the floor is usable as a regression gate — just not at the documented 32/32.

[JustVugg]

Good catch, and worth clarifying because it affects how you use the gate for #442. On the reference build here (gcc, make glm, WSL/Linux) current dev gives 32/32 — I re-ran it while merging #481/#477. So this isn't a forward-pass regression; it's what you named it: a near-tie at 2 positions. At those positions the top-1 and top-2 logits are within FP epsilon, so the argmax flips depending on FMA contraction, reduction order, -march/-ffp-contract, and libm — i.e. it's compiler/flags-dependent, not a correctness bug. The model is computing the right distribution; the oracle just does exact-argmax match and those 2 ties can land either way.

Practical implications:

• For [Performance]: hot-path float reductions are scalar and don't auto-vectorize (router matmul + MLA-absorb attention) #442 as a correctness gate: a fixed-order SIMD reduction stays bit-identical to the scalar fixed-order baseline, so it won't move these ties — perf: SIMD MLA-absorb score + value-mix reductions (#442, +4.7x) #481/attn: vectorize the MLA absorption helpers — +13% decode on AVX-512 hosts, byte-identical output #477 confirmed 32/32 held. That's still the right gate; just compare against your own pre-change run, not the hardcoded "32/32", since your toolchain sits at 30/32.
• Doc fix worth doing: the self-test string at colibri.c should say "expect ~30–32/32 (2 positions are FP near-ties; exact count is toolchain-dependent)" rather than a hard 32/32.

Which compiler/flags/OS gave you 30/32? If you want, I'll add the doc-string softening as a tiny PR.