GLM-5.2 speculative drafting: native MTP head beats GLM-4-9B, + opt-in MLA-latent KV cache

[joelovestech]

Hi leyten — findings plus one additive engine change from running your dense sm120 ring against GLM-5.2, packaged as a single commit on top of master. Opt-in and non-breaking: your original paths are untouched when the new flag is off.

TL;DR

1. GLM-5.2's native MTP head beats a standalone GLM-4-9B draft at every context, and the margin widens. Per-position greedy acceptance on the ring: 1k 0.849 vs 0.793, 8k 0.879 vs 0.820, 32k 0.883 vs 0.745 (accept-len 8.5 vs 3.9 — >2x accepted tokens per traversal), 100k 0.881 (the 9B is 32k-trained and can't reach it at all). MTP is also cheaper — shares the target's embed/lm_head, no separate model on the coordinator.
2. MLA-latent KV cache (MLA_LATENT=1). The dense ring caches decompressed full-head K/V and tops out ~16–24k context on an 8×96GB pod (OOM at 32k, any chunk size). The standard DeepSeek "absorb" caches the kv-latent + shared rope key and runs attention in latent space — ~70x smaller KV (32k: ~88GB → ~1.3GB/stage), clears 32k at 38GB vs a 94GB OOM. Opt-in env flag; full-head path is byte-identical when unset, and the absorb was proven equivalent offline (errors ~1e-13) before any GPU.

These are two independent wins — MTP is the draft, MLA is the target-side cache; one does not depend on the other.

What's in the diff (one commit, +~2.25k / -29)

• SHARDV2.md — overview + reproduce commands
• docs/research/GLM52-draft-and-MLA-findings-for-peers.md — full methodology, tables, the convention-bug warning, and why the 9B is architecturally absent past 32k
• docs/research/{H1-process-and-results,H1-test-plan-v2,serving-optimization-plan}.md
• research/ — mla_latent.py (offline equivalence proofs), ring_mtp.py (MTP head + --diag convention sweep), ring_long.py (chunked driver, blocked argmax), h1_bench.py, draft_accept_bench.py, pod_9b_dump.py, ring_target_dump.py, dumps + tests
• research/glm_swarm_nvfp4_kv.py — the MLA_LATENT=1 latent-cache attention path (additive)

One reproducibility trap

GLM-5.2's MTP wants the post-model.norm hidden, concat [enorm(emb); hnorm(h)]. The pre-norm hidden gives 0.53 — a plausible-but-wrong "9B wins" number. HF transformers doesn't implement 5.2's NEXTN, so we settled the convention by sweep (ring_mtp.py --diag).

Start at SHARDV2.md. Happy to walk through any of it.