Do diffusion LLMs tolerate extreme weight quantization better than autoregressive LLMs? Measured on matched pairs — same authors, data, tokenizer, scale, recipe — with pre-registered thresholds (timestamped before each run). All experiments on a single RTX 2080 Super (8 GB).
The publishable finding, with caveats and limitations: technical whitepaper (PDF).
MDLM-owt 130M dLLM versus its matched AR baseline. Values are relative to each model's own FP16 perplexity; negative excess means less dLLM degradation.
| evaluation | precision | AR degradation | dLLM degradation | excess (dLLM − AR) |
|---|---|---|---|---|
| PTB likelihood | INT8 | −0.03% | +0.62% | +0.65 pp |
| Wikitext103 likelihood | INT8 | +0.21% | +0.56% | +0.35 pp |
| LAMBADA likelihood | INT8 | −0.03% | +0.27% | +0.30 pp |
| PTB likelihood | INT4 | +31.81% | +20.85% | −10.97 pp |
| Wikitext103 likelihood (primary) | INT4 | +26.84% | +11.59% | −15.25 pp |
| LAMBADA likelihood | INT4 | +24.77% | +9.02% | −15.74 pp |
| GPT-2-large generative anchor | INT4 | +96.2% | +45.4% | −50.8 pp |
| Likelihood | ternary | ~10⁴× FP16 | ~10¹²× FP16 | uninformative (pre-registered) |
→ Pre-registered dllm_more_robust verdict fired: the diffusion model
absorbs INT4 weight noise roughly 2× better than its AR twin, unanimous
across PTB / Wikitext103 / LAMBADA.
Exp 0 — generative anchor (64 samples/condition, GPT-2-large oracle, entropy guard passing): INT4 costs the AR model +96% generative perplexity vs +45% for the dLLM. Ternary-PTQ models fail to produce a single valid sample. Corroborates the likelihood result — not a perplexity artifact.
Exp 2 — native ternary QAT replication (12 matched tiny models trained from scratch, BitNet-style STE, identical recipe, three seeds):
| metric | result |
|---|---|
| AR ternary tax | +16.1% to +30.7% |
| dLLM ternary tax | +4.2% to +15.4% |
| gap ratio R (dLLM tax / AR tax) | 0.883 to 0.898 |
| geometric mean R (95% t CI) | 0.890 [0.872, 0.908] |
→ The pre-registered no-extra-dLLM-tax criterion fires (upper CI 0.908
≤ 1.25). The stronger pre-registered dllm_better criterion does not: its
required upper CI below 0.80 was not met. This is a three-seed, 7M-scale,
Wikitext103 feasibility result—not evidence of superior diffusion tolerance.
Details, caveats, raw values, and checkpoint hashes: experiments/exp0/RESULTS.md, experiments/exp2/RESULTS-replication.md.
At a fixed position and commit set, a sub-margin logit perturbation can leave an argmax token unchanged. That limited identity preservation is not general error absorption: in 96 native commit-and-freeze trajectories, changing eight fixed tokens caused 65.7% FP16 and 66.3% ternary-QAT later-token disagreement; the QAT-minus-FP16 interval was [-2.2, +3.3] points. A wrong committed token remains conditioning context. See the exploratory result. This concerns the local commit-and-freeze sampler, not DiffusionGemma: its entropy-bound sampler re-noises non-accepted canvas tokens and recomputes acceptance each step. A custom post-hoc revisable-remasking probe also failed to absorb perturbations (~98% disagreement), but it was not trained under that sampler and is not evidence about DiffusionGemma. Next: train and evaluate a matched native revisable model under one frozen corruption/sampler pair.
Measurement-then-build, defined in the whitepaper (LaTeX) and the thesis program:
- Exp 0 (done): PTQ degradation with matched AR controls — the excess is the only admissible evidence about diffusion.
- Exp 1 (two exploratory sampler conditions): native commit-and-freeze and a post-hoc custom revisable sampler both propagated perturbations, with no resolved QAT effect. Next: train a matched native revisable condition.
- Exp 2 (three-seed replication done): native ternary QAT with matched AR controls — the gap-of-gaps supports no extra dLLM tax at toy scale, not superior tolerance. End goal: a natively ternary (1.58-bit) masked diffusion model via AR-to-diffusion conversion at ~2–3B, where BitNet says ternary reaches parity.
Every experiment carries pre-registered thresholds and kill criteria (configs/), frozen and tagged before runs; amendments are new tagged commits with justification.
Not "a 25B ternary dLLM in 6 GB" — activations, logits, and buffers dominate at small scale; in the current QAT checkpoint only 21.5% of model-state parameters are ternarized, for an idealized weight-only reduction of 1.23×. Not a claim about 7B+ models. Not calibrated PTQ — naive RTN by design, so the AR control carries the comparison.
experiments/exp0/ (PTQ grid + generative anchors) and experiments/exp2/
(QAT pilot) contain drivers, sanity tests, and resumable grid scripts; both
run against pinned bd3lms with an
sm75 flash-attn→SDPA shim (experiments/exp0/shim/, numerically validated).
See experiments/exp0/PREP.md for the full bring-up log, including why
zero-shot numbers must be reproduced through bd3lms (insert_valid_eos=False,
per mdlm#22).
- LLaDA/Dream "low-confidence remasking" is commit-and-freeze: committed tokens are frozen forever. True remasking needs ReMDM-style retrofits.
- The MDLM release ships a matched AR baseline — the confound-free control comes for free; its published zero-shot numbers reproduce only under the bd3lms eval protocol.
- Ternary PTQ fails on every architecture (BitNet's premise) — only native ternary training is informative, and no dLLM QAT existed in the literature as of 2026-07 (verified sweep in whitepaper §2).
See CITATION.cff for citation metadata.
- Diffusal-authored source code is licensed under the MIT License.
- The paper, whitepaper, figures, and other research prose are licensed under CC BY 4.0.
third_party/remains under its own upstream licenses.