GGUF compatibility CI: 'zoo' suite covering Llama / Qwen / Gemma / Phi / Mistral × Q4_0 / Q4_K_M / Q5_K_M / Q6_K / IQ4_NL

[sauravpanda]

Type: strategic direction.
Origin: codex review of v0.9.13.

Problem

Forge's pitch is "compile any GGUF, get a fast standalone binary". But the product promise dies on the first unsupported tensor name, RoPE mode, tokenizer field, or architecture metadata mismatch — and we discover those one model at a time, manually.

We've validated:

• Llama-3.2-1B (Q8_0, Q4_0, Q4_K, Q6_K, Q4_K_M)
• SmolLM2 135M (Q8_0, Q4_0)
• Qwen2.5 0.5B (Q8_0)
• Phi-3 mini (Q8_0)

That leaves a long tail of "the GGUF the user actually downloaded" untested.

Proposal

Build a CI-runnable GGUF compatibility harness. For each (architecture × quant) cell:

1. Download a small public GGUF (or generate a synthetic one with the right metadata).
2. Run forge compile + cargo build.
3. Run the AOT binary on a fixed prompt and a corpus-perplexity check.
4. Assert: build succeeds, output is non-NaN, perplexity is within tolerance of the per-cell baseline.

Coverage matrix

	Q4_0	Q4_K	Q4_K_M	Q5_K_M	Q6_K	Q8_0	IQ4_NL	IQ4_XS	F16
Llama-3.2-1B	✓	✓	✓		✓	✓
Llama-3.2-3B
Qwen2.5-0.5B						✓
Qwen2.5-1.5B
Qwen3-0.6B
Gemma-2-2B
Phi-3-mini						✓
Mistral-7B
SmolLM2-135M	✓					✓

(Boxes are aspirational; ✓ = currently validated.)

Priorities

1. Q5_K_M — second most common K-quant after Q4_K_M; trivial extension of the Q4_K/Q6_K codegen since it's another superblock layout.
2. Qwen3 — newer arch increasingly used; need to verify our RoPE / tokenizer / metadata parsing.
3. Gemma-2 — different architecture (SWA + logit softcap).
4. IQ4_NL / IQ4_XS — lossy int4 with lookup tables; popular in newer quants.
5. Mistral 7B Q4_K_M — larger than the validated set; verifies long-context paths.

Out of scope

• MoE / vision / audio architectures (separate epic).
• Cross-compilation targets (separate issue).
• llama.cpp-binary-compatible tokenizer parity (#issue 215 / future correctness work).

Test infrastructure notes

• Models too large for git LFS; fetch from HF on-demand into a CI cache.
• Use the smallest possible GGUF per (arch, quant) cell — we're validating the codepath, not training a new model.
• Skip the cell if the file isn't downloadable in CI; warn instead of fail so external contributors aren't blocked.
• Synthesize tiny GGUFs (4-layer, vocab=32) for cells where no public file exists — same parser path, far cheaper.