Benchmark datapoint: GLM-5.2 744B on Ryzen AI Max+ 395 (Strix Halo, 128 GB, Ubuntu) — ~1.0 tok/s CPU-only

[SimonRafferty]

Adding a datapoint for a hardware class not yet in the benchmark table: AMD Ryzen AI Max+ 395 "Strix Halo" with 128 GB unified memory, running headless CPU-only. Short version: frontier-quality answers at ~0.8–1.0 tok/s sustained, i.e. usable for single well-scoped questions, not for interactive/agentic loops.

Hardware / OS

Machine	GMKtec EVO-X2
CPU	AMD Ryzen AI Max+ 395 (Zen 5, 16C/32T; AVX2 + AVX-512 + AVX-VNNI)
Memory	128 GB unified (124 GB usable; iGPU VRAM carve-out set to 1 GB minimum in BIOS to free RAM)
Storage	Gen4 NVMe, ext4
OS	Ubuntu 24.04.4, kernel 7.0.0-28-generic
GPU path	none — CPU-only (no NVIDIA; iGPU unused by the engine)
Model	mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp, 383.8 GB / 150 files; int8 MTP heads verified
Engine	coli v1.0.0, make glm ARCH=native, gcc 13.3

Disk bandwidth (iobench, 19 MB random blocks)

Mode	Threads	GB/s
O_DIRECT	8 / 16 / 32	5.50 / 5.94 / 5.59
buffered (cold)	8	6.90
buffered (fully cached)	16	54.19

Placement (coli plan, quality policy)

RAM   113.3 GB budget · 10.9 GB dense · 6.1 GB runtime · 96.3 GB warm experts · cap 66/layer
disk  276.6 GB cold experts · 1282.7 GB free

Results

All decode figures are 128-token generations (sustained) unless noted. Cold = empty .coli_usage.

Scenario	tok/s	expert hit rate	pin%	RSS
Cold (7-token burst)	1.06	61.7%	0.0%	98.8 GB
Novel question (warm store, unseen prompt)	0.78	65.2%	39.3%	106 GB
Revisited warmed prompt	0.78–0.80	66–67%	39–40%	106 GB
--topp 0.85	0.86	66.9%	40.0%	105 GB
DIRECT=1	0.89	67.3%	40.9%	106 GB
DIRECT=1 + --topp 0.85	0.99	67.9%	42.4%	105 GB

Speculative decoding (int8 MTP heads): engaged, 2.0–3.5 tokens/forward, MTP acceptance 35–83% across runs. (Confirmed the int8 head sizes before running — the int4 head would have disabled this.)

Observations

1. Cache warming does not improve wall-clock here. Across ~11 turns the pin store rose from 3% to ~40%, but sustained tok/s stayed ~0.78 and expert hit-rate plateaued at ~66% almost immediately. A revisited prompt (0.80) was no faster than a novel one (0.78). With ~698 experts routed per token, the ~33% that miss must stream from disk regardless of pinning, and 96 GB of warm budget can't blanket the 372 GB expert space. Net: on this hardware, warming shifts hits from LRU to pin but not the clock.
2. The I/O knobs matter more than warming. DIRECT=1 (+14%) and --topp 0.85 (+10%) stack to +27% (≈1.0 tok/s). DIRECT=1 being the bigger lever is consistent with observed memory pressure: at the default RAM budget the box sits at ~1 GB free with kswapd active, so bypassing the page cache avoids reclaim overhead. (I did not test a lower --ram to leave page-cache headroom — that may be worth trying.)
3. Decode decays with context length: ~0.89 tok/s at token 16 → ~0.74 at token 96 (KV growth). Short answers are fastest.
4. Answer quality is excellent — coherent, accurate, well-structured, correct LaTeX/math. This is a genuine frontier model; the only cost is speed.

Practical takeaway

For a ~128-token answer, budget ~2.5–3 minutes. That makes it viable for single, surgical questions (my intended use is offline/low-connectivity fieldwork where a slow-but-capable model beats none) but not for agentic coding loops or long-context prompts. Recommended run flags on this hardware: DIRECT=1 ... --topp 0.85.

Happy to run additional configurations (lower --ram, --policy balanced, longer contexts, PIPE/PILOT toggles) if useful for the table.

[JustVugg]

Great datapoint — thank you. ~1.0 tok/s CPU-only for GLM-5.2 744B on the Strix Halo (Ryzen AI Max+ 395, 128 GB) is a useful reference for the unified-memory APU class. A couple of levers worth trying on that box, all measured to help elsewhere: COLI_NUMA=1 if the APU exposes multiple memory domains; --topp 0.85 (fewer expert bytes/token, no quality loss); and PIN a large hot set since 128 GB can hold a big chunk of the 370 GB expert set resident. If you rerun with those, the before/after would be a valuable addition to the tracker. Does the 395's iGPU show up to the planner (./coli plan)? Vulkan (#418) could eventually tap it.

[SimonRafferty]

Thanks @JustVugg — much appreciated. Great suggestions (COLI_NUMA=1, --topp 0.85, a bigger PIN set), and I'll check the planner/iGPU question too. I've run out of time to take it further for now, but I'll pick it back up and post the before/after with those levers when I next get a free day. The MTP-over-HTTP direction in #492 is exactly the kind of thing that'll pull me back to it. Thanks again for engaging so thoroughly.