[Performance]: GLM-5.2 (744B MoE) on a single AMD R9700 32 GB: 0.15 tok/s, 80% expert hit — capacity-bound benchmark

[limcheekin]

Commit

81f08a0

Hardware and storage

Hi there,

I use Opus 4.8 to drive the benchmark, it add the following benchmark results to the docs/benchmarks.md, appreciate if you have any idea to improve the performance:

Community benchmarks (measured)

Real numbers from real machines, stock build (setup.sh, gcc 13), greedy decoding, --ngen 32, MTP active:

machine	disk (iobench, 19 MB × 64, 8 threads)	config	measured
Intel i5-8600K (6C/6T, AVX2) · native Linux · 125 GB RAM · SATA SSD · AMD Radeon AI PRO R9700 32 GB (gfx1201/RDNA4, ROCm 7.2.2 HIP backend), measured 2026-07-24	— (SATA SSD, not iobench'd)	HIP build · CUDA_EXPERT_GB=30 CUDA_RELEASE_HOST=1 DIRECT=0 · warm cache · 1586 pinned experts (30 GB VRAM)	0.15 tok/s · hit 80.2% · RSS 97.7 GB (64-tok, MTP acc 0%) · at 26 GB budget → 0.12 tok/s / 77%

AMD single-card setup (ROCm/HIP, R9700) — measured 2026-07-24

First end-to-end AMD-discrete datapoint (the GPU backends matrix
had a hip-test pass on gfx1201 but no tok/s). Two setup notes not in the build docs:

# 1) Build the HIP backend. The documented command links as a PIE and fails
#    (backend_cuda.o is not -fPIC), so force position-independent code:
rm -f c/backend_cuda.o c/colibri c/glm
ROCM_HOME=/opt/rocm-7.2.2 make -C c glm HIP=1 HIP_ARCH=gfx1201 \
  HIPCCFLAGS="-O3 -std=c++17 -x hip --offload-arch=gfx1201 -fPIC -Wall -Wextra"

# 2) The `coli` wrapper is NVIDIA-only: cuda_binary() greps ldd for libcudart
#    (a HIP build links libamdhip64), and resource_plan.discover_gpus() only calls
#    nvidia-smi. So `coli run --gpu` / doctor report "no GPU". Drive the engine
#    directly — it enumerates the AMD device via HIP itself:
SNAP=/path/to/glm52_i4 NGEN=64 \
  PROMPT='[gMASK]<sop><|user|>...<|assistant|><think></think>' \
  COLI_CUDA=1 COLI_GPUS=0 HIP_VISIBLE_DEVICES=0 \
  CUDA_EXPERT_GB=30 CUDA_RELEASE_HOST=1 DIRECT=0 ./c/colibri 8

Result: HIP init clean on gfx1201/ROCm 7.2.2, coherent greedy output. Raising the
VRAM expert budget 26 → 30 GB (1359 → 1586 resident experts) lifted decode
0.12 → 0.15 tok/s and hit rate 77 → 80%; a warm-cache priming pass cut
prefill 188 → 137 s (buffered pread, DIRECT=0) but barely moved steady-state
decode. Hit rate plateaus at ~80% even at 30 GB: this prompt's working set
(~2757 hot experts) exceeds what one 32 GB card holds, so ~20% of routes stay cold
disk reads. It is capacity-bound, not compute-bound (decode wall ≈ 280 s
disk-wait + 160 s compute over 64 tokens); the compute-only floor is ~0.4 tok/s.
More residency needs more VRAM (more cards), not tuning.

Software environment

OS Ubuntu 24.04.4 LTS · kernel 6.8.0-134-generic (x86-64)
CPU Intel Core i5-8600K (6C/6T, AVX2) @ 3.60 GHz
RAM 125 GiB
Disk 3.6 TB SATA SSD (/dev/sda1), model on /mnt/data (2.3 TB free)
GPU AMD Radeon AI PRO R9700, 32 GB VRAM (34.2 GB reported) · gfx1201 (RDNA4)
ROCm / HIP ROCm 7.2.2 · HIP 7.2.53211 · AMD clang 22.0.0git (roc-7.2.2)
Host compiler gcc 13.3.0 (Ubuntu)
Engine colibri 1.1.1 (c/colibri), built with the HIP backend
Model GLM-5.2 int4 + int8 MTP — /mnt/data/models/GLM-5.2-colibri-int4-with-int8-mtp
(~384 GB on disk, 144 shards, 9.9 GB dense resident, 19,456 experts)

Build and benchmark commands

# --- 1. Build the colibri HIP engine for gfx1201 -----------------------------
# NOTE: the documented `make ... HIP=1` links as a PIE and fails
# (backend_cuda.o is not position-independent), so add -fPIC explicitly.
cd /mnt/data/colibri
rm -f c/backend_cuda.o c/colibri c/glm
ROCM_HOME=/opt/rocm-7.2.2 make -C c glm HIP=1 HIP_ARCH=gfx1201 \
  HIPCCFLAGS="-O3 -std=c++17 -x hip --offload-arch=gfx1201 -fPIC -Wall -Wextra"
ldd c/colibri | grep amdhip     # verify libamdhip64 is linked

# --- 2. Run the benchmark (max VRAM budget + warm page cache) ----------------
# The `coli` wrapper is NVIDIA-only (cuda_binary() greps ldd for libcudart;
# resource_plan.discover_gpus() only calls nvidia-smi), so it refuses --gpu on a
# HIP build. Drive the engine binary directly — it enumerates the AMD GPU via HIP.
# argv[1] = prompt cap; everything else is read from the environment.
export SNAP=/mnt/data/models/GLM-5.2-colibri-int4-with-int8-mtp
export PROMPT='[gMASK]<sop><|user|>Explain what a mixture-of-experts model is in two sentences.<|assistant|><think></think>'
export COLI_CUDA=1 COLI_GPUS=0 HIP_VISIBLE_DEVICES=0   # select the AMD device
export CUDA_EXPERT_GB=30       # max expert tier on a 32 GB card (2 GB reserve)
export CUDA_RELEASE_HOST=1     # drop RAM copies of VRAM experts -> more warm RAM
export DIRECT=0                # buffered pread -> OS page cache is used (warmable)
export COLI_POLICY=quality

# prime pass (warms page cache + expert placement / .coli_usage), then measure:
NGEN=32 ./c/colibri 8          # prime
NGEN=64 ./c/colibri 8          # measured (read the steady-state tail [t=N ... tok/s])

Results

# --- Result -------------------------------------------------------------------
# CUDA_EXPERT_GB=30, warm: 0.15 tok/s decode · 80.2% expert hit · 1586 experts
#   resident in 30.00 GB VRAM · peak RSS 97.7 GB · prefill 137 s (warmed from 188 s).
# Baseline CUDA_EXPERT_GB=26: 0.12 tok/s · 76.9% hit · 1359 experts (25.99 GB).
# Capacity-bound at ~80% hit on one 32 GB card (working set ~2757 hot experts);
# compute-only floor ~0.4 tok/s. More throughput needs more VRAM (more GPUs).

Baseline comparison

No response

[noobdev-ph]

Try using vuklan first. You can refer to this discussion - #523

[JustVugg]

This is a genuinely excellent datapoint — the first AMD-discrete tok/s on the tracker (RDNA4/gfx1201), and your capacity-bound diagnosis is exactly right. Confirming and adding to it:

Your diagnosis is correct. 80% hit plateau at 30 GB + decode wall ≈ 280 s disk-wait vs 160 s compute over 64 tokens = capacity-bound, not compute-bound. The ~2757-expert working set doesn't fit one 32 GB card, so ~20% of routes are cold SATA reads. "More residency needs more VRAM, not tuning" is the honest conclusion. The compute-only floor you derived (~0.4 tok/s) is the ceiling a second card would approach.

Two real tooling bugs you hit — both ours, both worth fixing:

1. make ... HIP=1 links as PIE and fails because backend_cuda.o isn't built -fPIC. Your -fPIC workaround is the fix; the Makefile should pass it for HIP builds so the documented command just works.
2. The coli wrapper is NVIDIA-only — cuda_binary() greps ldd for libcudart and discover_gpus() only calls nvidia-smi, so coli run --gpu / coli doctor report "no GPU" on a HIP build and you had to drive ./c/colibri directly. Detection should recognise libamdhip64 + rocm-smi.

I'll put a PR up on dev for both so the next AMD user doesn't reverse-engineer the invocation. Thank you for documenting it precisely enough to fix.

One lever you're leaving on the table: MTP acc 0%. On a disk-bound decode, accepted MTP tokens are worth the most — a verified draft amortizes the same expert working-set load across two tokens, so at your 280 s-disk / 64-tok ratio, non-zero MTP acceptance would move throughput more than any VRAM tuning short of a second card. 0% acceptance on a -with-int8-mtp container is unexpected (the int8 head exists precisely so MTP works, see #8). Could you run a quick check — does the log show [MTP] active and draft=N (N>0)? If MTP is active but every draft is rejected on the HIP path, that's a separate GPU-MTP bug I'd like to chase (it may be that MTP verification isn't wired on the HIP backend yet). A CPU-only run of the same prompt (COLI_CUDA=0, small --ngen) that shows non-zero acc would confirm it's HIP-path-specific.

For the 20% cold reads: a second SSD via COLI_MODEL_MIRROR (#421) or COLI_MODEL_DIRS (#469) raises cold-read bandwidth directly; NVMe over SATA is the bigger single win. But you're right that the structural fix is VRAM.

Would you like me to add this row to docs/benchmarks.md myself, or are you opening a PR with it? Either works — it's a valuable measurement to keep.

[RDouglasSharp]

I filed PR #585 because I think it gates having realistic numbers for your request on METAL. I plan to work on implementing METAL-based fmt:4 attention, at which point I can send you a PROF=1 breakdown of timing. Do you have a specific preferred task that you run to obtain your other timings?