Datapoints: GLM-5.2 int4 fully RAM-resident on 2-socket Ice Lake (48C, 661 GiB) — 3.85 tok/s; DRAFT=0 beats MTP at full residency

[veerareddyvishal144]

Benchmark datapoints from a 2-socket Ice Lake host with the whole int4 container RAM-resident — posting per the README's request for real-machine numbers. Also independently confirms two existing findings (#82 NUMA gains; the 6×5090 experiment log's "speculation loses at full residency") on pure-CPU hardware.

Machine

• Azure Standard_E96d_v5: 2× Intel Xeon Platinum 8370C (Ice Lake), 48 physical cores / 96 SMT, 2 NUMA nodes, 661 GiB RAM, avx512-vnni IDOT kernel
• Local NVMe: 4.18 GB/s O_DIRECT (iobench 19 MB × 64, 8 threads); buffered 26.9 GB/s (page cache)
• colibri main @ 00ed8d5, gcc 13.3, setup.sh, self-test 32/32
• Model: mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp (int8 MTP heads verified by size)

Method

Greedy (TEMP=0), --ngen 256, single request, KVSAVE=0, fresh process per run. Full residency via PIN_FILL=1 + PIN_GB=380 (all 19,456 experts pinned at startup, ~47 s warm / ~2 min cold page cache; expert hit 100%, disk service 0.000 s in every run below). 48 physical cores, OMP_PROC_BIND=spread OMP_PLACES=cores with the COLI_OMP_TUNED=1 workaround for the re-exec affinity issue (#471). Run-to-run noise observed ±0.1 tok/s.

Results (each row changes one thing vs the row above)

config	tok/s	notes
baseline (defaults + full pin)	2.14	expert-matmul 58% of wall, 48.9 GB/s effective
+ COLI_NUMA=1	2.65	+24% — matches the 2-socket selective-interleave gains in #82
+ DRAFT=0 (MTP off)	3.65	see below
+ I4S=1 (int4 IDOT at S=1)	3.85	+5.5%; expert-matmul 67.8→89.5 GB/s
+ TOPP=0.7 (lossy)	4.85	for completeness

Neutral on this host: CACHE_ROUTE=1 (route_agree 100.0%, route_kl 0.0000 at full residency — nothing to substitute), I4_ACC512=0 vs 1 (3.70 vs 3.65, within noise — the S=1 down-proj is the IDOT path's job once I4S=1), SMT 96 threads (3.43, −6% vs 48 physical), PILOT untested-by-design (disk already 0%).

MTP speculation at full residency (CPU-only confirmation)

int8 head works (not the #8 defect): DRAFT=3 → 2.39 tok/forward, DRAFT=1 → 78% acceptance. It still loses:

DRAFT	tok/s	tok/forward
3 (auto default)	2.65	2.39
1	3.53	1.78
0	3.65	1.00

Same structural cause the 6×5090 log identified: with disk out of the path the cost is per-unique-expert, and verify positions route to mostly different experts, so a verify batch scales cost ~linearly with S. Worth considering: at 100% expert hit rate the engine could warn (or auto-suggest DRAFT=0) — the DRAFT=-1 default silently costs ~27% on hosts like this.

Where the time goes at 3.85 tok/s

PROF=1: expert-I/O 0% | expert-matmul 49% | attention 37% | other 12%. Expert-matmul runs at ~89 GB/s effective vs ~350+ GB/s of platform DRAM bandwidth — the per-expert OpenMP fork/join cadence looks like the cap (2 regions/expert × 8 experts × 75 layers, ~200 KB touched per thread per region). An experimental one-region-per-block restructuring measured +24% here (4.77 tok/s); happy to open a separate discussion/PR with the patch if there's interest.

[JustVugg]

Excellent datapoints — thank you for posting real-machine numbers per the README's ask. Two things it settles:

Confirms #82 (NUMA): COLI_NUMA=1 2.14 → 2.65 tok/s on 2-socket Ice Lake is right in the measured +7–40% band. Good independent replication on Xeon.

Confirms — and usefully sharpens — the MTP story. You've shown DRAFT=0 beats MTP at full residency on pure CPU. That's consistent with, and complements, #467/#389: MTP has two failure modes, and this is the second one. On a streaming box it loses because the S=4 verify widens the expert union → more disk reads (#467, −32% at 85% hit). On a compute-bound box — which a fully-resident 2-socket CPU is — it loses because the S=4 batch does ~4× the matmul work per forward and acceptance <100% doesn't repay it (#389, −42%). So MTP only pays in the narrow window where you're both resident and have spare compute (a GPU with idle cycles), which your CPU host isn't. The planner's DRAFT=0 default is now confirmed correct across both regimes — disk-bound and compute-bound. If you set COLI_CUDA_MTP/DRAFT you'll now find the opt-in honoured (#470, just merged), for anyone who wants to probe the crossover on a GPU box.

The 3.85 tok/s full-residency headline is a great reference point for the tracker. Please keep them coming.

[veerareddyvishal144]

Correction to the last paragraph: the "+24%" figure for the experimental cross-expert restructuring came from a non-interleaved pair and overstated the gain — the baseline session was a low outlier (baseline placement drifts a few % with the accumulated .coli_usage pin ordering). A proper ABAB battery (3 prompts × 2 reps, same config) measures +11.6% mean (4.20 → 4.68 tok/s, worst prompt +11.3%), byte-identical output. Now submitted with full methodology as #475.