llama-cpp-overdoing

Extra patches for llama-cpp-rdna-boosts on AMD RDNA4 (gfx1201, Radeon AI PRO R9700), for the changes that are not part of rdna-boosts: mostly lossy prefill speed-ups, a faster decode / draft-verification matmul, a guard against replies that end inside the reasoning, an opt-in lenience for sampled drafts, faster wide draft verification, a smaller recurrent-state cache for draft verification, and a smaller FFN compute buffer. Most other bit-identical work goes to rdna-boosts itself. Patches only; MIT. Written with Claude (AI).

Patches

File names: <serial>-<upstream base>-<name>-v<version>.patch. The files in patches/ are for rdna-boosts v16-6050b467c-r195 (upstream base 6050b467c).

patch	what	switch	type
0001-605-k1-v7	K-quant prefill matmuls via hipBLASLt	GGML_LF_KQ_BLAS (default on, fp8)	speed/ lossy
0002-605-f8-v6	fp8 prefill attention	GGML_LF_FA_FP8 (default on)	speed/ lossy
0003-605-m4-v4	MXFP4 weights to fp8 for 0001	GGML_LF_MXFP4_DIRECT (default on)	speed/ lossy
0004-605-reasoning-guard-v2	no reply left inside the reasoning	LLAMA_REASONING_GUARD (default on)	guard
0005-605-state-bands-v1	saving and restoring a slot above 256 MiB through the fast batched path	LLAMA_STATE_ASYNC_BANDS (default on, one device)	speed
0006-605-mmvk-v4	faster 1-8 token K-quant matmuls	GGML_CUDA_MMVK (default on)	speed/ diff
0007-605-spec-lenience-v1	keep more sampled-draft tokens	LLAMA_SPEC_LENIENCE (default off)	speed/ lossy
0008-605-glu-inplace-v1	an FFN's gate, up and GLU output not all live at once (150 MiB less compute buffer at -ub 2048)	GGML_GLU_INPLACE (default on)	memory
0009-605-fa-band-split-v1	faster 9-16 row draft verification	GGML_CUDA_FA_BAND_SPLIT (default on)	speed/ diff
0012-605-dflash-adaptive-width-v2	DFlash2 verify width from the acceptance	LLAMA_DFLASH_ADAPTIVE (default on)	speed
0013-605-gdn-verify-anchor-v4	Qwen3.x draft verification keeps two recurrent-state planes, not one per draft token (about 1 GiB less VRAM)	LLAMA_RS_ANCHOR (default on); GGML_CUDA_GDN_NT (default on)	speed
0014-605-ngram-prompt-only-v2	n-gram drafts from the current prompt only, with a key check: for rewriting whole files	LLAMA_SPEC_NGRAM_PROMPT_ONLY (default on, with ngram-mod)	speed/ trade-off
0015-605-spec-align-v4	slots verifying together send drafts of one length	LLAMA_SPEC_ALIGN, LLAMA_SPEC_ALIGN_CUT (default on)	speed

_{0010, 0011 (Temporarily)}

What each patch does, what it needs, the types, and the change log: PATCHES.md.

Every patch can be turned off with its switch; left unset, all are on except 0007, and GGML_LF_ALL=0 turns off every switch left unset (SWITCHES.md). Prefill speed-ups measured on Linux, R9700, Qwen3.8-27B UD-Q4_K_XL, DFlash2 draft, 8K / 32K prompt, on v16-a55e952b8-r8; 0001 v7 is faster since (+7.8 % at depth 0, +4.7 % at 32K on 2048-token batches, the same output); KLD measured on r144 (2026-10-27 report):

	prefill	KLD vs official UD-Q5_K_M on r144 (zh / en)
reference (no patch)	—	0.0018 / 0.0006 (run-to-run floor)
K1 f16	+14% / +15%	0.0018 / 0.0005 (≈ floor)
K1 fp8 (the default)	+16% / +22% on top of f16	0.0026 / 0.0007
F8 (v5)	+3% / +17%	0.0018 / 0.0005
M4 (MXFP4 models only)	+55% at 8K, +44% at 110K	+0.002 / +0.0006 over the MXFP4 weights themselves
mmvk (0006)	— (decode: 4.5-6% less time per step with drafts)	0.0015 / 0.0003 (below the floor)
all defaults		0.0025 / 0.0009

K1, F8 and M4 leave decode unchanged; 0006 changes decode only (for its KLD the decode path runs it at one token, GGML_CUDA_MMVK=4). K1 fp8 with F8 also raises the perplexity of the prompt's own logits (every position scored on the prefill path): by 0.1-0.3% on new text in Chinese, English or Thai and on a German novel, 0.6-0.8% on texts the model knows by heart, and 2.4-3.6% on a text it half-remembers (Hong Lou Meng, our Chinese test text; this note used to read that as a Chinese effect). Generation reads the prompt through the KV cache and sees far less (the KLD column; Thai at the level of Chinese and English). For prompt-scoring workloads set GGML_LF_KQ_BLAS=f16. Details: 2026-10-12 report.

F8 v5 counts the row sum from the fp8 probabilities themselves: on a 170K-token reasoning conversation F8's own KLD drops by a fifth (0.000396 to 0.000307, floor 0.00021); with all defaults the change is below the floor, at the same prefill speed (report). On the table's 48K test v5 is within the floor of v4 (F8 alone 0.0019 / 0.0007 with v4; all defaults 0.0024 / 0.0008 with v4). F8 v6 gives the same output as v5 (LDS-only barriers, and the WMMAs of two accumulators issued in turn where that is faster): F8's kernel +2.7..+4.3 % on Qwen3.8-27B's shape, prefill +0.5 % at 32K and +0.6 % at 110K (change log).

0009 changes only draft-verification batches of 9-16 rows (two slots verifying at once, or a long n-gram draft), which neither the KLD run nor the standard benchmark has; each of those rows then gets the result a batch of 8 rows or fewer gives. Two slots decoding at once at 50K each: 13-19% slower than one after the other without it, 33-38% faster with it.

0008: the graph allocator lets a GLU write into its gate's memory, so an FFN's gate, up and GLU output are no longer all live at once: at -ub 2048 the compute buffer is 150 MiB smaller on Qwen3.8-27B and 70 MiB on Qwen3.6-35B-A3B. The output is the same, and prefill speed did not change.

0013 (LLAMA_RS_ANCHOR=0 turns the anchor off): draft verification on qwen35 / qwen35moe models (Qwen3.5 to 3.8) keeps, per slot, the recurrent state before and after the last batch plus small per-token replay factors, instead of one full state per draft token, and replays the accepted part when a draft is partly rejected. The output is the same. Qwen3.8-27B with DFlash2: 1.03 GiB less VRAM with 2 slots, 0.51 GiB with 1 (-c 131072); each decode step 0.3-0.4% faster, two slots decoding at once 2-4% faster.

Switches

Every switch, its values, and the server parameters it needs: SWITCHES.md.

On Linux we also set ROCm's own HSA_USERPTR_FOR_PAGED_MEM=0. By default ROCm allocates pinned host memory (hipHostMalloc, llama.cpp's ROCm_Host buffers) as ordinary pageable pages registered with the GPU; when the kernel swaps out or moves one of them (memory pressure, compaction), the driver stops all GPU queues of the process until the page is back (5-440 ms at a time, often several in a row). With 0 those buffers are allocated by the driver instead and cannot be swapped or moved. Under the same memory pressure: 23 stalls (1.16 s in total over 19 requests) became none, with the same output and speed (report). The cost: those buffers (about 0.7 GB with every layer on the GPU) stay in RAM; with large weights left in host memory (partial offload without mmap) they count against the driver's limit. Not needed on Windows.

Applicable versions

Tested on rdna-boosts v16-6050b467c-r195: the patches in patches/ apply in number order on the plain release, build, and give the History numbers. On another release, check with git apply --check first. A release on a new upstream base (not 6050b467c) needs a new patch set.

History

Speed on the standard benchmark (Qwen3.8-27B, 32K-token prompt, R9700, Linux), tokens/s. original: plain rdna-boosts. Each dated row: this repo's patches of that date, all on, on the release it names (or else the original above it); the date links to the report (setup and KLD).

date	build	Q5 prefill	Q5 decode	MXFP4 prefill	MXFP4 decode
original	v16-a55e952b8-r8	983	47.4	1151	54.7
2026‑10‑07	+ K1 v2, F8 v3.1, M4 v2	1958	47.5	2393	63.0
2026‑10‑12	+ K1 v3, F8 v3.2, M4 v2	1954	49.0	2388	62.9
2026‑10‑13	v16-097cf33d9-r50 + K1 v3, F8 v3.2, M4 v2	1959	50.6	2397	66.2
2026‑10‑15	v12-097cf33d9-r59 + K1 v3.2, F8 v3.3, M4 v2	2197	50.4	2365	66.2
2026‑10‑18	v12-097cf33d9-r59 + K1 v3.2, F8 v3.3, M4 v2, mmvk v2	2207	53.8	2389	67.2
original	v15-6050b467c-r156	1155	52.8	1384	57.9
2026‑11‑01	v16-6050b467c-r195 + K1 v7, F8 v5, M4 v4, mmvk v4 (the defaults)	2510	59.4	2531	61.2

Q5 = official UD-Q5_K_M; MXFP4 = FreedomAISVR MXFP4.

Build

• Linux
• Windows
• Then: run, switches

Run

The patches are on by default; each can be turned off with its switch. Speculative decoding needs a draft model and these server parameters. An example command, the one behind the measurements here (R9700, Qwen3.8-27B, Linux; on Windows the same in PowerShell, without HSA_USERPTR_FOR_PAGED_MEM and with --poll 0):

HSA_USERPTR_FOR_PAGED_MEM=0 ./llama-server -m Qwen3.8-27B-UD-Q4_K_XL.gguf -ngl 99 -c 131072 --kv-unified \
  -ub 2048 -b 2048 -fa on -ctk q8_0 -ctv q8_0 \
  -md Qwen3.8-27B-DFlash2-Q4_K_M.gguf -ngld 99 --spec-draft-n-max 5 \
  --spec-type draft-dflash \
  --spec-draft-sampling probabilistic

• -ub 2048 -b 2048 -fa on -ctk q8_0 -ctv q8_0: K1 and F8, the prefill speed-ups, need them (SWITCHES.md).
• -md ... -ngld 99 --spec-draft-n-max 5: the DFlash2 draft model, for Qwen3.8-27B only (z-lab/Qwen3.8-27B-DFlash2-GGUF, Q4_K_M). With any other model leave out the last three lines.
• --spec-type draft-dflash: DFlash2 drafts. 0014 acts only with an n-gram type (SWITCHES.md).
• --spec-draft-sampling probabilistic: llama.cpp's own; sampled drafts, verified exactly by rejection sampling (it matters at temperature > 0). 0007 (LLAMA_SPEC_LENIENCE, lossy, off by default) builds on it.
• HSA_USERPTR_FOR_PAGED_MEM=0: Linux only (Switches).
• --poll 0: Windows. The Windows build has no OpenMP, so llama.cpp's own CPU thread pool spins after the CPU part of every step: on a Ryzen 7 5700G with an R9700 the server used 4.2-4.4 cores while generating, 0.85-0.91 with --poll 0, at the same speed and with the same text. The Linux build uses OpenMP and does not need it.
• The GitHub Actions workflow can build a Windows package when it is run by hand (no package is published here); it includes run-server-sample.ps1: these settings, with the two model paths to fill in.

Use

These patches are offered for research and evaluation. Every patch can be turned off with its switch (SWITCHES.md). Users are responsible for checking the rights that apply where they use them.

本 repo 的 patch 供研究與評估。每個 patch 都能用它的開關關掉（SWITCHES.md）。使用者請自行確認所在地適用的權利。

Thanks

To stew675 and the rdna-boosts contributors, and to the llama.cpp / ggml community.

Note

Issues and pull requests may not get an answer here. For problems, try a frontier cloud model first.

本 repo 不一定會回應 issue 與 PR。遇到問題，建議先請雲端前沿模型協助處理。