Feature request: speculative decoding (MTP) in the HTTP `coli serve` path for single-slot (`--kv-slots 1`)

[SimonRafferty]

TL;DR

coli serve unconditionally disables MTP/speculative decoding, so the OpenAI-compatible HTTP API decodes at ~1 tok/s, while interactive coli chat on the same box gets 2–3 tokens/forward from MTP. The disable is correct for the multi-slot mux path (speculation isn't ragged-safe across KV slots), but with --kv-slots 1 there is only ever one non-ragged decoding sequence, so speculation ought to be expressible. Enabling MTP for the single-slot serve path would roughly 2–3× decode throughput for every API/agentic client (aider, opencode, Cline, OpenAI SDK, …) — the clients that can't use coli chat.

Motivation

Agentic/coding use requires the HTTP server, not the interactive TUI. In multi-turn tool-calling against coli serve, I confirmed the server's KV-prefix reuse keeps prefill cheap on every turn — e.g. across one agent loop:

turn 1:  prefix 0/2751     prefill 2751   (cold)
turn 2:  prefix 2765/2804  prefill 39
turn 3:  prefix 2834/2868  prefill 34
turn 4:  prefix 2887/3068  prefill 181

So after the first turn, prefill is no longer the bottleneck — decode is. And decode over HTTP is stuck at ~1 tok/s precisely because MTP is off. Interactive coli chat (with MTP) reaches 2–3 tokens/forward on the same model/box, but it can't drive tools or an OpenAI-API client. Closing this gap is the single biggest decode win available to agentic users.

Current behaviour (code trace)

coli serve hardcodes SERVE_BATCH=1 into the engine's environment:

# c/openai_server.py:470
child_env = dict(env or os.environ, SNAP=str(model), SERVE="1", SERVE_BATCH="1",
                 NGEN=str(max_tokens), KV_SLOTS=str(kv_slots))

and the engine then force-disables the draft/verify path whenever SERVE + SERVE_BATCH are set — independent of --kv-slots:

// c/glm.c:6479-6480
int mux_will_disable_mtp = getenv("SERVE") && getenv("SERVE_BATCH") && atoi(getenv("SERVE_BATCH"));
int eff_draft = mux_will_disable_mtp ? 0 : g_draft;

// c/glm.c:5334
g_draft=0; /* one scheduler owns every forward; MTP/speculation is not ragged-safe */

// c/glm.c:6401
int kv_limit=(getenv("SERVE_BATCH")&&atoi(getenv("SERVE_BATCH")))?512:16;

So there is no serve configuration — including --kv-slots 1 — that keeps MTP.

What I tried (and why it confirms the guard is needed as written)

I patched openai_server.py:470 to make SERVE_BATCH overridable and ran the server with SERVE_BATCH=0 --kv-slots 1:

• MTP engaged: startup logged [MTP] active: native speculative decoding (draft=3) and, on the first request, [SPEC_PIN] draft+verify pinned to the S=1 kernel family.
• Prefill completed normally, but the model then produced corrupted output — an unrelated, incoherent response — which the server rejected:

[api] request failed: invalid engine response: I don't understand what you mean by that input...
[api] "POST /v1/chat/completions HTTP/1.1" 500

So even at single-slot, the serve decode path (the ragged/mux KV route) is not compatible with the current speculation code — the not ragged-safe comment holds in practice, not just in theory. I reverted the patch. This is a feature request, not a bug report: the guard is correct; what's missing is a non-ragged single-slot decode path that speculation can run on.

Proposed direction

When --kv-slots 1 (no multiplexing), route decode through the single-sequence, non-ragged kernel family that coli chat already uses for MTP, rather than the mux/ragged path — so draft+verify is expressible and correct. Concretely, one of:

1. Gate eff_draft on slot count: allow g_draft when KV_SLOTS==1, and select the non-ragged decode kernel for that case in the serve loop; or
2. Make the mux decode ragged-safe for speculation (harder — general fix that also helps multi-slot).

Option 1 seems the lower-risk, high-value path: single-client agentic use is the common case, and it wouldn't touch the multi-slot behaviour.

Impact

• ~2–3× decode throughput for all OpenAI-compatible clients (the ones that can't use coli chat).
• Combined with the already-working KV-prefix reuse (above), this makes multi-turn agentic coding meaningfully faster, since decode is the remaining per-turn cost.

Environment

• GMKtec EVO-X2 — Ryzen AI Max+ 395 (Strix Halo), 128 GB unified, CPU-only, Gen4 NVMe / ext4, Ubuntu 24.04, kernel 7.0.0-28.
• Model: GLM-5.2-colibri-int4-with-int8-mtp (int8 MTP heads verified). coli v1.0.0.
• Same box as the benchmark datapoint in Benchmark datapoint: GLM-5.2 744B on Ryzen AI Max+ 395 (Strix Halo, 128 GB, Ubuntu) — ~1.0 tok/s CPU-only #486.

Happy to test patches on this hardware — I can A/B single-slot serve with/without speculation and report decode tok/s + output-correctness.

[SimonRafferty]

Correction to my own report above — I misdiagnosed the SERVE_BATCH=0 experiment, and the corrected finding actually widens the issue. Apologies for the noise.

What I got wrong

I wrote that forcing SERVE_BATCH=0 --kv-slots 1 "engages MTP but corrupts output," and concluded the ragged-safety guard "is correct even at single-slot." That conclusion is unsupported — my test never actually exercised speculation in the servable path.

What actually happens

SERVE_BATCH doesn't just toggle the draft/verify path — it selects a different engine entry point:

• SERVE_BATCH=1 → run_serve_mux (c/glm.c:5331) — the multiplexed path, and the only one that parses the HTTP server's SUBMIT line protocol (coli_submit_parse, glm.c:5281).
• SERVE_BATCH=0 → run_serve (c/glm.c:5417) — a non-mux path that does not understand the SUBMIT framing.

So when I set SERVE_BATCH=0, openai_server.py kept sending its mux SUBMIT lines, but the engine was now running run_serve, which fed that framing to the model as prompt text. The "corrupted output" was the model reacting to raw protocol, not a speculation bug:

[api] request failed: invalid engine response: I'm sorry, but I'm not quite
sure what you mean by the numerical sequence "1 0 71 60 0.7 0.9" or the
instruction "SUBMIT."

That's the SUBMIT <id> <slot> <len> <ngen> <temp> <topp> header reaching the model verbatim. So: protocol mismatch, not ragged-unsafe MTP. Whether speculation is actually safe at single-slot in the mux path remains untested — my experiment never got there.

The finding this surfaced (widens the request)

While confirming the above I found the same split affects grammar-constrained decoding: grammar_setup() is called in run_serve (glm.c:5437) but not in run_serve_mux. Confirmed empirically — with GRAMMAR=<file.gbnf> set and the env verified as reaching the engine (/proc/<pid>/environ), the mux path silently ignored it (free-form output), while run_serve loaded it ([GRAMMAR] …: 1 rules, forced span capped at 24 tokens/forward) but then hit the same SUBMIT-as-prompt failure.

So the HTTP-servable path (run_serve_mux) supports neither speculative decoding (MTP) nor grammar/GBNF-constrained generation — both live only in run_serve, which the OpenAI server can't drive.

Restated ask

Two routes to close it, either of which would help:

1. Bring speculation (and grammar) into run_serve_mux — with a ragged-safe single-slot verify (--kv-slots 1 decode is a single non-ragged row, so it seems tractable there); or
2. Teach run_serve the SUBMIT protocol (or point openai_server.py at it when KV_SLOTS==1) so the feature-rich path becomes HTTP-servable.

Grammar-constrained tool-call output would be a nice bonus of either — it both accelerates structured spans (up to 24 forced tokens/forward) and would fix malformed tool-args, which matters a lot for agentic clients. Same box as before; happy to test patches.

[JustVugg]

Reasonable ask, and single-slot is the case where it's actually safe. MTP is disabled in coli serve today because speculation isn't ragged-safe across multiple KV slots (SERVE_BATCH>1) — the draft/verify batching breaks when slots are at different positions. But with --kv-slots 1 there's no raggedness, so MTP could run exactly as it does in coli chat. The engine already resolves g_draft from COLI_CUDA_MTP/the MTP head; the serve path just force-sets g_draft=0 under the MUX. Gating that on SERVE_BATCH==1 (or --kv-slots 1) to keep MTP is a small, clean change. This overlaps #358 — let's track it there and here. Worth a PR; the correctness gate is: same tokens as coli chat on the same prompt.