[trap] Reload-to-reload score drift was autotune, not the checkpoint: two flags fix it at zero cost (extends trap 35)

[scottleimroth]

Scrub check

I have removed hostnames, IPs, absolute paths, usernames, tokens and keys from anything pasted below.

What broke

We nearly rejected a checkpoint for being irreproducible. It was the launch flags, and pinning them cost nothing.

This is offered as an extension to trap 35 (identical weights do not give identical scores), which documents the symptom well and stops short of a remedy. On this stack there is one, it is two flags, and it is free on both quality and speed. That seems worth having beside the symptom.

What you saw

Teacher-forced likelihood scoring of a 43-text corpus, same model, same weights, same serve command, across cold reloads:

stock flags     23 of 43 texts moved between two loads, max |delta NLL| 8.81
                perplexity 3.48822 (load 1) vs 3.471547 (load 2)

That is a checkpoint that looks unusable for any temperature-0 comparison, and it is exactly the size of effect that would manufacture or erase a published result. Our first reading was "this checkpoint is not reproducible, prefer the sibling".

What fixed it

Two launch flags:

--enable-deterministic-inference --disable-flashinfer-autotune

Re-measured under the same protocol, quiet box, conditions recorded per cell:

pinned flags     0 of 43 texts moved, max |delta NLL| 0.0000
                 perplexity 3.473925 on BOTH loads, bit-identical
                 decode 45.45 tok/s vs stock 45.34 tok/s (within noise)
                 perplexity sits inside the stock band, so no quality cost

Speed protocol for those decode figures: 3 prompt kinds by 3 repeats, 512 output tokens each, single stream, batch 1, context 131072, idle box.

Three things follow, and the second one is the one we would have got wrong:

1. Before attributing temperature-0 irreproducibility to a MODEL, test the same model with determinism pinning. The unpinned wobble comes from per-load kernel and autotune choices, not from the weights.
2. A pinned serve is its own fixed point. It is not yesterday's stock behaviour frozen, it is one arbitrary variant frozen. Turning pinning on is a one-time behaviour transition and should be planned like a model switch, not slipped in.
3. It did not transfer to a different model on a different engine in our estate: a large MoE on vLLM has a DIFFERENT signature (texts moving WITHIN a single load, not only across loads), so do not assume these flags tame that. Untested, stated as untested.

What were you serving

• Engine: SGLang, build 0.0.0.dev1+g5f55db35e, image lmsysorg/sglang:dev-cu13-qwen38-27b-dflash2
• Model: RadixArk/Qwen3.8-27B-NVFP4, NVFP4, context length 131072, --mem-fraction-static 0.80, --kv-cache-dtype fp8_e4m3, --attention-backend flashinfer
• Hardware: NVIDIA GB10 Grace Blackwell, 128 GB unified, Ubuntu 24.04, aarch64
• Protocol: pre-registered before the run, quiet box, one model loaded at a time, conditions recorded per cell

Optional diagnostic evidence

Per-text sum_nll for all 43 texts on all 4 loads (stock x2, pinned x2) plus the two speed files are retained as score-stock-load{1,2}.json, score-pinned-load{1,2}.json, speed-stock.json, speed-pinned.json. The moved-text counts above are re-derivable from those files by comparing sum_nll per tag between the two loads of each arm. Happy to share the raw rows.

I am happy for this data and the credit to be published. Credit as @scottleimroth.

[scottleimroth]

Correction to this report. The "free on both quality and speed" claim above is wrong, and I would like to withdraw it before anyone acts on it.

--enable-deterministic-inference is not free on this stack. It cost us a production outage six days after I filed this, on the same engine, the same model and the same attention backend measured above.

What the flag actually does in this build

Three things, none of them documented at the flag:

1. It hard-codes SGLANG_FLASHINFER_WORKSPACE_SIZE to 2 GiB via .set(), which overrides the environment variable of the same name. Raising that variable does nothing while the flag is on.
2. It forces fixed 4096-token KV splits for prefill, so the number of split partial-result buffers grows with prompt length instead of being bounded by SM count as the default planner does.
3. It disables the radix cache ("radix cache is not compatible with flashinfer attention backend for deterministic inference"), so every request is a full fresh prefill, including repeated agent system prompts.

The failure

FlashInfer sizes batch_prefill_tmp_v as num_qo_heads x padded_batch_size x cta_tile_q x head_dim x 4 bytes. For this model (24 query heads, 4 KV heads, head_dim 256, so cta_tile_q 64) a single 8,192-token prefill at kv_len 8,192 gives 1,536 work items, and 24 x 1536 x 64 x 256 x 4 = 2,415,919,104 bytes. That is the exact figure in our tracebacks.

The 2 GiB ceiling is crossed at any fresh prompt of roughly 7,275 tokens or more, and earlier when several mid-sized prompts land in one prefill batch. Above 8,192 tokens later chunks need more, so no fixed buffer is safe at the 131072 context this report was measured at. Patching the constant to 8 GiB and sending a 95k prompt showed it wanting 9.6 GiB.

The result is not a rejected request. The whole server process dies, taking every other in-flight request with it, and nothing in the crash names the flag.

There is also a large capacity cost that has nothing to do with crashes: at an identical memory fraction, removing the flag moved the KV pool from 93,574 to 324,186 tokens. Roughly three and a half times.

Why my measurement missed it

This is the part worth having in the registry. My speed protocol was three prompt kinds by three repeats, 512 output tokens, single stream, batch 1. Every prompt in it sat far below the threshold where the flag becomes fatal. The protocol could not have detected the cost it reported as absent. A short-prompt speed check is not evidence that a prefill-path flag is free, and I presented it as though it were.

What still stands

The finding this report was actually about is unaffected: the reload-to-reload score movement was per-load kernel and autotune choices, not the checkpoint, and pinning does collapse it to zero. That part reproduced cleanly and I stand behind it.

What does not stand is the recommendation. If you want reload-stable scoring on this stack, --disable-flashinfer-autotune carries none of the above. --enable-deterministic-inference should be treated as a measurement-only flag: fine in a benchmark harness with bounded prompt lengths, not something to put in a serving config. We now keep it in our determinism experiment script deliberately and have removed it from production permanently.

Apologies for the noise. I would rather correct this here than leave a "costs nothing" line sitting under my name in a registry that exists to stop exactly this.

[scottleimroth]

Amending my correction above, because it was itself incomplete on the point that matters most to anyone reading this issue.

I said the flag should be treated as measurement-only and left it there, which reads as though you must choose between reproducible scoring and serving long prompts. You do not. The triton and fa3 attention backends both support deterministic inference by other means (SGLANG_TRITON_DECODE_SPLIT_TILE_SIZE, set_batch_invariant), and neither touches the hard-coded workspace constant. If reload-stable scoring is what you came to this issue for, switching backend keeps it without the failure mode.

Also worth correcting the direction of the original claim: removing the flag was measured slightly faster, decode 44.05 to 45.72 tok/s on a matched comparison, with a considerably larger KV pool. So "free on quality and speed" was not merely an overstatement of a neutral change, it had the sign wrong.

Full detail is on #87.