[memory] KV-capacity accounting: runner-init buffers escape the budget; capture residue double-counted; per-graph extrapolation over-reserves; GLM page-size branch asymmetry

[malaiwah]

Provenance: found during a memory review of the v20 r12 release image voipmonitor/vllm:gilded-gnosis-v20-vllmb46c3aa-si35aebc6-fi801d57a-cu132-20260730-r12. Code refs are against the composed r12 vllm tree (base f978d009 + PR heads per the release locks). vllm/utils/mem_utils.py, vllm/v1/worker/gpu_model_runner.py, and vllm/v1/kv_cache_interface.py are byte-identical to base f978d009 (permalinks below use that sha); vllm/v1/worker/gpu_worker.py line refs are against r12-composed (the file was last touched by the PR-198 head 703af3ffd, "profile repeatable serving peak after warmup"), so its code is quoted inline only.

Deployment used for impact numbers: GLM-5.2 753B MoE EXL3 3.0bpw, 4x RTX 6000 Pro Blackwell 96GB, TP4; production turnkey = DCP2, nvfp4_ds_mla KV (368 B/tok/layer, 78 layers -> 14,352 B/token/GPU at DCP2), V1 runner, max_num_batched_tokens=3072, max_num_seqs=8, MTP k=5, GMU 0.957 (~4.1 GiB total margin/GPU). These are proposals from a static source review, not demands — happy to be corrected on any of them.

1. Persistent runner-__init__ buffers escape every KV-budget term (OOM-side, ~36 MiB/GPU on the turnkey)

Where:

• vllm/v1/worker/gpu_worker.py:407 — init_snapshot is taken in init_device() before the model runner is constructed at lines 428-442:

# take current memory snapshot
self.init_snapshot = init_snapshot = MemorySnapshot(device=self.device)
self.requested_memory = request_memory(init_snapshot, self.cache_config)
...
# Construct the model runner
if self.use_v2_model_runner:
    ...
else:
    self.model_runner = GPUModelRunnerV1(self.vllm_config, self.device)

• GPUModelRunnerV1.__init__ then allocates persistent GPU tensors: the placeholder InputBatch/block tables (gpu_model_runner.py#L705-L741) and the CUDA-graph persistent buffers (gpu_model_runner.py#L773-L820), dominated by:

self.inputs_embeds = self._make_buffer(
    self.max_num_tokens, self.inputs_embeds_size, dtype=self.dtype, numpy=False
)

• memory_profiling entry (mem_utils.py#L284-L294) resets peak stats and takes before_profile, so before_profile.torch_peak == bytes currently allocated == weights + all runner-init buffers.

What/Why: the final budget is (gpu_worker.py:566-573, 598-602):

profile_result.torch_peak_increase = (
    profile_torch_peak - profile_result.before_profile.torch_peak
)
profile_result.non_kv_cache_memory = (
    profile_result.non_torch_increase
    + profile_result.torch_peak_increase
    + profile_result.weights_memory
)
...
self.available_kv_cache_memory_bytes = (
    self.requested_memory
    - profile_result.non_kv_cache_memory
    - cudagraph_memory_estimate_applied
)

The runner-init buffers are (a) not in weights_memory (that is only model_memory_usage from load_model), (b) cancelled out of torch_peak_increase because they sit in both profile_torch_peak and before_profile.torch_peak, and (c) not in non_torch_increase because they are torch-allocator memory (inside memory_reserved, hence excluded from non_torch_memory, mem_utils.py#L156-L163). The PR-198 mid-profile reset_peak_memory_stats (second profile_run pass) does not change this: the reset re-bases the peak to current allocated, which still includes the init buffers, so they cancel out of the increase exactly as before; only first-pass transients are excluded (as intended). Net: KV capacity is over-budgeted by exactly the size of the runner-init buffers, eroding the thin 0.957-GMU margin from the OOM side.

Impact (recomputed): turnkey V1 runner (max_num_tokens=3072, max_num_reqs=8, hidden=6144, bf16, max_model_len=524288, block 64):

• inputs_embeds 3072x6144x2 B = 36.0 MiB
• placeholder block table 8 x cdiv(524288,64)=8192 x int32 = 0.25 MiB (DCP-unaware until may_reinitialize_input_batch)
• positions/input_ids/req_indices/query_pos/slot_mapping/is_token_ids + req-sized buffers ≈ 0.11 MiB
• Total ≈ 36.4 MiB/GPU ≈ 2,650 KV tokens/GPU at DCP2/nvfp4.

The V2 runner (release-gate config) has the same snapshot-ordering exposure but no inputs_embeds buffer; its InputBuffers (vllm/v1/worker/gpu/input_batch.py:24-34) at 8192 tokens is only ~105 KiB, so the material impact is on the V1 turnkey path.

Suggested fix: either take init_snapshot after runner construction, or add (before_profile.torch_peak - weights_memory) (equivalently before_profile allocated minus weights) as an explicit term of non_kv_cache_memory.

Verification: log profile_result.before_profile.torch_peak - model_memory_usage right after memory_profiling entry on the turnkey config; it reports ~36 MiB that appears in no subtraction term. Alternatively compare available_kv_cache_memory_bytes before/after moving the snapshot below runner construction.

2. Non-torch CUDA-graph capture residue is double-counted (over-reserve)

Where: gpu_worker.py:541-573 — profile_cudagraph_memory() runs inside the memory_profiling context:

with memory_profiling(
    self.init_snapshot,
    weights_memory=int(self.model_runner.model_memory_usage),
) as profile_result:
    self._profile_model_with_kernel_warmup()

    profile_torch_peak = torch.accelerator.memory_stats(self.device).get(
        "allocated_bytes.all.peak", 0
    )
    ...
    cudagraph_memory_estimate = self.model_runner.profile_cudagraph_memory()

# Use the pre-cudagraph torch peak to avoid double-counting.
profile_result.torch_peak_increase = (
    profile_torch_peak - profile_result.before_profile.torch_peak
)

What/Why: the pre-capture peak read protects only the torch side. non_torch_increase is computed at context exit (mem_utils.py#L298-L306), i.e. after the throwaway capture probe. The probe's per-graph costs are measured as get_memory_info free-memory deltas (gpu_model_runner.py#L6754-L6770), which include non-torch (driver/NCCL) allocations made during capture. The cleanup path (gpu_model_runner.py#L6802-L6819) destroys the graphs and the profiling KV cache, and memory_profiling exit runs gc + empty_cache, which releases the torch pool — but non-torch residue that survives graph destruction (NCCL per-communicator graph-capture channels first allocated on the first captured collective; cuBLAS/cuDNN capture-mode workspaces; driver context growth) stays resident and lands in non_torch_increase and was already inside cudagraph_memory_estimate. Both are subtracted from the budget at gpu_worker.py:598-602 when VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1 (the default, envs.py:2242-2244), so the residue is charged twice → KV under-allocation.

Impact: mechanism is unambiguous from the code; the magnitude (residue size after graph destruction, TP4/NCCL) is not statically determinable — plausibly tens of MiB on this deployment, and it is pure over-reserve (safe side, but wasted capacity while the turnkey margin is only ~4.1 GiB total).

Suggested fix: snapshot non_torch_memory immediately before the capture probe and subtract the post-cleanup non-torch delta from cudagraph_memory_estimate (or from non_torch_increase).

Verification: log MemorySnapshot(...).non_torch_memory immediately before line 563 and after the with block; any positive delta is currently counted in both subtraction terms.

3. Per-graph cudagraph extrapolation charges the 2nd-largest graph's cost to all remaining graphs (over-reserve)

Where: gpu_model_runner.py#L6750-L6779:

for mode, descs in capture_descs:
    profile_descs = descs[:2]
    ...
    first_capture = mem_samples[0]
    # Use at least 1 MiB per graph for driver overhead
    per_graph = max(
        mem_samples[1] if len(mem_samples) > 1 else 0, 1 << 20
    )
    shared_memory_estimate[mode] = first_capture
    per_graph_estimate[mode] = per_graph * (len(descs) - 1)

descs are sorted largest-first (cudagraph_dispatcher.py:340-348), so the two largest graphs per mode are sampled and every remaining (smaller) graph is charged the 2nd-largest graph's marginal cost, floored at 1 MiB.

What/Why: graph memory cost is roughly monotone in num_tokens (input/output pool blocks scale with batch), so charging the 2nd-largest cost to all smaller graphs is a systematic over-estimate whenever the true marginal cost of small captures is below the 2nd-largest sample (pool-block reuse makes small captures often cost near the 1 MiB driver floor).

Impact (recomputed, turnkey): with max_num_seqs=8, MTP k=5 (uniform_decode_query_len=6), default max_cudagraph_capture_size = min(8*6*2, 512) = 96 → capture sizes [1,2,4,8,16,...,96] = 15. FULL_AND_PIECEWISE gives 15 PIECEWISE descs + 6 FULL descs (uniform-decode sizes in [6,48]: 8,16,24,32,40,48) = 21 graphs, of which 19 are extrapolated. Every 1 MiB by which the 2nd-largest sample exceeds the true small-graph cost over-reserves 19 MiB (≈1,400 DCP2/nvfp4 KV tokens/GPU); a 4 MiB gap ≈ 76 MiB.

Suggested fix: also sample the smallest desc per mode (3 probes instead of 2) and interpolate per-graph cost by num_tokens between the smallest and 2nd-largest samples.

Verification: enable the existing logger.debug at line 6781 and compare per_graph against the actual free-memory delta of the last (smallest) capture during the real capture_model() pass.

4. Page-size branch asymmetry for MLA records: glm_fp8_rope uses block_size, deepseek_v4 uses storage_block_size (latent)

Where: kv_cache_interface.py#L477-L500:

@property
def real_page_size_bytes(self) -> int:
    if self.cache_dtype_str == "nvfp4_ds_mla":
        ...
        if self.model_version == "deepseek_v4":
            return self.storage_block_size * 432
        if self.model_version == "glm_fp8_rope":
            ...
            return self.block_size * 368
        return self.block_size * 432

with storage_block_size = block_size // compress_ratio (line 472-474) and compress_ratio: int = 1 as the dataclass default (line 465).

What/Why: today this is byte-identical for GLM: the GLM spec is built at mla_attention.py:1963-1972 (model_version="glm_fp8_rope" when KV_FP8_ROPE=1 + nvfp4_ds_mla + glm_moe_dsa) without passing compress_ratio, so it stays 1 and storage_block_size == block_size; compress_ratio > 1 is only ever set on the DeepseekV4 layer path (models/deepseek_v4/attention.py:826-842). The asymmetry is therefore harmless in the shipped r12 image — but it is a landmine: if a compressor is ever enabled for GLM (e.g. cr=4), the glm_fp8_rope branch would silently size pages at 64*368 = 23,552 B instead of (64/4)*368 = 5,888 B, over-allocating the entire KV cache 4x. (The fp8_ds_mla V3.2 branch at line 500, block_size * 656, has the same shape.)

Impact: none today (verified compress_ratio=1 on the GLM path); 4x KV over-allocation if the compressor is ever turned on for GLM without touching this file.

Suggested fix: use storage_block_size uniformly in all real_page_size_bytes branches; with compress_ratio=1 this is a no-op for current deployments.

Verification: assert spec.storage_block_size == spec.block_size for all GLM specs today (passes); unit-test real_page_size_bytes for a hypothetical glm_fp8_rope spec with compress_ratio=4 — currently returns 23,552 instead of 5,888.

[malaiwah]

Independent second-pass re-review of all four claims against the same trees (composed r12 vLLM = base f978d009 for mem_utils.py / gpu_model_runner.py / kv_cache_interface.py, r12-composed for gpu_worker.py; sparkinfer 35aebc6), done by a reviewer who did not author the original claims. Two corrections are material and one suggested fix does not work; retractions are stated plainly.

Claim	Verdict	Note
1. Runner-__init__ buffers escape the budget (OOM-side)	HOLDS (magnitude confirmed 36.34 MiB)	Direction re-derived independently; may_reinitialize_input_batch timing checked. But 36.0 of the 36.34 MiB is the same inputs_embeds that #200/GG-11 asks to delete — do not count the win twice.
1b. Suggested fix "take init_snapshot after runner construction"	RETRACT	Provably a no-op: the cancellation is via before_profile.torch_peak, not init_snapshot. Only the explicit-term fix works.
2. Non-torch capture residue double-counted	HOLDS-WITH-CORRECTION (mechanism) / UNPROVABLE-STATICALLY (magnitude)	The residue sources we named (A2A / PCIe-AR channels) are minted before the measurement window, so they are single-counted. What is left is small. Separate real finding below: the V1 probe leaks an A2A channel that V2 already rolls back.
3. Per-graph extrapolation over-reserves	HOLDS-WITH-CORRECTION	Graph count was derived from vLLM defaults. Real turnkey: 18 graphs, 16 extrapolated (not 21/19), capture sizes [6,12,…,60] (not [1,2,4,8,…,96]). Over-reserve is bounded by ~16 MiB; the "4 MiB gap ⇒ 76 MiB" scenario is unreachable here. Not applicable to the release gate at all (V2 runner has no extrapolation path).
3b. Suggested fix (3rd probe + interpolate)	HOLDS-WITH-CORRECTION	The samples are order-dependent, not per-graph costs; interpolating can flip the bias to under-reserve (OOM side). Prefer porting V2's whole-set gross measurement.
4. glm_fp8_rope uses block_size, deepseek_v4 uses storage_block_size	HOLDS (latent, zero impact today)	compress_ratio=1 on the GLM path re-verified.
4b. Suggested fix "use storage_block_size uniformly"	HOLDS-WITH-CORRECTION	It converts a fail-safe 4× over-allocation into a 4× under-allocation (OOB) unless the whole GLM write path also honours compress_ratio. A hard reject is the safer hardening and equally a no-op today.

---

Claim 1 — cross-issue double-count warning (the important part)

Recomputed the turnkey V1 buffer set from self.max_num_tokens = scheduler_config.max_num_batched_tokens = 3072, self.max_num_reqs = max_num_seqs = 8 (gpu_model_runner.py:516-517):

Buffer	Bytes
inputs_embeds 3072×6144 bf16	36.00 MiB
placeholder block table 8 × cdiv(524288,64)=8192 × int32	0.25 MiB
input_ids/positions/req_indices/is_token_ids/slot_mapping + req-sized	0.09 MiB
total	36.34 MiB ≈ 2,659 KV tokens/GPU at DCP2/nvfp4 (14,352 B/tok/GPU)

inputs_embeds is 99.1% of it, and it is the same buffer #200 (GG-11) asks to gate off for a text-only draft. If #200 lands, this issue's magnitude collapses to 0.34 MiB ≈ 24 KV tokens/GPU. Also note the two fixes move reported KV capacity in opposite directions: fixing #201 alone reduces advertised KV by 36 MiB (correctly — that memory is physically occupied today), fixing #200 alone frees the 36 MiB. Landing both is capacity-neutral versus today and strictly safer. There is no +72 MiB anywhere.

(Aside for #200: CpuGpuBuffer (v1/utils.py:113-126) allocates a pinned host mirror of the same shape, so deleting inputs_embeds also frees 36 MiB of mlocked host RAM per rank.)

Direction re-verified (it really is OOM-side, not freed before allocation). may_reinitialize_input_batch is called at gpu_model_runner.py:7812, inside initialize_kv_cache, i.e. after determine_available_memory() returns — so the placeholder batch is live for the whole profile and the DCP-aware re-init cannot shrink anything before the budget is fixed. inputs_embeds is never re-initialized at all.

Arithmetic, with A = torch bytes allocated at profile entry = W (weights) + B (base buffers), P = peak during profile:

• before_profile.torch_peak = A (reset at mem_utils.py:286, measured at :294)
• torch_peak_increase = P − A ; + weights_memory = P − A + W = P − B
• B is not in non_torch_increase (torch-allocator memory is excluded at mem_utils.py:156-163)

⇒ the budget reserves P − B where P is required. Confirmed, and the PR-198 mid-profile reset_peak_memory_stats (gpu_worker.py:493) does not change it (it re-bases to current allocated, which still contains B; it does correctly capture persistent warmup allocations, since profile_torch_peak ≥ post-warmup allocated).

Retraction of the first suggested fix. "Take init_snapshot after runner construction" cannot fix this. init_snapshot feeds only (a) requested_memory = ceil(total_memory × gpu_memory_utilization) (v1/worker/utils.py:425-427 — free_memory is used only for a sanity check, never in the amount) and (b) the non_torch_increase baseline. The base buffers are torch-allocator memory, so they are invisible to both. Moving the snapshot later is a no-op on the 36 MiB and is marginally worse, because the small non-torch driver allocations made by the runner constructor (streams, events) would then drop out of non_torch_increase too.

Claim 2 — correction to the named residue, plus a separate real leak

Mechanism holds as stated: non_torch_increase is computed at context exit (mem_utils.py:301-306), after the probe, while cudagraph_memory_estimate includes non-torch bytes in its get_memory_info deltas. Any non-torch allocation made inside the sampled captures that survives cleanup is in both terms.

But the residue sources the issue names are wrong. Both SparkInfer's A2A channel and the cpp-PCIe-AR channel are created by graph_capture() on context entry (parallel_state.py:1666-1686, custom_all_reduce.py:671-672), which in profile_cudagraph_memory happens at gpu_model_runner.py:6743-6745 — before the first mem_before = torch.accelerator.get_memory_info()[0] at :6755. They therefore land in non_torch_increase only, single-counted (and correctly, since they persist). What actually falls inside the measurement window is driver graph metadata (released by cudaGraphExecDestroy via clear_all_graphs()) and lazily-loaded CUDA modules (never released → genuinely double-counted, but small). So: mechanism HOLDS, "plausibly tens of MiB" is not supported, and the honest verdict on magnitude is UNPROVABLE-STATICALLY — needs the MemorySnapshot(...).non_torch_memory probe the issue already proposes, taken immediately before gpu_worker.py:563 and after the with block.

Separate finding found while checking this, worth more than the double count. PCIeDCPA2APool.capture() unconditionally mints a new channel (pcie_dcp_a2a.py:766-789; single_channel=False is set at dcp_alltoall.py:132) and retains it in _all_channels/_channels; the finally only pops the stack. The V2 runner guards against exactly this — checkpoint_b12x_graph_channels() / rollback_b12x_graph_channels() at v1/worker/gpu/model_runner.py:953-957 and :986-989, with the comment "so profiling cannot leave stale channels behind". The V1 profile_cudagraph_memory (gpu_model_runner.py:6741-6819) has no such pair, so the throwaway probe permanently strands one raw-cudaMalloc IPC slab (pcie_oneshot.py:468). Size from _staging_layout (pcie_dcp_a2a.py:61-106), GLM geometry 96 heads × max(512,576):

• with VLLM_DCP_A2A_MAX_TOKENS=64 (what the gate launcher sets, serve-glm52-v16.sh:21): 2 × align(64·96·576·2 + 64·96·4) ≈ 13.5 MiB/rank ≈ 990 KV tokens, charged to non_kv_cache_memory for a channel that is never used again;
• if the turnkey leaves VLLM_DCP_A2A_MAX_TOKENS at the source default 0 (envs.py:1166), max_batch_size falls back to max_num_batched_tokens (mla_attention.py:832-835) = 3072 → the stranded channel is ~650 MiB/rank. Worth checking the turnkey value regardless of this issue.

Porting the two V2 calls into V1's finally block is a two-line change with existing coverage (tests/distributed/test_dcp_a2a.py:653-656).

Claim 3 — graph count recomputed from the real configs

The filed numbers came from vLLM defaults (max_cudagraph_capture_size = min(8·6·2, 512) = 96, sizes [1,2,4,8,16,…,96]). The turnkey sets the list explicitly (CUDAGRAPH_CAPTURE_SIZES → [4,8,…,64], 16 entries; MAX_CUDAGRAPH_CAPTURE_SIZE=64), and V1 then rewrites it: CompilationConfig.adjust_cudagraph_sizes_for_spec_decode (config/compilation.py:1473-1481, 1510-1593) rounds every size up to a multiple of uniform_decode_query_len = 1 + num_spec_tokens = 6 and drops anything above the max:

[4,8,12,…,64]  --round_up(·,6)-->  {6,12,18,24,30,36,42,48,54,60}   (64 → 66 > 64, dropped)
∪ small_decode_sizes {6·n ≤ 64}  =  [6,12,18,24,30,36,42,48,54,60]   → max_cudagraph_capture_size = 60

With FULL_AND_PIECEWISE: 10 PIECEWISE descs; FULL descs are the sizes in [udql, udql·max_num_seqs] = [6,48] (cudagraph_dispatcher.py:211-231) = [6,12,18,24,30,36,42,48] → 8 descs (num_reqs 1…8). 18 graphs, 16 extrapolated, not 21/19. (Sanity check that this is the shipped path: with pass_config.enable_sp on, sp_multiple_of = max(6,4) = 6 and 6 % 4 != 0 would raise at compilation.py:1533 — so SP is off at TP4/MTP5, consistent with the turnkey booting.)

Magnitude is smaller than filed, not larger. descs are sorted largest-first, so mem_samples[1] is the incremental new physical pages a 54-token capture needs on top of a 60-token capture in the same pool — almost certainly ~0, which means per_graph sits on the 1 << 20 floor and per_graph_estimate = 1 MiB × (9 + 7) = 16 MiB ≈ 1,170 KV tokens, which is also the hard upper bound on the over-reserve. The issue's "a 4 MiB gap ≈ 76 MiB" requires mem_samples[1] ≥ 4 MiB at a 54-token capture; at these sizes the per-graph pool delta is dominated by size-independent graph metadata, so that is not reachable. Also drop the "pool-block reuse makes small captures near the 1 MiB floor" argument as stated — every shipped capture is ≤60 tokens; there is no tiny-vs-huge spread.

Release gate: this claim does not apply. The gate runs the V2 runner (VLLM_USE_V2_MODEL_RUNNER=1), whose profile_cudagraph_memory (v1/worker/gpu/model_runner.py:918-1020) captures the entire descriptor set into current_platform.get_global_graph_pool() and measures one gross free-memory delta — no 2-sample extrapolation exists there. For completeness, had the gate run V1: at GRAPH=6/seq=1, MTP3 gives graph_step = 4 → sizes [4] → 1 PIECEWISE + 1 FULL desc → 0 extrapolated graphs; MTP0 gives sizes [1..6] and udql=1, so FULL is limited to [1, 1·max_num_seqs] = [1] → 7 graphs, 5 extrapolated, ≤5 MiB. Either way immaterial.

Claim 4 — the suggested fix has the wrong failure direction

compress_ratio=1 on the GLM path re-confirmed: mla_attention.py:1963-1972 builds MLAAttentionSpec without compress_ratio, and the only non-1 assignments in the tree are DeepseekV4 (models/deepseek_v4/attention.py:839, :985, compressor.py:153 — which asserts compress_ratio in [4, 128]). So storage_block_size == block_size and the branch is byte-identical today. Latent claim HOLDS.

But "use storage_block_size uniformly" is not the safe hardening. Under the hypothetical it protects against (a compressor enabled for GLM with cr=4), it would shrink real_page_size_bytes to 16 × 368 while the GLM write path has no compressed slot mapping — get_compressed_slot_mapping is only wired for the DeepseekV4 layers and for DCP localisation — so the writer would still address 64 records per page into a 4×-too-small page: out-of-bounds stores instead of a 4× over-allocation. Today's asymmetry fails safe; the proposed fix makes it fail unsafe. Prefer raising in the glm_fp8_rope branch (and in the V3.2 block_size * 656 branch at :500) when compress_ratio != 1. That is equally a no-op on every shipped config and equally catches the landmine.

---

Fix-safety notes

Claim 1. Adding the base-buffer term cannot double-subtract against weights — it is exactly cancelling:
non_torch_increase + (before_profile.torch_peak − W) + (P − before_profile.torch_peak) + W = non_torch_increase + P.
So the whole fix reduces to non_kv_cache_memory = profile_result.non_torch_increase + profile_torch_peak, which never mentions weights_memory and is therefore immune to any mis-measurement of it. Worth preferring that form: weights_memory comes from DeviceMemoryProfiler.consumed_memory (mem_utils.py:86-105), and although it is a resident delta rather than a load-time peak (each current_memory_usage() resets the peak before reading it, so the ~0.9 GiB transient draft embed/lm_head allocation does not inflate it), an explicit before_peak − W term would go negative and increase the KV budget if that ever changed. No capture-safety, IPC or numerics exposure: it is arithmetic on already-collected counters, evaluated after the profiling context exits. What to measure: log before_profile.torch_peak − model_memory_usage at gpu_worker.py:569; expect ≈36.3 MiB on turnkey V1 and ≈0.1 MiB on the V2 gate (V2's InputBuffers, v1/worker/gpu/input_batch.py:24-34, is 8192·13 B + 100 B ≈ 104 KiB — the filed ~105 KiB is right). Then confirm Available KV cache memory drops by that amount and nothing else moves.

Claim 2. Subtracting the measured non-torch delta from cudagraph_memory_estimate is not safe in general: part of that delta (lazily-loaded CUDA modules) is permanent and shared with the real capture, but part of what the real capture_model() will allocate (a fresh A2A channel per graph_capture() entry) is not in the estimate at all — subtract blindly and you move to the under-reserve/OOM side. The clean fix is to free the probe's residue rather than to net it out: port checkpoint_b12x_graph_channels()/rollback_b12x_graph_channels() into V1's finally. That is capture-safe (V2 runs it after set_cudagraph_capturing_enabled(False) and after graph destruction, so no allocation or free happens during capture) and IPC-safe provided it stays collective-symmetric — rollback_channels funnels through _coordinated_close_channels(..., exchange_group=...) (pcie_dcp_a2a.py:661-665), and every rank enters graph_capture() the same number of times, so the closes match. It changes no numerics and no dtype contract. Ordering requirement, already honoured by V2 and by the docstring at pcie_dcp_a2a.py:637-641: graphs must be destroyed and the device synchronized before rollback, i.e. rollback must sit after _cleanup_profiling_kv_cache/clear_all_graphs() and inside a finally that cannot be skipped. What to measure: MemorySnapshot(...).non_torch_memory immediately before gpu_worker.py:563 and after the with block, plus the pcie_dcp_a2a channel count before/after the probe.

Claim 3. A third probe is capture-safe (all probes are outside the real capture, in a throwaway pool) but the interpolation is not sound: mem_samples[i] is a free-memory delta given the pool state left by samples 0..i−1, not an isolated per-graph cost, so a smallest-desc sample taken last will read ≈0 and interpolation would extrapolate downward — under-reserve, OOM side, on a config whose entire margin is ~4.1 GiB. Safer: adopt V2's approach (capture the full set, measure the gross delta once, no extrapolation). That has a hard prerequisite — do it in current_platform.get_global_graph_pool(), not a disposable pool, exactly as V2 documents at v1/worker/gpu/model_runner.py:928-931: destroying graphs leaves physical pages retained by their pool, and a disposable pool strands them where production cannot reuse them. Startup cost is one extra full capture pass. What to measure: enable the logger.debug at gpu_model_runner.py:6781 and compare per_graph against the actual free-memory delta of the last (smallest) capture during the real capture_model(); if per_graph is pinned at 1.00 MiB, the ceiling on this claim is 16 MiB and it should be dropped.

Claim 4. Both variants are compile-time-only page arithmetic — no capture, IPC or numerics exposure. The raise-on-compress_ratio != 1 variant is strictly safe (dead branch today, verified). The storage_block_size variant silently changes the KV page stride and therefore the num_blocks↔writer contract, so it must not land without also auditing every nvfp4_ds_mla GLM writer and slot-mapping consumer.

Priority triage

• Trivial + safe, land now: claim 1 in the reduced form non_kv_cache_memory = non_torch_increase + profile_torch_peak (but land it with [memory] MTP spec-decode buffers: fp32 draft-logits arena, dead inputs_embeds, and redundant full-vocab fp32 copies on the GLM-5.2 verify path #200/GG-11, otherwise it is a pure −36 MiB to advertised KV capacity); claim 4 as a raise, not a rewrite; the two-line A2A checkpoint/rollback port into V1's profile_cudagraph_memory (drop the init_snapshot relocation entirely).
• Needs measurement before any change: claim 2's residue magnitude (probe the non-torch delta); claim 3 (log per_graph; if it sits at the 1 MiB floor the whole claim is ≤16 MiB and the code change is not worth the risk of an order-dependent interpolation).
• Drop: the init_snapshot-relocation fix (no-op); "tens of MiB" for claim 2 and the 76 MiB scenario for claim 3; the 21-graph/19-extrapolated count; any claim 3 impact at the release gate (V2 has no extrapolation path).