[correctness] PCIe DCP A2A: signed int slot_ overflows so staging_[-1] aliases signals_/self_signal_ — eager channel writes staged attention over a peer's barrier region and null-derefs

[malaiwah]

Found during a source review of the v20 r12 release image voipmonitor/vllm:gilded-gnosis-v20-vllmb46c3aa-si35aebc6-fi801d57a-cu132-20260730-r12. The cited sparkinfer file comes from the composed r12 tree, which is byte-identical to public commit de04125508c75e748016493f851e6bd4513b515b (git diff de04125 r12-composed -- sparkinfer/comm/pcie/pcie_dcp_a2a.cu sparkinfer/comm/pcie/pcie_dcp_a2a.py is empty, tree clean), so permalinks use that SHA; code is quoted inline regardless. vLLM caller refs are from the composed r12 vLLM tree (base b46c3aa). Deployment: GLM-5.2 753B MoE, EXL3 3.0bpw, 4x RTX 6000 Pro (sm120), TP4 with DCP4 (release gate) / DCP2 (production turnkey), nvfp4_ds_mla KV, FULL_AND_PIECEWISE CUDA graphs.

This was first noted as §5 NEW inside a review comment on #95 and never filed on its own. This report validates it independently, confirms the core mechanism (verified by compiling the exact pattern, not by reading), and corrects the consequence: in both shipped world sizes the out-of-bounds slot also produces a guaranteed null dereference, so the failure is cross-rank barrier corruption plus a device-side illegal memory access — not purely silent corruption. Purely silent corruption is the DCP8 case.

Where

sparkinfer/comm/pcie/pcie_dcp_a2a.cu#L336-L346 (member layout), #L399 (run, i.e. lse_reduce_scatter), #L460 (all_gather_heads).

class PCIeDCPA2A {
public:
  int rank_;                        // :338
  int world_size_;                  // :339
  RankSignals signals_{};           // :340   struct { Signal *signals[8]; }
  Signal *self_signal_;             // :341
  RankStaging staging_[2]{};        // :342   struct { void *ptrs[8]; }
  int64_t output_capacity_elems_;   // :343
  int64_t lse_offset_;              // :344
  int64_t lse_capacity_;            // :345
  int slot_ = 0;                    // :346   <-- signed, unbounded

    // Staging happens inside the kernel (warp-per-row, before the start
    // barrier); no host staging memcpys are issued.
    const int slot = slot_++ % 2;   // :399 (run) and :460 (all_gather_heads)

staging_[slot] is then passed by value into the kernel, which uses it both as the write target and as the peer read source:

  auto *staging_out = reinterpret_cast<Pack<T> *>(staging.ptrs[rank]);        // :172
  auto *staging_lse = reinterpret_cast<float *>(
      reinterpret_cast<char *>(staging.ptrs[rank]) + lse_offset);             // :173-174
  ...
    rot_packs[i] = src == rank ? local_packs
                 : reinterpret_cast<const Pack<T> *>(staging.ptrs[src]);      // :203-205

slot_ is written and read nowhere else — confirmed by grep over the whole repo: only :346, :399, :460. There is no Python-side accessor in pcie_dcp_a2a.py, and no code depends on the counter being monotonic.

Mechanism

1. Both A2A ops resolve to the same pool key and therefore the same channel object (established in [correctness] DCP A2A capture channel adoption: graphs sharing one channel bake host-side staging-slot parity; stale capture stream-key mappings can hand a graph's channel to later captures #95: total_heads, head_dim=512, query_head_dim=576, max_batch_size all match between mla_attention.py:1429 and :1545), so one slot_ serves 2 ops per MLA layer.
2. slot_++ is a signed increment with no wrap guard. After 2147483647 calls it overflows: formally UB, in practice wraps to INT_MIN.
3. The expression takes % of the pre-increment value, and C++ % truncates toward zero, so the produced slot sequence across the wrap is:

call	slot_ before	slot_++ % 2
2^31-1	2147483646	0
2^31	2147483647	1
2^31+1	-2147483648	0 (INT_MIN is even)
2^31+2	-2147483647	-1
2^31+3	-2147483646	0
2^31+4	-2147483645	-1

So the first out-of-bounds index appears on call 2147483650 and on every second call for the remaining life of the process. Slot 1 is never used again; the double buffer degenerates to {0, -1}.
4. staging_[-1] is 64 bytes below staging_[0]. Verified independently with offsetof on the exact declarations (no CUDA needed): sizeof(Signal)=133120, sizeof(RankSignals)=sizeof(RankStaging)=64, sizeof(PCIeDCPA2A)=240, offsets rank_=0, world_size_=4, signals_=8, self_signal_=72, staging_=80, output_capacity_elems_=208, lse_offset_=216, lse_capacity_=224, slot_=232. staging_[-1] therefore covers bytes 16..79, and its ptrs[] alias:

staging_[-1].ptrs[k]	byte offset	aliases
ptrs[0]	16	signals_.signals[1]
ptrs[1]	24	signals_.signals[2]
...	...	...
ptrs[6]	64	signals_.signals[7]
ptrs[7]	72	self_signal_

The report's claim (c) holds exactly: staging_[-1] is the barrier-pointer region, shifted by one slot.

5. Correction the original note missed. RankSignals signals_{} is value-initialized and the constructor fills only signals[0 .. world_size-1] (:355-359), so signals_.signals[world_size .. 7] are nullptr. Because the alias shifts by one, the used index range [0, world_size) of staging_[-1] contains exactly one null for any world_size < 8:

• DCP2 (turnkey): ptrs[0] = signals[1], ptrs[1] = nullptr.
  • rank 0 stages into ptrs[0] = rank 1's Signal region (valid IPC pointer, real cross-rank write); then post-barrier reads ptrs[1] = nullptr → illegal address.
  • rank 1 stages into ptrs[1] = nullptr → illegal address in the pre-barrier write phase, so its barrier flags never post; it would have read ptrs[0] = its own Signal region as "peer 0's staged output".
• DCP4 (gate): ptrs[0..2] = signals[1..3], ptrs[3] = nullptr. Ranks 0/1/2 stage over the Signal region of ranks 1/2/3 respectively; rank 3 null-writes; every rank 0..2 also null-reads ptrs[3].
• DCP8 (not shipped): all eight entries are non-null (ptrs[7] = self_signal_). No fault at all — every rank writes its staged attention output over a peer's barrier flags and reads a peer's barrier flags back as attention output. Fully silent cross-rank corruption.

6. The cross-rank writes do not fault. signal_ptrs are the IPC slab base pointers and staging follows the signal region in the same allocation (pcie_dcp_a2a.py:279-285, _staging_layout :95-105: slab_bytes = align_up(signal_bytes) + 2*slot_bytes). A staging write is capacity-checked to <= slot_bytes from its base (.cu:371-373), so writing slot_bytes starting at the peer's slab base stays inside the peer's own allocation — it silently overwrites Signal.self_counter (2 KiB) + Signal.peer_counter (128 KiB) and runs on into staging0. The peer spinning in start_barrier (.cu:103-118) then reads corrupted flags: while (load_flag(mine) != value) either never matches (hang) or matches spuriously (barrier released before peers staged → reduction over unstaged data).

7. Empirically verified, not inferred. Compiling the verbatim pattern in the same class shape with the host compiler at -O3 (g++ 13.3, no -fwrapv), with the object laundered through an int64_t exactly as fptr_t does:

slot_= 2147483645 -> staging_[slot].ptrs[0] = 0x2000   (slot 1)
slot_= 2147483646 -> staging_[slot].ptrs[0] = 0x1000   (slot 0)
slot_= 2147483647 -> staging_[slot].ptrs[0] = 0x2000   (slot 1)
slot_=-2147483648 -> staging_[slot].ptrs[0] = 0x1000   (slot 0)
slot_=-2147483647 -> staging_[slot].ptrs[0] = 0xaa01   <-- signals_[1] sentinel
slot_=-2147483646 -> staging_[slot].ptrs[0] = 0x1000   (slot 0)

The emitted body is the sign-correcting signed remainder (shrl $31 / addl / andl $1 / subl), i.e. the compiler does not narrow it to & 1, and the base offset is leaq 10(%rcx,%rax,8) = byte 80, matching offsetof(staging_). The disproof attempt "UB lets the optimizer assume slot_ >= 0 and emit & 1, accidentally fixing it" therefore fails on the shipped toolchain — and it cannot succeed in general, because the object pointer arrives through reinterpret_cast<PCIeDCPA2A *>(fptr_t) (.cu:519, :600), which destroys any value-range provenance for slot_.

Reachability

Per shipped config, eager only. run()/all_gather_heads() contain no cudaStreamIsCapturing check (unlike pcie_oneshot.cu), so the increment executes on the host at capture time and the selected staging_[slot] pointer is baked into the captured kernel node's arguments. Graph replays never touch slot_. Consequently:

• FULL graphs (uniform decode, capture sizes 4..64) bake 156 ops once and then replay for free — they cannot reach the overflow.
• PIECEWISE passes bake zero A2A ops: attention is in splitting_ops by default (vllm/config/compilation.py:1147, splitting_ops = list(self._attention_ops)), so MLA attention — and with it both A2A calls — runs eagerly on the capture/replay stream. This eager traffic lands on the long-lived default-stream channel created at dcp_alltoall.py:134 (single_channel=False, pool.for_stream()), which is never recreated for the life of the worker.
• Both ops are gated on dcp_use_b12x = dcp_b12x and num_mqa_tokens <= VLLM_DCP_A2A_MAX_TOKENS (mla_attention.py:1391-1395), with DCP_A2A_MAX_TOKENS=64 effective in both shipped paths (serve-glm52-v16.sh:21, VLLM_USE_B12X_DCP_A2A=1 at :240). Chunked-prefill-only batches (num_mqa_tokens up to 3072) take ag_rs and never touch this channel.

Net: the reachable driver is mixed prefill+decode steps under the turnkey config (8 seqs, MTP k=5 → 6..48 MQA tokens, well under the 64-token cap, dispatched PIECEWISE → eager attention). Every such step advances slot_ by 156.

• Turnkey (DCP2, 8 seqs, chunked prefill): reachable. Requires only continuous uptime, no unusual input.
• Release gate (DCP4, max_num_seqs=1, graph=6): effectively unreachable. With one sequence there are no mixed prefill+decode steps, so essentially all A2A-bearing decode passes are FULL-graph replays; eager A2A traffic is limited to warmup. This is a correction to the original note, which did not separate the two configs.
• A process restart resets the counter, so only long-lived servers are exposed.

The corrupting code is inside the eager dcp_lse_reduce_kernel / all_gather_heads_kernel launches themselves (not inside a captured graph), so nothing about graph replay masks or delays it.

Severity

correctness — silent cross-rank memory corruption, followed by an unrecoverable crash or hang. Ranks 0..W-2 write staged attention data straight into a peer's barrier-flag arrays across the IPC mapping; rank W-1 dereferences a null device pointer. On the shipped DCP2/DCP4 the crash is guaranteed and roughly concurrent with the corruption (illegal address → sticky CUDA error → engine death; or a permanent spin in start_barrier on smashed flags). At DCP8 there is no fault, so it degrades to purely silent wrong attention output on every second op forever. It is a slow-fuse bug — nothing is observable until it isn't — but the fuse is uptime, not a rare input, and the corruption crosses the rank boundary, so it can also manifest as an unexplained peer hang rather than a local error.

Impact (recomputed independently)

• Ops per model pass: 2 per DCP-attention MLA layer (all_gather_heads at mla_attention.py:1429 + lse_reduce_scatter at :1545), both gated on the same flag → 2 x 78 = 156 for GLM-5.2 (78 decoder layers, of which 75 MoE), consistent with the count established in [correctness] DCP A2A capture channel adoption: graphs sharing one channel bake host-side staging-slot parity; stale capture stream-key mappings can hand a graph's channel to later captures #95.
• Calls to the first negative index: 2147483650 (= 2^31 + 2). Eager passes: 2147483650 / 156 = 13,765,921.
• Uptime required (eager, PIECEWISE-dispatched, ≤64-MQA-token passes only):

eager passes/s	time to first staging_[-1]
40	4.0 days
20	8.0 days
15	10.6 days
8	19.9 days
4	39.8 days

The original note quoted "~1.6 days at 100 eager passes/s"; 100 eager forward passes/s is not plausible for a 753B MoE on 4x sm120, and only the mixed fraction of steps counts. A realistic turnkey band is ~4–40 days of continuous serving, i.e. weeks, not hours — still comfortably inside the lifetime of a production deployment that is not restarted.

• Bytes corrupted per bad op: one contiguous run of batch * total_heads * head_dim * 2 bytes from the peer's slab base, plus an LSE write at +lse_offset. sizeof(Signal) = 133,120 B (130 KiB), so with total_heads = 64 a 48-token decode step writes ~3 MiB — about 24x the entire peer barrier region. Even a 16-head, 8-token step writes 128 KiB, i.e. the whole region. There is no partial-damage regime.

Suggested fix

Mirror the sibling channel in this same directory, which already gets it right — pcie_dcp_topk.cu:161-198 uses uint32_t next_slot_ = 0; with next_slot_ ^= uint32_t{1};:

  unsigned slot_ = 0;                     // :346
  ...
  const int slot = static_cast<int>(slot_);
  slot_ ^= 1u;                            // :399 and :460

This preserves today's slot sequence exactly (0,1,0,1,...), so the first-call slot and the alternation the double buffer relies on are unchanged. slot_ = (slot_ + 1) & 1 also removes the overflow but flips the phase to 1,0,1,0,...; harmless, only worth noting so it is not mistaken for a no-op. Plain unsigned slot_ with the existing slot_++ % 2 is also correct (unsigned wrap is defined, % stays in {0,1}), but keeps a pointless 32-bit counter.

Fix safety: host-side arithmetic only, so it is capture-safe (nothing new executes under capture, no host sync, no allocation), numerics-neutral (same bytes, same buffer), and ABI-neutral — the whole extension is a single JIT-compiled TU (torch.utils.cpp_extension.load, pcie_dcp_a2a.py:109-119), and slot_ is not exposed to Python. Nothing reads slot_ expecting monotonicity (only :346/:399/:460 touch it), so no caller changes. This is orthogonal to and compatible with the device-side slot derivation proposed as F1 in #95, which would remove the host counter altogether.

Same overflow pattern, out of scope here, worth a separate look: pcie_twoshot.cu:264 with const int s = slot_ % 2; slot_++; at :315/:332, and pcie_oneshot.cu:569 with int slot = dbuf_slot_ % 2; dbuf_slot_++; at :689/:767. Both take % of a signed counter and are subject to the identical wrap; the aliasing target of [-1] in those classes was not analysed in this report.

Verification / repro hint

No GPU and no 2^31-op soak needed — the interesting behavior is entirely host-side pointer selection:

1. Offset check (seconds): compile the Signal/RankSignals/RankStaging/PCIeDCPA2A declarations verbatim and print offsetof for every member plus reinterpret_cast<char*>(&o.staging_[0]) - 64 - base. Expect staging_ = 80 and staging_[-1] = 16 == offsetof(signals_) + 8.
2. Alias check (seconds): in the same program, fill signals_[k] and self_signal_ with sentinels, new the object, launder the pointer through int64_t, set slot_ = 2147483645, and call a noinline method containing the verbatim const int slot = slot_++ % 2; return staging_[slot].ptrs[rank_];. Build with -O3 and without -fwrapv. Expect the signals_[1] sentinel on the 5th call. (Done above; reproduced on g++ 13.3.)
3. End-to-end (optional, GPU): add a temporary test-only setter for slot_, run the existing 2-rank tests/comm/test_pcie_dcp_a2a_gpu.py::test_pcie_dcp_a2a_eager_and_cuda_graph_correctness eager loop with slot_ pre-seeded to 2147483645, and observe the illegal-memory-access on rank 1 within a handful of iterations plus mismatching output on rank 0. With the fix the loop is clean.
4. Nothing to measure for the fix itself — it is behaviour-preserving for all slot_ < 2^31.

Provenance of the observation

First noted as §5 — new: int slot_ overflow indexes staging_[-1] in #95 (comment) (a second-reviewer comment on #95). It was never filed as its own issue; #95's body covers a different slot_ concern (host-baked slot parity under graph replay) and does not mention the overflow. Parent issue for context: #95. Related memory-side report: #94.

[malaiwah]

Implemented as PR #101: #101

The patch mirrors the proven pcie_dcp_topk.cu lifetime pattern: one uint32_t two-state toggle shared by both DCP A2A operations. It preserves the exact 0,1,0,1,... sequence and capture-time host selection while making negative/OOB staging indices unreachable.

Verification:

• -O3 host reproduction: original 1 0 1 0 -1 0; replacement remained exact for 10,000,000 calls
• exact modified CUDA extension compiled and loaded in a CPU-isolated r12 container
• targeted DCP A2A suite: 10 passed
• independent adversarial reviewer: APPROVE, 0.99 confidence, no finding
• CodeRabbit: pass, no actionable comments

The active production container was not rebuilt or restarted and remains healthy. Similar counters in one-shot/two-shot are intentionally separate follow-ups because #97 establishes the concrete aliasing and production reachability only for DCP A2A.

[malaiwah]

Completion update: PR #101 now covers every surviving signed two-slot counter finding, not only DCP A2A: #101

• Shared closed two-state selector is used by DCP A2A, one-shot (plain/fused), and two-shot (RS/AG); no overflow-capable arithmetic remains.
• Host-only -O3 regression executed 10,000,003 transitions: 1 passed.
• DCP A2A, one-shot, and two-shot CUDA extensions all compiled and loaded against the pulled r12 image dependencies.
• Changed-file Ruff passes; independent adversarial review: APPROVE (0.99).

PR #103 is stacked on #101 and supersedes DCP's host selector with execution-owned device parity so graph replay advances staging slots; #101 remains the complete one-shot/two-shot fix and an independently mergeable DCP overflow fix. No live-GPU result is claimed because all four GPUs remain occupied by the unchanged production service.

[malaiwah]

Final implementation update:

• PR #101 head b518259 fixes signed selector overflow and replay-frozen reusable staging ownership.
• DCP remains independently mergeable with the bounded host selector in fix(pcie): prevent two-slot selector overflow #101; stacked PR fix(pcie): retire capture channel aliases #103 migrates DCP to device parity and removes that helper afterward.
• One-shot plain/fused and two-shot RS/AG select one launch-global slab inside each existing kernel at execution time. No host take() and no added selector GPU launch remain.

Verification: 33 passed, 2 opt-in GPU skips; one-shot/two-shot CUDA extensions compiled+loaded from a fresh cache; Ruff passes. Live multi-GPU execution is assigned to the separate field-test agent.

[malaiwah]

Rental revalidation correction — FAIL pending safe replacement

The earlier PASS at PR #101 head 0959fe807d366628380f41933244e3ccce0a8ae0 proved the corrected numerical replay oracle, but a later strict synchronization/lifecycle audit found P1 defects in the implementation around this overflow fix. In particular, DCP still capture-bakes one staging slab, partial IPC setup/GC teardown can free or unmap live peer pointers, subgroup order is sorted incorrectly, and fixed resident-grid barriers are not proved safe on smaller/MIG devices.

The signed-overflow regression itself passes, but this implementation cannot be accepted as a safe resolution. I corrected the PR validation comment with exact heads, GPU evidence, AB/BA cost, blockers, and immutable logs: #101 (comment)

A collective lifecycle + operation-wide DCP control-node replacement is in fresh-cache DCP2/DCP4 testing now. This issue remains open pending that candidate and independent review.

[malaiwah]

Replacement qualification update: the frozen composed successor
31fa6a48116471ce423f0338047453ec1032c202
passes the live four-GPU gates relevant to this issue: bounded-selector CPU
coverage, DCP4 one-shot eager/graph/multistream torture with 1,025 graph
replays, opposite rank-local graph order with 1,025 replays, fused RMSNorm, and
two-shot correctness/capture/slot/teardown. The two-shot diagnostic was slower
than NCCL BF16 on this host, so no performance gain is claimed.

This does not retroactively pass public #101 head 0959fe8; it supersedes
the unsafe repair heads and is published separately as draft successor #105.
Exact commands, results, host/toolchain, raw-log hashes, and the supersession
chain are recorded in the #101 evidence comment
and the exact-head #105 validation.

I recommend keeping this issue open until #105 is reviewed and merged.

[malaiwah]

Superseding field result: the frozen semantic-channel successor is not yet
deployable.

The focused four-rank eager/capture/torture and atomic vLLM/SparkInfer suites
all passed, but the first full GLM-5.2 startup gate found a contract the focused
tests missed. During determine_available_memory()'s uncaptured warm-up, a
compiled TP all-reduce requested vllm:eager:allreduce on a physical CUDA
stream that was already owned by the active vllm:target:profile semantic
scope. Its descriptor warm-ups run before CUDA itself reports the stream as
capturing, so SparkInfer did not apply its active-scope override, selected the
static eager channel, and correctly refused to alias the two channels:

RuntimeError: CUDA stream key ... is already bound to another logical PCIe oneshot channel

The engine failed before KV allocation/health, so there is no performance or
deployability claim for SparkInfer 31fa6a4 + vLLM be1e289. The SHA-only
appliance image containing them will not be promoted.

The complete call trace makes the repair direction deliberately narrow:

1. while a semantic capture scope is active, route a pre-capture warm-up on
   that scope's owner stream to its top logical channel;
2. do not extend that routing to unrelated side streams, and retain hard
   failures for genuine out-of-scope stream/channel mismatches;
3. apply the rule symmetrically to one-shot and DCP pools; and
4. make failure-path teardown idempotent when Torch has already removed the
   process group from the world-group map.

Re-qualification will include the same full-model boot/traffic/shutdown gate,
not just focused tests.

Complete immutable server log and analysis:

• full rejected log
• campaign section L

[malaiwah]

Resolution candidate is now complete: SparkInfer #105 at bc62980 plus exact field vLLM f99e1e7b8636ca3811ab6d23084ac6da63420dc3 passed the full GLM-5.2 graph/traffic/shutdown gate; clean upstream caller PR local-inference-lab/vllm#216 is patch-identical. The successor retains bounded execution-owned two-slot selection for DCP, one-shot, and two-shot paths together with collective setup and retryable ownership. No PCIe speedup is claimed; this is correctness/lifecycle work. Exact evidence: https://github.com/malaiwah/glm52-exl3-vast/blob/5c76a2536e7fc9a5f1cb6bf182531889f5385e65/docs/field-review-results/2026-07-30-vast-46335896/UPSTREAM-REPAIR-CAMPAIGN.md Reproduction: https://github.com/malaiwah/glm52-exl3-vast/blob/5c76a2536e7fc9a5f1cb6bf182531889f5385e65/docs/field-review-results/2026-07-30-vast-46335896/COUNTER-VALIDATION.md. Keeping this issue open until #105 merges.

[malaiwah]

PR #105 review follow-up is complete at caa643c8546121214493d052221f16c62f69f719: all ten CodeRabbit threads are resolved, with nine valid fixes and one documented cuda-bindings false positive. Portable validation is 128 targeted passes plus 243 passes / 18 expected skips across the locally collectable communication suite.

The exact GPU evidence boundary remains bc62980543b3ca59a9bee971df1b19ce6181964c paired with vLLM f99e1e7b8636ca3811ab6d23084ac6da63420dc3; the newer review layer is not being mislabeled as GPU-run. Full provenance and reproduction instructions: https://github.com/malaiwah/glm52-exl3-vast/blob/d821001f97dd15f0ae6ace626cc1f334a2069553/docs/field-review-results/2026-07-30-vast-46335896/COUNTER-VALIDATION.md

This issue should remain open until the reviewed successor merges through PR #105.