[None][fix] Guard V2 mamba state-index tables against ADP dummy-request overflow#2
Open
lingjiew wants to merge 1 commit into
Conversation
…st overflow _pad_attention_dp_dummy_request -> add_dummy_requests extends the previous iteration's self.requests, so _setup_state_indices can see len(self.requests) > max_batch and index past the max_batch-sized cuda/_host_state_indices tables (IndexError at index == max_batch). Reproduced on Qwen3.5-397B-A17B hybrid disagg CTX workers with attention-DP (max_batch 8 and 16, concurrency >= 8). Grow the tables dynamically with a warning instead of crashing. The guard fired ~7x per run across a 37-point ctx-only sweep and a 52-point gen-only sweep on GB300 with no further incident; the underlying sizing fix belongs in the add_dummy_requests bookkeeping. Signed-off-by: Lingjie Wu <lingjiew@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug
PyExecutor._pad_attention_dp_dummy_request->add_dummy_requestsextends the previous iteration'sself.requests, soV2MambaHybridCacheManager._setup_state_indicescan seelen(self.requests) > max_batchand index past themax_batch-sizedcuda_state_indices/_host_state_indicestables:Repro
Qwen3.5-397B-A17B-NVFP4 disagg CTX worker with attention-DP on GB300, this branch @ c78b793:
max_batch8 or 16, concurrency >= 8, AgentX multi-turn trace replay — crashes within minutes (three independent jobs, all dying at the same line with index == max_batch).Fix
Defensive guard: grow the two state-index tables with a warning instead of crashing. The underlying sizing fix belongs in the
add_dummy_requestsbookkeeping (dummy slots should stay within the table bounds) — happy to rework it there if you prefer.Validation
Image built from this branch + this patch: the guard fired ~7x per run with no further incident across a 37-point ctx-only sweep and a 52-point gen-only sweep on GB300 — full combo (V2MambaHybridCacheManager + python transceiver + KV cache reuse at realized 93-94% + conversation-affinity ADP router + MTP forced-AL) all green.
🤖 Generated with Claude Code