Skip to content

bugfix: prevent incompatible npu graph replays. - #2065

Open
shifengmin wants to merge 1 commit into
xLLM-AI:mainfrom
shifengmin:codex/bugfix-npu-dp-graph-replay-signature
Open

bugfix: prevent incompatible npu graph replays.#2065
shifengmin wants to merge 1 commit into
xLLM-AI:mainfrom
shifengmin:codex/bugfix-npu-dp-graph-replay-signature

Conversation

@shifengmin

Copy link
Copy Markdown
Collaborator

Description

ACL graph keys are based on the rounded or exact maximum DP token bucket. Two decode steps can therefore select the same graph while carrying different per-rank token distributions.

The persistent DP/EP padding buffers are refreshed before replay, but the graph keeps the tensor descriptors and empty-rank branches recorded during capture. Replaying with another distribution can expose stale tails, truncate new padding metadata, or reuse the wrong q_cu_seq_lens binding.

This change:

  • records the complete dp_global_token_nums vector for each captured ACL graph;
  • skips double-buffer input preparation when the next request is incompatible;
  • falls back to eager execution before replay when the distribution changed; and
  • adds coverage for rounded buckets, decode_no_padding, and empty-to-nonempty rank transitions.

The distribution vector is global and identical on all DP ranks, so the fallback decision remains collective-safe.

Dependency

Depends on #2024.

#2024 preserves raw DP token counts for empty shards so fake execution rows are excluded from LM-head/output compaction. This PR fixes graph persistence consistency but intentionally does not duplicate that accounting change. Without #2024, graph and eager execution follow the same metadata after fallback, but both can still count an empty shard's fake row.

Please merge #2024 first, or rebase this PR after #2024 lands.

Related Issues

Change Type

  • Bug fix
  • New feature
  • Performance improvement
  • Refactor
  • Documentation
  • Test
  • Build or CI

Pull Request Checklist

PR Title and Commit Messages

  • The PR title and each commit message follow the xLLM commit format: <type>: <subject>.

Pre-commit Checks

  • I have installed pre-commit by running pip install pre-commit or an equivalent command.
  • I have installed the hooks with pre-commit install.
  • I have run pre-commit run --all-files and fixed any reported issues.

Changed-file validation passed:

pre-commit run --files tests/core/runtime/acl_graph_executor_test.cpp \
  xllm/core/runtime/acl_graph_executor_impl.cpp \
  xllm/core/runtime/acl_graph_executor_impl.h
clang-format.............................................................Passed

Self Review

  • I have self-reviewed the code according to .agents/skills/code-review/references/custom-code-style.md, especially code written or assisted by AI.
  • I have rebased this PR onto the latest main branch.

Build and Test Coverage

  • Tests have been added or updated as needed.
  • CUDA: python setup.py build test has passed on a CUDA machine.
  • NPU: a full python setup.py build --device npu completed successfully on an Ascend development machine.
  • MLU: python setup.py build test has passed on an MLU machine.

NPU test results:

  • targeted graph replay tests: 3/3 passed;
  • complete acl_graph_executor_test: 24/24 passed.

Reviewer Notes

This intentionally falls back to eager execution instead of caching one graph per DP distribution. The latter can create an unbounded graph cache for normal scheduling variance.

Please review together with #2024 for the complete empty-shard data flow.

- Track the per-rank DP token distribution captured by each ACL graph.\n- Fall back to eager execution before preparing or replaying incompatible persistent padding metadata.\n- Cover rounded buckets, decode-no-padding buckets, and empty-rank transitions.\n\nDepends on xLLM-AI#2024 for correct empty-shard raw token accounting.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant