Skip to content

bugfix: pin NPU python runtime to the process own device. - #2095

Open
yingxudeng wants to merge 1 commit into
xLLM-AI:mainfrom
yingxudeng:bugfix/npu-python-runtime-die0-context
Open

bugfix: pin NPU python runtime to the process own device.#2095
yingxudeng wants to merge 1 commit into
xLLM-AI:mainfrom
yingxudeng:bugfix/npu-python-runtime-die0-context

Conversation

@yingxudeng

Copy link
Copy Markdown
Collaborator

init_npu_python_runtime() ran _npu_setDevice on the first visible logical device for every NPU process. In multi-process single-card serving each process sees all TP cards and selects its own via node_rank, so every node_rank != 0 process created an extra small context on die0 before its worker ran on the real card, piling ~112MB per rank onto die0.

Select the target device with the same get_device_idx logic used by the Master constructor so each process initializes its context on its own card.

Description

Related Issues

Change Type

  • Bug fix
  • New feature
  • Performance improvement
  • Refactor
  • Documentation
  • Test
  • Build or CI

Pull Request Checklist

Thank you for contributing to xLLM. Before requesting review, please make sure the following items are complete.

PR Title and Commit Messages

  • The PR title and each commit message follow the xLLM commit format: <type>: <subject>.

Allowed types: feat, bugfix, docs, test, refactor, chore, style, revert, perf, model, build, release.
The subject should use clear English, start with a verb, include at least 4 words, and end with ..

Pre-commit Checks

  • I have installed pre-commit by running pip install pre-commit or an equivalent command.
  • I have installed the hooks with pre-commit install.
  • I have run pre-commit run --all-files and fixed any reported issues.

If you are unsure how to set up pre-commit, see the pre-commit documentation.

Self Review

  • I have self-reviewed the code according to .agents/skills/code-review/references/custom-code-style.md, especially code written or assisted by AI.
  • I have rebased this PR onto the latest main branch.

Build and Test Coverage

  • Tests have been added or updated as needed.
  • CUDA: python setup.py build test has passed on a CUDA machine.
  • NPU: python setup.py build test has passed on an NPU machine.
  • MLU: python setup.py build test has passed on an MLU machine.

Reviewer Notes

init_npu_python_runtime() ran _npu_setDevice on the first visible logical
device for every NPU process. In multi-process single-card serving each
process sees all TP cards and selects its own via node_rank, so every
node_rank != 0 process created an extra small context on die0 before its
worker ran on the real card, piling ~112MB per rank onto die0.

Select the target device with the same get_device_idx logic used by the
Master constructor so each process initializes its context on its own card.
@yingxudeng

Copy link
Copy Markdown
Collaborator Author
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants