Skip to content

pip-installing vLLM/torch silently downgrades NCCL below the 2.30.4 floor; import dies at undefined symbol and check_nccl_so() cannot catch it #710

Description

@dmvevents

Problem

import deep_ep fails with a raw dynamic-linker error after installing vLLM (or anything that pins
torch==2.11.0) into an environment that satisfied the documented NCCL floor:

ImportError: /opt/DeepEP/deep_ep/_C.cpython-312-x86_64-linux-gnu.so: undefined symbol: ncclGetLsaDevicePointer

The chain (each step verified on H200/EFA, deep_ep 2.0.0, torch 2.11.0+cu130, vllm 0.26.0):

  1. The environment starts at nvidia-nccl-cu13==2.30.4 (the README floor: "NCCL 2.30.4 and above").
  2. pip install vllm==0.26.0 pins torch==2.11.0, and torch 2.11.0 pins nvidia-nccl-cu13==2.28.9
    (exact ==) — pip silently downgrades NCCL as a side effect.
  3. NCCL 2.28.x lacks the GIN/LSA device-API symbols _C.so was built against, so the next
    import deep_ep dies at the linker with no version hint.
  4. pip install --no-deps --force-reinstall nvidia-nccl-cu13==2.30.4 fully heals it — deep_ep + vllm
    0.26.0 + torch 2.11.0 then import and run together cleanly.

Why check_nccl_so() cannot catch this case

The guard in deep_ep/__init__.py compares the loaded libnccl.so (from /proc/self/maps) against
the lib at find_nccl_root() — i.e. the NCCL currently installed. A wheel downgrade moves both sides
together
: torch loads 2.28.9 and find_nccl_root() now also resolves to 2.28.9. Loaded == reference, the
guard passes, and the failure surfaces one line later at import deep_ep._C as an opaque linker error.
The invariant that actually broke is "loaded == what _C.so was built against", which the build does
not record anywhere the guard can read.

Suggested fixes (either or both)

  1. Declare the floor in wheel metadatainstall_requires=["nvidia-nccl-cu13>=2.30.4"] (per CUDA
    variant). pip would then refuse the torch-driven downgrade loudly instead of performing it silently.
    Today setup.py locates NCCL at build time via find_pkgs.find_nccl_root() but ships no constraint.
  2. Record the built-against NCCL version into the wheel (e.g. write it into the generated envs.py
    next to the existing persistent envs) and extend check_nccl_so() to compare the loaded lib against
    that, so this failure mode produces the guard's existing clear message instead of
    undefined symbol: ....

Environment

  • deep_ep 2.0.0 (wheel built against nvidia-nccl-cu13 2.30.4)
  • torch 2.11.0+cu130, vllm 0.26.0, python 3.12
  • 2× p5en.48xlarge (H200), EFA — though the failure is host-side and hardware-independent

Happy to provide the full install transcript.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions