Problem
import deep_ep fails with a raw dynamic-linker error after installing vLLM (or anything that pins
torch==2.11.0) into an environment that satisfied the documented NCCL floor:
ImportError: /opt/DeepEP/deep_ep/_C.cpython-312-x86_64-linux-gnu.so: undefined symbol: ncclGetLsaDevicePointer
The chain (each step verified on H200/EFA, deep_ep 2.0.0, torch 2.11.0+cu130, vllm 0.26.0):
- The environment starts at
nvidia-nccl-cu13==2.30.4 (the README floor: "NCCL 2.30.4 and above").
pip install vllm==0.26.0 pins torch==2.11.0, and torch 2.11.0 pins nvidia-nccl-cu13==2.28.9
(exact ==) — pip silently downgrades NCCL as a side effect.
- NCCL 2.28.x lacks the GIN/LSA device-API symbols
_C.so was built against, so the next
import deep_ep dies at the linker with no version hint.
pip install --no-deps --force-reinstall nvidia-nccl-cu13==2.30.4 fully heals it — deep_ep + vllm
0.26.0 + torch 2.11.0 then import and run together cleanly.
Why check_nccl_so() cannot catch this case
The guard in deep_ep/__init__.py compares the loaded libnccl.so (from /proc/self/maps) against
the lib at find_nccl_root() — i.e. the NCCL currently installed. A wheel downgrade moves both sides
together: torch loads 2.28.9 and find_nccl_root() now also resolves to 2.28.9. Loaded == reference, the
guard passes, and the failure surfaces one line later at import deep_ep._C as an opaque linker error.
The invariant that actually broke is "loaded == what _C.so was built against", which the build does
not record anywhere the guard can read.
Suggested fixes (either or both)
- Declare the floor in wheel metadata —
install_requires=["nvidia-nccl-cu13>=2.30.4"] (per CUDA
variant). pip would then refuse the torch-driven downgrade loudly instead of performing it silently.
Today setup.py locates NCCL at build time via find_pkgs.find_nccl_root() but ships no constraint.
- Record the built-against NCCL version into the wheel (e.g. write it into the generated
envs.py
next to the existing persistent envs) and extend check_nccl_so() to compare the loaded lib against
that, so this failure mode produces the guard's existing clear message instead of
undefined symbol: ....
Environment
- deep_ep 2.0.0 (wheel built against nvidia-nccl-cu13 2.30.4)
- torch 2.11.0+cu130, vllm 0.26.0, python 3.12
- 2× p5en.48xlarge (H200), EFA — though the failure is host-side and hardware-independent
Happy to provide the full install transcript.
Problem
import deep_epfails with a raw dynamic-linker error after installing vLLM (or anything that pinstorch==2.11.0) into an environment that satisfied the documented NCCL floor:The chain (each step verified on H200/EFA, deep_ep 2.0.0, torch 2.11.0+cu130, vllm 0.26.0):
nvidia-nccl-cu13==2.30.4(the README floor: "NCCL 2.30.4 and above").pip install vllm==0.26.0pinstorch==2.11.0, and torch 2.11.0 pinsnvidia-nccl-cu13==2.28.9(exact
==) — pip silently downgrades NCCL as a side effect._C.sowas built against, so the nextimport deep_epdies at the linker with no version hint.pip install --no-deps --force-reinstall nvidia-nccl-cu13==2.30.4fully heals it — deep_ep + vllm0.26.0 + torch 2.11.0 then import and run together cleanly.
Why
check_nccl_so()cannot catch this caseThe guard in
deep_ep/__init__.pycompares the loadedlibnccl.so(from/proc/self/maps) againstthe lib at
find_nccl_root()— i.e. the NCCL currently installed. A wheel downgrade moves both sidestogether: torch loads 2.28.9 and
find_nccl_root()now also resolves to 2.28.9. Loaded == reference, theguard passes, and the failure surfaces one line later at
import deep_ep._Cas an opaque linker error.The invariant that actually broke is "loaded == what
_C.sowas built against", which the build doesnot record anywhere the guard can read.
Suggested fixes (either or both)
install_requires=["nvidia-nccl-cu13>=2.30.4"](per CUDAvariant). pip would then refuse the torch-driven downgrade loudly instead of performing it silently.
Today
setup.pylocates NCCL at build time viafind_pkgs.find_nccl_root()but ships no constraint.envs.pynext to the existing persistent envs) and extend
check_nccl_so()to compare the loaded lib againstthat, so this failure mode produces the guard's existing clear message instead of
undefined symbol: ....Environment
Happy to provide the full install transcript.