contrib: exclude rdma-core libs from wheel packaging#1929
Open
GuangguanWang wants to merge 1 commit into
Open
Conversation
|
👋 Hi GuangguanWang! Thank you for contributing to ai-dynamo/nixl. Your PR reviewers will review your contribution then trigger the CI to test your changes. 🚀 |
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Enterprise Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThe wheel build script expands its ChangesWheel build configuration
Estimated code review effort: 1 (Trivial) | ~2 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Add libibverbs and libmlx5 to AUDITWHEEL_EXCLUDES so auditwheel does not bundle host RDMA userspace libraries into NIXL wheels. This avoids loading a wheel-private libibverbs alongside the system libibverbs used by other RDMA consumers such as NCCL, and lets UCX resolve rdma-core providers from the host environment. Signed-off-by: Guangguan Wang <guangguan.wang@linux.alibaba.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add libibverbs and libmlx5 to AUDITWHEEL_EXCLUDES so auditwheel does not bundle host RDMA userspace libraries into NIXL wheels. This avoids loading a wheel-private libibverbs alongside the system libibverbs used by other RDMA consumers such as NCCL, and lets UCX resolve rdma-core providers from the host environment.
What?
The NIXL wheel currently bundles two rdma-core shared libraries —
libibverbsandlibmlx5— because they are not listed inAUDITWHEEL_EXCLUDESincontrib/build-wheel.sh. Afterauditwheel repair, these appear as renamed files in the wheel, forexample:
(Verified from
nixl-cu13==1.3.1on PyPI.)These libraries should be excluded from the wheel and resolved from the host system at runtime.
Why?
1. rdma-core must match the host kernel's RDMA subsystem
rdma-core libraries (
libibverbs, provider plugins likelibmlx5, etc.) communicate directly with kernel RDMA drivers via/dev/infiniband/device nodes and kernel ABI. Bundling a fixed version in the wheel can produce silent failures or undefined behavior when deployed on hosts running a different kernel RDMA stack.Furthermore, different deployment environments require different RDMA stacks — some use upstream rdma-core, while others require MLNX_OFED >= 5.0 for features like GPUDirect RDMA or SHARP. Bundling a specific rdma-core version in the wheel forces a single choice and prevents users from selecting the RDMA stack that matches their hardware and deployment requirements.
2. Dual-libibverbs conflict when NCCL and UCX coexist
NCCL is not bundled with the wheel, so when a process uses NCCL's IB net plugin, it resolves
libibverbs.sofrom the system at runtime. Meanwhile, the UCX transport modulelibuct_ib.soinside the wheel has been patched byauditwheelto load the bundledlibibverbs-<hash>.so.If a process uses both NCCL and NIXL simultaneously:
Two independent instances of
libibverbsin the same process maintain separate device lists and internal state, which can cause unpredictable RDMA behavior.3. Bundling
libmlx5locks out non-Mellanox RDMA providersThe wheel currently bundles
libmlx5.so(a Mellanox/NVIDIA-specific userspace driver) as a build-time artifact. On hosts with non-Mellanox RDMA hardware — such as Alibaba eRDMA, Intel/Cornelis OPX, Broadcom, or other ibverbs-capable devices. These hosts provide their own vendor-specific provider libraries (e.g.,liberdma.so,libhfi1.so) through the system's rdma-core installation.By excluding rdma-core from the wheel, NIXL would use whatever ibverbs providers the host system offers, enabling RDMA support across a wider range of hardware.
Summary by CodeRabbit