From 232e9b7bff8b501ab6e8d9b913825665f3293831 Mon Sep 17 00:00:00 2001 From: Xuan Jiang Date: Mon, 24 Aug 2026 23:59:00 +0000 Subject: [PATCH] fix(deepep-v2): use the EFA installer bundled aws-ofi-nccl plugin Installer 1.50.0 bundles an EFA-GDA-capable aws-ofi-nccl (verified in PR 1234 review), so drop the source build and apply the ncclGinPlugin_v14 gate to the bundled binary instead. Fold the libfabric and aws-ofi-nccl version bullets into the EFA installer bullet, and state explicitly that the installer must run on the node itself (not the container) to provide the kernel driver. Addresses post-merge review comments on PR 1234. Co-authored-by: Vladimir Aerov --- .../deepep-v2-benchmark/README.md | 40 ++++++++------ .../deepep-v2-benchmark/deepep.Dockerfile | 53 ++++--------------- .../deepep-v2-benchmark/setup_deepep_gin.sh | 3 +- 3 files changed, 36 insertions(+), 60 deletions(-) diff --git a/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/README.md b/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/README.md index 43d287735..a06ca178b 100644 --- a/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/README.md +++ b/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/README.md @@ -1,22 +1,24 @@ # DeepEP V2 Benchmark (NCCL GIN / EFA-GDA) [DeepEP](https://github.com/deepseek-ai/DeepEP) is a communication library for -Mixture-of-Experts (MoE) **expert parallelism** — its dispatch and combine kernels perform the +Mixture-of-Experts (MoE) **expert parallelism** where its dispatch and combine kernels perform the GPU all-to-all that routes tokens to experts and gathers the results back. -This directory provides **[`setup_deepep_gin.sh`](./setup_deepep_gin.sh)**, which installs the -**DeepEP V2** Python package — the version whose internode transport is **NCCL GIN -(GPU-Initiated Networking)** — into your container or environment (for example a vLLM image +This directory provides: + +- **[`setup_deepep_gin.sh`](./setup_deepep_gin.sh)**, which installs the +**DeepEP V2** Python package, the version whose internode transport is **NCCL GIN +(GPU-Initiated Networking)**, into your container or environment (for example a vLLM image that already ships a DeepEP): it uninstalls any existing `deep_ep`, validates that the NCCL it is pointed at is GIN-capable (`include/nccl_device.h`, errors out otherwise), and builds + -installs DeepEP against your torch. No arguments needed for the default flow; see `--help`. +installs DeepEP against your torch. No arguments needed for the default flow, see `--help`. Everything else here is reference material for using the script and testing that the installation works: - **[`deepep.Dockerfile`](./deepep.Dockerfile)** — a reference image showing the full stack the - script needs (EFA userspace, a GIN-capable NCCL, aws-ofi-nccl with the EFA-GDA backend, - torch), built from a bare CUDA base; it can also be used directly for testing. + script needs (the EFA installer, which bundles the EFA-GDA-capable aws-ofi-nccl plugin, a + GIN-capable NCCL, torch), built from a bare CUDA base; it can also be used directly for testing. - **[`slurm/`](./slurm/)** — launchers that run DeepEP's own `tests/elastic/test_ep.py` to validate the installation: @@ -29,15 +31,15 @@ installation works: > - **NCCL >= 2.31** — the GIN device API (`nccl_device.h`) DeepEP V2 links against. > The Dockerfile builds NCCL from source and fails the build if the header is missing. -> - **EFA installer >= 1.50** — EFA-GDA requires **libfabric >= 2.5**; installers up to -> 1.49.0 ship libfabric 2.4 and will not work (the image build fails loudly on a -> too-old installer). -> - **aws-ofi-nccl built against that libfabric** — required for the **EFA-GDA** GIN backend -> (GIN type 5) this benchmark measures. DeepEP's kernels can also run over the CPU-proxy -> GIN backend, which has no such floor. +> - **EFA installer >= 1.50** — provides everything EFA-GDA needs in the container (the +> userspace runtime and the aws-ofi-nccl plugin with the EFA-GDA GIN backend, type 5, +> this benchmark measures). Installers up to 1.49.0 will not work; the image build fails +> loudly on a too-old installer. DeepEP's kernels can also run over the CPU-proxy GIN +> backend, which has no such floor. > - A host EFA kernel driver **>= 3.3.0**, the first release with the completion-counter API -> (`efadv_create_comp_cntr`). Check with `modinfo efa | grep ^version`; stock AMIs may ship -> older and need a driver upgrade. The container ships only the userspace stack. +> (`efadv_create_comp_cntr`). Run the EFA installer **on the node itself** (not in the +> container) to install/upgrade the kernel driver; the container ships only the userspace +> stack. Check with `modinfo efa | grep ^version`; stock AMIs may ship older. > - The **gdrcopy kernel module (`gdrdrv`) loaded on compute nodes** — the launchers > bind-mount `/dev/gdrdrv`, and the plugin's GIN initialization opens a gdr handle. > Check with `lsmod | grep gdrdrv`. @@ -51,7 +53,7 @@ in two kinds: GPU-initiated ones, where the NIC work queues are mapped into GPU kernels post RDMA themselves, and the CPU-proxy backend (available on EFA as well as IB/RoCE), where the GPU hands work to a proxy thread. On EFA, the **aws-ofi-nccl** plugin provides the GPU-initiated backend, **EFA-GDA** (GIN type 5). The launchers pin `NCCL_GIN_TYPE=5` to select -it; left unset, NCCL selects the CPU-proxy backend. +it, if left unset, NCCL selects the CPU-proxy backend when running on EFA. | Variable | Value | Purpose | |----------|-------|---------| @@ -79,7 +81,11 @@ it; left unset, NCCL selects the CPU-proxy backend. The script refuses to run if the NCCL at `--nccl-root` lacks the GIN device API, so it cannot silently produce a DeepEP that falls back to a slower path. Runtime still requires the -aws-ofi-nccl GIN plugin built with EFA-GDA support. +EFA-GDA-capable aws-ofi-nccl GIN plugin (bundled with EFA installer >= 1.50): run the EFA +installer **inside your container** too (with `--skip-kmod`, as the reference Dockerfile does) +to provide the userspace stack and the plugin, then point `NCCL_NET_PLUGIN` and +`NCCL_GIN_PLUGIN` at `/opt/amazon/ofi-nccl/lib/libnccl-net-ofi.so`. The host installer run +(previous section) provides only the kernel driver. ## Building the reference image diff --git a/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/deepep.Dockerfile b/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/deepep.Dockerfile index e18180533..a6a9e3d52 100644 --- a/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/deepep.Dockerfile +++ b/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/deepep.Dockerfile @@ -16,9 +16,9 @@ ARG TORCH_VERSION=2.11.0 ARG TORCH_CUDA_INDEX=cu130 ARG GDRCOPY_VERSION=v2.5.2 -# EFA installer: provides OpenMPI, the EFA userspace runtime, and libfabric. -# EFA-GDA requires libfabric >= 2.5, so installer >= 1.50 is required (1.49.0 and -# older ship libfabric 2.4; the build fails loudly on a too-old installer). +# EFA installer: provides the complete EFA userspace stack (OpenMPI, the EFA +# runtime, and the aws-ofi-nccl plugin). EFA-GDA requires installer >= 1.50; +# the build fails loudly on a too-old installer (plugin gate below). # EFA_INSTALLER_TARBALL overrides the download with a local tarball staged next # to this Dockerfile. ARG EFA_INSTALLER_VERSION=1.50.0 @@ -29,14 +29,6 @@ ARG EFA_INSTALLER_TARBALL= ARG NCCL_REPO=https://github.com/NVIDIA/nccl.git ARG NCCL_REF=v2.31.2-1 -# aws-ofi-nccl: built from source at a pinned tag so the plugin version is -# controlled by this file rather than by whatever the EFA installer bundles. -# EFA-GDA is auto-detected by configure from -# the libfabric >= 2.5 GDA ops + hardware counters + CUDA; the build gate below -# fails the image if the detection did not fire. -ARG AWS_OFI_NCCL_REPO=https://github.com/aws/aws-ofi-nccl.git -ARG AWS_OFI_NCCL_REF=v1.21.1 - # DeepEP V2. The repo is pinned to amazon-contributing/DeepEP inside # setup_deepep_gin.sh; only the ref is overridable. ARG DEEPEP_REF=main @@ -110,9 +102,9 @@ RUN --mount=type=bind,target=/ctx \ && cd aws-efa-installer \ && ./efa_installer.sh --disable-ngc -y --skip-kmod --skip-limit-conf --no-verify \ && ldconfig \ - && FI_VER=$(/opt/amazon/efa/bin/fi_info --version | grep -oP "libfabric: \K[0-9]+\.[0-9]+") \ - && { [ "$(printf "%s\n2.5\n" "$FI_VER" | sort -V | head -1)" = "2.5" ] \ - || { echo "ERROR: installer libfabric $FI_VER < 2.5; EFA-GDA requires >= 2.5 (use installer >= 1.50)" >&2; exit 1; }; } \ + && { nm -D /opt/amazon/ofi-nccl/lib/libnccl-net-ofi.so 2>/dev/null | grep -qw ncclGinPlugin_v14 \ + && echo "OK: bundled plugin is EFA-GDA-capable (ncclGinPlugin_v14 present)" \ + || { echo "ERROR: installer's plugin has no ncclGinPlugin_v14 export: EFA-GDA requires installer >= 1.50" >&2; exit 1; }; } \ && rm -rf /tmp/aws-efa-installer* /var/lib/apt/lists/* ENV LD_LIBRARY_PATH=/opt/amazon/openmpi/lib:$LD_LIBRARY_PATH ENV PATH=/opt/amazon/openmpi/bin:/opt/amazon/efa/bin:$PATH @@ -128,34 +120,11 @@ RUN git clone ${NCCL_REPO} /opt/nccl \ || (echo "ERROR: nccl_device.h missing -- this NCCL is not GIN-capable" >&2 && exit 1) ENV LD_LIBRARY_PATH="${NCCL_HOME}/lib:${LD_LIBRARY_PATH}" -## aws-ofi-nccl. EFA-GDA support is auto-detected by configure from the libfabric -## above; the ncclGinPlugin_v14 export check makes the detection authoritative -## (v11/v13 op-tables are exported by every build; the v14 table is EFA-GDA-only). -ENV OFI_HOME=/opt/aws-ofi-nccl -RUN set -e; \ - git clone --recursive ${AWS_OFI_NCCL_REPO} /tmp/aws-ofi-nccl \ - && cd /tmp/aws-ofi-nccl \ - && git checkout ${AWS_OFI_NCCL_REF} \ - && git submodule sync && git submodule update --init --recursive \ - && ./autogen.sh \ - && { ./configure \ - --prefix=${OFI_HOME} \ - --with-mpi=/opt/amazon/openmpi \ - --with-libfabric=${EFA_PREFIX} \ - --with-nccl=${NCCL_HOME} \ - --with-cuda=${CUDA_HOME} \ - --with-gdrcopy=${GDRCOPY_PREFIX} \ - --enable-platform-aws \ - || { tail -n 160 config.log 2>/dev/null; exit 1; }; } \ - && make -j"$(nproc)" \ - && make install \ - && if nm -D "${OFI_HOME}/lib/libnccl-net-ofi.so" 2>/dev/null | grep -qw ncclGinPlugin_v14; then \ - echo "OK: EFA-GDA (ncclGinPlugin_v14) present"; \ - else \ - echo "ERROR: no ncclGinPlugin_v14 export -- configure did not detect EFA-GDA (libfabric >= 2.5 with hw counters + CUDA required)" >&2; exit 1; \ - fi \ - && echo "${OFI_HOME}/lib" > /etc/ld.so.conf.d/aws-ofi-nccl.conf && ldconfig \ - && rm -rf /tmp/aws-ofi-nccl +## aws-ofi-nccl: the EFA installer (>= 1.50) bundles the plugin with EFA-GDA +## support; the installer layer above gates on the plugin's ncclGinPlugin_v14 +## export (v11/v13 op-tables are exported by every build; the v14 table is +## EFA-GDA-only), so a too-old installer fails that layer immediately. +ENV OFI_HOME=/opt/amazon/ofi-nccl ENV LD_LIBRARY_PATH="${OFI_HOME}/lib:${LD_LIBRARY_PATH}" ENV NCCL_NET_PLUGIN="${OFI_HOME}/lib/libnccl-net-ofi.so" ENV NCCL_GIN_PLUGIN="${OFI_HOME}/lib/libnccl-net-ofi.so" diff --git a/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/setup_deepep_gin.sh b/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/setup_deepep_gin.sh index cfa9eceb1..72b6395c4 100755 --- a/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/setup_deepep_gin.sh +++ b/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/setup_deepep_gin.sh @@ -250,7 +250,8 @@ print_completion_notes() { log "" log " export LD_LIBRARY_PATH=${NCCL_ROOT}/lib:\${LD_LIBRARY_PATH}" log " export FI_PROVIDER=efa" - log " export NCCL_NET_PLUGIN=/path/to/libnccl-net-ofi.so # aws-ofi-nccl (also set NCCL_GIN_PLUGIN)" + log " export NCCL_NET_PLUGIN=/opt/amazon/ofi-nccl/lib/libnccl-net-ofi.so # from the EFA installer (>= 1.50) run in this container" + log " export NCCL_GIN_PLUGIN=/opt/amazon/ofi-nccl/lib/libnccl-net-ofi.so" log "" log "EFA-GDA also requires the HOST EFA kernel driver to support the" log "comp-counter API (efadv_create_comp_cntr). The container ships only the"