fix(deepep-v2): use the EFA installer bundled aws-ofi-nccl plugin

micro-benchmarks/expert-parallelism/deepep-v2-benchmark/README.md

@@ -1,22 +1,24 @@
 # DeepEP V2 Benchmark (NCCL GIN / EFA-GDA)
 
 [DeepEP](https://github.com/deepseek-ai/DeepEP) is a communication library for
-Mixture-of-Experts (MoE) **expert parallelism** — its dispatch and combine kernels perform the
+Mixture-of-Experts (MoE) **expert parallelism** where its dispatch and combine kernels perform the
 GPU all-to-all that routes tokens to experts and gathers the results back.
 
-This directory provides **[`setup_deepep_gin.sh`](./setup_deepep_gin.sh)**, which installs the
-**DeepEP V2** Python package — the version whose internode transport is **NCCL GIN
-(GPU-Initiated Networking)** — into your container or environment (for example a vLLM image
+This directory provides:
+
+- **[`setup_deepep_gin.sh`](./setup_deepep_gin.sh)**, which installs the
+**DeepEP V2** Python package, the version whose internode transport is **NCCL GIN
+(GPU-Initiated Networking)**, into your container or environment (for example a vLLM image
 that already ships a DeepEP): it uninstalls any existing `deep_ep`, validates that the NCCL it
 is pointed at is GIN-capable (`include/nccl_device.h`, errors out otherwise), and builds +
-installs DeepEP against your torch. No arguments needed for the default flow; see `--help`.
+installs DeepEP against your torch. No arguments needed for the default flow, see `--help`.
 
 Everything else here is reference material for using the script and testing that the
 installation works:
 
 - **[`deepep.Dockerfile`](./deepep.Dockerfile)** — a reference image showing the full stack the
-  script needs (EFA userspace, a GIN-capable NCCL, aws-ofi-nccl with the EFA-GDA backend,
-  torch), built from a bare CUDA base; it can also be used directly for testing.
+  script needs (the EFA installer, which bundles the EFA-GDA-capable aws-ofi-nccl plugin, a
+  GIN-capable NCCL, torch), built from a bare CUDA base; it can also be used directly for testing.
 - **[`slurm/`](./slurm/)** — launchers that run DeepEP's own `tests/elastic/test_ep.py` to
   validate the installation:
 
@@ -29,15 +31,15 @@ installation works:
 
 > - **NCCL >= 2.31** — the GIN device API (`nccl_device.h`) DeepEP V2 links against.
 >   The Dockerfile builds NCCL from source and fails the build if the header is missing.
-> - **EFA installer >= 1.50** — EFA-GDA requires **libfabric >= 2.5**; installers up to
->   1.49.0 ship libfabric 2.4 and will not work (the image build fails loudly on a
->   too-old installer).
-> - **aws-ofi-nccl built against that libfabric** — required for the **EFA-GDA** GIN backend
->   (GIN type 5) this benchmark measures. DeepEP's kernels can also run over the CPU-proxy
->   GIN backend, which has no such floor.
+> - **EFA installer >= 1.50** — provides everything EFA-GDA needs in the container (the
+>   userspace runtime and the aws-ofi-nccl plugin with the EFA-GDA GIN backend, type 5,
+>   this benchmark measures). Installers up to 1.49.0 will not work; the image build fails
+>   loudly on a too-old installer. DeepEP's kernels can also run over the CPU-proxy GIN
+>   backend, which has no such floor.
 > - A host EFA kernel driver **>= 3.3.0**, the first release with the completion-counter API
->   (`efadv_create_comp_cntr`). Check with `modinfo efa | grep ^version`; stock AMIs may ship
->   older and need a driver upgrade. The container ships only the userspace stack.
+>   (`efadv_create_comp_cntr`). Run the EFA installer **on the node itself** (not in the
+>   container) to install/upgrade the kernel driver; the container ships only the userspace
+>   stack. Check with `modinfo efa | grep ^version`; stock AMIs may ship older.
 > - The **gdrcopy kernel module (`gdrdrv`) loaded on compute nodes** — the launchers
 >   bind-mount `/dev/gdrdrv`, and the plugin's GIN initialization opens a gdr handle.
 >   Check with `lsmod | grep gdrdrv`.
@@ -51,7 +53,7 @@ in two kinds: GPU-initiated ones, where the NIC work queues are mapped into GPU
 kernels post RDMA themselves, and the CPU-proxy backend (available on EFA as well as IB/RoCE),
 where the GPU hands work to a proxy thread. On EFA, the **aws-ofi-nccl** plugin provides the
 GPU-initiated backend, **EFA-GDA** (GIN type 5). The launchers pin `NCCL_GIN_TYPE=5` to select
-it; left unset, NCCL selects the CPU-proxy backend.
+it, if left unset, NCCL selects the CPU-proxy backend when running on EFA.
 
 | Variable | Value | Purpose |
 |----------|-------|---------|
@@ -79,7 +81,11 @@ it; left unset, NCCL selects the CPU-proxy backend.
 
 The script refuses to run if the NCCL at `--nccl-root` lacks the GIN device API, so it cannot
 silently produce a DeepEP that falls back to a slower path. Runtime still requires the
-aws-ofi-nccl GIN plugin built with EFA-GDA support.
+EFA-GDA-capable aws-ofi-nccl GIN plugin (bundled with EFA installer >= 1.50): run the EFA
+installer **inside your container** too (with `--skip-kmod`, as the reference Dockerfile does)
+to provide the userspace stack and the plugin, then point `NCCL_NET_PLUGIN` and
+`NCCL_GIN_PLUGIN` at `/opt/amazon/ofi-nccl/lib/libnccl-net-ofi.so`. The host installer run
+(previous section) provides only the kernel driver.
 
 ## Building the reference image
 

micro-benchmarks/expert-parallelism/deepep-v2-benchmark/deepep.Dockerfile

@@ -16,9 +16,9 @@ ARG TORCH_VERSION=2.11.0
 ARG TORCH_CUDA_INDEX=cu130
 ARG GDRCOPY_VERSION=v2.5.2
 
-# EFA installer: provides OpenMPI, the EFA userspace runtime, and libfabric.
-# EFA-GDA requires libfabric >= 2.5, so installer >= 1.50 is required (1.49.0 and
-# older ship libfabric 2.4; the build fails loudly on a too-old installer).
+# EFA installer: provides the complete EFA userspace stack (OpenMPI, the EFA
+# runtime, and the aws-ofi-nccl plugin). EFA-GDA requires installer >= 1.50;
+# the build fails loudly on a too-old installer (plugin gate below).
 # EFA_INSTALLER_TARBALL overrides the download with a local tarball staged next
 # to this Dockerfile.
 ARG EFA_INSTALLER_VERSION=1.50.0
@@ -29,14 +29,6 @@ ARG EFA_INSTALLER_TARBALL=
 ARG NCCL_REPO=https://github.com/NVIDIA/nccl.git
 ARG NCCL_REF=v2.31.2-1
 
-# aws-ofi-nccl: built from source at a pinned tag so the plugin version is
-# controlled by this file rather than by whatever the EFA installer bundles.
-# EFA-GDA is auto-detected by configure from
-# the libfabric >= 2.5 GDA ops + hardware counters + CUDA; the build gate below
-# fails the image if the detection did not fire.
-ARG AWS_OFI_NCCL_REPO=https://github.com/aws/aws-ofi-nccl.git
-ARG AWS_OFI_NCCL_REF=v1.21.1
-
 # DeepEP V2. The repo is pinned to amazon-contributing/DeepEP inside
 # setup_deepep_gin.sh; only the ref is overridable.
 ARG DEEPEP_REF=main
@@ -110,9 +102,9 @@ RUN --mount=type=bind,target=/ctx \
     && cd aws-efa-installer \
     && ./efa_installer.sh --disable-ngc -y --skip-kmod --skip-limit-conf --no-verify \
     && ldconfig \
-    && FI_VER=$(/opt/amazon/efa/bin/fi_info --version | grep -oP "libfabric: \K[0-9]+\.[0-9]+") \
-    && { [ "$(printf "%s\n2.5\n" "$FI_VER" | sort -V | head -1)" = "2.5" ] \
-         || { echo "ERROR: installer libfabric $FI_VER < 2.5; EFA-GDA requires >= 2.5 (use installer >= 1.50)" >&2; exit 1; }; } \
+    && { nm -D /opt/amazon/ofi-nccl/lib/libnccl-net-ofi.so 2>/dev/null | grep -qw ncclGinPlugin_v14 \
+         && echo "OK: bundled plugin is EFA-GDA-capable (ncclGinPlugin_v14 present)" \
+         || { echo "ERROR: installer's plugin has no ncclGinPlugin_v14 export: EFA-GDA requires installer >= 1.50" >&2; exit 1; }; } \
     && rm -rf /tmp/aws-efa-installer* /var/lib/apt/lists/*
 ENV LD_LIBRARY_PATH=/opt/amazon/openmpi/lib:$LD_LIBRARY_PATH
 ENV PATH=/opt/amazon/openmpi/bin:/opt/amazon/efa/bin:$PATH
@@ -128,34 +120,11 @@ RUN git clone ${NCCL_REPO} /opt/nccl \
        || (echo "ERROR: nccl_device.h missing -- this NCCL is not GIN-capable" >&2 && exit 1)
 ENV LD_LIBRARY_PATH="${NCCL_HOME}/lib:${LD_LIBRARY_PATH}"
 
-## aws-ofi-nccl. EFA-GDA support is auto-detected by configure from the libfabric
-## above; the ncclGinPlugin_v14 export check makes the detection authoritative
-## (v11/v13 op-tables are exported by every build; the v14 table is EFA-GDA-only).
-ENV OFI_HOME=/opt/aws-ofi-nccl
-RUN set -e; \
-    git clone --recursive ${AWS_OFI_NCCL_REPO} /tmp/aws-ofi-nccl \
-    && cd /tmp/aws-ofi-nccl \
-    && git checkout ${AWS_OFI_NCCL_REF} \
-    && git submodule sync && git submodule update --init --recursive \
-    && ./autogen.sh \
-    && { ./configure \
-        --prefix=${OFI_HOME} \
-        --with-mpi=/opt/amazon/openmpi \
-        --with-libfabric=${EFA_PREFIX} \
-        --with-nccl=${NCCL_HOME} \
-        --with-cuda=${CUDA_HOME} \
-        --with-gdrcopy=${GDRCOPY_PREFIX} \
-        --enable-platform-aws \
-      || { tail -n 160 config.log 2>/dev/null; exit 1; }; } \
-    && make -j"$(nproc)" \
-    && make install \
-    && if nm -D "${OFI_HOME}/lib/libnccl-net-ofi.so" 2>/dev/null | grep -qw ncclGinPlugin_v14; then \
-         echo "OK: EFA-GDA (ncclGinPlugin_v14) present"; \
-       else \
-         echo "ERROR: no ncclGinPlugin_v14 export -- configure did not detect EFA-GDA (libfabric >= 2.5 with hw counters + CUDA required)" >&2; exit 1; \
-       fi \
-    && echo "${OFI_HOME}/lib" > /etc/ld.so.conf.d/aws-ofi-nccl.conf && ldconfig \
-    && rm -rf /tmp/aws-ofi-nccl
+## aws-ofi-nccl: the EFA installer (>= 1.50) bundles the plugin with EFA-GDA
+## support; the installer layer above gates on the plugin's ncclGinPlugin_v14
+## export (v11/v13 op-tables are exported by every build; the v14 table is
+## EFA-GDA-only), so a too-old installer fails that layer immediately.
+ENV OFI_HOME=/opt/amazon/ofi-nccl
 ENV LD_LIBRARY_PATH="${OFI_HOME}/lib:${LD_LIBRARY_PATH}"
 ENV NCCL_NET_PLUGIN="${OFI_HOME}/lib/libnccl-net-ofi.so"
 ENV NCCL_GIN_PLUGIN="${OFI_HOME}/lib/libnccl-net-ofi.so"

micro-benchmarks/expert-parallelism/deepep-v2-benchmark/setup_deepep_gin.sh

@@ -250,7 +250,8 @@ print_completion_notes() {
     log ""
     log "  export LD_LIBRARY_PATH=${NCCL_ROOT}/lib:\${LD_LIBRARY_PATH}"
     log "  export FI_PROVIDER=efa"
-    log "  export NCCL_NET_PLUGIN=/path/to/libnccl-net-ofi.so   # aws-ofi-nccl (also set NCCL_GIN_PLUGIN)"
+    log "  export NCCL_NET_PLUGIN=/opt/amazon/ofi-nccl/lib/libnccl-net-ofi.so   # from the EFA installer (>= 1.50) run in this container"
+    log "  export NCCL_GIN_PLUGIN=/opt/amazon/ofi-nccl/lib/libnccl-net-ofi.so"
     log ""
     log "EFA-GDA also requires the HOST EFA kernel driver to support the"
     log "comp-counter API (efadv_create_comp_cntr). The container ships only the"