diff --git a/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/README.md b/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/README.md index a06ca178b..9a97a888a 100644 --- a/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/README.md +++ b/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/README.md @@ -19,8 +19,10 @@ installation works: - **[`deepep.Dockerfile`](./deepep.Dockerfile)** — a reference image showing the full stack the script needs (the EFA installer, which bundles the EFA-GDA-capable aws-ofi-nccl plugin, a GIN-capable NCCL, torch), built from a bare CUDA base; it can also be used directly for testing. -- **[`slurm/`](./slurm/)** — launchers that run DeepEP's own `tests/elastic/test_ep.py` to - validate the installation: +- **[`slurm/`](./slurm/)** - launchers that run DeepEP's own `tests/elastic/test_ep.py` to + validate the installation. +- **[`kubernetes/`](./kubernetes/)** - launchers that run DeepEP's own `tests/elastic/test_ep.py` to + validate the installation. | Benchmark | Nodes | Measures | |-----------|-------|----------| @@ -30,7 +32,7 @@ installation works: ## ⚠️ Version constraints > - **NCCL >= 2.31** — the GIN device API (`nccl_device.h`) DeepEP V2 links against. -> The Dockerfile builds NCCL from source and fails the build if the header is missing. +> The Dockerfile install NVIDIA's precompiled NCCL from PyPi as a python package. > - **EFA installer >= 1.50** — provides everything EFA-GDA needs in the container (the > userspace runtime and the aws-ofi-nccl plugin with the EFA-GDA GIN backend, type 5, > this benchmark measures). Installers up to 1.49.0 will not work; the image build fails @@ -38,7 +40,7 @@ installation works: > backend, which has no such floor. > - A host EFA kernel driver **>= 3.3.0**, the first release with the completion-counter API > (`efadv_create_comp_cntr`). Run the EFA installer **on the node itself** (not in the -> container) to install/upgrade the kernel driver; the container ships only the userspace +> container) to install/upgrade the kernel driver; the provided Dockerfile ships only the userspace > stack. Check with `modinfo efa | grep ^version`; stock AMIs may ship older. > - The **gdrcopy kernel module (`gdrdrv`) loaded on compute nodes** — the launchers > bind-mount `/dev/gdrdrv`, and the plugin's GIN initialization opens a gdr handle. @@ -58,18 +60,16 @@ it, if left unset, NCCL selects the CPU-proxy backend when running on EFA. | Variable | Value | Purpose | |----------|-------|---------| | `NCCL_GIN_TYPE` | `5` | Pins the EFA-GDA GIN backend; unset falls back to the CPU-proxy backend. | -| `EP_NCCL_ROOT_DIR` | `$NCCL_HOME` | Points DeepEP's build/runtime checks at the GIN-capable NCCL. | -| `LD_PRELOAD` | `$NCCL_HOME/lib/libnccl.so.2` | Guarantees the GIN-capable NCCL wins over any other on the loader path. | ## Prerequisites - EFA-enabled GPU nodes with the EFA kernel driver installed (section below). -- **GPU architecture:** the image builds for **Hopper (`sm_90`) and Blackwell (`sm_100`)** by +- **GPU architecture:** the image builds for **Hopper (`sm_90`) and Blackwell (`sm_100`, `sm_103`)** by default, so one image runs on `p5`/`p5en` and `p6`. Earlier DeepEP revisions required **CUDA >= 13.1** on p6/Blackwell (ptxas 13.0.88 rejects the 32-bit `st.bulk` size operand); DeepEP main carries the fix ([amazon-contributing/DeepEP#3](https://github.com/amazon-contributing/DeepEP/pull/3)), - so CUDA 13.0 works as well. Override - `--build-arg TORCH_CUDA_ARCH_LIST` / `--build-arg NVCC_GENCODE` for a single-arch image. + so CUDA 13.0 works. + Override `--build-arg TORCH_CUDA_ARCH_LIST` / `--build-arg NVCC_GENCODE` for a single-arch image. - Docker with BuildKit, and enroot + pyxis on the cluster for the Slurm flow. ## Using the script in your own container @@ -81,11 +81,10 @@ it, if left unset, NCCL selects the CPU-proxy backend when running on EFA. The script refuses to run if the NCCL at `--nccl-root` lacks the GIN device API, so it cannot silently produce a DeepEP that falls back to a slower path. Runtime still requires the -EFA-GDA-capable aws-ofi-nccl GIN plugin (bundled with EFA installer >= 1.50): run the EFA -installer **inside your container** too (with `--skip-kmod`, as the reference Dockerfile does) -to provide the userspace stack and the plugin, then point `NCCL_NET_PLUGIN` and -`NCCL_GIN_PLUGIN` at `/opt/amazon/ofi-nccl/lib/libnccl-net-ofi.so`. The host installer run -(previous section) provides only the kernel driver. +EFA-GDA-capable aws-ofi-nccl GIN plugin (bundled with EFA installer >= 1.50). + +> NOTE: The aws-ofi-nccl plugin has to be in your container, run the EFA Installer inside your container with the `-y --skip-kmod --skip-limit-conf --no-verify` arguments. \ +> For installing the EFA Driver on the host, the EFA Installer bundles the driver, run the EFA Installer on the host with the `--minimal -y` arguments. ## Building the reference image @@ -96,11 +95,11 @@ DOCKER_BUILDKIT=1 docker build --progress=plain -f ./deepep.Dockerfile \ -t deepep-v2:gin . ``` -Override `NCCL_REF` or `EFA_INSTALLER_VERSION` for different versions: +Override `NCCL_VERSION` or `EFA_INSTALLER_VERSION` for different versions: ```bash DOCKER_BUILDKIT=1 docker build --progress=plain -f ./deepep.Dockerfile \ - --build-arg NCCL_REF=v2.31.2-1 \ + --build-arg NCCL_VERSION=2.31.2 \ --build-arg EFA_INSTALLER_VERSION=1.50.0 \ -t deepep-v2:gin . ``` diff --git a/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/deepep.Dockerfile b/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/deepep.Dockerfile index a6a9e3d52..1fdaaab6f 100644 --- a/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/deepep.Dockerfile +++ b/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/deepep.Dockerfile @@ -1,164 +1,128 @@ # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. # SPDX-License-Identifier: MIT-0 -# -# DeepEP V2 image for AWS EFA clusters. -# See README.md for build and run instructions. - -# CUDA 13.1 is the default. Earlier DeepEP revisions needed >= 13.1 on -# p6/Blackwell (ptxas 13.0.88 rejects the 32-bit st.bulk size operand); DeepEP -# main carries the 64-bit fix (amazon-contributing/DeepEP#3), so 13.0 works too. -ARG CUDA_VERSION=13.1.2 +ARG CUDA_VERSION=13.0.2 + FROM nvidia/cuda:${CUDA_VERSION}-devel-ubuntu22.04 -ARG TORCH_VERSION=2.11.0 -# Torch wheel index. Kept explicit rather than derived from nvcc: pytorch.org has -# no cu131 index, and cu130 wheels run on a 13.1 toolkit (same CUDA major). -ARG TORCH_CUDA_INDEX=cu130 ARG GDRCOPY_VERSION=v2.5.2 - -# EFA installer: provides the complete EFA userspace stack (OpenMPI, the EFA -# runtime, and the aws-ofi-nccl plugin). EFA-GDA requires installer >= 1.50; -# the build fails loudly on a too-old installer (plugin gate below). -# EFA_INSTALLER_TARBALL overrides the download with a local tarball staged next -# to this Dockerfile. ARG EFA_INSTALLER_VERSION=1.50.0 -ARG EFA_INSTALLER_TARBALL= - -# NCCL: defaults to the public NVIDIA release. For internal use, override with -# the upstream fork + staging branch (see build instructions in README.md). -ARG NCCL_REPO=https://github.com/NVIDIA/nccl.git -ARG NCCL_REF=v2.31.2-1 - -# DeepEP V2. The repo is pinned to amazon-contributing/DeepEP inside -# setup_deepep_gin.sh; only the ref is overridable. -ARG DEEPEP_REF=main +ARG DEEPEP_REF="main" +ARG NCCL_VERSION=2.31.2 +ARG PYTHON_VERSION=3.10 +ARG TORCH_VERSION=2.11.0 +ARG TORCH_CUDA_INDEX=cu130 # CUDA architecture(s), semicolon-separated: -# 9.0 = Hopper (H100/H200, sm_90); 10.0/10.3 = Blackwell (B200/B300, sm_100/sm_103). -# Defaults to Hopper + Blackwell so one image runs on p5/p5en and p6. -ARG TORCH_CUDA_ARCH_LIST="9.0;10.0" -ARG NVCC_GENCODE="-gencode=arch=compute_90,code=sm_90 -gencode=arch=compute_100,code=sm_100" +# 9.0 = Hopper (H100, sm_90), 10.0 and 10.3 = Blackwell (B200/B300, sm_100,sm_103). Defaults to +# both so one image runs on Hopper and Blackwell; override with e.g. +# --build-arg TORCH_CUDA_ARCH_LIST=9.0 \ +# --build-arg NVCC_GENCODE=-gencode=arch=compute_90,code=sm_90 to build a smaller Hopper-only image. +ARG TORCH_CUDA_ARCH_LIST="9.0;10.0;10.3" +ARG NVCC_GENCODE="-gencode=arch=compute_90,code=sm_90 -gencode=arch=compute_100,code=sm_100 -gencode=arch=compute_103,code=sm_103" + ARG CUDA_HOME="/usr/local/cuda" ENV DEBIAN_FRONTEND=noninteractive -## Remove conflicting distro packages -RUN apt-get update -y \ - && apt-get remove -y --allow-change-held-packages \ +RUN apt-get update -y && \ + apt-get remove -y --allow-change-held-packages \ ibverbs-utils \ libibverbs-dev \ libibverbs1 \ libmlx5-1 \ libnccl2 \ - libnccl-dev || true -RUN rm -rf /opt/hpcx /usr/local/mpi /etc/ld.so.conf.d/hpcx.conf && ldconfig - -## Toolchain -RUN apt-get update -y && apt-get install -y \ - apt-utils autoconf automake build-essential check cmake curl debhelper \ - devscripts git gcc gdb libsubunit-dev libtool ninja-build meson pandoc \ - pkg-config vim wget \ - python3.10-dev python3.10-venv python3-distutils \ - cython3 \ - libnl-3-dev libnl-route-3-dev libudev-dev libsystemd-dev \ - libhwloc-dev \ - && rm -rf /var/lib/apt/lists/* -RUN apt-get purge -y cuda-compat-* || true + libnccl-dev + +RUN rm -rf /opt/hpcx \ + && rm -rf /usr/local/mpi \ + && rm -f /etc/ld.so.conf.d/hpcx.conf \ + && ldconfig + +RUN apt-get install -y --no-install-recommends \ + apt-utils \ + autoconf \ + automake \ + build-essential \ + check \ + cmake \ + curl \ + debhelper \ + devscripts \ + git \ + gcc \ + gdb \ + libsubunit-dev \ + libtool \ + openssh-client \ + openssh-server \ + pkg-config \ + python3-distutils \ + vim \ + python${PYTHON_VERSION}-dev \ + python${PYTHON_VERSION}-venv + +RUN apt-get purge -y cuda-compat-* + +RUN mkdir -p /var/run/sshd +RUN sed -i 's/[ #]\(.*StrictHostKeyChecking \).*/ \1no/g' /etc/ssh/ssh_config && \ + echo " UserKnownHostsFile /dev/null" >> /etc/ssh/ssh_config && \ + sed -i 's/#\(StrictModes \).*/\1no/g' /etc/ssh/sshd_config RUN curl https://bootstrap.pypa.io/get-pip.py -o /tmp/get-pip.py \ && python3 /tmp/get-pip.py \ - && pip3 install awscli nvidia-ml-py ninja Cython - -## GDRCopy -# Required by aws-ofi-nccl: the plugin's GIN initialization opens a gdr handle -# regardless of backend, so building or running without GDRCopy leaves the GIN -# backends unable to initialize (verified: DeepEP then fails with "NCCL GIN is -# unavailable"). Built from source because no distro package ships it. The -# matching gdrdrv kernel module must be loaded on the host (see README). + && pip3 install awscli nvidia-ml-py ninja + +################################################# +## Install NVIDIA GDRCopy ARG GDRCOPY_PREFIX="/opt/gdrcopy" RUN git clone -b ${GDRCOPY_VERSION} https://github.com/NVIDIA/gdrcopy.git /tmp/gdrcopy \ && cd /tmp/gdrcopy \ && make prefix="${GDRCOPY_PREFIX}" install \ && rm -rf /tmp/gdrcopy + ENV LD_LIBRARY_PATH="${GDRCOPY_PREFIX}/lib:$LD_LIBRARY_PATH" ENV LIBRARY_PATH="${GDRCOPY_PREFIX}/lib:$LIBRARY_PATH" ENV PATH="${GDRCOPY_PREFIX}/bin:$PATH" -## EFA installer (OpenMPI + EFA runtime; local tarball wins when set) -RUN --mount=type=bind,target=/ctx \ - apt-get update -y \ - && cd /tmp \ - && if [ -n "${EFA_INSTALLER_TARBALL}" ]; then \ - echo "=== EFA installer: LOCAL tarball /ctx/${EFA_INSTALLER_TARBALL} ==="; \ - if [ ! -f "/ctx/${EFA_INSTALLER_TARBALL}" ]; then \ - echo "ERROR: ${EFA_INSTALLER_TARBALL} not found in build context" >&2; exit 1; \ - fi; \ - tar -xf "/ctx/${EFA_INSTALLER_TARBALL}"; \ - else \ - echo "=== EFA installer: downloading v${EFA_INSTALLER_VERSION} ==="; \ - curl -fsSL -O "https://efa-installer.amazonaws.com/aws-efa-installer-${EFA_INSTALLER_VERSION}.tar.gz" \ - && tar -xf "aws-efa-installer-${EFA_INSTALLER_VERSION}.tar.gz"; \ - fi \ - && cd aws-efa-installer \ - && ./efa_installer.sh --disable-ngc -y --skip-kmod --skip-limit-conf --no-verify \ - && ldconfig \ - && { nm -D /opt/amazon/ofi-nccl/lib/libnccl-net-ofi.so 2>/dev/null | grep -qw ncclGinPlugin_v14 \ - && echo "OK: bundled plugin is EFA-GDA-capable (ncclGinPlugin_v14 present)" \ - || { echo "ERROR: installer's plugin has no ncclGinPlugin_v14 export: EFA-GDA requires installer >= 1.50" >&2; exit 1; }; } \ - && rm -rf /tmp/aws-efa-installer* /var/lib/apt/lists/* +################################################# +## Install EFA installer +RUN cd $HOME && \ + curl --retry 3 --retry-delay 2 -fsSL -o aws-efa-installer-${EFA_INSTALLER_VERSION}.tar.gz \ + https://efa-installer.amazonaws.com/aws-efa-installer-${EFA_INSTALLER_VERSION}.tar.gz && \ + tar -xf aws-efa-installer-${EFA_INSTALLER_VERSION}.tar.gz && \ + cd aws-efa-installer && \ + apt-get update && \ + ./efa_installer.sh -y --skip-kmod --skip-limit-conf --no-verify && \ + cd .. && rm -rf aws-efa-installer* && \ + ldconfig + ENV LD_LIBRARY_PATH=/opt/amazon/openmpi/lib:$LD_LIBRARY_PATH -ENV PATH=/opt/amazon/openmpi/bin:/opt/amazon/efa/bin:$PATH -ENV EFA_PREFIX=/opt/amazon/efa - -## NCCL -ENV NCCL_HOME=/opt/nccl/build -RUN git clone ${NCCL_REPO} /opt/nccl \ - && cd /opt/nccl \ - && git checkout ${NCCL_REF} \ - && make -j"$(nproc)" src.build CUDA_HOME="${CUDA_HOME}" NVCC_GENCODE="${NVCC_GENCODE}" \ - && test -f ${NCCL_HOME}/include/nccl_device.h \ - || (echo "ERROR: nccl_device.h missing -- this NCCL is not GIN-capable" >&2 && exit 1) -ENV LD_LIBRARY_PATH="${NCCL_HOME}/lib:${LD_LIBRARY_PATH}" - -## aws-ofi-nccl: the EFA installer (>= 1.50) bundles the plugin with EFA-GDA -## support; the installer layer above gates on the plugin's ncclGinPlugin_v14 -## export (v11/v13 op-tables are exported by every build; the v14 table is -## EFA-GDA-only), so a too-old installer fails that layer immediately. -ENV OFI_HOME=/opt/amazon/ofi-nccl -ENV LD_LIBRARY_PATH="${OFI_HOME}/lib:${LD_LIBRARY_PATH}" -ENV NCCL_NET_PLUGIN="${OFI_HOME}/lib/libnccl-net-ofi.so" -ENV NCCL_GIN_PLUGIN="${OFI_HOME}/lib/libnccl-net-ofi.so" - -# Keep NCCL's bootstrap off the docker/loopback/veth interfaces. -ENV NCCL_SOCKET_IFNAME=^docker,lo,veth - -## PyTorch. -# The torch wheel drags in pip's stock NCCL (nvidia-nccl-cu*), which predates GIN. -# We remove it rather than keep both: DeepEP refuses to import when it detects two -# NCCL runtimes, and with both on the loader path which one wins is load-order -# luck. The GIN capability itself is enforced above at NCCL build time -# (nccl_device.h check errors the build), so this cannot silently downgrade: the -# image ends up with exactly one NCCL, the GIN-capable one it was built and -# tested with. -RUN pip3 install torch==${TORCH_VERSION} numpy --index-url https://download.pytorch.org/whl/${TORCH_CUDA_INDEX} \ - && pip3 uninstall -y nvidia-nccl-cu13 nvidia-nccl-cu12 nvidia-nccl 2>/dev/null || true - -## No NVSHMEM: DeepEP V2's transport is NCCL GIN, so the NVSHMEM backend is not -## built (the DeepEP fork's build only links NVSHMEM when NVSHMEM_DIR points at an -## install). To additionally build the legacy NVSHMEM backend, install the -## libnvshmem3-*-cuda-13 packages and set NVSHMEM_DIR before running the setup script. - -## DeepEP V2. -ARG DEEPEP_PREFIX="/opt/amazon/deepep" +ENV PATH=/opt/amazon/openmpi/bin/:/opt/amazon/efa/bin:$PATH + +RUN rm -rf /var/lib/apt/lists/* + +## Set Open MPI variables to exclude network interface and conduit. +ENV OMPI_MCA_pml=^ucx \ + OMPI_MCA_btl=tcp,self \ + OMPI_MCA_btl_tcp_if_exclude=lo,docker0,veth_def_agent\ + OPAL_PREFIX=/opt/amazon/openmpi \ + NCCL_SOCKET_IFNAME=^docker,lo,veth + +## Turn off PMIx Error https://github.com/open-mpi/ompi/issues/7516 +ENV PMIX_MCA_gds=hash + +################################ PyTorch ######################################## +RUN pip3 install torch==${TORCH_VERSION} numpy --index-url https://download.pytorch.org/whl/${TORCH_CUDA_INDEX} + +################################ DeepEP v2 ######################################## RUN --mount=type=bind,source=setup_deepep_gin.sh,target=/tmp/setup_deepep_gin.sh \ - set -e; \ - TORCH_CUDA_ARCH_LIST="${TORCH_CUDA_ARCH_LIST}" \ + pip3 uninstall -y nvidia-nccl-cu13 nvidia-nccl-cu12 nvidia-nccl 2>/dev/null || true && \ + CUDA_MAJOR=$(nvcc --version | grep -oP 'release \K[0-9]+') && \ + pip3 install --no-deps nvidia-nccl-cu${CUDA_MAJOR}==${NCCL_VERSION} && \ /tmp/setup_deepep_gin.sh \ - --deepep-ref "${DEEPEP_REF}" \ - --deepep-prefix "${DEEPEP_PREFIX}" \ - --nccl-root "${NCCL_HOME}" + --deepep-ref $DEEPEP_REF \ + --nccl-root "/usr/local/lib/python${PYTHON_VERSION}/dist-packages/nvidia/nccl" -## Runtime env ENV NVIDIA_GDRCOPY=enabled -ENV NCCL_OFI_RDMA_GDR_FLUSH_DISABLE=0 -WORKDIR /root +ENV NCCL_NET_PLUGIN=ofi +ENV NCCL_TUNER_PLUGIN=ofi diff --git a/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/kubernetes/README.md b/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/kubernetes/README.md new file mode 100644 index 000000000..f92660bbc --- /dev/null +++ b/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/kubernetes/README.md @@ -0,0 +1,117 @@ +# DeepEP v2 Benchmark — Kubernetes (EKS) + +Run the DeepEP v2 micro-benchmarks on Amazon EKS as [MPIJobs](https://www.kubeflow.org/docs/components/training/user-guides/mpi/). +These are the Kubernetes equivalents of the [Slurm](../slurm/) `sbatch` scripts. + +Each job launches **one MPI process per node** (`slotsPerWorker: 1`); that process spawns 8 +local GPU ranks itself via `torch.multiprocessing`. The launcher passes the EFA / NCCL +environment (including `NCCL_SOCKET_IFNAME` and `LD_PRELOAD`) through `mpirun -x` because +SSH-launched workers do not inherit the image `ENV`. + +## Prerequisites + +### EKS cluster with EFA + GPUs + +An EKS cluster with EFA-enabled GPU nodes (e.g. `p6-b300.48xlarge`), the +[NVIDIA device plugin](https://github.com/NVIDIA/k8s-device-plugin) and the +[AWS EFA device plugin](https://github.com/aws/eks-charts/tree/master/stable/aws-efa-k8s-device-plugin) +installed. See the [EKS architectures](../../../../1.architectures) in this repo. + +> NOTE: To expose the GDRCopy gdrdrv kernel driver, install the NVIDIA device plugin with `gdrcopyEnabled=true`. \ +> [AI/ML on EKS - NVIDIA device plugin](https://docs.aws.amazon.com/eks/latest/userguide/device-management-nvidia-dra-device-plugin.html#eks-nvidia-device-plugin) + +```bash +aws eks update-kubeconfig --name +kubectl config current-context +``` + +### MPI Operator + +The `MPIJob` CRD is provided by the standalone [Kubeflow MPI Operator](https://github.com/kubeflow/mpi-operator) +(not the Training Operator): + +```bash +kubectl apply -f https://raw.githubusercontent.com/kubeflow/mpi-operator/refs/tags/v0.8.2/deploy/v2beta1/mpi-operator.yaml +kubectl get crd mpijobs.kubeflow.org # confirm the CRD exists +``` + +## Build & push the container image + +Build the image from [`../deepep.Dockerfile`](../deepep.Dockerfile) and push it to ECR. The +default image is built for **both Hopper (`sm_90`) and Blackwell (`sm_100`, `sm_103`)**, so it runs on +`p5`/`p5en` and `p6` instances family without a rebuild. + +```bash +export AWS_REGION=$(aws ec2 describe-availability-zones --output text --query 'AvailabilityZones[0].[RegionName]') +export ACCOUNT=$(aws sts get-caller-identity --query Account --output text) +export REGISTRY=${ACCOUNT}.dkr.ecr.${AWS_REGION}.amazonaws.com +export IMAGE_URI=${REGISTRY}/deepep-v2:efa1.50.0-nccl2.31.2 + +aws ecr create-repository --repository-name deepep-v2 --region ${AWS_REGION} 2>/dev/null || true +aws ecr get-login-password --region ${AWS_REGION} | docker login --username AWS --password-stdin ${REGISTRY} + +cd .. # build context is the deepep-v2-benchmark/ directory +DOCKER_BUILDKIT=1 docker build --progress=plain -f ./deepep.Dockerfile -t ${IMAGE_URI} . +docker push ${IMAGE_URI} +``` + +## Configure & launch + +Copy the committed template to `env_vars` (gitignored) and edit it with your image URI, instance +type, and per-node device counts, then `envsubst` the manifest into `kubectl`. Restrict +substitution to the known variables so the launcher's runtime shell vars +(`$OMPI_COMM_WORLD_RANK`, `$MASTER_ADDR`, `$PATH`, …) are left intact: + +```bash +cd kubernetes +cp env_vars.example env_vars # then edit env_vars +source env_vars + +# Single-node intranode (NVLink) test (intranode is fixed at 1 node; NUM_NODES is unused) +envsubst '$IMAGE_URI $INSTANCE_TYPE $GPU_PER_NODE $EFA_PER_NODE $NUM_NODES' \ + < test-intranode.yaml | kubectl apply -f - + +# Two-node internode (RDMA over EFA) test — set NUM_NODES=2 +envsubst '$IMAGE_URI $INSTANCE_TYPE $GPU_PER_NODE $EFA_PER_NODE $NUM_NODES' \ + < test-internode.yaml | kubectl apply -f - +``` + +## Monitor + +```bash +kubectl get mpijob +watch kubectl get pods -o wide +kubectl logs -l training.kubeflow.org/job-name=deepep-v2-internode -f +``` + +## Clean up + +MPIJob names are fixed, so delete a run before re-applying it: + +```bash +envsubst '$IMAGE_URI $INSTANCE_TYPE $GPU_PER_NODE $EFA_PER_NODE $NUM_NODES' \ + < test-internode.yaml | kubectl delete -f - +``` + +## Notes for `p6-b300.48xlarge` (B300 / Blackwell) + +- **Image:** the default image already includes `sm_103`, so no rebuild is needed. +- **EFA count:** `EFA_PER_NODE=16` (a `p6-b300.48xlarge` exposes 16 EFA NICs). `p6-b200.48xlarge` + exposes 8 — adjust per instance type. +- **Tolerations:** the manifests tolerate `nvidia.com/gpu`, `workload=bench`, and + `capacity-reservation` taints. If your nodes use different taints, edit the `tolerations` + block; if they are untainted, the extra tolerations are harmless. +- **Multi-node DNS:** the launcher sets `MASTER_ADDR` from the first entry of the + mpi-operator hostfile (`/etc/mpi/hostfile`), which is worker-0's cluster-resolvable name. If + `init_process_group` hangs, confirm name resolution from a peer: + `kubectl exec deepep-v2-internode-worker-1 -- getent hosts `. +- **GPU profiling (internode / low-latency tuning):** the `internode` and `intranode` tests + profile their kernels with the Kineto/CUPTI profiler during the tuning phase. + Two requirements, both already handled here for `p6-b300`: + 1. *Driver profiling access.* These nodes load the driver with + `NVreg_RestrictProfilingToAdminUsers=1` (`grep RmProfiling /proc/driver/nvidia/params`), so + the test pods are granted `SYS_ADMIN` (see the worker `securityContext` in the internode / + intranode manifests). Remove it if your nodes already permit non-admin profiling. + 2. *A CUPTI that supports the GPU.* The image is built on **CUDA 13** — its CUPTI supports + Blackwell. CUDA ≤ 12.9 returns `CUPTI_ERROR_INVALID_DEVICE` on B300 and the tuning phase + fails after the correctness checks pass, so do not downgrade the image's CUDA for B300. diff --git a/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/kubernetes/env_vars.example b/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/kubernetes/env_vars.example new file mode 100644 index 000000000..49578f630 --- /dev/null +++ b/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/kubernetes/env_vars.example @@ -0,0 +1,19 @@ +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: MIT-0 + +# Copy this file to env_vars (gitignored), edit the values, then source it: +# cp env_vars.example env_vars +# source env_vars +# envsubst '$IMAGE_URI $INSTANCE_TYPE $GPU_PER_NODE $EFA_PER_NODE $NUM_NODES' \ +# < test-internode.yaml | kubectl apply -f - + +# ECR image built from ../deepep.Dockerfile (default image supports Hopper + Blackwell). +export IMAGE_URI=.dkr.ecr..amazonaws.com/deepep-v2: + +# GPU node instance type and per-node device counts. +export INSTANCE_TYPE=p6-b300.48xlarge +export GPU_PER_NODE=8 # Must stay 8, test_ep.py --num-processes is 8. +export EFA_PER_NODE=16 # p6-b300.48xlarge has 16 EFA NICs; p5.48xlarge has 32 + +# Number of nodes: 1 for intranode; 2 for internode. +export NUM_NODES=2 diff --git a/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/kubernetes/test-internode.yaml b/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/kubernetes/test-internode.yaml new file mode 100644 index 000000000..63aa2c7b4 --- /dev/null +++ b/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/kubernetes/test-internode.yaml @@ -0,0 +1,148 @@ +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: MIT-0 +apiVersion: kubeflow.org/v2beta1 +kind: MPIJob +metadata: + name: deepep-v2-internode +spec: + runPolicy: + cleanPodPolicy: Running + backoffLimit: 3 + # One MPI slot per node: each worker process spawns 8 local GPU ranks itself + # (torch.multiprocessing.spawn). The test reads WORLD_SIZE = number of nodes + # and RANK = node rank, so MPI launches exactly one process per node. + slotsPerWorker: 1 + mpiReplicaSpecs: + Launcher: + replicas: 1 + template: + spec: + # The launcher runs the full benchmark image (~20GB) just to invoke + # mpirun. Co-locate it on the GPU node pool, whose nodes have enough + # ephemeral storage; small CPU nodes hit DiskPressure and evict it. + affinity: + podAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchExpressions: + - key: training.kubeflow.org/job-name + operator: In + values: + - deepep-v2-internode + topologyKey: kubernetes.io/hostname + nodeSelector: + node.kubernetes.io/instance-type: ${INSTANCE_TYPE} + restartPolicy: OnFailure + tolerations: + - key: nvidia.com/gpu + operator: Exists + effect: NoSchedule + - key: workload + value: bench + operator: Equal + effect: NoSchedule + - key: capacity-reservation + operator: Exists + effect: NoSchedule + containers: + - image: ${IMAGE_URI} + imagePullPolicy: IfNotPresent + name: deepep-v2-internode-launcher + resources: + requests: + cpu: "1" + memory: 2Gi + command: + - /bin/bash + - -c + - | + # MASTER_ADDR must be worker-0 and must resolve from the other workers + # (the torch rendezvous runs on the worker pods). Read it from the + # mpi-operator-injected hostfile so we use exactly the DNS name the + # operator made cluster-resolvable. + # Wait for each worker's sshd to accept connections before launching. + # mpi-operator starts the launcher as soon as workers are Running, but + # their sshd may not be ready yet; without this wait the launcher loses + # the SSH race, exits non-zero, and burns Job retries (backoffLimit). + for host in $(awk '{print $1}' /etc/mpi/hostfile); do + until ssh -o ConnectTimeout=3 -o BatchMode=yes -o StrictHostKeyChecking=no "$host" true 2>/dev/null; do + echo "waiting for sshd on $host"; sleep 3 + done + done + MASTER_ADDR=$(head -n1 /etc/mpi/hostfile | awk '{print $1}') + # SSH-launched workers do not inherit Dockerfile ENV, so pass every + # needed variable through `mpirun -x`. + /opt/amazon/openmpi/bin/mpirun \ + --allow-run-as-root --tag-output \ + -np ${NUM_NODES} -N 1 --bind-to none \ + -x PATH -x LD_LIBRARY_PATH \ + -x NCCL_SOCKET_IFNAME \ + -x NCCL_DEBUG=INFO \ + -x NCCL_NET_PLUGIN=ofi -x NCCL_TUNER_PLUGIN=ofi \ + -x TORCHDYNAMO_DISABLE=1 \ + -x TORCH_COMPILE_DISABLE=1 \ + -x NCCL_GIN_TYPE=5 \ + -x NCCL_SYM_GIN_KERNELS_ENABLE=0 \ + -x MASTER_ADDR=$MASTER_ADDR -x MASTER_PORT=8361 \ + bash -c 'RANK=$OMPI_COMM_WORLD_RANK WORLD_SIZE=$OMPI_COMM_WORLD_SIZE \ + python3 /opt/amazon/deepep/tests/elastic/test_ep.py --test-first-only --num-tokens=8192 --num-topk=8 --num-sms=12' + Worker: + replicas: ${NUM_NODES} + template: + spec: + # Schedule all workers in the same availability zone + affinity: + podAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchExpressions: + - key: training.kubeflow.org/job-name + operator: In + values: + - deepep-v2-internode + - key: training.kubeflow.org/job-role + operator: In + values: + - worker + topologyKey: topology.kubernetes.io/zone + nodeSelector: + node.kubernetes.io/instance-type: ${INSTANCE_TYPE} + tolerations: + - key: nvidia.com/gpu + operator: Exists + effect: NoSchedule + - key: workload + value: bench + operator: Equal + effect: NoSchedule + - key: capacity-reservation + operator: Exists + effect: NoSchedule + containers: + - image: ${IMAGE_URI} + imagePullPolicy: IfNotPresent + name: deepep-v2-internode-worker + env: + - name: NVIDIA_GDRCOPY + value: "enabled" + # The tuning phase profiles kernels via CUPTI. These nodes set the + # driver's RmProfilingAdminOnly=1, so the workload needs SYS_ADMIN to + # profile. Remove if your cluster allows non-admin GPU profiling. + securityContext: + capabilities: + add: ["SYS_ADMIN", "IPC_LOCK"] + volumeMounts: + - name: shmem + mountPath: /dev/shm + resources: + limits: + nvidia.com/gpu: ${GPU_PER_NODE} + vpc.amazonaws.com/efa: ${EFA_PER_NODE} + requests: + nvidia.com/gpu: ${GPU_PER_NODE} + vpc.amazonaws.com/efa: ${EFA_PER_NODE} + volumes: + - name: shmem + emptyDir: + medium: Memory + sizeLimit: 200Gi diff --git a/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/kubernetes/test-intranode.yaml b/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/kubernetes/test-intranode.yaml new file mode 100644 index 000000000..d00295766 --- /dev/null +++ b/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/kubernetes/test-intranode.yaml @@ -0,0 +1,129 @@ +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: MIT-0 +apiVersion: kubeflow.org/v2beta1 +kind: MPIJob +metadata: + name: deepep-v2-intranode +spec: + runPolicy: + cleanPodPolicy: Running + backoffLimit: 3 + # Single node: one MPI process that spawns 8 local GPU ranks itself + # (torch.multiprocessing.spawn). WORLD_SIZE defaults to 1, RANK to 0. + slotsPerWorker: 1 + mpiReplicaSpecs: + Launcher: + replicas: 1 + template: + spec: + restartPolicy: OnFailure + # The launcher runs the full benchmark image (~20GB) just to invoke + # mpirun. Co-locate it on the same GPU node as the worker (which has + # enough ephemeral storage); small CPU nodes hit DiskPressure and evict + # it. Use pod affinity on the hostname key so the launcher lands on + # whichever node the worker pod is scheduled to. + affinity: + podAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchExpressions: + - key: training.kubeflow.org/job-name + operator: In + values: + - deepep-v2-intranode + topologyKey: kubernetes.io/hostname + tolerations: + - key: nvidia.com/gpu + operator: Exists + effect: NoSchedule + - key: workload + value: bench + operator: Equal + effect: NoSchedule + - key: capacity-reservation + operator: Exists + effect: NoSchedule + containers: + - image: ${IMAGE_URI} + imagePullPolicy: IfNotPresent + name: deepep-v2-intranode-launcher + resources: + requests: + cpu: "1" + memory: 2Gi + command: + - /bin/bash + - -c + - | + # SSH-launched workers do not inherit Dockerfile ENV, so pass every + # needed variable through `mpirun -x`. Single node, so no MASTER_ADDR + # is needed (the test defaults to 127.0.0.1). + # Wait for each worker's sshd to accept connections before launching. + # mpi-operator starts the launcher as soon as workers are Running, but + # their sshd may not be ready yet; without this wait the launcher loses + # the SSH race, exits non-zero, and burns Job retries (backoffLimit). + for host in $(awk '{print $1}' /etc/mpi/hostfile); do + until ssh -o ConnectTimeout=3 -o BatchMode=yes -o StrictHostKeyChecking=no "$host" true 2>/dev/null; do + echo "waiting for sshd on $host"; sleep 3 + done + done + MASTER_ADDR=$(head -n1 /etc/mpi/hostfile | awk '{print $1}') + /opt/amazon/openmpi/bin/mpirun \ + --allow-run-as-root --tag-output \ + -np 1 -N 1 --bind-to none \ + -x PATH -x LD_LIBRARY_PATH \ + -x NCCL_SOCKET_IFNAME \ + -x NCCL_DEBUG=INFO \ + -x NCCL_NET_PLUGIN=ofi -x NCCL_TUNER_PLUGIN=ofi \ + -x TORCHDYNAMO_DISABLE=1 \ + -x TORCH_COMPILE_DISABLE=1 \ + -x NCCL_GIN_TYPE=5 \ + -x NCCL_SYM_GIN_KERNELS_ENABLE=0 \ + -x MASTER_ADDR=$MASTER_ADDR -x MASTER_PORT=8361 \ + bash -c 'RANK=$OMPI_COMM_WORLD_RANK WORLD_SIZE=$OMPI_COMM_WORLD_SIZE \ + python3 /opt/amazon/deepep/tests/elastic/test_ep.py --test-first-only --num-tokens=8192 --num-topk=8 --num-sms=64' + Worker: + replicas: 1 + template: + spec: + nodeSelector: + node.kubernetes.io/instance-type: ${INSTANCE_TYPE} + tolerations: + - key: nvidia.com/gpu + operator: Exists + effect: NoSchedule + - key: workload + value: bench + operator: Equal + effect: NoSchedule + - key: capacity-reservation + operator: Exists + effect: NoSchedule + containers: + - image: ${IMAGE_URI} + imagePullPolicy: IfNotPresent + name: deepep-v2-intranode-worker + env: + - name: NVIDIA_GDRCOPY + value: "enabled" + volumeMounts: + - name: shmem + mountPath: /dev/shm + resources: + limits: + nvidia.com/gpu: ${GPU_PER_NODE} + vpc.amazonaws.com/efa: ${EFA_PER_NODE} + requests: + nvidia.com/gpu: ${GPU_PER_NODE} + vpc.amazonaws.com/efa: ${EFA_PER_NODE} + # The tuning phase profiles kernels via CUPTI. These nodes set the + # driver's RmProfilingAdminOnly=1, so the workload needs SYS_ADMIN to + # profile. Remove if your cluster allows non-admin GPU profiling. + securityContext: + capabilities: + add: ["SYS_ADMIN", "IPC_LOCK"] + volumes: + - name: shmem + emptyDir: + medium: Memory + sizeLimit: 200Gi diff --git a/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/slurm/test-internode.sbatch b/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/slurm/test-internode.sbatch index 8f9a661e1..81f1ffd00 100644 --- a/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/slurm/test-internode.sbatch +++ b/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/slurm/test-internode.sbatch @@ -34,8 +34,6 @@ srun --mpi=pmix --cpu-bind=none \ # Pin the EFA-GDA GIN backend (type 5); unset, NCCL selects the CPU-proxy backend. export NCCL_GIN_TYPE=5 export NCCL_SYM_GIN_KERNELS_ENABLE=0 - export EP_NCCL_ROOT_DIR=$NCCL_HOME - export LD_PRELOAD=$NCCL_HOME/lib/libnccl.so.2 export TORCHDYNAMO_DISABLE=1 export TORCH_COMPILE_DISABLE=1 diff --git a/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/slurm/test-intranode.sbatch b/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/slurm/test-intranode.sbatch index 41adc961c..49c1de3d4 100644 --- a/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/slurm/test-intranode.sbatch +++ b/micro-benchmarks/expert-parallelism/deepep-v2-benchmark/slurm/test-intranode.sbatch @@ -29,7 +29,6 @@ srun --mpi=pmix --cpu-bind=none \ set -x export RANK=0 - export EP_NCCL_ROOT_DIR=$NCCL_HOME export TORCHDYNAMO_DISABLE=1 export TORCH_COMPILE_DISABLE=1