diff --git a/examples/inference/sglang/dsr1-deepep-efa/README.md b/examples/inference/sglang/dsr1-deepep-efa/README.md index 386542b05..082d243c3 100644 --- a/examples/inference/sglang/dsr1-deepep-efa/README.md +++ b/examples/inference/sglang/dsr1-deepep-efa/README.md @@ -174,11 +174,14 @@ Three things to know before running this on Blackwell: `setup/env_vars` derives `IFACE` from the default route, so the interface name is not something you need to know per instance family. -**Serving on Blackwell is not validated here.** The DeepEP-EFA kernels are — the same -`567632d` + EFA patch, same NVSHMEM 3.7.0, is measured out to 256 ranks on `p6-b300` in -[`ep-backend-comparison`](../../../../micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md). -The serving-side gap matters, because **on B300 the published SGLang comparison goes the other -way**: see [Blackwell: expect DeepEP to lose at 2 nodes](./benchmarks/README.md#blackwell-expect-deepep-to-lose-at-2-nodes). +**Serving on Blackwell is not validated here.** The older B300 backend comparison previously cited +here has been retired because its backend-native timing and byte accounting did not support a +cross-backend ranking. The replacement +[`ep-backend-comparison`](../../../../micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md) +defines common boundaries for synthetic Decode-like and Prefill-like communication workloads at +EP16 and EP32 on B200. Its replacement result matrix is pending and does not establish B300 serving +behavior. See +[Blackwell serving needs a matched measurement](./benchmarks/README.md#blackwell-serving-needs-a-matched-measurement). ## Smoke-test the EFA transport before loading the model diff --git a/examples/inference/sglang/dsr1-deepep-efa/benchmarks/README.md b/examples/inference/sglang/dsr1-deepep-efa/benchmarks/README.md index 15f2412a6..51758dd82 100644 --- a/examples/inference/sglang/dsr1-deepep-efa/benchmarks/README.md +++ b/examples/inference/sglang/dsr1-deepep-efa/benchmarks/README.md @@ -814,60 +814,35 @@ it was added for was real on some host during bring-up, just not one that has be since. Treat it as a knob to try if NVSHMEM init fails on a `normal` role, not as a requirement — and do not conclude from a passing `normal` run that your host disagrees with this table. -# Blackwell: expect DeepEP to lose at 2 nodes - -The serving tables in this document are H200. The question comes up — *"we benchmarked DeepEP vs the -NCCL all-to-all on 2× B300 and DeepEP was slower in every configuration; is that expected?"* — and -the answer from the data that **is** here is **yes at 2 nodes, and it is not an EFA problem.** -Reported shape of such a result: output throughput −7% to −26%, median TTFT +17% to +82%, P99 ITL -1.2–1.9 s vs 0.8–0.9 s, `normal` (HT) slowest where it ran, at TP16/EP16 across two nodes, 8K input -/ 1K output, concurrency 128. - -Read this section as a **mechanism argument extrapolated from H200 serving data plus B300 kernel -data** — not as a Blackwell serving measurement. No B300 serving sweep exists here yet. - -1. **16 ranks is DeepEP's worst case on the hardware measured here, and the mechanism is not - GPU-specific.** Every table in this document says so: the colocated decode sweep has DeepEP at - 0.23–0.75× the baseline's throughput and 1.4–4.3× its TPOT; the 2P2D sweep 0.55–0.71× at 1.4–1.8× - TPOT. The reason — 256 experts over 16 ranks means 16 experts per GPU, so the fan-out is small and - mostly intra-node NVLink, and DeepEP's per-layer dispatch/combine cost is not amortised — is a - function of EP width and expert count, not of the GPU generation. That makes a −7% to −26% - *aggregate* regression on a mixed 8K/1K workload **milder than what we measure on Hopper**, though - confirming the H200 magnitude carries to B300 requires the B300 sweep. - **One Blackwell configuration note either way:** on B200 every rank logs `Only use 20 SMs for DeepEP - communication ... Consider using --deepep-config`, so any Blackwell number taken with the default - config is a **floor**, not DeepEP's best (thanks @KeitaW for the observation). -2. **The published B300 kernel numbers are healthy, which localises the gap above the transport.** - At 2 nodes / 16 ranks on `p6-b300`, DeepEP-over-EFA dispatch/combine is **126.6 / 106.4 GB/s** — - best of the three backends there, *above* the NCCL all-to-all's 104.9 GB/s at matched payload - ([`ep-backend-comparison`](../../../../../micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md)). - So fabric and kernels are fine at that scale; the serving regression is per-layer - launch/scheduling overhead and MoE-runner choice, not bytes on the wire. -3. **`normal`/HT being slowest at decode-heavy concurrency is by design.** On a 1K-output workload - the run is TPOT-dominated, where `low_latency` is the intended mode. - -Before concluding anything from such a run, eliminate these: - -- **The four settings in [How to measure this correctly](#how-to-measure-this-correctly)** — the - harness defaults and the mode pinning in particular. `--random-range-ratio` alone moved a nominal - 256/512/conc-64 point from 374 to 1127 tok/s. -- **The MoE runner is not held constant** in the usual formulation: DeepEP rows run - `--moe-runner-backend deep_gemm` while the no-DeepEP rows resolve `auto` to `flashinfer_trtllm` on - Blackwell. That is two variables, and TRT-LLM's Blackwell MoE kernels are heavily tuned. Re-run - DeepEP against `flashinfer_trtllm` (or the baseline against `deep_gemm`) before attributing the - delta to the all-to-all. -- **DeepGEMM JIT warmup.** It inflates early TTFT and P99 ITL specifically — the two metrics that - move most in reports like this. Pre-warm on **both** nodes (`recipe/serve-pd.sh precompile`). -- **An HT-path hang that does not reproduce.** At ≥128 ranks the NVSHMEM-libfabric host proxy - exhausts libfabric retries (`EAGAIN` in `nvshmemi_process_multisend_rma`) and kills a different - pair of ranks each run — a documented statistical fan-out limit, not a bad node. At 16 ranks it - should not fire, but a non-reproducing hang on the HT path has the same signature. - -**The load-bearing point for a large fleet: 2 nodes measures the wrong thing.** DeepEP is built for -EP domains where experts are spread thin enough that every token crosses the fabric. Kernel scaling -to 256 ranks on `p6-b300` is already characterised: the useful envelope is ~64–160 ranks, with hard -implementation caps past that (HT: 160 PEs at `deep_ep.cpp:158`; low-latency: between 64 and 128 -PEs). **A production-EP-width run — EP32 or EP64, not EP16 — is the measurement that decides this.** +# Blackwell serving needs a matched measurement + +The serving tables in this document are H200. No matched B200 or B300 serving sweep is reported +here, so the H200 direction must not be assumed to carry across GPU generations. + +The current +[`ep-backend-comparison`](../../../../../micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md) +defines common CUDA timing boundaries and common logical payload accounting for synthetic +Decode-like and Prefill-like dispatch-plus-combine workloads. The replacement result matrix is +pending. It excludes expert compute, scheduling, communication/computation overlap, and +request-level serving behavior, and it does not provide a B300 serving result. + +Before attributing a Blackwell serving delta to the all-to-all backend, control these variables: + +- Apply the 4 settings in [How to measure this correctly](#how-to-measure-this-correctly), including + the harness defaults and explicit mode pinning. `--random-range-ratio` alone moved a nominal + 256-token/512-token/concurrency-64 point from 374 tokens/s to 1,127 tokens/s in the H200 campaign. +- Hold the MoE runner constant. DeepEP rows commonly select `deep_gemm`, while no-DeepEP rows can + resolve `auto` to `flashinfer_trtllm` on Blackwell. Changing both the MoE kernel and communication + backend does not isolate either effect. +- Pre-warm DeepGEMM JIT on every node with `recipe/serve-pd.sh precompile` before collecting TTFT or + P99 ITL. +- Match model, EP and TP widths, input and output lengths, concurrency, request seeds, image digests, + runtime versions, and named nodes. Rotate backend order across independent process starts. +- Compare request-level throughput and latency directly. Treat each backend's native GB/s fields as + diagnostics unless they share the same timing boundary and byte numerator. +- Test the production EP width. An EP16 result cannot decide an EP32 or EP64 deployment. + +The Blackwell serving conclusion remains unmeasured until that matched sweep is run. # Reproduce diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md b/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md index fbaab92f3..171a4e925 100644 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/README.md @@ -1,177 +1,203 @@ -# Expert-Parallelism Backend Comparison (NCCL vs UCCL vs NVSHMEM) on EKS +# Expert-Parallelism Backend Comparison on EFA -Head-to-head MoE dispatch/combine micro-benchmark across three communication backends, run -at the **same EP world size** on the same GPU nodes (designed for 8× `p6-b300.48xlarge`, -64 ranks; also exercised at **32 nodes / 256 ranks** — see -[Scaling beyond 8 nodes](#scaling-beyond-8-nodes-256-rank-findings) for the hard backend -limits that appear there). This directory is the orchestration layer; the benchmarks -themselves live in the sibling directories. +This directory compares 3 expert-parallel dispatch/combine backends through common Decode-like and Prefill-like communication workloads: -| Config | What it is | Source benchmark | -|---|---|---| -| **NCCL** (baseline) | Raw all-to-all over EFA. The transport-level **reference ceiling** — moves bytes, but does *not* do token routing or combine-reduction. | [`nccl-alltoall.yaml`](nccl-alltoall.yaml) (built from [`../../nccl-tests`](../../nccl-tests)) | -| **UCCL** | DeepEP-style dispatch/combine over the UCCL all-to-all backend. | [`../uccl-ep-benchmark/kubernetes`](../uccl-ep-benchmark/kubernetes) | -| **NVSHMEM** | DeepEP dispatch/combine over NVSHMEM (libfabric/EFA). | [`../deepep-benchmark/kubernetes`](../deepep-benchmark/kubernetes) | +| Backend | Implementation and transport | +|---|---| +| UCCL | DeepEP-compatible dispatch/combine over UCCL all-to-all and EFA | +| DeepEP V1 NVSHMEM | DeepEP V1 `Buffer` over NVSHMEM, libfabric, and EFA | +| DeepEP V2 NCCL GIN | DeepEP V2 `ElasticBuffer` over NCCL GIN EFA-GDA | + +Each workload uses one external CUDA timing boundary and one logical payload definition. Backend-native latency and bandwidth fields remain diagnostics because they do not share one timing boundary or byte numerator. + +The B200 report and backend box plots are in [RESULTS.md](RESULTS.md). EP32 means 32 GPU ranks on 4 B200 nodes, not 32 B200 nodes. + +## Workload profiles + +| Profile | Tokens | UCCL and DeepEP V1 API | DeepEP V2 API | Primary metric | +|---|---:|---|---|---| +| Decode-like | 128 tokens/rank | `low_latency_dispatch` and `low_latency_combine` | `ElasticBuffer.dispatch` and `ElasticBuffer.combine` | Slowest-rank latency, in ms | +| Prefill-like | 4,096 tokens/rank | Normal `Buffer.dispatch` and `Buffer.combine` | `ElasticBuffer.dispatch` and `ElasticBuffer.combine` | Slowest-rank latency, in ms | + +The Prefill-like timing boundary starts with the dispatch input and exact route ready. It includes the dispatch layout required by the normal UCCL and DeepEP V1 APIs, dispatch, and combine completion. The Decode-like boundary starts with the dispatch input ready and includes dispatch and combine completion. Host-side FP8 conversion runs once before either boundary, so no arm is charged for its Python quantizer. In the Decode-like FP8 cell the low-latency UCCL and DeepEP V1 kernels still quantize internally, inside the boundary; each backend is measured from its own API's dispatch-ready entry point. -> **Why no "DeepEP without a backend"?** DeepEP at the pinned commit (`567632d`, pre-EPv2) has -> no internode dispatch/combine path without a transport backend, so a literal "no-backend -> DeepEP on 8 nodes" does not exist. The NCCL all-to-all stands in as the neutral baseline and -> is labelled as a transport ceiling, not as an equal dispatch/combine number. +Both profiles hold the following controls constant across backends: -## Matched configuration (what makes the numbers comparable) +| Control | Rule | +|---|---| +| Input | One deterministic BF16 tensor per profile and EP size | +| Routing | One exact top-k route and one set of weights, verified by SHA-256 across all arms and starts | +| Model shape | Hidden size 7,168, 256 experts, top-k 8 experts/token | +| Operations | FP8 or BF16 dispatch followed by BF16 combine | +| Rank reduction | Maximum elapsed time across all ranks for each measured iteration | +| SM count | Each result records the communication-kernel SM count (`num_sms`) where the backend exposes one; `EP_NUM_SMS` pins it explicitly, and 0 keeps each backend's automatic choice | +| Warmup | 20 warmup iterations per dtype and process start | +| Measurement | 100 measured iterations per dtype and process start | +| Replication | `INDEPENDENT_STARTS` independent process starts per arm and workload cell, default 20; the reported campaigns used 16 and 4 | +| Order | Backend, dtype, and workload-profile order rotate across starts | +| Hardware | The same named nodes serve every arm at a given EP size | +| Runtime | Every result reports the same GPU, PyTorch, CUDA, and torch-built NCCL versions, plus the NCCL library actually loaded (consistent within each arm) | +| Correctness | Every rank passes the common identity-expert result before timing | + +Each process start contributes its median of 100 slowest-rank iteration measurements. The report then takes the median across the independent process starts. Iterations within one process are not treated as independent replicates. + +## Common logical throughput + +Each valid expert assignment contributes the dispatch tensor, FP8 scales when FP8 dispatch is selected, and the BF16 combine tensor. Backend metadata is excluded. Scale-out logical bytes include only assignments whose destination expert is on another node. + +```text +logical GB/s/rank = average logical bytes/rank / median slowest-rank latency +scale-out logical GB/s/rank = average remote logical bytes/rank / median slowest-rank latency +``` + +These are secondary logical efficiency metrics, not observed wire bandwidth. Within one profile, EP size, and dtype cell the byte numerator is a shared constant, so they rank backends identically to the primary latency metric. Compare across EP sizes with aggregate input tokens/s instead: the scale-out numerator counts every remote assignment as a full tensor, while a backend may send one copy per destination node and forward locally, so its over-count factor changes with node count and the scale-out column is comparable only within one EP size. -All runs use the **same EP problem size** — otherwise the table is meaningless: +## Files -| Parameter | Value | +| File | Purpose | |---|---| -| World size | 8 nodes × 8 GPU = **64 ranks** | -| `num-tokens` | 4096 (internode) / 128 (low-latency) | -| `hidden` | 7168 | -| `num-topk` | 8 | -| `num-experts` | 256 (divides evenly across 64 ranks) | -| dtype | bf16 | +| [`ep_benchmark.py`](ep_benchmark.py) | Workload profiles, backend adapters, correctness checks, CUDA timing, and logical-byte accounting | +| [`run_ep_rank.sh`](run_ep_rank.sh) | Per-node `torchrun` entry point and backend-specific transport environment | +| [`run_ep_comparison.sh`](run_ep_comparison.sh) | EKS admission, shared-Lease coordination, rotated matrix, durable harvest, and verified teardown | +| [`result_io.py`](result_io.py) | Robust result-marker parsing from interleaved native output | +| [`extract_results.py`](extract_results.py) | Canonical JSONL extraction from a rank-zero log | +| [`summarize_results.py`](summarize_results.py) | Matrix validation, per-start aggregation, paired deltas, and bootstrap intervals | +| [`plot_results.py`](plot_results.py) | Box plots of the independent-start primary values for every backend arm | +| [`RESULTS.md`](RESULTS.md) | Human-readable result, provenance, and scope limits | + +## Requirements + +The scored B200 matrix requires: + +- enough named, Ready `p6-b200.48xlarge` nodes (`EP_INSTANCE_TYPE`) in one EKS cluster for the largest configured EP size (`EP_WORLD_SIZES`, default `16 32`, needs 4 nodes; a 2-node cluster can run `EP_WORLD_SIZES=16`); `EP_REGION` and `EP_CLUSTER_NAME` label the campaign provenance; +- 8 allocatable GPUs and 8 allocatable EFA devices on every selected node; +- no active GPU requests on the selected nodes before each arm; +- the NVIDIA and EFA Kubernetes device plugins; +- `uvm_disable_hmm=Y` or `uvm_disable_hmm=1` on every selected host; +- `/dev/gdrdrv` as a character device on every selected host; +- `aws`, `kubectl`, `jq`, `rg`, Python 3, and Bash on the launch host; and +- access to the 3 digest-pinned backend images. -The UCCL manifests bake these args into the `python3 bench/test_*.py` invocation; the DeepEP -test hard-codes its config in-image. **Before running, confirm the DeepEP image's config is the -anchor** and align UCCL to it: +DeepEP V2 receives an INFO-level Decode-like admission run at the smallest configured EP size before the scored matrix. All 3 backends then receive a Prefill-like admission run at that size. A missing HMM mitigation, GDRCopy device, GDAKI proof, or profile correctness result stops the campaign before scoring. + +## Run on an exclusive node set + +Use a unique namespace and durable artifact directory. `KUBECTL_CONTEXT` is required explicitly so a concurrent process changing the default context cannot redirect the campaign. ```bash -# Read the DeepEP test config from the NVSHMEM image and match UCCL's CLI args to it. -docker run --rm ${NVSHMEM_IMAGE_URI} sed -n '1,60p' /DeepEP/tests/test_internode.py +campaign_id=ep-b200-$(date -u +%Y%m%d%H%M%S) +CAMPAIGN_ID="${campaign_id}" \ +EP_BENCHMARK_NODES=node-a,node-b,node-c,node-d \ +PROTECTED_NODES_CSV="" \ +ARTIFACT_ROOT="/shared/artifacts/${campaign_id}" \ +KUBECTL_CONTEXT=aps1-shared \ +LOCK_MODE=exclusive \ +./run_ep_comparison.sh ``` -If the DeepEP values differ from 4096/7168/8/256, edit the bench args in -`../uccl-ep-benchmark/kubernetes/test-*.yaml` to match. +`LOCK_MODE=exclusive` claims the configured shared Lease only when its holder is empty. The runner releases only a Lease that it still owns. + +## Run beside a coordinated campaign -## Prerequisites +Observe mode is allowed only when the other campaign has a known Lease holder and a disjoint named node set: -- EKS cluster with EFA + GPU nodes; NVIDIA device plugin + AWS EFA device plugin; Kubeflow MPI - Operator (`kubectl get crd mpijobs.kubeflow.org`). See each benchmark's `kubernetes/README.md`. -- The container images in ECR: - - NVSHMEM: `../deepep-benchmark/deepep.Dockerfile` (CUDA 13, `sm_90`+`sm_100`) - - UCCL: `../uccl-ep-benchmark/uccl-ep.Dockerfile` (CUDA 13; pinned UCCL commit; Hopper + Blackwell via PTX) - - NCCL: **reuse the NVSHMEM/DeepEP image** — it already builds `/opt/nccl-tests/build/alltoall_perf` - with `sm_100` gencode, so no separate `nccl-tests` build is needed for the baseline. +```bash +campaign_id=ep-b200-$(date -u +%Y%m%d%H%M%S) +CAMPAIGN_ID="${campaign_id}" \ +EP_BENCHMARK_NODES=node-a,node-b,node-c,node-d \ +PROTECTED_NODES_CSV=foreign-node-a,foreign-node-b \ +ARTIFACT_ROOT="/shared/artifacts/${campaign_id}" \ +KUBECTL_CONTEXT=aps1-shared \ +LOCK_MODE=observe \ +EXPECTED_LOCK_HOLDER=foreign-campaign-id \ +./run_ep_comparison.sh +``` -## Account / cluster safety (run first) +Observe mode never mutates the shared Lease. It verifies the exact holder before every arm and again before aggregation. Any selected/protected node overlap or Lease-holder change stops the run. + +## Execution matrix + +The scored order uses 20 independent starts (`INDEPENDENT_STARTS`). Three rotation patterns cycle across the starts: + +| Start index mod 3 | Backend order | Profile order | Dtype order | +|---:|---|---|---| +| 1 | UCCL, DeepEP V1, DeepEP V2 | Decode-like, Prefill-like | FP8, BF16 | +| 2 | DeepEP V2, UCCL, DeepEP V1 | Prefill-like, Decode-like | BF16, FP8 | +| 0 | DeepEP V1, DeepEP V2, UCCL | Decode-like, Prefill-like | FP8, BF16 | + +The runner executes the rotation at each configured EP size in `EP_WORLD_SIZES` order, by default first at 16 ranks on 2 nodes and then at 32 ranks on 4 nodes. Arms run serially, and every StatefulSet and its GPU pods must be gone before the next arm is admitted. The default matrix contains 240 distributed process starts and 480 scored dtype results. + +## Durable artifacts and teardown + +The campaign writes the following layout under `ARTIFACT_ROOT`: + +```text +control/ + aws-caller-identity.json + fleet-nodes-before.json + fleet-nodes-after.json + fleet-pods-before.json + fleet-pods-after.json + provenance.json + selected-nodes.txt +runs/ + decode/ep16/{admission,measurement}-repeat-*// + decode/ep32/measurement-repeat-*// + prefill/ep16/{admission,measurement}-repeat-*// + prefill/ep32/measurement-repeat-*// +summary/ + summary.json + summary.md +teardown/ + namespace-delete.log + remaining-resources.json + shared-lease-after.json +CAMPAIGN_COMPLETE +SHA256SUMS +STATUS +``` + +Every rank log, rendered Pod manifest, Pod description, canonical rank-zero JSONL result, case status, input/route hash, and immutable image reference is retained. `CAMPAIGN_COMPLETE` is written only after the full scored matrix succeeds and teardown verifies that the owned namespace and labeled resources are absent. `SHA256SUMS` is generated after the final status markers. + +## Re-aggregate preserved logs ```bash -aws sts get-caller-identity # confirm the target account -kubectl config current-context # confirm the target cluster -kubectl get nodes -l node.kubernetes.io/instance-type=p6-b300.48xlarge # confirm $NUM_NODES schedulable -kubectl get crd mpijobs.kubeflow.org # confirm MPI Operator +python3 summarize_results.py /path/to/artifacts/runs \ + --starts=20 \ + --provenance=/path/to/artifacts/control/provenance.json \ + --json=/path/to/artifacts/summary/summary.json \ + --markdown=/path/to/artifacts/summary/summary.md ``` -## Run order (serial — each config needs all 8 nodes) +Pass `--world-sizes` for a reduced matrix, for example `--world-sizes=16` for a 2-node EP16-only campaign. Without the flag the summarizer derives the EP sizes from the loaded logs and validates the full arm/dtype/profile/start matrix for each derived size. -**Smoke first.** Before any 8-node job, run the single-node `test-intranode.yaml` for each EP -image. It validates the image, that `sm_100` actually runs on B300, and the launch path in -minutes instead of failing eight nodes deep. Intranode is NVLink-only (same for every backend), -so it is a smoke test, not a comparison row. +If a native library appends a diagnostic to the JSON marker's physical line, use the repository parser: ```bash -cp env_vars.example env_vars # then edit image URIs / topology -source env_vars - -# 1) NVSHMEM (DeepEP) -( cd ../deepep-benchmark/kubernetes - IMAGE_URI=$NVSHMEM_IMAGE_URI NUM_NODES=$NUM_NODES \ - envsubst '$IMAGE_URI $INSTANCE_TYPE $GPU_PER_NODE $EFA_PER_NODE $NUM_NODES' < test-internode.yaml | kubectl apply -f - - # ...wait, save logs, delete. Then low-latency -- see the override note below. ) - -# 2) UCCL (UCCL-EP) — one MPI rank per GPU (NP = NUM_NODES * GPU_PER_NODE) -( cd ../uccl-ep-benchmark/kubernetes - IMAGE_URI=$UCCL_IMAGE_URI NUM_NODES=$NUM_NODES NP=$NP \ - envsubst '$IMAGE_URI $INSTANCE_TYPE $GPU_PER_NODE $EFA_PER_NODE $NUM_NODES $NP' < test-internode.yaml | kubectl apply -f - - # ...then test-low-latency.yaml (already pinned to --num-experts=256) ) - -# 3) NCCL baseline (reuses the DeepEP image's alltoall_perf) -IMAGE_URI=$NCCL_IMAGE_URI \ -envsubst '$IMAGE_URI $INSTANCE_TYPE $GPU_PER_NODE $EFA_PER_NODE $NUM_NODES $NP' < nccl-alltoall.yaml | kubectl apply -f - +python3 extract_results.py rank-zero.log results.jsonl ``` -> **DeepEP low-latency at 8 nodes — required override.** The merged DeepEP low-latency manifest -> runs `python3 /DeepEP/tests/test_low_latency.py` with no args, so it uses the upstream default -> `--num-experts=288`. The test asserts `num_experts % num_ranks == 0`; at 8 nodes (64 ranks), -> `288 % 64 ≠ 0` and it aborts. Match the comparison's 256 by patching the rendered manifest: -> -> ```bash -> cd ../deepep-benchmark/kubernetes -> IMAGE_URI=$NVSHMEM_IMAGE_URI NUM_NODES=$NUM_NODES \ -> envsubst '$IMAGE_URI $INSTANCE_TYPE $GPU_PER_NODE $EFA_PER_NODE $NUM_NODES' < test-low-latency.yaml \ -> | sed 's#test_low_latency.py#test_low_latency.py --num-experts 256#' | kubectl apply -f - -> ``` -> -> (DeepEP internode defaults are already 4096/7168/8/**256**, so internode needs no override.) - -Save each launcher log (`kubectl logs <…-launcher> > .log`) and **delete the job before -the next run** (MPIJob names are fixed; re-applying collides, and each job needs all 8 nodes): +The summarizer rejects an incomplete matrix, correctness failure, mutable image tag, route/input mismatch, runtime-stack mismatch, or disagreement in common logical payload accounting. + +Regenerate the box plots from the committed machine-readable summary: ```bash -kubectl delete mpijob deepep-internode uccl-ep-internode nccl-alltoall # etc. +python3 plot_results.py results/b200-us-east-1-2026-09-01-ep16-n16.json \ + --output=results/b200-us-east-1-2026-09-01-ep16-n16-boxplots.png ``` -## Collate +Plot generation requires Matplotlib. Each box uses the independent process-start medians for one backend and workload cell. The plot also shows every underlying point. + +## Local validation ```bash -python3 collect_results.py \ - --nvshmem-internode nvshmem_internode.log \ - --nvshmem-lowlat nvshmem_lowlat.log \ - --uccl-internode uccl_internode.log \ - --uccl-lowlat uccl_lowlat.log \ - --nccl nccl_alltoall.log +python3 -m pytest -q test_ep_benchmark.py test_summarize_results.py +python3 -m py_compile \ + ep_benchmark.py result_io.py extract_results.py summarize_results.py plot_results.py +bash -n run_ep_comparison.sh run_ep_rank.sh +shellcheck run_ep_comparison.sh run_ep_rank.sh ``` -The parser reports, for internode, the **RDMA** leg of the "Best dispatch/combine" line (the -cross-node bottleneck — *not* the intra-node NVL number printed on the same line), and for the -NCCL baseline the busbw **at the EP per-rank payload size** (`num_tokens*hidden*2`, ~56 MiB -target; the power-of-two sweep reports the nearest sampled row, 64 MiB) as -well as the asymptotic peak. Record the table in [`RESULTS.md`](RESULTS.md) with image tags, -date, and any config deltas. Eyeball one real launcher log against the parser before trusting it. - -Results are recorded per platform: [`RESULTS.md`](RESULTS.md) (B300) and -[`RESULTS-p5.md`](RESULTS-p5.md) (P5/H100). For other instance types set `INSTANCE_TYPE` and -`EFA_PER_NODE` accordingly (e.g. `p5.48xlarge` exposes **32** EFA NICs vs **16** on `p6-b300`). - -## Scaling beyond 8 nodes (256-rank findings) - -The full matrix was pushed to 16 and 32 nodes (128 / 256 ranks) on a 32× `p6-b300` Capacity -Block on 2026-07-14. **Every DeepEP-class kernel hits a hard implementation limit between -65 and 256 ranks; only the NCCL reference runs at 256.** Details and the per-limit source -citations are in [`RESULTS.md`](RESULTS.md) ("32 / 16 nodes" section). Operational notes for -anyone re-running at scale: - -- **HT internode**: DeepEP asserts at >160 ranks (`NUM_MAX_NVL_PEERS 8 × NUM_MAX_RDMA_PEERS 20`, - `kernels/configs.cuh`) and its stock combine tuning tables already abort at 16 nodes; UCCL - overflows an `int32` buffer bound above 64 ranks. Treat the HT comparison as an - **8-nodes-per-EP-domain benchmark** — which matches how training deploys these kernels - (EP32/EP64 groups inside a larger world). -- **Low-latency**: both implementations cap between 64 and 128 ranks (UCCL: compile-time - signaling-buffer arena; NVSHMEM/DeepEP: libfabric host-proxy retry exhaustion with moving - victims per run). -- **GDRCopy at scale (NVSHMEM)**: past ~1 GiB of LL buffer, NVSHMEM grows its symmetric heap - dynamically and must register each chunk over libfabric via **GDRCopy inside the container**. - The manifests set `NVIDIA_GDRCOPY=enabled`, but some clusters' nvidia container toolkit - ignores it — if every rank dies at `mem_heap.cpp:1361 register_mem_handle failed` after a - `GDRCopy support not enabled` warning, hostPath-mount `/dev/gdrdrv` into the worker - (requires `privileged: true`) and ensure the host loads `gdrdrv` (gdrcopy-loader DaemonSet - or DLAMI). -- **NCCL at 32 nodes** works unmodified (`NUM_NODES=32`, `NP=256`); expect matched-size busbw - to drop vs 8 nodes (fan-out cost). - -## Caveats - -- **NCCL is a reference, not an equal.** `alltoall_perf` busbw is pure transport throughput; the - EP dispatch/combine numbers carry routing + reduction overhead, so they should sit *below* the - NCCL ceiling. Compare against the **matched-size** busbw, not the asymptotic peak. -- **Internode = RDMA leg.** DeepEP/UCCL print both an RDMA (cross-node) and an NVL (intra-node) - bandwidth on the same line; only the RDMA number reflects the inter-node transport being - compared. -- **`num-experts` must divide the world size.** Both tests assert `num_experts % num_ranks == 0`. - At 8 nodes (64 ranks) the comparison uses 256 (= 4/rank). The DeepEP low-latency default (288) - is not divisible by 64 and must be overridden (see the run-order note). -- **Toolchain.** All three images are CUDA 13 (NVSHMEM/NCCL share the DeepEP image; UCCL is - CUDA 13 per `uccl-ep.Dockerfile`), so there is no CUDA skew across backends. -- **UCCL bench scripts** are pulled from upstream `uccl/ep/bench` at image-build time and pinned - via `UCCL_COMMIT`. If upstream renames CLI flags, adjust the bench args in the UCCL - manifests. +## Scope limits + +These profiles measure synthetic dispatch-plus-combine communication. Prefill-like does not measure TTFT, and Decode-like does not measure TPOT. Neither profile measures expert compute, communication/computation overlap, end-to-end training, serving throughput, or end-to-end latency. Results apply only to the reported profile, EP size, routing distribution, hardware, and runtime stack. diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS-p5.md b/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS-p5.md deleted file mode 100644 index d9184ead4..000000000 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS-p5.md +++ /dev/null @@ -1,76 +0,0 @@ -# EP-Backend Comparison Results — P5 (H100) - -Same harness and matched config as [`RESULTS.md`](RESULTS.md) (B300), run on `p5.48xlarge` -(H100). Use this to compare the three dispatchers **across GPU generations**. - -## Environment - -| Field | Value | -|---|---| -| Date | 2026-06-22 | -| Hardware | `p5.48xlarge` on EKS (H100, 8 GPU + **32 EFA** / node) | -| EP config | num-tokens=4096 (LL: 128), hidden=7168, num-topk=8, num-experts=256, bf16 (identical to B300) | -| NVSHMEM image | `deepep:efa1.48.0-nvshmem3.7.0-deepep567632d-cuda13` (sm_90) | -| UCCL image | UCCL `0dc87eb`, CUDA 13 (Hopper sm_90 path of the committed `uccl-ep.Dockerfile`) | -| NCCL image | DeepEP image (`/opt/nccl-tests/build/alltoall_perf`) | - -> **Node bring-up note.** One of the 8 freshly-deployed P5 nodes initially had its NVSwitch -> **Fabric Manager stuck in `In Progress`** (every CUDA op on it failed with `error 802: -> system not yet initialized`, including a trivial 1-GPU test). The 4-node run was done on the -> 7 healthy nodes meanwhile; after that node was recycled the 8-node run completed on all 8. - -## 8 nodes — 64 ranks - -| Backend | Mode | Dispatch (GB/s) | Combine (GB/s) | -|---|---|---:|---:| -| NVSHMEM (DeepEP) | internode (RDMA) | 39.6 | 39.2 | -| NVSHMEM (DeepEP) | low-latency | 6.5 | 15.6 | -| UCCL (UCCL-EP) | internode (RDMA) | 29.9 | 26.7 | -| UCCL (UCCL-EP) | low-latency | 3.1¹ | 3.7¹ | - -| Reference (NCCL all-to-all) | Metric | GB/s | -|---|---|---:| -| busbw at EP payload (~64 MiB) | matched-size | 40.4 | -| busbw peak (asymptotic) | peak | 51.1 | - -¹ Same as B300, the standard FP8 low-latency path (`round_scale=False`) **passes** correctness at -64 ranks (max diff 1.07e-4 vs the 9e-4 FP8 tolerance — 8× margin) and gives these numbers. The -unpatched test aborts *first* on the coarser `round_scale=True` FP8 sub-case, which upstream DeepEP -exempts via `if not round_scale`; matching that gating recovers the bandwidth. The per-sub-case -errors are **identical to B300** (the reference is generated from fixed seeds), confirming this is a -quantization-recipe property, not GPU-arch. See [`RESULTS.md`](RESULTS.md) for the sub-case table. - -## 4 nodes — 32 ranks - -| Backend | Mode | Dispatch (GB/s) | Combine (GB/s) | -|---|---|---:|---:| -| NVSHMEM (DeepEP) | internode (RDMA) | 43.7 | 46.1 | -| NVSHMEM (DeepEP) | low-latency | 9.6 | 20.7 | -| UCCL (UCCL-EP) | internode (RDMA) | 38.2 | 30.2 | -| UCCL (UCCL-EP) | low-latency | 5.0 | 5.0 | - -| Reference (NCCL all-to-all) | Metric | GB/s | -|---|---|---:| -| busbw at EP payload (~64 MiB) | matched-size | 45.1 | -| busbw peak (asymptotic) | peak | 53.9 | - -## Observations (and the B300 contrast) - -- **The winner flips by GPU generation.** On **B300** (see `RESULTS.md`) UCCL matches/beats - NVSHMEM at 4 nodes and clearly wins at 8. On **P5/H100 the order reverses at both scales**: - NVSHMEM leads UCCL on internode (4n 44/46 vs 38/30; 8n **40/39 vs 30/27** GB/s RDMA) and on - low-latency (8n LL 6.5/15.6 vs 3.1/3.7). UCCL-EP's kernels lean on SM90+ features tuned for - Blackwell; on H100 they trail NVSHMEM here. **Pick the dispatcher per target GPU, not globally.** -- **Absolute bandwidth is ~half of B300.** P5 internode tops out ~40–46 GB/s (RDMA) and the NCCL - reference ~40–45 (matched) / ~51–54 (peak), versus ~73–96 / ~103–117 on B300 — a combination of - EFA throughput and NVLink-generation differences. -- **UCCL low-latency's 64-rank abort is a test-gate divergence, not a kernel fault.** The standard - `round_scale=False` FP8 path passes at 64 ranks (max diff 1.07e-4, identical on P5 and B300); the - default test aborts only on the coarser `round_scale=True` sub-case that DeepEP exempts via - `if not round_scale`. Matching that gating recovers the LL bandwidth (P5 3.1/3.7, B300 28.4/24.8). - See [`RESULTS.md`](RESULTS.md) for the sub-case breakdown. - -## Reproduce - -Identical to [`README.md`](README.md), with `INSTANCE_TYPE=p5.48xlarge` and `EFA_PER_NODE=32` -(p5.48xlarge exposes 32 EFA NICs vs 16 on p6-b300). diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md b/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md index b83549a34..47241118e 100644 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/RESULTS.md @@ -1,183 +1,123 @@ -# EP-Backend Comparison Results — B300 - -Generated by running the three backends at a matched EP config on `p6-b300.48xlarge` -nodes and collating with [`collect_results.py`](collect_results.py). For the **P5 (H100)** run -of the same matrix — and the cross-generation contrast (the winner flips by GPU) — see -[`RESULTS-p5.md`](RESULTS-p5.md). - -## Environment - -| Field | Value | -|---|---| -| Date | 2026-06-21 | -| Hardware | `p6-b300.48xlarge` on EKS (Blackwell B300, 8 GPU + 16 EFA / node) | -| EP config | num-tokens=4096 (LL: 128), hidden=7168, num-topk=8, num-experts=256, bf16 | -| NVSHMEM image | `deepep:efa1.48.0-nvshmem3.7.0-deepep567632d-cuda13` (CUDA 13) | -| UCCL image | UCCL `0dc87eb`, CUDA 13, Hopper+Blackwell — see image note below | -| NCCL image | DeepEP image (reuses `/opt/nccl-tests/build/alltoall_perf`, sm_100) | - -`num-experts=256` divides evenly at both 32 and 64 ranks, so the config is identical across the -4-node and 8-node runs. - -> **UCCL image note.** The UCCL dispatch/combine numbers below were measured with a prebuilt -> image of the **same UCCL commit (`0dc87eb`) and the same CUDA-13 `setup.py`+PTX build method** -> as the committed [`uccl-ep.Dockerfile`](../uccl-ep-benchmark/uccl-ep.Dockerfile). That committed -> Dockerfile is independently validated to **build** and to **run the benchmark on B300** — a -> 2-node internode run from it gives dispatch ~92 / combine ~60 GB/s (RDMA), consistent with the -> table — so it reproduces these results. - -## 32 / 16 nodes — 256 / 128 ranks (2026-07-14 scale run) - -A 32× `p6-b300.48xlarge` Capacity Block (**us-east-1-atl-2a local zone**, EKS, same images -as below) was used to sweep the same matched config from 2 to 32 nodes. Headline: -**every DeepEP-class kernel hits a hard implementation limit between 65 and 256 ranks — -at 256 ranks only the NCCL all-to-all reference runs.** - -**HT internode scaling (like-for-like, this cluster, 2026-07-14).** The matched config -(`num-experts=256`) is only *runnable* at power-of-2 node counts: -`tests/test_internode.py:30` asserts `num_experts % num_ranks == 0`, and 256 divides -16/32/64/128/256 ranks but not the 24/48/96/144/160 of DeepEP's other instantiated -shapes ({3, 6, 12, 18, 20} nodes — a menu cut for 288-expert models, where those counts -do divide). Dispatch/combine are the RDMA leg in GB/s: - -| nodes | ranks | NVSHMEM (DeepEP) disp / comb | UCCL disp / comb | NCCL matched / peak | -|---|---|---|---|---| -| 2 | 16 | **126.6 / 106.4** | 91.9 / 59.6 | 104.9 / 179.6 | -| 4 | 32 | 97.1 / 95.4 | **102.1 / 95.3** | 94.0 / 116.9 | -| 8 | 64 | 84.2 / 73.1 | **93.9 / 90.5** | 74.0 / 103.2 | -| 16 | 128 | 74.7 / tuning abort¹ | constructor abort³ | 73.8 / 84.0 | -| 32 | 256 | constructor abort² | constructor abort³ | 54.7 / 74.4 | - -Two reads: **(a)** the 8-node row reproduces the June `us-west-2` primary table below -within ~1% on a different cluster (NVSHMEM 84.2/73.1 vs 83.7/72.7; UCCL 93.9/90.5 vs -93.4/90.7; NCCL 74.0 vs 72.5) — strong cross-cluster reproducibility for these -benchmarks. **(b)** the winner flips with scale: NVSHMEM leads at 2 nodes, UCCL from -4 nodes up — per-rank bandwidth decays smoothly for all three as fan-out grows. - -**Low-latency kernels** cap between 64 and 128 ranks on both implementations -(⁴ and ⁵ below); at 256 ranks nothing but NCCL runs: - -| Backend / kernel | 128 ranks (16n) | 256 ranks (32n) | Limit (source) | -|---|---|---|---| -| NVSHMEM (DeepEP) low-latency | init traffic abort⁴ | init traffic abort⁴ | 64 < cap ≤ 128 PEs (host-proxy) | -| UCCL (UCCL-EP) low-latency | buffer-config abort⁵ | buffer-config abort⁵ | 64 < cap ≤ 128 ranks | - -¹ `internode.cu:2363` — `num_max_nvl_chunked_recv_tokens / num_rdma_ranks > max(send chunk)`: -the shipped chunk configs were tuned for ≤8-node EP domains; at 16 RDMA peers the combine -sweep violates the constraint. Dispatch (74.7 GB/s RDMA, ~143 GB/s NVL) is a valid datapoint. -² `deep_ep.cpp:158` — `num_ranks <= NUM_MAX_NVL_PEERS(8) × NUM_MAX_RDMA_PEERS(20) = 160 or -low_latency_mode`. DeepEP v1's HT kernels decompose every rank into `(rdma_rank, nvl_rank)` -against fixed 8×20 compile-time tables (`kernels/configs.cuh`). The same assert block also -bounds `num_rdma_bytes <= INT_MAX` for HT — waiting behind the peer cap even if it were raised. -³ `uccl_ep.cc:431` — HT `num_rdma_bytes` (∝ ranks at matched config) exceeds `INT_MAX` -somewhere between 64 ranks (8n, June: passes at 93 GB/s) and 128 ranks. -⁴ NVSHMEM 3.7 libfabric **host-proxy** retry exhaustion: `Max amount of libfabric retries -reached, -11 (EAGAIN)` in `nvshmemi_process_multisend_rma`, killing ~2 nodes per run with -**different victims each run** (4 runs: ranks {12,29}, {9,12}, {12}, {6,11}) — a statistical -fan-out limit of the single proxy thread at ≥128 PEs on EFA, not a bad node and not geometry. -Registration itself is solvable: the ≥1 GiB LL buffer forces dynamic (CUDA-VMM) heap growth -whose remote-chunk registration needs **GDRCopy in-container** — on clusters whose toolkit -ignores `NVIDIA_GDRCOPY=enabled`, hostPath-mount `/dev/gdrdrv` (privileged); a >2 GiB static -heap is no workaround (exceeds EFA's single-MR registration limit). -⁵ `ep_config.hpp:279` — LL per-peer signaling buffer (∝ ranks) exceeds the compile-time -`kAtomicBufferSize` arena at ≥128 ranks. - -**Reading.** DeepEP-class dispatchers are engineered for EP domains of ~64–160 ranks — -matching how training actually deploys them (EP32/EP64 groups inside a larger world; the -[`kimi-k2`](../../../examples/training/megatron-bridge/kimi-k2/benchmarks/RESULTS.md) -NVSHMEM arm ran clean on 256 GPUs precisely because its `deep_ep` domains are 32-rank EP -groups). A *flat* EP domain >64 ranks is already off the map for both low-latency paths on -EFA, and >160 for HT; NCCL all-to-all is the only working option there. NCCL's per-rank -busbw at the matched ~56 MiB payload decays smoothly with fan-out across the whole sweep: -104.9 → 94.0 → 74.0 → 73.8 → 54.7 GB/s over 16 → 256 ranks (like-for-like, this cluster). - -## 8 nodes — 64 ranks (primary) - -| Backend | Mode | Dispatch (GB/s) | Combine (GB/s) | -|---|---|---:|---:| -| NVSHMEM (DeepEP) | internode (RDMA) | 83.7 | 72.7 | -| NVSHMEM (DeepEP) | low-latency | 9.2 | 18.6 | -| UCCL (UCCL-EP) | internode (RDMA) | 93.4 | 90.7 | -| UCCL (UCCL-EP) | low-latency | 28.4¹ | 24.8¹ | - -¹ The standard FP8 low-latency path (`round_scale=False`) **passes** correctness at 64 ranks and -gives these numbers. The unpatched test still aborts *first* on the coarser `round_scale=True` FP8 -sub-case — which upstream DeepEP exempts from the same check — so reproducing with the committed -manifest shows the abort; matching DeepEP's gating recovers the bandwidth. See the note below. - -| Reference (NCCL all-to-all) | Metric | GB/s | -|---|---|---:| -| busbw at EP payload (~64 MiB) | matched-size | 72.5 | -| busbw peak (asymptotic) | peak | 103.1 | - -> **UCCL low-latency at 64 ranks: the abort is a test-gate divergence, not a kernel precision regression.** -> `bench/test_low_latency.py` validates a dispatch→(identity GEMM)→combine round-trip against the -> closed-form reference `current_x · Σ(top-k gate weights)`, using the global similarity error -> `calc_diff(x,y) = 1 − 2·⟨x,y⟩ / (‖x‖²+‖y‖²)`. UCCL asserts `diff < 9e-4` for **every** FP8 -> sub-case: -> -> ```python -> assert diff < (9e-4 if dispatch_use_fp8_case else 1e-5) # uccl ep/bench/test_low_latency.py:373 -> ``` -> -> whereas upstream DeepEP gates the *same* assert with `if not round_scale` -> (`deepseek-ai/DeepEP tests/test_low_latency.py:178`) — it deliberately exempts the coarse, -> power-of-2-scale (`round_scale=True`) FP8 path. Instrumenting the run to log every sub-case's error -> at 64 ranks (256 samples per sub-case across the 64 ranks) shows the split exactly: -> -> | dispatch_use_fp8 | round_scale | max diff | tolerance | result | -> |---|---|---:|---:|---| -> | False (bf16) | — | 1.7e-6 | 1e-5 | pass | -> | True | **False** | **1.07e-4** | 9e-4 | **pass (8× margin)** | -> | True | **True** | **1.80e-3** | 9e-4 | fail | -> -> So the **standard `round_scale=False` FP8 path is numerically correct at 64 ranks** (and at 32); -> the default test aborts only because UCCL applies the tight FP8 tolerance to the `round_scale=True` -> sub-case that DeepEP does not gate. `round_scale=True` is a coarser (power-of-2 / UE8M0-family) -> scaling recipe whose larger quantization error is expected — hence DeepEP's exemption. The -> dispatch/combine numbers in the table are the `round_scale=False` FP8 timed phase, recovered by -> matching DeepEP's gating (skipping the `round_scale=True` assert) — **not** by loosening the -> tolerance on the path the table measures. - -## 4 nodes — 32 ranks (scaling reference) - -| Backend | Mode | Dispatch (GB/s) | Combine (GB/s) | -|---|---|---:|---:| -| NVSHMEM (DeepEP) | internode (RDMA) | 97.2 | 95.6 | -| NVSHMEM (DeepEP) | low-latency | 11.2 | 25.2 | -| UCCL (UCCL-EP) | internode (RDMA) | 101.3 | 94.4 | -| UCCL (UCCL-EP) | low-latency | 44.5 | 45.3 | - -| Reference (NCCL all-to-all) | Metric | GB/s | -|---|---|---:| -| busbw at EP payload (~64 MiB) | matched-size | 93.9 | -| busbw peak (asymptotic) | peak | 116.9 | - -## Observations - -- **Internode high-throughput: a tie at 4 nodes, a UCCL win at 8.** At 32 ranks NVSHMEM and UCCL - are within a few percent (97/96 vs 101/94). At 64 ranks UCCL pulls ahead on both legs — - dispatch 93 vs 84 and **combine 91 vs 73 GB/s (RDMA)** — i.e. UCCL's combine degrades far less - as the RDMA peer count doubles. Bandwidth drops with scale for both backends (more RDMA peers, - more contention). -- **Low-latency (decode path): UCCL wins.** At 32 ranks UCCL delivers ~4× dispatch and ~1.8× - combine over NVSHMEM (44/45 vs 11/25). At 64 ranks UCCL leads on both legs too — **28.4/24.8 vs - 9.2/18.6** (~3× dispatch, ~1.3× combine) — once the test's `round_scale=True` gate (which DeepEP - exempts) is matched; the standard `round_scale=False` FP8 path passes at this scale (see note). -- **NCCL is a reference, not an equal.** `alltoall_perf` busbw is pure data movement (no token - routing / combine-reduction) and is measured differently from the EP kernels' RDMA-only, - NVL-overlapped bandwidth — so the EP numbers can sit *above* the NCCL matched-size busbw (e.g. - UCCL 93 vs 72.5 at 8 nodes) while staying below the asymptotic peak (103). Treat it as a - transport context line, not a hard ceiling. - -## Reproduce - -See [`README.md`](README.md). Logs were collected with `kubectl logs > .log` -and collated with: - -```bash -python3 collect_results.py \ - --nvshmem-internode nvshmem_internode.log --nvshmem-lowlat nvshmem_lowlat.log \ - --uccl-internode uccl_internode.log --uccl-lowlat uccl_lowlat.log \ - --nccl nccl_alltoall.log -``` +# EP Backend Comparison Results on B200 + +Status: `PASS` + +This is a synthetic expert-parallel communication microbenchmark. It measures one dispatch followed by one combine under a single external CUDA timing boundary, with the same deterministic input and route for every backend. + +Two campaigns are reported. The primary campaign covers EP16 with 16 independent process starts, which is the replication level at which the bootstrap intervals below carry information. The companion campaign covers EP16 and EP32 with 4 starts and exists to show the EP-size axis; its intervals are weak and its EP32 rows should be read as a direction, not a measurement. + +## Campaigns + +| | Primary | Companion | +|---|---|---| +| EP sizes | 16 ranks on 2 nodes | 16 ranks on 2 nodes, 32 ranks on 4 nodes | +| Independent starts per cell | 16 | 4 | +| Scored records | 192 | 96 | +| Campaign ID | `ep-b200-use1-n16-20260901054057` | `ep-b200-use1-20260901023226` | +| Harness commit | `40990df2` | `df6d2943` | +| Machine-readable summary | [`b200-us-east-1-2026-09-01-ep16-n16.json`](results/b200-us-east-1-2026-09-01-ep16-n16.json) | [`b200-us-east-1-2026-09-01-ep16-ep32-n4.json`](results/b200-us-east-1-2026-09-01-ep16-ep32-n4.json) | + +Both campaigns ran on the same 4 `p6-b200.48xlarge` nodes in EKS cluster `ml-clusters-shared-us-east-1`, `us-east-1c`, each node with 8 NVIDIA B200 GPUs and 8 allocatable EFA devices. Both used the same three digest-pinned images, hidden size 7,168 dimensions, 256 experts, top-k 8 experts/token, 20 warmup and 100 measured iterations per dtype and start, and PyTorch 2.13.0+cu130 with CUDA 13.0. Every arm loaded NCCL 2.31.2; the PyTorch build constant reports 2.29.7, which is why the summaries record both. + +Each process start contributes the median of its 100 slowest-rank iteration measurements, and each cell reports the median across independent starts. Iterations inside one process are not treated as independent replicates. + +## Relationship to the earlier ap-south-1 tables + +The previous report measured a different harness on a different cluster, so its numbers are not comparable row by row with the tables below and have been retired to [`b200-ap-south-1-2026-08-25.json`](results/b200-ap-south-1-2026-08-25.json) as a dated historical artifact. Three harness changes affect what the boundary contains: the host-side FP8 cast now runs once before the timed region instead of inside it, `EP_BUFFER_DEBUG` is off for scored runs instead of enabled on the DeepEP V2 arm only, and slowest-rank latency is the primary metric for both profiles. The FP8-slower-than-BF16 decode pattern visible in the earlier tables does not appear in these measurements. + +## Primary results: EP16, 16 independent starts + +![Box plots comparing the three backend arms for every Decode-like and Prefill-like workload cell at EP16](results/b200-us-east-1-2026-09-01-ep16-n16-boxplots.png) + +Each box spans Q1 to Q3, the center line is the median, the whiskers are the minimum and maximum, and the markers are the 16 per-start medians. + +DeepEP V2 has the lowest slowest-rank latency in all four cells. Against UCCL its margin is about 10% for FP8 decode and about 19% for BF16 decode, and roughly 39% to 50% for prefill; against DeepEP V1 it ranges from about 5% for BF16 prefill to about 54% for FP8 decode. Every direction is supported at the 5% run-to-run CV gate. DeepEP V2 is also faster with FP8 than with BF16 in both profiles, which is the expected ordering because FP8 puts fewer bytes on the wire. + +### Decode-like latency, 128 tokens/rank + +| EP size | Dispatch dtype | Backend | Latency (ms) | 95% bootstrap CI (ms) | Run-to-run CV (%) | Input throughput (tokens/s) | Logical throughput (GB/s/rank) | Scale-out logical throughput (GB/s/rank) | +|---:|:---:|:---|---:|:---:|---:|---:|---:|---:| +| 16 ranks | FP8 | UCCL | 0.5258 ms | [0.5237, 0.5284] ms | 0.93% | 3,895,017.08 tokens/s | 42.32 GB/s/rank | 21.16 GB/s/rank | +| 16 ranks | FP8 | DeepEP V1 NVSHMEM | 1.0264 ms | [1.0243, 1.0346] ms | 1.30% | 1,995,323.99 tokens/s | 21.68 GB/s/rank | 10.84 GB/s/rank | +| 16 ranks | FP8 | DeepEP V2 NCCL GIN | 0.4712 ms | [0.4664, 0.4792] ms | 2.45% | 4,346,582.46 tokens/s | 47.22 GB/s/rank | 23.61 GB/s/rank | +| 16 ranks | BF16 | UCCL | 0.5986 ms | [0.5943, 0.6012] ms | 0.73% | 3,421,595.90 tokens/s | 49.05 GB/s/rank | 24.53 GB/s/rank | +| 16 ranks | BF16 | DeepEP V1 NVSHMEM | 1.0358 ms | [1.0327, 1.0392] ms | 0.66% | 1,977,201.79 tokens/s | 28.35 GB/s/rank | 14.17 GB/s/rank | +| 16 ranks | BF16 | DeepEP V2 NCCL GIN | 0.4839 ms | [0.4810, 0.5229] ms | 4.47% | 4,231,973.48 tokens/s | 60.67 GB/s/rank | 30.33 GB/s/rank | + +### Prefill-like latency, 4,096 tokens/rank + +| EP size | Dispatch dtype | Backend | Latency (ms) | 95% bootstrap CI (ms) | Run-to-run CV (%) | Input throughput (tokens/s) | Logical throughput (GB/s/rank) | Scale-out logical throughput (GB/s/rank) | +|---:|:---:|:---|---:|:---:|---:|---:|---:|---:| +| 16 ranks | FP8 | UCCL | 5.2691 ms | [5.2664, 5.2789] ms | 0.24% | 12,437,731.75 tokens/s | 135.12 GB/s/rank | 67.56 GB/s/rank | +| 16 ranks | FP8 | DeepEP V1 NVSHMEM | 2.9742 ms | [2.9700, 2.9863] ms | 0.58% | 22,034,892.87 tokens/s | 239.39 GB/s/rank | 119.69 GB/s/rank | +| 16 ranks | FP8 | DeepEP V2 NCCL GIN | 2.6578 ms | [2.6517, 2.6614] ms | 0.25% | 24,657,617.21 tokens/s | 267.88 GB/s/rank | 133.94 GB/s/rank | +| 16 ranks | BF16 | UCCL | 5.5088 ms | [5.5038, 5.5136] ms | 0.17% | 11,896,550.74 tokens/s | 170.55 GB/s/rank | 85.27 GB/s/rank | +| 16 ranks | BF16 | DeepEP V1 NVSHMEM | 3.5448 ms | [3.5397, 3.5513] ms | 0.39% | 18,487,721.74 tokens/s | 265.04 GB/s/rank | 132.52 GB/s/rank | +| 16 ranks | BF16 | DeepEP V2 NCCL GIN | 3.3709 ms | [3.3659, 3.3740] ms | 0.28% | 19,441,899.67 tokens/s | 278.72 GB/s/rank | 139.36 GB/s/rank | + +### Paired DeepEP V2 improvements + +Positive values mean DeepEP V2 had lower slowest-rank latency. A direction is supported only when the paired bootstrap interval excludes 0% and both arms have at most 5% run-to-run CV. + +| Profile | EP size | Dispatch dtype | Baseline | Median improvement (%) | 95% bootstrap CI (%) | Direction supported | +|:---|---:|:---:|:---|---:|:---:|:---:| +| decode | 16 ranks | FP8 | UCCL | 10.28% | [9.08, 11.28]% | yes | +| decode | 16 ranks | FP8 | DeepEP V1 NVSHMEM | 54.24% | [53.07, 55.00]% | yes | +| decode | 16 ranks | BF16 | UCCL | 18.78% | [12.02, 19.87]% | yes | +| decode | 16 ranks | BF16 | DeepEP V1 NVSHMEM | 53.25% | [49.48, 53.62]% | yes | +| prefill | 16 ranks | FP8 | UCCL | 49.58% | [49.44, 49.65]% | yes | +| prefill | 16 ranks | FP8 | DeepEP V1 NVSHMEM | 10.77% | [10.45, 11.04]% | yes | +| prefill | 16 ranks | BF16 | UCCL | 38.81% | [38.75, 38.92]% | yes | +| prefill | 16 ranks | BF16 | DeepEP V1 NVSHMEM | 4.95% | [4.76, 5.15]% | yes | + +## Companion campaign: the EP-size axis at 4 starts + +The companion campaign repeats EP16 and adds EP32. Its EP16 medians agree with the primary campaign to within 1.3% in every arm and cell except BF16 decode for DeepEP V2, which differs by 2.5% and is also the cell with the highest run-to-run variability in both campaigns. That agreement is the available evidence that the two campaigns measured the same thing. The EP32 rows come from 4 starts, so they indicate a direction rather than establishing a margin. + +At EP32 the decode ranking differs from EP16: UCCL leads both decode cells, at 0.7758 ms against 0.8361 ms for DeepEP V2 in FP8 and 0.8718 ms against 0.9423 ms in BF16. DeepEP V2 leads both prefill cells, at 6.6226 ms against 8.0894 ms for UCCL in FP8 and 8.7098 ms against 9.8898 ms in BF16. An EP16 decode result therefore does not carry to EP32 for these backends. + +| Profile | EP size | Dispatch dtype | Backend | Latency (ms) | 95% bootstrap CI (ms) | Run-to-run CV (%) | Input throughput (tokens/s) | +|:---|---:|:---:|:---|---:|:---:|---:|---:| +| decode | 32 ranks | FP8 | UCCL | 0.7758 ms | [0.7701, 0.7799] ms | 0.55% | 5,279,730.97 tokens/s | +| decode | 32 ranks | FP8 | DeepEP V1 NVSHMEM | 1.5568 ms | [1.5428, 1.5656] ms | 0.62% | 2,631,129.46 tokens/s | +| decode | 32 ranks | FP8 | DeepEP V2 NCCL GIN | 0.8361 ms | [0.8296, 0.8670] ms | 2.00% | 4,898,703.91 tokens/s | +| decode | 32 ranks | BF16 | UCCL | 0.8718 ms | [0.8695, 0.8757] ms | 0.30% | 4,698,111.16 tokens/s | +| decode | 32 ranks | BF16 | DeepEP V1 NVSHMEM | 1.5598 ms | [1.5465, 1.5696] ms | 0.69% | 2,626,007.14 tokens/s | +| decode | 32 ranks | BF16 | DeepEP V2 NCCL GIN | 0.9423 ms | [0.9300, 0.9612] ms | 1.54% | 4,347,061.52 tokens/s | +| prefill | 32 ranks | FP8 | UCCL | 8.0894 ms | [8.0691, 8.1183] ms | 0.26% | 16,202,922.98 tokens/s | +| prefill | 32 ranks | FP8 | DeepEP V1 NVSHMEM | 12.7493 ms | [12.6702, 12.8505] ms | 0.59% | 10,280,762.55 tokens/s | +| prefill | 32 ranks | FP8 | DeepEP V2 NCCL GIN | 6.6226 ms | [6.6190, 6.6343] ms | 0.11% | 19,791,699.50 tokens/s | +| prefill | 32 ranks | BF16 | UCCL | 9.8898 ms | [9.8781, 9.9021] ms | 0.10% | 13,253,295.68 tokens/s | +| prefill | 32 ranks | BF16 | DeepEP V1 NVSHMEM | 13.1769 ms | [13.1605, 13.2159] ms | 0.18% | 9,947,071.17 tokens/s | +| prefill | 32 ranks | BF16 | DeepEP V2 NCCL GIN | 8.7098 ms | [8.7034, 8.7265] ms | 0.11% | 15,048,865.63 tokens/s | + +Compare EP16 against EP32 through aggregate input tokens/s rather than through the logical throughput columns. Those columns are logical efficiency metrics, not observed wire bandwidth: the scale-out numerator counts every remote assignment as a full tensor, while a backend may send one copy per destination node and forward locally, so its over-count factor changes with node count and the column is comparable only inside one EP size. + +## Controls that differ across arms + +Each result now records the communication-kernel SM count where the backend exposes one, and the recorded values are not equal across arms. In the prefill cells DeepEP V2 resolved 64 SMs from its own bandwidth heuristic, while the normal-mode UCCL and DeepEP V1 buffers report 20. The prefill comparison is therefore not SM-matched, and part of the margin may be a consequence of that difference rather than of the transport. In the decode cells the low-latency UCCL and DeepEP V1 APIs take no SM count and record none, while DeepEP V2 again resolved 64. Pin the count with `EP_NUM_SMS` to measure the arms at equal SM budgets. + +The decode profile is also not API-symmetric. UCCL and DeepEP V1 use purpose-built low-latency kernels that quantize FP8 internally, inside the timed boundary, while DeepEP V2 has only `ElasticBuffer` on EFA and receives input that was quantized once before the boundary. Each backend is measured from its own API's dispatch-ready entry point. + +The heuristic input is also recorded: `detected_rdma_gigabytes_per_second` is 50.0 on these nodes, which is one 400 Gb/s EFA device and matches the one device per GPU that `p6-b200.48xlarge` provides. + +## Qualification and custody + +- Primary campaign: 4 admission cases and 96 scored distributed starts passed; 192 scored records passed the common correctness check; the durable checksum manifest holds 1,124 entries. +- Companion campaign: 4 admission cases and 48 scored distributed starts passed; 96 scored records passed the common correctness check; the durable checksum manifest holds 740 entries. +- Backend, profile, and dtype order rotated across starts. Every arm at a given EP size used the same named nodes. +- All three container images were pinned by SHA-256 digest and are the same images used for the retired ap-south-1 campaign. The DeepEP and UCCL source commits inside those images are not recorded, so a reader can establish that both campaigns used identical backend builds but cannot establish which upstream commit each contains. +- Runtime stack, input hash, route hash, payload accounting, and derived metrics were validated during aggregation. The loaded NCCL version is required to agree within each arm, not across arms, because each image carries its own build. +- With the pinned UCCL image, normal-mode explicit proxy destruction invalidates the CUDA context after result emission. Prefill-like UCCL workers therefore synchronize, flush their results, and exit the worker process. This cleanup happens after all timed iterations and does not change the timing boundary. +- The machine-readable summaries have SHA-256 `60d2917ad4fb1f4662b4c522c740927d1403ba6a6f23c762937562e524a6b7c1` (primary) and `f4edd9a172d9b93e0903fbb3d43b499bf17aaee62e4dd3db3d212bab440173ff` (companion). +- Both campaigns reported a teardown failure. Every owned Pod, StatefulSet, and Service was removed and the shared Lease was released, but the owned namespace stayed in `Terminating` because two stale `visibility.kueue.x-k8s.io` APIServices in this cluster fail namespace deletion discovery. The condition is a property of the cluster, not of a campaign, and no scored record depends on it. + +## Interpretation limits + +Prefill-like does not measure time to first token, and Decode-like does not measure time per output token. Neither profile measures expert computation, communication and computation overlap, end-to-end training, end-to-end serving throughput, or end-to-end latency. + +The route is balanced by construction: every token reaches 8 distinct experts and every expert carries identical load. Rankings under a skewed or group-limited gate are untested here. + +Every iteration synchronizes and reduces across ranks before the next one starts, so these are single-shot latencies rather than pipelined steady-state figures. They are not comparable against kernel-time figures from a profiler-based benchmark. + +Conclusions apply only to the reported payload, routing distribution, EP size, B200 hardware, and runtime stack. diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/collect_results.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/collect_results.py deleted file mode 100644 index 86406ad7c..000000000 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/collect_results.py +++ /dev/null @@ -1,162 +0,0 @@ -#!/usr/bin/env python3 -# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. -# SPDX-License-Identifier: MIT-0 -"""Collate EP-backend comparison logs into a single markdown table. - -Parses the launcher logs from the three benchmarks run at the same world size: - - * NVSHMEM (DeepEP) -- dispatch/combine bandwidth - * UCCL (UCCL-EP) -- dispatch/combine bandwidth - * NCCL (nccl-tests alltoall_perf) -- busbw (transport reference) - -Output formats parsed (DeepEP and UCCL print identically -- UCCL's bench is -derived from DeepEP's tests): - - internode (test_internode.py): - [tuning] Best dispatch (BF16): ... BW: GB/s (RDMA), GB/s (NVL) - [tuning] Best combine: ... BW: GB/s (RDMA), GB/s (NVL) - -> we report the RDMA leg (the cross-node bottleneck). Reporting the NVL leg - here would be wrong: it is the intra-node ~hundreds-GB/s number. - - low-latency (test_low_latency.py): - [rank N] Dispatch bandwidth: GB/s, avg_t=... | Combine bandwidth: GB/s, avg_t=... - -> single bandwidth per dispatch/combine (no RDMA/NVL split). - - NCCL alltoall_perf: the standard size-sweep table; busbw is column 7 - (out-of-place) / 11 (in-place). We report busbw at the sampled row whose size - is closest to the EP per-rank dispatch payload (num_tokens * hidden * 2 bytes, - ~56 MiB target; the power-of-two sweep lands on the 64 MiB row) AND the - asymptotic peak, because the peak overstates the transport ceiling relative to - EP's smaller messages. - -Both EP formats vary slightly across versions; if a value comes back N/A, print -the raw log and adjust the regexes. - -Usage: - python3 collect_results.py \ - --nvshmem-internode nvshmem_internode.log \ - --nvshmem-lowlat nvshmem_lowlat.log \ - --uccl-internode uccl_internode.log \ - --uccl-lowlat uccl_lowlat.log \ - --nccl nccl_alltoall.log \ - --nccl-target-bytes 58720256 -""" -import argparse -import re -import sys - -_BW = r"([0-9]+(?:\.[0-9]+)?)" -# internode: pull the RDMA leg from the "Best dispatch/combine" summary lines. -BEST_DISPATCH_RDMA = re.compile(r"Best dispatch[^\n]*?" + _BW + r"\s*GB/s\s*\(RDMA\)", re.I) -BEST_COMBINE_RDMA = re.compile(r"Best combine[^\n]*?" + _BW + r"\s*GB/s\s*\(RDMA\)", re.I) -# low-latency: "Dispatch bandwidth: X GB/s" / "Combine bandwidth: Y GB/s". -# Case-SENSITIVE on purpose: the aggregate line "Dispatch + combine bandwidth: Z" -# uses a lowercase "combine" and must NOT match the per-leg "Combine bandwidth:". -LL_DISPATCH = re.compile(r"Dispatch bandwidth:\s*" + _BW + r"\s*GB/s") -LL_COMBINE = re.compile(r"Combine bandwidth:\s*" + _BW + r"\s*GB/s") -# nccl-tests data row: size count type redop root time algbw busbw #wrong ... -NCCL_ROW_RE = re.compile(r"^\s*\d+\s+\d+\s+\w+") -NCCL_BUSBW_COLS = (7, 11) -# mpirun --tag-output prefixes every line with e.g. "[1,0]:". -TAG_PREFIX_RE = re.compile(r"^\[\d+,\d+\]:") - - -def _last(text, regex): - m = regex.findall(text) - return float(m[-1]) if m else None - - -def _max(text, regex): - m = regex.findall(text) - return max(float(x) for x in m) if m else None - - -def parse_internode(path): - """(dispatch_rdma, combine_rdma) GB/s from the Best-config summary lines.""" - if not path: - return None, None - with open(path) as f: - text = f.read() - # The "Best" line is printed once per config after tuning; take the last. - return _last(text, BEST_DISPATCH_RDMA), _last(text, BEST_COMBINE_RDMA) - - -def parse_lowlat(path): - """(dispatch, combine) GB/s. Per-rank lines; take the max across ranks.""" - if not path: - return None, None - with open(path) as f: - text = f.read() - return _max(text, LL_DISPATCH), _max(text, LL_COMBINE) - - -def parse_nccl(path, target_bytes): - """Return (busbw_at_target, size_at_target, busbw_peak) GB/s.""" - if not path: - return None, None, None - peak = None - best_at = None # (abs_size_delta, size, busbw) - with open(path) as f: - for line in f: - line = TAG_PREFIX_RE.sub("", line) - if line.lstrip().startswith("#") or not NCCL_ROW_RE.match(line): - continue - cols = line.split() - try: - size = int(cols[0]) - except ValueError: - continue - for idx in NCCL_BUSBW_COLS: - if idx >= len(cols): - continue - try: - bw = float(cols[idx]) - except ValueError: - continue - peak = bw if peak is None else max(peak, bw) - delta = abs(size - target_bytes) - if best_at is None or delta < best_at[0]: - best_at = (delta, size, bw) - if best_at is None: - return None, None, peak - return best_at[2], best_at[1], peak - - -def fmt(v): - return f"{v:.1f}" if isinstance(v, float) else "N/A" - - -def main(): - p = argparse.ArgumentParser() - p.add_argument("--nvshmem-internode") - p.add_argument("--nvshmem-lowlat") - p.add_argument("--uccl-internode") - p.add_argument("--uccl-lowlat") - p.add_argument("--nccl", help="NCCL alltoall_perf log (transport reference)") - p.add_argument("--nccl-target-bytes", type=int, default=4096 * 7168 * 2, - help="EP per-rank dispatch payload to read busbw at (default num_tokens*hidden*2)") - args = p.parse_args() - - nv_i_d, nv_i_c = parse_internode(args.nvshmem_internode) - nv_l_d, nv_l_c = parse_lowlat(args.nvshmem_lowlat) - uc_i_d, uc_i_c = parse_internode(args.uccl_internode) - uc_l_d, uc_l_c = parse_lowlat(args.uccl_lowlat) - nccl_at, nccl_size, nccl_peak = parse_nccl(args.nccl, args.nccl_target_bytes) - - out = sys.stdout - out.write("| Backend | Mode | Dispatch (GB/s) | Combine (GB/s) |\n") - out.write("|---|---|---:|---:|\n") - out.write(f"| NVSHMEM (DeepEP) | internode (RDMA) | {fmt(nv_i_d)} | {fmt(nv_i_c)} |\n") - out.write(f"| NVSHMEM (DeepEP) | low-latency | {fmt(nv_l_d)} | {fmt(nv_l_c)} |\n") - out.write(f"| UCCL (UCCL-EP) | internode (RDMA) | {fmt(uc_i_d)} | {fmt(uc_i_c)} |\n") - out.write(f"| UCCL (UCCL-EP) | low-latency | {fmt(uc_l_d)} | {fmt(uc_l_c)} |\n") - out.write("\n") - sz_mib = f"{nccl_size / 2**20:.0f} MiB" if isinstance(nccl_size, int) else "N/A" - out.write("| Reference (NCCL all-to-all, transport ceiling) | Metric | GB/s |\n") - out.write("|---|---|---:|\n") - out.write(f"| busbw at EP payload (~{sz_mib}) | matched-size | {fmt(nccl_at)} |\n") - out.write(f"| busbw peak (asymptotic, overstates ceiling) | peak | {fmt(nccl_peak)} |\n") - - -if __name__ == "__main__": - main() diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/env_vars.example b/micro-benchmarks/expert-parallelism/ep-backend-comparison/env_vars.example deleted file mode 100644 index 1a653a5f4..000000000 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/env_vars.example +++ /dev/null @@ -1,25 +0,0 @@ -# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. -# SPDX-License-Identifier: MIT-0 - -# Shared topology for the EP-backend comparison. Copy to env_vars (gitignored), -# edit, then source it before applying the per-backend manifests. -# cp env_vars.example env_vars && source env_vars - -# GPU node instance type and per-node device counts (p6-b300.48xlarge => 16 EFA). -export INSTANCE_TYPE=p6-b300.48xlarge -export GPU_PER_NODE=8 -export EFA_PER_NODE=16 -# 8 = primary 3-way comparison. 32 (256 ranks) works for NCCL and the low-latency kernels -# only — both HT internode paths hit hard caps past 64-160 ranks (see README "Scaling -# beyond 8 nodes"). -export NUM_NODES=8 -export NP=$((NUM_NODES * GPU_PER_NODE)) # ranks; used by the NCCL baseline (64 at 8 nodes) - -# Per-backend image URIs. -# NVSHMEM: built from ../deepep-benchmark/deepep.Dockerfile (CUDA 13, sm_90+sm_100). -# UCCL: built from ../uccl-ep-benchmark/uccl-ep.Dockerfile (CUDA 13; Hopper + Blackwell via PTX). -# NCCL: the DeepEP image already ships /opt/nccl-tests/build/alltoall_perf built with -# sm_100 gencode, so reuse it for the baseline -- no separate nccl-tests build. -export NVSHMEM_IMAGE_URI=.dkr.ecr..amazonaws.com/deepep:efa1.48.0-nvshmem3.7.0-deepep567632d-cuda13 -export UCCL_IMAGE_URI=.dkr.ecr..amazonaws.com/uccl-ep:efa1.48.0-uccl0dc87eb-cu13 -export NCCL_IMAGE_URI=$NVSHMEM_IMAGE_URI diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/ep_benchmark.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/ep_benchmark.py new file mode 100755 index 000000000..5e22566c1 --- /dev/null +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/ep_benchmark.py @@ -0,0 +1,871 @@ +#!/usr/bin/env python3 +"""Common-boundary DeepEP-compatible dispatch/combine benchmark. + +The benchmark deliberately avoids each backend's native timing and byte +accounting. Every arm receives the same deterministic BF16 input, exact route +indices, and top-k weights. A CUDA event pair surrounds the complete dispatch +followed by combine operation, and the slowest rank is the iteration latency. + +The decode profile uses each backend's low-latency path with 128 tokens/rank. +The prefill profile uses the normal high-throughput path with 4,096 tokens/rank +and includes any required dispatch-layout work. Host-side FP8 conversion runs +once before the timed region; low-latency kernels that quantize internally +still do so inside it. +""" + +from __future__ import annotations + +import argparse +import ctypes +import hashlib +import json +import math +import os +import statistics +import sys +import traceback +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +import torch +import torch.distributed as dist + + +RESULT_PREFIX = "ADAI_EP_RESULT " +SCHEMA_VERSION = 2 +NORMAL_NUM_NVL_BYTES = 2_000_000_000 +NORMAL_NUM_RDMA_BYTES = 1_000_000_000 +NORMAL_NUM_QPS_PER_RANK = 24 + + +def deepep_v2_build_lib(root: Path = Path("/opt/amazon/deepep-v2")) -> Path: + """Locate the single built DeepEP V2 package containing its C extension.""" + + candidates = [ + path + for path in sorted((root / "build").glob("lib.*")) + if any((path / "deep_ep").glob("_C*.so")) + ] + if len(candidates) != 1: + rendered = ", ".join(str(path) for path in candidates) or "none" + raise RuntimeError( + f"expected exactly one built DeepEP V2 package, found: {rendered}" + ) + return candidates[0] + + +def preload_backend(arm: str) -> None: + """Load V2 before NCCL initializes its OFI plugin and tuner libraries.""" + + if arm == "deepep-v2-gin-gda": + sys.path.insert(0, str(deepep_v2_build_lib())) + __import__("deep_ep") + + +def decode_nccl_version(value: int) -> list[int]: + """Split ncclGetVersion's integer encoding into major, minor, and patch.""" + + if value < 20_900: + return [value // 1_000, value % 1_000 // 100, value % 100] + return [value // 10_000, value % 10_000 // 100, value % 100] + + +def loaded_nccl_version() -> list[int] | None: + """Read the version of the NCCL library actually loaded in this process. + + torch.cuda.nccl.version() reports torch's compile-time constant, which + differs from the loaded library in the source-built-NCCL images GIN + requires. Returns None when no loaded libnccl can be queried. + """ + + try: + with open("/proc/self/maps") as maps: + paths = sorted({line.split()[-1] for line in maps if "libnccl.so" in line}) + for path in paths: + library = ctypes.CDLL(path) + version = ctypes.c_int(0) + if library.ncclGetVersion(ctypes.byref(version)) == 0: + return decode_nccl_version(version.value) + except (OSError, AttributeError, ValueError): + pass + return None + + +def percentile(values: list[float], quantile: float) -> float: + """Return a linearly interpolated percentile without a NumPy dependency.""" + + if not values: + raise ValueError("percentile requires at least one value") + ordered = sorted(values) + position = (len(ordered) - 1) * quantile + lower = math.floor(position) + upper = math.ceil(position) + if lower == upper: + return ordered[lower] + fraction = position - lower + return ordered[lower] * (1 - fraction) + ordered[upper] * fraction + + +def tensor_sha256(tensor: torch.Tensor) -> str: + raw = tensor.detach().contiguous().view(torch.uint8).cpu().numpy().tobytes() + return hashlib.sha256(raw).hexdigest() + + +def global_digest(local_digest: str, device: torch.device) -> str: + digest_tensor = torch.tensor( + list(bytes.fromhex(local_digest)), dtype=torch.uint8, device=device + ) + gathered = [torch.empty_like(digest_tensor) for _ in range(dist.get_world_size())] + dist.all_gather(gathered, digest_tensor) + payload = b"".join(bytes(item.cpu().tolist()) for item in gathered) + return hashlib.sha256(payload).hexdigest() + + +def make_route( + rank: int, + num_tokens: int, + num_experts: int, + top_k: int, + seed: int, + device: torch.device, +) -> torch.Tensor: + """Create a balanced, backend-independent route with unique experts/token.""" + + stride = 31 + if math.gcd(stride, num_experts) != 1: + raise ValueError("route stride must be coprime with num_experts") + global_token = rank * num_tokens + torch.arange( + num_tokens, dtype=torch.int64, device=device + ) + slots = torch.arange(top_k, dtype=torch.int64, device=device) + route = (global_token[:, None] * 17 + slots[None, :] * stride + seed) % num_experts + if top_k > num_experts: + raise ValueError("top_k cannot exceed num_experts") + return route.contiguous() + + +def make_input( + rank: int, num_tokens: int, hidden: int, device: torch.device +) -> torch.Tensor: + """Create deterministic, bounded BF16 data without backend RNG state.""" + + row = rank * num_tokens + torch.arange(num_tokens, dtype=torch.int64, device=device) + column = torch.arange(hidden, dtype=torch.int64, device=device) + values = (row[:, None] * 17 + column[None, :] * 13 + 19) % 257 + return ((values - 128).to(torch.float32) / 128.0).to(torch.bfloat16) + + +def normalized_diff(actual: torch.Tensor, expected: torch.Tensor) -> float: + actual64 = actual.double() + 1 + expected64 = expected.double() + 1 + denominator = (actual64.square() + expected64.square()).sum() + similarity = 2 * (actual64 * expected64).sum() / denominator + return float((1 - similarity).item()) + + +def logical_payload_bytes_per_rank( + route: torch.Tensor, + rank: int, + world_size: int, + local_world_size: int, + hidden: int, + dispatch_dtype: str, + num_experts: int, +) -> tuple[int, int, int]: + """Return all, scale-out, and valid route selections' logical payload bytes. + + Each valid expert assignment contributes its useful dispatch tensor, FP8 + scales when applicable, and one BF16 combine tensor. Backend metadata is + excluded. Scale-out bytes count assignments owned by a different node. + """ + + if num_experts % world_size: + raise ValueError("num_experts must divide world_size") + dispatch_bytes = hidden * 2 + if dispatch_dtype == "fp8": + dispatch_bytes = hidden + math.ceil(hidden / 128) * 4 + elif dispatch_dtype != "bf16": + raise ValueError(f"unsupported dispatch dtype: {dispatch_dtype}") + bytes_per_selection = dispatch_bytes + hidden * 2 + valid = route >= 0 + valid_selections = int(valid.sum().item()) + expert_owner = route // (num_experts // world_size) + source_node = rank // local_world_size + destination_node = expert_owner // local_world_size + remote_selections = int(((destination_node != source_node) & valid).sum().item()) + return ( + valid_selections * bytes_per_selection, + remote_selections * bytes_per_selection, + valid_selections, + ) + + +@dataclass(frozen=True) +class WorkloadProfile: + name: str + tokens_per_rank: int + api_mode: str + primary_metric: str + timing_boundary: str + + +WORKLOAD_PROFILES = { + "decode": WorkloadProfile( + name="decode", + tokens_per_rank=128, + api_mode="low-latency", + primary_metric="slowest-rank latency in milliseconds", + timing_boundary=( + "dispatch input ready through dispatch and combine completion; " + "slowest rank CUDA elapsed time; host-side FP8 conversion precedes " + "the boundary" + ), + ), + "prefill": WorkloadProfile( + name="prefill", + tokens_per_rank=4_096, + api_mode="normal", + primary_metric="slowest-rank latency in milliseconds", + timing_boundary=( + "dispatch input and route ready through required layout, dispatch, " + "and combine completion; slowest rank CUDA elapsed time; host-side " + "FP8 conversion precedes the boundary" + ), + ), +} + + +@dataclass +class DispatchState: + recv_x: Any + recv_topk_idx: torch.Tensor | None + recv_topk_weights: torch.Tensor | None + handle: Any + + +class BackendAdapter: + def __init__( + self, + arm: str, + group: dist.ProcessGroup, + profile: WorkloadProfile, + num_tokens: int, + hidden: int, + num_experts: int, + top_k: int, + num_sms: int = 0, + ) -> None: + self.arm = arm + self.group = group + self.profile = profile + self.num_tokens = num_tokens + self.hidden = hidden + self.num_experts = num_experts + self.world_size = dist.get_world_size(group) + self.buffer: Any + self._cast_back: Any = None + self._cast_to_fp8: Any = None + self.resolved_num_sms: int | None = None + self.detected_rdma_gbs: float | None = None + + if arm == "uccl": + sys.path.insert(0, "/opt/uccl/ep/bench") + from buffer import Buffer # type: ignore[import-not-found] + from utils import ( # type: ignore[import-not-found] + per_token_cast_back, + per_token_cast_to_fp8, + ) + + self._cast_back = per_token_cast_back + self._cast_to_fp8 = per_token_cast_to_fp8 + if profile.api_mode == "low-latency": + rdma_bytes = Buffer.get_low_latency_rdma_size_hint( + num_tokens, hidden, self.world_size, num_experts + ) + self.buffer = Buffer( + group, + num_rdma_bytes=rdma_bytes, + low_latency_mode=True, + num_qps_per_rank=num_experts // self.world_size, + allow_nvlink_for_low_latency_mode=True, + explicitly_destroy=True, + ) + else: + self.buffer = Buffer( + group, + num_nvl_bytes=NORMAL_NUM_NVL_BYTES, + num_rdma_bytes=NORMAL_NUM_RDMA_BYTES, + low_latency_mode=False, + num_qps_per_rank=NORMAL_NUM_QPS_PER_RANK, + explicitly_destroy=True, + ) + elif arm == "deepep-v1-nvshmem": + sys.path.insert(0, "/opt/amazon/deepep/tests") + sys.path.insert(0, "/opt/amazon/deepep") + import deep_ep # type: ignore[import-not-found] + from utils import ( # type: ignore[import-not-found] + per_token_cast_back, + per_token_cast_to_fp8, + ) + + self._cast_back = per_token_cast_back + self._cast_to_fp8 = per_token_cast_to_fp8 + if profile.api_mode == "low-latency": + rdma_bytes = deep_ep.Buffer.get_low_latency_rdma_size_hint( + num_tokens, hidden, self.world_size, num_experts + ) + self.buffer = deep_ep.Buffer( + group, + num_rdma_bytes=rdma_bytes, + low_latency_mode=True, + num_qps_per_rank=num_experts // self.world_size, + allow_nvlink_for_low_latency_mode=True, + explicitly_destroy=True, + allow_mnnvl=False, + ) + else: + self.buffer = deep_ep.Buffer( + group, + num_nvl_bytes=NORMAL_NUM_NVL_BYTES, + num_rdma_bytes=NORMAL_NUM_RDMA_BYTES, + low_latency_mode=False, + num_qps_per_rank=NORMAL_NUM_QPS_PER_RANK, + explicitly_destroy=True, + allow_mnnvl=False, + ) + elif arm == "deepep-v2-gin-gda": + import deep_ep # type: ignore[import-not-found] + from deep_ep.utils.math import ( # type: ignore[import-not-found] + per_token_cast_back, + per_token_cast_to_fp8, + ) + + self._cast_back = per_token_cast_back + self._cast_to_fp8 = per_token_cast_to_fp8 + self.buffer = deep_ep.ElasticBuffer( + group, + num_max_tokens_per_rank=num_tokens, + hidden=hidden, + num_topk=top_k, + deterministic=False, + allow_hybrid_mode=True, + allow_multiple_reduction=True, + prefer_overlap_with_compute=False, + sl_idx=0, + num_allocated_qps=0, + explicitly_destroy=True, + num_gpu_timeout_secs=180, + num_cpu_timeout_secs=180, + ) + else: + raise ValueError(f"unsupported arm: {arm}") + + # Resolve the SM count each timed kernel will actually use so it can be + # recorded with the result instead of staying an implicit control. + if arm == "deepep-v2-gin-gda": + from deep_ep.utils.envs import ( # type: ignore[import-not-found] + get_rdma_gbs, + ) + + self.detected_rdma_gbs = float(get_rdma_gbs()) + self.resolved_num_sms = ( + num_sms + if num_sms > 0 + else int(self.buffer.get_theoretical_num_sms(num_experts, top_k)) + ) + elif profile.api_mode == "normal": + buffer_class = type(self.buffer) + if num_sms > 0: + if not hasattr(buffer_class, "set_num_sms"): + raise RuntimeError( + f"--num-sms is not supported by the {arm} buffer" + ) + buffer_class.set_num_sms(num_sms) + resolved = getattr(buffer_class, "num_sms", None) + self.resolved_num_sms = int(resolved) if resolved is not None else None + elif num_sms > 0 and dist.get_rank(group) == 0: + # The low-latency kernels take no SM count; the record keeps null. + print( + f"ADAI_EP_NOTICE num_sms pin ignored by the {arm} low-latency path", + flush=True, + ) + + @property + def is_elastic(self) -> bool: + return self.arm == "deepep-v2-gin-gda" + + def prepare_dispatch_input(self, x: torch.Tensor, dispatch_dtype: str) -> Any: + if dispatch_dtype == "bf16": + return x + if dispatch_dtype != "fp8": + raise ValueError(f"unsupported dispatch dtype: {dispatch_dtype}") + if self.is_elastic or self.profile.api_mode == "normal": + return self._cast_to_fp8(x) + return x + + def dispatch( + self, + prepared_x: Any, + topk_idx: torch.Tensor, + topk_weights: torch.Tensor, + dispatch_dtype: str, + ) -> DispatchState: + if self.is_elastic: + recv_x, recv_idx, recv_weights, handle, event = self.buffer.dispatch( + x=prepared_x, + topk_idx=topk_idx, + topk_weights=topk_weights, + num_experts=self.num_experts, + num_max_tokens_per_rank=self.num_tokens, + expert_alignment=1, + num_sms=self.resolved_num_sms, + async_with_compute_stream=True, + allocate_on_comm_stream=False, + do_handle_copy=True, + do_cpu_sync=True, + ) + event.current_stream_wait() + return DispatchState(recv_x, recv_idx, recv_weights, handle) + + if self.profile.api_mode == "normal": + ( + num_tokens_per_rank, + num_tokens_per_rdma_rank, + num_tokens_per_expert, + is_token_in_rank, + _, + ) = self.buffer.get_dispatch_layout(topk_idx, self.num_experts) + ( + recv_x, + recv_idx, + recv_weights, + _, + handle, + event, + ) = self.buffer.dispatch( + x=prepared_x, + num_tokens_per_rank=num_tokens_per_rank, + num_tokens_per_rdma_rank=num_tokens_per_rdma_rank, + is_token_in_rank=is_token_in_rank, + num_tokens_per_expert=num_tokens_per_expert, + topk_idx=topk_idx, + topk_weights=topk_weights, + expert_alignment=1, + async_finish=True, + allocate_on_comm_stream=False, + ) + event.current_stream_wait() + return DispatchState(recv_x, recv_idx, recv_weights, handle) + + recv_x, _, handle, event, _ = self.buffer.low_latency_dispatch( + prepared_x, + topk_idx, + self.num_tokens, + self.num_experts, + use_fp8=dispatch_dtype == "fp8", + async_finish=True, + return_recv_hook=False, + ) + event.current_stream_wait() + return DispatchState(recv_x, None, None, handle) + + def received_as_bf16(self, recv_x: Any, dispatch_dtype: str) -> torch.Tensor: + if dispatch_dtype == "bf16": + return recv_x + if self.is_elastic: + return self._cast_back(recv_x[0], recv_x[1]) + fp8, scales = recv_x + return self._cast_back( + fp8.reshape(-1, self.hidden), + scales.reshape(-1, self.hidden // 128), + ).reshape(fp8.shape) + + def identity_expert_output( + self, state: DispatchState, dispatch_dtype: str + ) -> torch.Tensor: + output = self.received_as_bf16(state.recv_x, dispatch_dtype) + if state.recv_topk_idx is None or state.recv_topk_weights is None: + return output + local_weights = state.recv_topk_weights.masked_fill( + state.recv_topk_idx < 0, 0 + ).sum(dim=1, keepdim=True) + return output * local_weights.to(output.dtype) + + def combine( + self, + combine_input: torch.Tensor, + state: DispatchState, + topk_idx: torch.Tensor, + topk_weights: torch.Tensor, + ) -> torch.Tensor: + if self.is_elastic: + combined, _, event = self.buffer.combine( + x=combine_input, + handle=state.handle, + topk_weights=state.recv_topk_weights, + async_with_compute_stream=True, + allocate_on_comm_stream=False, + ) + event.current_stream_wait() + return combined + + if self.profile.api_mode == "normal": + combined, _, event = self.buffer.combine( + x=combine_input, + handle=state.handle, + topk_weights=state.recv_topk_weights, + async_finish=True, + allocate_on_comm_stream=False, + ) + event.current_stream_wait() + return combined + + combined, event, _ = self.buffer.low_latency_combine( + combine_input, + topk_idx, + topk_weights, + state.handle, + use_logfmt=False, + async_finish=True, + return_recv_hook=False, + ) + event.current_stream_wait() + return combined + + def destroy(self) -> None: + self.buffer.destroy() + + +def initialize_distributed() -> tuple[int, int, int, torch.device, dist.ProcessGroup]: + local_rank = int(os.environ["LOCAL_RANK"]) + local_world_size = int(os.environ.get("LOCAL_WORLD_SIZE", "1")) + torch.cuda.set_device(local_rank) + device = torch.device(f"cuda:{local_rank}") + dist.init_process_group("nccl", device_id=device) + world_size = dist.get_world_size() + group = dist.new_group(list(range(world_size))) + torch.set_default_dtype(torch.bfloat16) + return dist.get_rank(), world_size, local_world_size, device, group + + +def run_dtype( + adapter: BackendAdapter, + x: torch.Tensor, + route: torch.Tensor, + topk_weights: torch.Tensor, + dispatch_dtype: str, + warmups: int, + iterations: int, + rank: int, + world_size: int, + local_world_size: int, + args: argparse.Namespace, + route_hash: str, + input_hash: str, +) -> dict[str, Any]: + prepared = adapter.prepare_dispatch_input(x, dispatch_dtype) + correctness_state = adapter.dispatch(prepared, route, topk_weights, dispatch_dtype) + # Normal and ElasticBuffer dispatch send one token per destination rank. + # Apply the local identity experts' gated reduction before combine. The + # low-latency API performs its corresponding reduction internally. + correctness_input = adapter.identity_expert_output( + correctness_state, dispatch_dtype + ) + correctness_output = adapter.combine( + correctness_input, correctness_state, route, topk_weights + ) + torch.cuda.synchronize() + expected = x * topk_weights.sum(dim=1, keepdim=True).to(torch.bfloat16) + diff = normalized_diff(correctness_output, expected) + max_abs_error = float( + (correctness_output.float() - expected.float()).abs().max().item() + ) + tolerance = 9e-4 if dispatch_dtype == "fp8" else 1e-5 + correctness_pass = diff <= tolerance and bool( + torch.isfinite(correctness_output).all().item() + ) + correctness_tensor = torch.tensor( + [1 if correctness_pass else 0], dtype=torch.int32, device=x.device + ) + dist.all_reduce(correctness_tensor, op=dist.ReduceOp.MIN) + if int(correctness_tensor.item()) != 1: + raise RuntimeError( + f"correctness failed for {dispatch_dtype}: diff={diff}, " + f"tolerance={tolerance}, max_abs_error={max_abs_error}" + ) + + # Timed combine uses a stable, preallocated expert-output tensor. This + # keeps expert computation and FP8 dequantization outside the communication + # boundary while retaining the handle created by each timed dispatch. + combine_input = torch.zeros_like(correctness_input, dtype=torch.bfloat16) + + # x is fixed across the timed loop, so the host-side FP8 cast runs once in + # prepare_dispatch_input above. Low-latency kernels that quantize + # internally still do so inside the timed boundary. + def iteration() -> None: + state = adapter.dispatch(prepared, route, topk_weights, dispatch_dtype) + adapter.combine(combine_input, state, route, topk_weights) + + for _ in range(warmups): + dist.barrier(group=adapter.group) + iteration() + torch.cuda.synchronize() + + start = torch.cuda.Event(enable_timing=True) + end = torch.cuda.Event(enable_timing=True) + max_rank_latency_ms: list[float] = [] + for _ in range(iterations): + dist.barrier(group=adapter.group) + start.record() + iteration() + end.record() + end.synchronize() + local_latency = torch.tensor( + [start.elapsed_time(end)], dtype=torch.float32, device=x.device + ) + dist.all_reduce(local_latency, op=dist.ReduceOp.MAX, group=adapter.group) + max_rank_latency_ms.append(float(local_latency.item())) + + logical_bytes, scaleout_bytes, valid_selections = logical_payload_bytes_per_rank( + route, + rank, + world_size, + local_world_size, + args.hidden, + dispatch_dtype, + args.experts, + ) + counters = torch.tensor( + [logical_bytes, scaleout_bytes, valid_selections], + dtype=torch.int64, + device=x.device, + ) + dist.all_reduce(counters, op=dist.ReduceOp.SUM, group=adapter.group) + avg_logical_bytes = int(counters[0].item()) / world_size + avg_scaleout_bytes = int(counters[1].item()) / world_size + global_valid_selections = int(counters[2].item()) + + median_ms = statistics.median(max_rank_latency_ms) + mean_ms = statistics.fmean(max_rank_latency_ms) + stdev_ms = statistics.stdev(max_rank_latency_ms) if iterations > 1 else 0.0 + result = { + "schema_version_dimensionless": SCHEMA_VERSION, + "benchmark": "common-boundary-dispatch-combine", + "workload_profile": args.profile, + "backend_api_mode": ( + "elastic" if adapter.is_elastic else adapter.profile.api_mode + ), + "primary_metric": adapter.profile.primary_metric, + "layout_in_timed_region": args.profile == "prefill", + "arm": args.arm, + "run_index_dimensionless": args.run_index, + "dispatch_dtype": dispatch_dtype, + "world_size_ranks": world_size, + "nodes": world_size // local_world_size, + "gpus_per_node": local_world_size, + "tokens_per_rank": args.tokens, + "global_input_tokens": args.tokens * world_size, + "hidden_dimensions": args.hidden, + "experts": args.experts, + "top_k_dimensionless": args.top_k, + "route_seed_dimensionless": args.seed, + "route_hash_sha256": route_hash, + "input_hash_sha256": input_hash, + "global_valid_expert_selections": global_valid_selections, + "warmup_iterations": warmups, + "measured_iterations": iterations, + "num_sms_dimensionless": adapter.resolved_num_sms, + "detected_rdma_gigabytes_per_second": adapter.detected_rdma_gbs, + "timing_boundary": adapter.profile.timing_boundary, + "logical_payload_definition": "per valid expert assignment: dispatch tensor plus FP8 scales when selected plus BF16 combine tensor; backend metadata excluded", + "avg_logical_payload_bytes_per_rank": avg_logical_bytes, + "avg_scaleout_logical_payload_bytes_per_rank": avg_scaleout_bytes, + "latency_ms": { + "median": median_ms, + "mean": mean_ms, + "p95": percentile(max_rank_latency_ms, 0.95), + "minimum": min(max_rank_latency_ms), + "maximum": max(max_rank_latency_ms), + "stdev": stdev_ms, + "cv_percent": stdev_ms / mean_ms * 100 if mean_ms else 0.0, + }, + "aggregate_input_tokens_per_second": ( + args.tokens * world_size / (median_ms / 1e3) + ), + "effective_logical_gigabytes_per_second_per_rank": ( + avg_logical_bytes / (median_ms / 1e3) / 1e9 + ), + "effective_scaleout_logical_gigabytes_per_second_per_rank": ( + avg_scaleout_bytes / (median_ms / 1e3) / 1e9 + ), + "correctness": { + "status": "PASS", + "normalized_diff_dimensionless": diff, + "tolerance_dimensionless": tolerance, + "max_abs_error_bf16_value": max_abs_error, + }, + "runtime": { + "image_reference": os.environ.get("ADAI_IMAGE_REFERENCE", "unknown"), + "torch_version": torch.__version__, + "cuda_version": torch.version.cuda, + # torch's compile-time constant; must stay uniform across arms. + "nccl_version": list(torch.cuda.nccl.version()), + # The library actually loaded in this process; may differ per arm. + "nccl_version_loaded": loaded_nccl_version(), + "gpu": torch.cuda.get_device_name(), + }, + } + return result + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument( + "--arm", + required=True, + choices=("uccl", "deepep-v1-nvshmem", "deepep-v2-gin-gda"), + ) + parser.add_argument("--profile", required=True, choices=tuple(WORKLOAD_PROFILES)) + parser.add_argument("--hidden", type=int, default=7168) + parser.add_argument("--top-k", type=int, default=8) + parser.add_argument("--experts", type=int, default=256) + parser.add_argument("--seed", type=int, default=20260824) + parser.add_argument("--warmups", type=int, default=20) + parser.add_argument("--iterations", type=int, default=100) + parser.add_argument( + "--num-sms", + type=int, + default=0, + help=( + "Pin the communication-kernel SM count; 0 keeps each backend's " + "automatic choice. Applies to the DeepEP V2 arm and the " + "normal-mode UCCL and DeepEP V1 APIs." + ), + ) + parser.add_argument("--run-index", type=int, required=True) + parser.add_argument( + "--dispatch-dtypes", + default="fp8,bf16", + help="Comma-separated dtype order; allowed values are fp8 and bf16", + ) + args = parser.parse_args() + args.tokens = WORKLOAD_PROFILES[args.profile].tokens_per_rank + args.dispatch_dtypes = args.dispatch_dtypes.split(",") + if sorted(args.dispatch_dtypes) != ["bf16", "fp8"]: + parser.error("--dispatch-dtypes must contain fp8 and bf16 exactly once") + if args.warmups < 1 or args.iterations < 2: + parser.error("at least 1 warmup and 2 measured iterations are required") + return args + + +def main() -> None: + args = parse_args() + preload_backend(args.arm) + rank, world_size, local_world_size, device, group = initialize_distributed() + if args.experts % world_size: + raise SystemExit("experts must divide the distributed world size") + route = make_route(rank, args.tokens, args.experts, args.top_k, args.seed, device) + x = make_input(rank, args.tokens, args.hidden, device) + topk_weights = torch.full( + (args.tokens, args.top_k), + 1.0 / args.top_k, + dtype=torch.float32, + device=device, + ) + route_hash = global_digest(tensor_sha256(route), device) + input_hash = global_digest(tensor_sha256(x), device) + route_histogram = torch.bincount(route.flatten(), minlength=args.experts).to( + torch.int64 + ) + dist.all_reduce(route_histogram, op=dist.ReduceOp.SUM, group=group) + + if rank == 0: + print( + "ADAI_EP_CONFIG " + + json.dumps( + { + "arm": args.arm, + "workload_profile": args.profile, + "world_size_ranks": world_size, + "tokens_per_rank": args.tokens, + "hidden_dimensions": args.hidden, + "experts": args.experts, + "top_k_dimensionless": args.top_k, + "route_hash_sha256": route_hash, + "input_hash_sha256": input_hash, + "route_histogram_min_selections": int(route_histogram.min().item()), + "route_histogram_max_selections": int(route_histogram.max().item()), + }, + sort_keys=True, + ), + flush=True, + ) + + adapter = BackendAdapter( + args.arm, + group, + WORKLOAD_PROFILES[args.profile], + args.tokens, + args.hidden, + args.experts, + args.top_k, + num_sms=args.num_sms, + ) + completed = False + try: + for dispatch_dtype in args.dispatch_dtypes: + result = run_dtype( + adapter, + x, + route, + topk_weights, + dispatch_dtype, + args.warmups, + args.iterations, + rank, + world_size, + local_world_size, + args, + route_hash, + input_hash, + ) + if rank == 0: + print(RESULT_PREFIX + json.dumps(result, sort_keys=True), flush=True) + completed = True + except BaseException: + if args.arm == "uccl" and args.profile == "prefill": + # The pinned UCCL high-throughput proxy teardown can strand the + # interpreter after an error. Preserve the traceback and let + # torchrun observe an unambiguous worker failure instead. + traceback.print_exc() + sys.stdout.flush() + sys.stderr.flush() + os._exit(1) + raise + finally: + if args.arm == "uccl" and args.profile == "prefill": + if completed: + # UCCL's pinned normal-mode proxy cleanup destroys the CUDA + # context while PyTorch still owns CUDA tensors, which hangs + # worker shutdown. All communication has completed here and + # this barrier keeps every rank alive through result output. + # Process exit then releases the CUDA context, QPs, and file + # descriptors outside the measured benchmark boundary. + if rank == 0: + print( + "ADAI_EP_PROCESS_LIFETIME_CLEANUP arm=uccl profile=prefill", + flush=True, + ) + dist.barrier(group=group) + sys.stdout.flush() + sys.stderr.flush() + os._exit(0) + else: + adapter.destroy() + dist.barrier(group=group) + dist.destroy_process_group() + + +if __name__ == "__main__": + main() diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/extract_results.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/extract_results.py new file mode 100644 index 000000000..15fc55440 --- /dev/null +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/extract_results.py @@ -0,0 +1,31 @@ +#!/usr/bin/env python3 +"""Extract canonical JSONL result records from a mixed benchmark log.""" + +from __future__ import annotations + +import argparse +import json +from pathlib import Path + +from result_io import PREFIX, load_result_log + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("input_log", type=Path) + parser.add_argument("output_jsonl", type=Path) + args = parser.parse_args() + + results = load_result_log(args.input_log) + if not results: + raise SystemExit(f"no benchmark results found in {args.input_log}") + args.output_jsonl.write_text( + "".join( + PREFIX + json.dumps(result, sort_keys=True) + "\n" for result in results + ) + ) + print(f"extracted {len(results)} benchmark results from {args.input_log}") + + +if __name__ == "__main__": + main() diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/nccl-alltoall.yaml b/micro-benchmarks/expert-parallelism/ep-backend-comparison/nccl-alltoall.yaml deleted file mode 100644 index 7f14fd384..000000000 --- a/micro-benchmarks/expert-parallelism/ep-backend-comparison/nccl-alltoall.yaml +++ /dev/null @@ -1,106 +0,0 @@ -# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. -# SPDX-License-Identifier: MIT-0 -# -# Baseline for the EP-backend comparison: a raw NCCL all-to-all over the same -# world size as the EP benchmarks (NUM_NODES x GPU_PER_NODE ranks). This is the -# transport-level reference ceiling -- it moves bytes but does NOT do MoE token -# routing or combine-reduction, so its busbw is an upper bound, not a -# like-for-like dispatch/combine number. -# -# Built from micro-benchmarks/nccl-tests (CUDA 13.0.2 image ships alltoall_perf -# and sm_100/sm_103). Set NP = NUM_NODES * GPU_PER_NODE in env_vars. -apiVersion: kubeflow.org/v2beta1 -kind: MPIJob -metadata: - name: nccl-alltoall -spec: - runPolicy: - cleanPodPolicy: Running - backoffLimit: 3 - slotsPerWorker: ${GPU_PER_NODE} - mpiReplicaSpecs: - Launcher: - replicas: 1 - template: - spec: - restartPolicy: OnFailure - nodeSelector: - node.kubernetes.io/instance-type: ${INSTANCE_TYPE} - tolerations: - - key: nvidia.com/gpu - operator: Exists - effect: NoSchedule - - key: workload - value: bench - operator: Equal - effect: NoSchedule - - key: capacity-reservation - operator: Exists - effect: NoSchedule - containers: - - image: ${IMAGE_URI} - imagePullPolicy: IfNotPresent - name: nccl-alltoall-launcher - # PATH / LD_LIBRARY_PATH come from the image ENV and are forwarded to - # workers via `mpirun -x PATH -x LD_LIBRARY_PATH` below (Kubernetes does - # not expand a bare $PATH in an env value, so we do not set one here). - command: - - /bin/bash - - -c - - | - # Wait for each worker's sshd before launching: mpi-operator starts - # the launcher as soon as workers are Running, but their sshd / DNS - # may not be ready yet, and the launcher would lose the SSH race. - for host in $(awk '{print $1}' /etc/mpi/hostfile); do - until ssh -o ConnectTimeout=3 -o BatchMode=yes -o StrictHostKeyChecking=no "$host" true 2>/dev/null; do - echo "waiting for sshd on $host"; sleep 3 - done - done - /opt/amazon/openmpi/bin/mpirun \ - --allow-run-as-root --tag-output \ - -np ${NP} -N ${GPU_PER_NODE} --bind-to none \ - -x PATH -x LD_LIBRARY_PATH \ - -x FI_PROVIDER=efa \ - -x FI_EFA_USE_DEVICE_RDMA=1 \ - -x FI_EFA_FORK_SAFE=1 \ - -x NCCL_DEBUG=INFO \ - /opt/nccl-tests/build/alltoall_perf -b 8 -e 8G -f 2 -g 1 -c 1 -n 100 - Worker: - replicas: ${NUM_NODES} - template: - spec: - nodeSelector: - node.kubernetes.io/instance-type: ${INSTANCE_TYPE} - tolerations: - - key: nvidia.com/gpu - operator: Exists - effect: NoSchedule - - key: workload - value: bench - operator: Equal - effect: NoSchedule - - key: capacity-reservation - operator: Exists - effect: NoSchedule - containers: - - image: ${IMAGE_URI} - imagePullPolicy: IfNotPresent - name: nccl-alltoall-worker - volumeMounts: - - name: shmem - mountPath: /dev/shm - resources: - limits: - nvidia.com/gpu: ${GPU_PER_NODE} - hugepages-2Mi: 5120Mi - vpc.amazonaws.com/efa: ${EFA_PER_NODE} - memory: 32000Mi - requests: - nvidia.com/gpu: ${GPU_PER_NODE} - hugepages-2Mi: 5120Mi - vpc.amazonaws.com/efa: ${EFA_PER_NODE} - memory: 32000Mi - volumes: - - name: shmem - hostPath: - path: /dev/shm diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/plot_results.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/plot_results.py new file mode 100644 index 000000000..a71cfbc4b --- /dev/null +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/plot_results.py @@ -0,0 +1,262 @@ +#!/usr/bin/env python3 +"""Render backend box plots from an EP comparison summary.""" + +from __future__ import annotations + +import argparse +import json +import math +from pathlib import Path +from typing import Any + +import matplotlib + +matplotlib.use("Agg") + +from matplotlib import pyplot as plt +from matplotlib.patches import Patch + + +ARMS = ("uccl", "deepep-v1-nvshmem", "deepep-v2-gin-gda") +ARM_STYLES = { + "uccl": { + "color": "#0072B2", + "hatch": "///", + "label": "UCCL", + "marker": "o", + }, + "deepep-v1-nvshmem": { + "color": "#E69F00", + "hatch": "\\\\", + "label": "DeepEP V1 NVSHMEM", + "marker": "s", + }, + "deepep-v2-gin-gda": { + "color": "#009E73", + "hatch": "...", + "label": "DeepEP V2 NCCL GIN", + "marker": "^", + }, +} +DTYPE_ORDER = ("fp8", "bf16") +PROFILE_CONFIG = { + "decode": { + "title": "Decode-like: slowest-rank latency (lower is better)", + "ylabel": "Latency (ms)", + }, + "prefill": { + "title": "Prefill-like: slowest-rank latency (lower is better)", + "ylabel": "Latency (ms)", + }, +} + + +def load_plot_data( + path: Path, +) -> tuple[ + int, + dict[tuple[str, int, str], dict[str, Any]], + tuple[tuple[int, str], ...], +]: + summary = json.loads(path.read_text()) + if summary.get("status") != "PASS": + raise ValueError("the input summary does not have PASS status") + + starts = summary.get("independent_starts_per_cell") + if not isinstance(starts, int) or starts < 1: + raise ValueError("independent_starts_per_cell must be a positive integer") + + world_sizes = summary.get("configuration", {}).get("world_sizes_ranks") + if ( + not isinstance(world_sizes, list) + or not world_sizes + or not all(isinstance(world, int) and world > 0 for world in world_sizes) + ): + raise ValueError("world_sizes_ranks must be a nonempty list of EP sizes") + cell_order = tuple( + (world_size, dtype) for world_size in world_sizes for dtype in DTYPE_ORDER + ) + + cells: dict[tuple[str, int, str], dict[str, Any]] = {} + for cell in summary.get("cells", []): + key = ( + cell.get("workload_profile"), + cell.get("world_size_ranks"), + cell.get("dispatch_dtype"), + ) + if key in cells: + raise ValueError(f"duplicate workload cell: {key}") + cells[key] = cell + + expected_keys = { + (profile, world_size, dtype) + for profile in PROFILE_CONFIG + for world_size, dtype in cell_order + } + if cells.keys() != expected_keys: + missing = sorted(expected_keys - cells.keys()) + extra = sorted(cells.keys() - expected_keys) + raise ValueError(f"workload cell mismatch; missing={missing}, extra={extra}") + + for key, cell in cells.items(): + arms = cell.get("arms", {}) + if arms.keys() != set(ARMS): + missing = sorted(set(ARMS) - arms.keys()) + extra = sorted(arms.keys() - set(ARMS)) + raise ValueError( + f"backend arm mismatch for {key}; missing={missing}, extra={extra}" + ) + for arm in ARMS: + values = arms[arm].get("per_start_primary_values") + if not isinstance(values, list) or len(values) != starts: + raise ValueError( + f"{key}/{arm} must contain {starts} per-start primary values" + ) + if not all( + isinstance(value, (int, float)) and math.isfinite(value) and value > 0 + for value in values + ): + raise ValueError(f"{key}/{arm} contains an invalid primary value") + return starts, cells, cell_order + + +def render_box_plots(summary_path: Path, output_path: Path) -> None: + starts, cells, cell_order = load_plot_data(summary_path) + matplotlib.rcParams.update( + { + "axes.edgecolor": "#333333", + "axes.labelcolor": "#222222", + "font.size": 10, + "savefig.facecolor": "white", + "text.color": "#222222", + "xtick.color": "#333333", + "ytick.color": "#333333", + } + ) + + figure, axes = plt.subplots(2, 1, figsize=(12, 8), sharex=True) + group_positions = list(range(1, len(cell_order) + 1)) + arm_offsets = (-0.24, 0.0, 0.24) + # Keep the per-start markers inside their 0.19-wide box for any start count. + point_spacing = min(0.025, 0.17 / max(starts - 1, 1)) + point_offsets = tuple( + (index - (starts - 1) / 2) * point_spacing for index in range(starts) + ) + + for axis, profile in zip(axes, PROFILE_CONFIG, strict=True): + all_values: list[float] = [] + for arm, arm_offset in zip(ARMS, arm_offsets, strict=True): + values_by_cell = [ + cells[(profile, world_size, dtype)]["arms"][arm][ + "per_start_primary_values" + ] + for world_size, dtype in cell_order + ] + all_values.extend(value for values in values_by_cell for value in values) + positions = [position + arm_offset for position in group_positions] + style = ARM_STYLES[arm] + box_plot = axis.boxplot( + values_by_cell, + positions=positions, + widths=0.19, + whis=(0, 100), + showfliers=False, + patch_artist=True, + manage_ticks=False, + boxprops={ + "facecolor": style["color"], + "edgecolor": "#222222", + "hatch": style["hatch"], + "linewidth": 1.0, + "alpha": 0.55, + }, + whiskerprops={"color": "#333333", "linewidth": 1.0}, + capprops={"color": "#333333", "linewidth": 1.0}, + medianprops={"color": "#111111", "linewidth": 1.8}, + ) + for median in box_plot["medians"]: + median.set_zorder(4) + for position, values in zip(positions, values_by_cell, strict=True): + axis.scatter( + [position + offset for offset in point_offsets], + values, + color=style["color"], + edgecolor="#111111", + linewidth=0.6, + marker=style["marker"], + s=29, + zorder=5, + ) + + config = PROFILE_CONFIG[profile] + axis.set_title(config["title"], loc="left", fontweight="bold", pad=10) + axis.set_ylabel(config["ylabel"]) + axis.set_ylim(0, max(all_values) * 1.12) + axis.set_xlim(0.5, len(cell_order) + 0.5) + axis.grid(axis="y", color="#D9D9D9", linewidth=0.8) + axis.set_axisbelow(True) + axis.spines["top"].set_visible(False) + axis.spines["right"].set_visible(False) + + axes[-1].set_xticks( + group_positions, + [f"{world_size} ranks\n{dtype.upper()}" for world_size, dtype in cell_order], + ) + axes[-1].set_xlabel("Expert-parallel size and dispatch dtype", labelpad=9) + + legend_handles = [ + Patch( + facecolor=ARM_STYLES[arm]["color"], + edgecolor="#222222", + hatch=ARM_STYLES[arm]["hatch"], + alpha=0.55, + label=ARM_STYLES[arm]["label"], + ) + for arm in ARMS + ] + figure.legend( + handles=legend_handles, + loc="upper center", + bbox_to_anchor=(0.5, 0.945), + frameon=False, + ncol=len(ARMS), + ) + figure.suptitle( + "B200 expert-parallel backend comparison", + fontsize=15, + fontweight="bold", + y=0.99, + ) + figure.text( + 0.5, + 0.012, + ( + f"Each arm has {starts} independent process starts. " + "Box: Q1 to Q3; line: median; whiskers: minimum to maximum; " + "markers: per-start medians." + ), + ha="center", + fontsize=9, + color="#444444", + ) + figure.subplots_adjust(left=0.1, right=0.98, top=0.88, bottom=0.12, hspace=0.34) + + output_path.parent.mkdir(parents=True, exist_ok=True) + figure.savefig( + output_path, + dpi=160, + metadata={"Software": "matplotlib via plot_results.py"}, + ) + plt.close(figure) + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("summary", type=Path, help="machine-readable summary JSON") + parser.add_argument("--output", type=Path, required=True, help="output image path") + args = parser.parse_args() + render_box_plots(args.summary, args.output) + + +if __name__ == "__main__": + main() diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/result_io.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/result_io.py new file mode 100644 index 000000000..1d53cee33 --- /dev/null +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/result_io.py @@ -0,0 +1,45 @@ +"""Parse benchmark result markers despite interleaved native stdout.""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any, Iterator + + +PREFIX = "ADAI_EP_RESULT " + + +def iter_result_objects(text: str, source: str = "") -> Iterator[dict[str, Any]]: + """Yield JSON objects following result markers in mixed process output. + + Native libraries can flush a diagnostic after Python has emitted a JSON + object but before its newline reaches the combined log. JSONDecoder's + raw_decode identifies the exact end of the object without treating that + trailing diagnostic as part of the result. + """ + + decoder = json.JSONDecoder() + for line_number, line in enumerate(text.splitlines(), 1): + search_from = 0 + while True: + marker = line.find(PREFIX, search_from) + if marker < 0: + break + payload_start = marker + len(PREFIX) + try: + result, consumed = decoder.raw_decode(line[payload_start:]) + except json.JSONDecodeError as error: + raise ValueError( + f"invalid benchmark result in {source}:{line_number}: {error}" + ) from error + if not isinstance(result, dict): + raise ValueError( + f"benchmark result in {source}:{line_number} is not a JSON object" + ) + yield result + search_from = payload_start + consumed + + +def load_result_log(path: Path) -> list[dict[str, Any]]: + return list(iter_result_objects(path.read_text(errors="replace"), str(path))) diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/results/b200-ap-south-1-2026-08-25-boxplots.png b/micro-benchmarks/expert-parallelism/ep-backend-comparison/results/b200-ap-south-1-2026-08-25-boxplots.png new file mode 100644 index 000000000..91fd47ad9 Binary files /dev/null and b/micro-benchmarks/expert-parallelism/ep-backend-comparison/results/b200-ap-south-1-2026-08-25-boxplots.png differ diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/results/b200-ap-south-1-2026-08-25.json b/micro-benchmarks/expert-parallelism/ep-backend-comparison/results/b200-ap-south-1-2026-08-25.json new file mode 100644 index 000000000..76a06e7fe --- /dev/null +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/results/b200-ap-south-1-2026-08-25.json @@ -0,0 +1,1114 @@ +{ + "bootstrap_samples_dimensionless": 20000, + "campaign_provenance": { + "campaign_id": "ep-b200-profiles-20260825t023933z", + "cluster": "ml-clusters-shared-ap-south-1", + "comparison": { + "experts": 256, + "hidden_dimensions": 7168, + "independent_starts": 3, + "measured_iterations": 100, + "profiles": { + "decode": { + "api_mode": "low-latency", + "primary_metric": "slowest-rank latency in milliseconds", + "tokens_per_rank": 128 + }, + "prefill": { + "api_mode": "normal", + "primary_metric": "effective logical gigabytes per second per rank", + "tokens_per_rank": 4096 + } + }, + "top_k_dimensionless": 8, + "warmup_iterations": 20 + }, + "created_at_utc": "2026-08-25T02:41:59Z", + "git_commit": "97c93ee33535ffb41b7635befc4a97617ab8acef", + "images": { + "deepep-v1-nvshmem": "159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-deepep-v1-nvshmem@sha256:175a8470e11bc7832024941d1775923d5e8ecae2c6dc06183d5f1653d66b4fac", + "deepep-v2-gin-gda": "159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-deepep-v2-gin-gda@sha256:4d07367ea290c5d6ec3c02b223ac819feed5240a46fd4a6492421e9c0853dbeb", + "uccl": "159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-uccl@sha256:a703a0d35916bbd51b2d34d968626d9617cbceb328d64b920f21ad159d93c91a" + }, + "region": "ap-south-1" + }, + "cells": [ + { + "arms": { + "deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_primary_value": [ + 1.0147680044174194, + 1.0428639650344849 + ], + "median_aggregate_input_tokens_per_second": 1984619.2206983268, + "median_effective_logical_gigabytes_per_second_per_rank": 21.560903213666624, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 10.780451606833312, + "median_latency_ms": 1.03193598985672, + "median_primary_value": 1.03193598985672, + "per_start_median_latency_ms": [ + 1.03193598985672, + 1.0428639650344849, + 1.0147680044174194 + ], + "per_start_primary_values": [ + 1.03193598985672, + 1.0428639650344849, + 1.0147680044174194 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 1.3752408024975253, + "starts": 3 + }, + "deepep-v2-gin-gda": { + "bootstrap_95_percent_ci_primary_value": [ + 0.6111679971218109, + 0.6488159894943237 + ], + "median_aggregate_input_tokens_per_second": 3329691.4185950146, + "median_effective_logical_gigabytes_per_second_per_rank": 36.17376757161624, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 18.08688378580812, + "median_latency_ms": 0.6150720119476318, + "median_primary_value": 0.6150720119476318, + "per_start_median_latency_ms": [ + 0.6150720119476318, + 0.6111679971218109, + 0.6488159894943237 + ], + "per_start_primary_values": [ + 0.6150720119476318, + 0.6111679971218109, + 0.6488159894943237 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 3.3121127855081234, + "starts": 3 + }, + "uccl": { + "bootstrap_95_percent_ci_primary_value": [ + 0.5123519897460938, + 0.5320799946784973 + ], + "median_aggregate_input_tokens_per_second": 3926862.247229175, + "median_effective_logical_gigabytes_per_second_per_rank": 42.661431453897755, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 21.330715726948878, + "median_latency_ms": 0.5215359926223755, + "median_primary_value": 0.5215359926223755, + "per_start_median_latency_ms": [ + 0.5123519897460938, + 0.5215359926223755, + 0.5320799946784973 + ], + "per_start_primary_values": [ + 0.5123519897460938, + 0.5215359926223755, + 0.5320799946784973 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 1.8911904515714288, + "starts": 3 + } + }, + "comparisons": { + "deepep-v2-gin-gda_vs_deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_improvement_percent": [ + 36.06262843625914, + 41.39523297253817 + ], + "direction_supported": true, + "median_paired_improvement_percent": 40.39630190308296, + "paired_improvement_percent_per_start": [ + 40.39630190308296, + 41.39523297253817, + 36.06262843625914 + ], + "primary_metric": "slowest-rank latency in milliseconds" + }, + "deepep-v2-gin-gda_vs_uccl": { + "bootstrap_95_percent_ci_improvement_percent": [ + -21.939557206311182, + -17.186158916616627 + ], + "direction_supported": true, + "median_paired_improvement_percent": -20.04872124190306, + "paired_improvement_percent_per_start": [ + -20.04872124190306, + -17.186158916616627, + -21.939557206311182 + ], + "primary_metric": "slowest-rank latency in milliseconds" + } + }, + "dispatch_dtype": "fp8", + "input_hash_sha256": "829eba61c7ee2f8a7ff38ac4a993ee4508b1d7e6dd1e2182be31943d13e58ed2", + "primary_metric": "slowest-rank latency in milliseconds", + "route_hash_sha256": "2faaa30925b7e296347a15963d52ab5f828cccbced161eac81c84c3134f26884", + "tokens_per_rank": 128, + "workload_profile": "decode", + "world_size_ranks": 16 + }, + { + "arms": { + "deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_primary_value": [ + 1.0214080214500427, + 1.052016019821167 + ], + "median_aggregate_input_tokens_per_second": 1962016.6651629754, + "median_effective_logical_gigabytes_per_second_per_rank": 28.127470911776417, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 14.063735455888208, + "median_latency_ms": 1.0438239574432373, + "median_primary_value": 1.0438239574432373, + "per_start_median_latency_ms": [ + 1.0438239574432373, + 1.052016019821167, + 1.0214080214500427 + ], + "per_start_primary_values": [ + 1.0438239574432373, + 1.052016019821167, + 1.0214080214500427 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 1.5249277826940286, + "starts": 3 + }, + "deepep-v2-gin-gda": { + "bootstrap_95_percent_ci_primary_value": [ + 0.48580799996852875, + 0.5008960068225861 + ], + "median_aggregate_input_tokens_per_second": 4161789.511411154, + "median_effective_logical_gigabytes_per_second_per_rank": 59.663414435590305, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 29.831707217795152, + "median_latency_ms": 0.4920960068702698, + "median_primary_value": 0.4920960068702698, + "per_start_median_latency_ms": [ + 0.48580799996852875, + 0.5008960068225861, + 0.4920960068702698 + ], + "per_start_primary_values": [ + 0.48580799996852875, + 0.5008960068225861, + 0.4920960068702698 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 1.5374847658084227, + "starts": 3 + }, + "uccl": { + "bootstrap_95_percent_ci_primary_value": [ + 0.5859040021896362, + 0.5971360206604004 + ], + "median_aggregate_input_tokens_per_second": 3474201.342724331, + "median_effective_logical_gigabytes_per_second_per_rank": 49.80615044929601, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 24.903075224648006, + "median_latency_ms": 0.5894879996776581, + "median_primary_value": 0.5894879996776581, + "per_start_median_latency_ms": [ + 0.5859040021896362, + 0.5971360206604004, + 0.5894879996776581 + ], + "per_start_primary_values": [ + 0.5859040021896362, + 0.5971360206604004, + 0.5894879996776581 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.9710265933603435, + "starts": 3 + } + }, + "comparisons": { + "deepep-v2-gin-gda_vs_deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_improvement_percent": [ + 51.8217992676751, + 53.45881874962169 + ], + "direction_supported": true, + "median_paired_improvement_percent": 52.38703618717386, + "paired_improvement_percent_per_start": [ + 53.45881874962169, + 52.38703618717386, + 51.8217992676751 + ], + "primary_metric": "slowest-rank latency in milliseconds" + }, + "deepep-v2-gin-gda_vs_uccl": { + "bootstrap_95_percent_ci_improvement_percent": [ + 16.116933246026264, + 17.08402773270526 + ], + "direction_supported": true, + "median_paired_improvement_percent": 16.521454696388034, + "paired_improvement_percent_per_start": [ + 17.08402773270526, + 16.116933246026264, + 16.521454696388034 + ], + "primary_metric": "slowest-rank latency in milliseconds" + } + }, + "dispatch_dtype": "bf16", + "input_hash_sha256": "829eba61c7ee2f8a7ff38ac4a993ee4508b1d7e6dd1e2182be31943d13e58ed2", + "primary_metric": "slowest-rank latency in milliseconds", + "route_hash_sha256": "2faaa30925b7e296347a15963d52ab5f828cccbced161eac81c84c3134f26884", + "tokens_per_rank": 128, + "workload_profile": "decode", + "world_size_ranks": 16 + }, + { + "arms": { + "deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_primary_value": [ + 1.543503999710083, + 1.5656799674034119 + ], + "median_aggregate_input_tokens_per_second": 2638821.590379041, + "median_effective_logical_gigabytes_per_second_per_rank": 14.334078878938953, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 10.750559159204213, + "median_latency_ms": 1.5522080063819885, + "median_primary_value": 1.5522080063819885, + "per_start_median_latency_ms": [ + 1.5522080063819885, + 1.5656799674034119, + 1.543503999710083 + ], + "per_start_primary_values": [ + 1.5522080063819885, + 1.5656799674034119, + 1.543503999710083 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.7190825412863633, + "starts": 3 + }, + "deepep-v2-gin-gda": { + "bootstrap_95_percent_ci_primary_value": [ + 0.9453760087490082, + 0.9715520143508911 + ], + "median_aggregate_input_tokens_per_second": 4248046.16583251, + "median_effective_logical_gigabytes_per_second_per_rank": 23.075386772802197, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 17.306540079601646, + "median_latency_ms": 0.9642079770565033, + "median_primary_value": 0.9642079770565033, + "per_start_median_latency_ms": [ + 0.9642079770565033, + 0.9453760087490082, + 0.9715520143508911 + ], + "per_start_primary_values": [ + 0.9642079770565033, + 0.9453760087490082, + 0.9715520143508911 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 1.405863396881843, + "starts": 3 + }, + "uccl": { + "bootstrap_95_percent_ci_primary_value": [ + 0.7613919973373413, + 0.7740640044212341 + ], + "median_aggregate_input_tokens_per_second": 5366426.29256539, + "median_effective_logical_gigabytes_per_second_per_rank": 29.150427621215194, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 21.862820715911397, + "median_latency_ms": 0.763264000415802, + "median_primary_value": 0.763264000415802, + "per_start_median_latency_ms": [ + 0.763264000415802, + 0.7613919973373413, + 0.7740640044212341 + ], + "per_start_primary_values": [ + 0.763264000415802, + 0.7613919973373413, + 0.7740640044212341 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.8926876291866787, + "starts": 3 + } + }, + "comparisons": { + "deepep-v2-gin-gda_vs_deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_improvement_percent": [ + 37.05542619044861, + 39.61882195396173 + ], + "direction_supported": true, + "median_paired_improvement_percent": 37.88152276678711, + "paired_improvement_percent_per_start": [ + 37.88152276678711, + 39.61882195396173, + 37.05542619044861 + ], + "primary_metric": "slowest-rank latency in milliseconds" + }, + "deepep-v2-gin-gda_vs_uccl": { + "bootstrap_95_percent_ci_improvement_percent": [ + -26.326929677180296, + -24.164164064644243 + ], + "direction_supported": true, + "median_paired_improvement_percent": -25.513137001806243, + "paired_improvement_percent_per_start": [ + -26.326929677180296, + -24.164164064644243, + -25.513137001806243 + ], + "primary_metric": "slowest-rank latency in milliseconds" + } + }, + "dispatch_dtype": "fp8", + "input_hash_sha256": "f60cdd82e9a2396f8e4ae7800e19205dd438d4a179b7d4185c479cf895ffacc3", + "primary_metric": "slowest-rank latency in milliseconds", + "route_hash_sha256": "96c8e609e9944e3b151c8ad8d4f5b52d102fbd4b2757f81cf65fe21b1cf7e0e2", + "tokens_per_rank": 128, + "workload_profile": "decode", + "world_size_ranks": 32 + }, + { + "arms": { + "deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_primary_value": [ + 1.5467199683189392, + 1.5694079995155334 + ], + "median_aggregate_input_tokens_per_second": 2633988.784611133, + "median_effective_logical_gigabytes_per_second_per_rank": 18.8804316080926, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 14.160323706069452, + "median_latency_ms": 1.5550559759140015, + "median_primary_value": 1.5550559759140015, + "per_start_median_latency_ms": [ + 1.5550559759140015, + 1.5694079995155334, + 1.5467199683189392 + ], + "per_start_primary_values": [ + 1.5550559759140015, + 1.5694079995155334, + 1.5467199683189392 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.7370410374246165, + "starts": 3 + }, + "deepep-v2-gin-gda": { + "bootstrap_95_percent_ci_primary_value": [ + 0.9447679817676544, + 0.9491199851036072 + ], + "median_aggregate_input_tokens_per_second": 4322061.081339236, + "median_effective_logical_gigabytes_per_second_per_rank": 30.980533831039647, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 23.235400373279735, + "median_latency_ms": 0.9476960003376007, + "median_primary_value": 0.9476960003376007, + "per_start_median_latency_ms": [ + 0.9491199851036072, + 0.9476960003376007, + 0.9447679817676544 + ], + "per_start_primary_values": [ + 0.9491199851036072, + 0.9476960003376007, + 0.9447679817676544 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.2342596155941813, + "starts": 3 + }, + "uccl": { + "bootstrap_95_percent_ci_primary_value": [ + 0.8620480000972748, + 0.8686240017414093 + ], + "median_aggregate_input_tokens_per_second": 4733640.246388126, + "median_effective_logical_gigabytes_per_second_per_rank": 33.93073328611009, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 25.44804996458257, + "median_latency_ms": 0.8652960062026978, + "median_primary_value": 0.8652960062026978, + "per_start_median_latency_ms": [ + 0.8620480000972748, + 0.8686240017414093, + 0.8652960062026978 + ], + "per_start_primary_values": [ + 0.8620480000972748, + 0.8686240017414093, + 0.8652960062026978 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.3799833309701129, + "starts": 3 + } + }, + "comparisons": { + "deepep-v2-gin-gda_vs_deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_improvement_percent": [ + 38.91796827356664, + 39.61442782054452 + ], + "direction_supported": true, + "median_paired_improvement_percent": 38.965542089521804, + "paired_improvement_percent_per_start": [ + 38.965542089521804, + 39.61442782054452, + 38.91796827356664 + ], + "primary_metric": "slowest-rank latency in milliseconds" + }, + "deepep-v2-gin-gda_vs_uccl": { + "bootstrap_95_percent_ci_improvement_percent": [ + -10.100595906087255, + -9.103133051546882 + ], + "direction_supported": true, + "median_paired_improvement_percent": -9.18436870103156, + "paired_improvement_percent_per_start": [ + -10.100595906087255, + -9.103133051546882, + -9.18436870103156 + ], + "primary_metric": "slowest-rank latency in milliseconds" + } + }, + "dispatch_dtype": "bf16", + "input_hash_sha256": "f60cdd82e9a2396f8e4ae7800e19205dd438d4a179b7d4185c479cf895ffacc3", + "primary_metric": "slowest-rank latency in milliseconds", + "route_hash_sha256": "96c8e609e9944e3b151c8ad8d4f5b52d102fbd4b2757f81cf65fe21b1cf7e0e2", + "tokens_per_rank": 128, + "workload_profile": "decode", + "world_size_ranks": 32 + }, + { + "arms": { + "deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_primary_value": [ + 213.2319192391771, + 214.41244335810896 + ], + "median_aggregate_input_tokens_per_second": 19687953.230441127, + "median_effective_logical_gigabytes_per_second_per_rank": 213.8899238955124, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 106.9449619477562, + "median_latency_ms": 3.3287360668182373, + "median_primary_value": 213.8899238955124, + "per_start_median_latency_ms": [ + 3.3287360668182373, + 3.320623993873596, + 3.339008092880249 + ], + "per_start_primary_values": [ + 213.8899238955124, + 214.41244335810896, + 213.2319192391771 + ], + "primary_metric": "effective logical gigabytes per second per rank", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.276628898113071, + "starts": 3 + }, + "deepep-v2-gin-gda": { + "bootstrap_95_percent_ci_primary_value": [ + 252.42471250242997, + 254.64928630242528 + ], + "median_aggregate_input_tokens_per_second": 23272859.416482452, + "median_effective_logical_gigabytes_per_second_per_rank": 252.83634470066536, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 126.41817235033268, + "median_latency_ms": 2.815984010696411, + "median_primary_value": 252.83634470066536, + "per_start_median_latency_ms": [ + 2.7959359884262085, + 2.8205760717391968, + 2.815984010696411 + ], + "per_start_primary_values": [ + 254.64928630242528, + 252.42471250242997, + 252.83634470066536 + ], + "primary_metric": "effective logical gigabytes per second per rank", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.4672510049725491, + "starts": 3 + }, + "uccl": { + "bootstrap_95_percent_ci_primary_value": [ + 128.59999990696588, + 129.14415760507953 + ], + "median_aggregate_input_tokens_per_second": 11853360.76013532, + "median_effective_logical_gigabytes_per_second_per_rank": 128.7749112981101, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 64.38745564905506, + "median_latency_ms": 5.52889609336853, + "median_primary_value": 128.7749112981101, + "per_start_median_latency_ms": [ + 5.536416053771973, + 5.52889609336853, + 5.51308798789978 + ], + "per_start_primary_values": [ + 128.59999990696588, + 128.7749112981101, + 129.14415760507953 + ], + "primary_metric": "effective logical gigabytes per second per rank", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.21561850838250798, + "starts": 3 + } + }, + "comparisons": { + "deepep-v2-gin-gda_vs_deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_improvement_percent": [ + 17.72857421377984, + 19.056233068194594 + ], + "direction_supported": true, + "median_paired_improvement_percent": 18.573403833159205, + "paired_improvement_percent_per_start": [ + 19.056233068194594, + 17.72857421377984, + 18.573403833159205 + ], + "primary_metric": "effective logical gigabytes per second per rank" + }, + "deepep-v2-gin-gda_vs_uccl": { + "bootstrap_95_percent_ci_improvement_percent": [ + 95.7783839311061, + 98.01655247795354 + ], + "direction_supported": true, + "median_paired_improvement_percent": 96.02010201977482, + "paired_improvement_percent_per_start": [ + 98.01655247795354, + 96.02010201977482, + 95.7783839311061 + ], + "primary_metric": "effective logical gigabytes per second per rank" + } + }, + "dispatch_dtype": "fp8", + "input_hash_sha256": "04a406d03e5e0a9d44119896e94ceed7d3257977c5de9474cf39480200b29084", + "primary_metric": "effective logical gigabytes per second per rank", + "route_hash_sha256": "ee2cd82f82a4f909f261f1d252146085a492a4a74a0e10000a1c48e717ce0b7a", + "tokens_per_rank": 4096, + "workload_profile": "prefill", + "world_size_ranks": 16 + }, + { + "arms": { + "deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_primary_value": [ + 262.95281518810515, + 265.06324509499836 + ], + "median_aggregate_input_tokens_per_second": 18395265.03985649, + "median_effective_logical_gigabytes_per_second_per_rank": 263.7145196113826, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 131.8572598056913, + "median_latency_ms": 3.5626559257507324, + "median_primary_value": 263.7145196113826, + "per_start_median_latency_ms": [ + 3.544528007507324, + 3.572975993156433, + 3.5626559257507324 + ], + "per_start_primary_values": [ + 265.06324509499836, + 262.95281518810515, + 263.7145196113826 + ], + "primary_metric": "effective logical gigabytes per second per rank", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.40496166504210196, + "starts": 3 + }, + "deepep-v2-gin-gda": { + "bootstrap_95_percent_ci_primary_value": [ + 276.15387615995263, + 277.7540250741766 + ], + "median_aggregate_input_tokens_per_second": 19347586.023543168, + "median_effective_logical_gigabytes_per_second_per_rank": 277.36699323351485, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 138.68349661675742, + "median_latency_ms": 3.387295961380005, + "median_primary_value": 277.36699323351485, + "per_start_median_latency_ms": [ + 3.3825759887695312, + 3.387295961380005, + 3.402176022529602 + ], + "per_start_primary_values": [ + 277.7540250741766, + 277.36699323351485, + 276.15387615995263 + ], + "primary_metric": "effective logical gigabytes per second per rank", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.30129296612441553, + "starts": 3 + }, + "uccl": { + "bootstrap_95_percent_ci_primary_value": [ + 170.41603779313468, + 171.27497910590617 + ], + "median_aggregate_input_tokens_per_second": 11926565.800798107, + "median_effective_logical_gigabytes_per_second_per_rank": 170.97924732024165, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 85.48962366012083, + "median_latency_ms": 5.494959831237793, + "median_primary_value": 170.97924732024165, + "per_start_median_latency_ms": [ + 5.494959831237793, + 5.513120174407959, + 5.485471963882446 + ], + "per_start_primary_values": [ + 170.97924732024165, + 170.41603779313468, + 171.27497910590617 + ], + "primary_metric": "effective logical gigabytes per second per rank", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.25534341659717424, + "starts": 3 + } + }, + "comparisons": { + "deepep-v2-gin-gda_vs_deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_improvement_percent": [ + 4.716978256222297, + 5.481659527051807 + ], + "direction_supported": true, + "median_paired_improvement_percent": 4.787830909800359, + "paired_improvement_percent_per_start": [ + 4.787830909800359, + 5.481659527051807, + 4.716978256222297 + ], + "primary_metric": "effective logical gigabytes per second per rank" + }, + "deepep-v2-gin-gda_vs_uccl": { + "bootstrap_95_percent_ci_improvement_percent": [ + 61.23421973340055, + 62.758738452895045 + ], + "direction_supported": true, + "median_paired_improvement_percent": 62.44896935003305, + "paired_improvement_percent_per_start": [ + 62.44896935003305, + 62.758738452895045, + 61.23421973340055 + ], + "primary_metric": "effective logical gigabytes per second per rank" + } + }, + "dispatch_dtype": "bf16", + "input_hash_sha256": "04a406d03e5e0a9d44119896e94ceed7d3257977c5de9474cf39480200b29084", + "primary_metric": "effective logical gigabytes per second per rank", + "route_hash_sha256": "ee2cd82f82a4f909f261f1d252146085a492a4a74a0e10000a1c48e717ce0b7a", + "tokens_per_rank": 4096, + "workload_profile": "prefill", + "world_size_ranks": 16 + }, + { + "arms": { + "deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_primary_value": [ + 53.399459764271676, + 54.41046183143514 + ], + "median_aggregate_input_tokens_per_second": 9953621.843209853, + "median_effective_logical_gigabytes_per_second_per_rank": 54.068073852315926, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 40.55105538923694, + "median_latency_ms": 13.168272018432617, + "median_primary_value": 54.068073852315926, + "per_start_median_latency_ms": [ + 13.333151817321777, + 13.168272018432617, + 13.085408210754395 + ], + "per_start_primary_values": [ + 53.399459764271676, + 54.068073852315926, + 54.41046183143514 + ], + "primary_metric": "effective logical gigabytes per second per rank", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.9529367915600195, + "starts": 3 + }, + "deepep-v2-gin-gda": { + "bootstrap_95_percent_ci_primary_value": [ + 105.10601540037308, + 105.3551119621716 + ], + "median_aggregate_input_tokens_per_second": 19368582.567262173, + "median_effective_logical_gigabytes_per_second_per_rank": 105.21014050536813, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 78.9076053790261, + "median_latency_ms": 6.767247915267944, + "median_primary_value": 105.21014050536813, + "per_start_median_latency_ms": [ + 6.757936000823975, + 6.773952007293701, + 6.767247915267944 + ], + "per_start_primary_values": [ + 105.3551119621716, + 105.1060154003731, + 105.21014050536813 + ], + "primary_metric": "effective logical gigabytes per second per rank", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.11889443898406217, + "starts": 3 + }, + "uccl": { + "bootstrap_95_percent_ci_primary_value": [ + 85.4062152365377, + 85.49400388769168 + ], + "median_aggregate_input_tokens_per_second": 15732818.609831655, + "median_effective_logical_gigabytes_per_second_per_rank": 85.46067068860556, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 64.09550301645416, + "median_latency_ms": 8.331120014190674, + "median_primary_value": 85.46067068860556, + "per_start_median_latency_ms": [ + 8.327871799468994, + 8.331120014190674, + 8.336431980133057 + ], + "per_start_primary_values": [ + 85.49400388769168, + 85.46067068860556, + 85.4062152365377 + ], + "primary_metric": "effective logical gigabytes per second per rank", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.05185948130512096, + "starts": 3 + } + }, + "comparisons": { + "deepep-v2-gin-gda_vs_deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_improvement_percent": [ + 93.36380718713903, + 97.29621315881221 + ], + "direction_supported": true, + "median_paired_improvement_percent": 94.39570880121344, + "paired_improvement_percent_per_start": [ + 97.29621315881221, + 94.39570880121344, + 93.36380718713903 + ], + "primary_metric": "effective logical gigabytes per second per rank" + }, + "deepep-v2-gin-gda_vs_uccl": { + "bootstrap_95_percent_ci_improvement_percent": [ + 22.987585462966464, + 23.23099535795548 + ], + "direction_supported": true, + "median_paired_improvement_percent": 23.187920473916623, + "paired_improvement_percent_per_start": [ + 23.23099535795548, + 22.987585462966464, + 23.187920473916623 + ], + "primary_metric": "effective logical gigabytes per second per rank" + } + }, + "dispatch_dtype": "fp8", + "input_hash_sha256": "f49abec913e14c19c444e1599a536dde42c0ecf5399444a229f780ebbb8b2181", + "primary_metric": "effective logical gigabytes per second per rank", + "route_hash_sha256": "9b0358f379e45666ab47399443462b55fd2be657fe4a0b0de5bc0581acfd8930", + "tokens_per_rank": 4096, + "workload_profile": "prefill", + "world_size_ranks": 32 + }, + { + "arms": { + "deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_primary_value": [ + 70.48387203078285, + 71.07303480105705 + ], + "median_aggregate_input_tokens_per_second": 9898441.780961027, + "median_effective_logical_gigabytes_per_second_per_rank": 70.95203068592863, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 53.21402301444648, + "median_latency_ms": 13.241680145263672, + "median_primary_value": 70.95203068592863, + "per_start_median_latency_ms": [ + 13.241680145263672, + 13.32963228225708, + 13.219135761260986 + ], + "per_start_primary_values": [ + 70.95203068592863, + 70.48387203078285, + 71.07303480105705 + ], + "primary_metric": "effective logical gigabytes per second per rank", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.43926775013658587, + "starts": 3 + }, + "deepep-v2-gin-gda": { + "bootstrap_95_percent_ci_primary_value": [ + 108.14282406101631, + 108.36234822283106 + ], + "median_aggregate_input_tokens_per_second": 15094422.788183186, + "median_effective_logical_gigabytes_per_second_per_rank": 108.19682254569707, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 81.14761690927281, + "median_latency_ms": 8.683472156524658, + "median_primary_value": 108.19682254569707, + "per_start_median_latency_ms": [ + 8.670207977294922, + 8.6878080368042, + 8.683472156524658 + ], + "per_start_primary_values": [ + 108.36234822283106, + 108.14282406101631, + 108.19682254569707 + ], + "primary_metric": "effective logical gigabytes per second per rank", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.10568432043857662, + "starts": 3 + }, + "uccl": { + "bootstrap_95_percent_ci_primary_value": [ + 95.25844126098058, + 95.3781989559693 + ], + "median_aggregate_input_tokens_per_second": 13296240.83483305, + "median_effective_logical_gigabytes_per_second_per_rank": 95.3074543040833, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 71.48059072806247, + "median_latency_ms": 9.857823848724365, + "median_primary_value": 95.3074543040833, + "per_start_median_latency_ms": [ + 9.850512027740479, + 9.857823848724365, + 9.862895965576172 + ], + "per_start_primary_values": [ + 95.3781989559693, + 95.3074543040833, + 95.25844126098058 + ], + "primary_metric": "effective logical gigabytes per second per rank", + "run_indices_dimensionless": [ + 1, + 2, + 3 + ], + "run_to_run_cv_percent": 0.06316609677985278, + "starts": 3 + } + }, + "comparisons": { + "deepep-v2-gin-gda_vs_deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_improvement_percent": [ + 52.23329473485195, + 53.429175987645 + ], + "direction_supported": true, + "median_paired_improvement_percent": 52.726211181326654, + "paired_improvement_percent_per_start": [ + 52.726211181326654, + 53.429175987645, + 52.23329473485195 + ], + "primary_metric": "effective logical gigabytes per second per rank" + }, + "deepep-v2-gin-gda_vs_uccl": { + "bootstrap_95_percent_ci_improvement_percent": [ + 13.467330389479418, + 13.613330309220672 + ], + "direction_supported": true, + "median_paired_improvement_percent": 13.582398697107667, + "paired_improvement_percent_per_start": [ + 13.613330309220672, + 13.467330389479418, + 13.582398697107667 + ], + "primary_metric": "effective logical gigabytes per second per rank" + } + }, + "dispatch_dtype": "bf16", + "input_hash_sha256": "f49abec913e14c19c444e1599a536dde42c0ecf5399444a229f780ebbb8b2181", + "primary_metric": "effective logical gigabytes per second per rank", + "route_hash_sha256": "9b0358f379e45666ab47399443462b55fd2be657fe4a0b0de5bc0581acfd8930", + "tokens_per_rank": 4096, + "workload_profile": "prefill", + "world_size_ranks": 32 + } + ], + "comparison_scope": "synthetic decode-like and prefill-like dispatch-plus-combine communication workloads; not end-to-end training or serving", + "configuration": { + "dispatch_dtypes": [ + "fp8", + "bf16" + ], + "experts_dimensionless": 256, + "hidden_dimensions": 7168, + "measured_iterations_dimensionless": 100, + "profiles": { + "decode": { + "api_mode": "low-latency", + "primary_metric": "slowest-rank latency in milliseconds", + "tokens_per_rank": 128 + }, + "prefill": { + "api_mode": "normal", + "primary_metric": "effective logical gigabytes per second per rank", + "tokens_per_rank": 4096 + } + }, + "top_k_dimensionless": 8, + "warmup_iterations_dimensionless": 20, + "world_sizes_ranks": [ + 16, + 32 + ] + }, + "images": { + "deepep-v1-nvshmem": "159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-deepep-v1-nvshmem@sha256:175a8470e11bc7832024941d1775923d5e8ecae2c6dc06183d5f1653d66b4fac", + "deepep-v2-gin-gda": "159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-deepep-v2-gin-gda@sha256:4d07367ea290c5d6ec3c02b223ac819feed5240a46fd4a6492421e9c0853dbeb", + "uccl": "159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-uccl@sha256:a703a0d35916bbd51b2d34d968626d9617cbceb328d64b920f21ad159d93c91a" + }, + "independent_starts_per_cell": 3, + "logical_payload_definition": "per valid expert assignment: dispatch tensor plus FP8 scales when selected plus BF16 combine tensor; backend metadata excluded", + "maximum_run_to_run_cv_percent_for_direction_support": 5.0, + "runtime": { + "cuda_version": "13.0", + "gpu": "NVIDIA B200", + "nccl_version": [ + 2, + 29, + 7 + ], + "torch_version": "2.13.0+cu130" + }, + "schema_version_dimensionless": 2, + "scored_result_records_dimensionless": 72, + "status": "PASS", + "timing_boundaries": { + "decode": "BF16 input ready through dispatch and combine completion; slowest rank CUDA elapsed time", + "prefill": "BF16 input and route ready through required layout, dispatch, and combine completion; slowest rank CUDA elapsed time" + } +} diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/results/b200-us-east-1-2026-09-01-ep16-ep32-n4-boxplots.png b/micro-benchmarks/expert-parallelism/ep-backend-comparison/results/b200-us-east-1-2026-09-01-ep16-ep32-n4-boxplots.png new file mode 100644 index 000000000..0d3def1c4 Binary files /dev/null and b/micro-benchmarks/expert-parallelism/ep-backend-comparison/results/b200-us-east-1-2026-09-01-ep16-ep32-n4-boxplots.png differ diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/results/b200-us-east-1-2026-09-01-ep16-ep32-n4.json b/micro-benchmarks/expert-parallelism/ep-backend-comparison/results/b200-us-east-1-2026-09-01-ep16-ep32-n4.json new file mode 100644 index 000000000..3aa2a53f1 --- /dev/null +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/results/b200-us-east-1-2026-09-01-ep16-ep32-n4.json @@ -0,0 +1,1243 @@ +{ + "bootstrap_samples_dimensionless": 20000, + "campaign_provenance": { + "campaign_id": "ep-b200-use1-20260901023226", + "cluster": "ml-clusters-shared-us-east-1", + "comparison": { + "experts": 256, + "hidden_dimensions": 7168, + "independent_starts": 4, + "measured_iterations": 100, + "profiles": { + "decode": { + "api_mode": "low-latency", + "primary_metric": "slowest-rank latency in milliseconds", + "tokens_per_rank": 128 + }, + "prefill": { + "api_mode": "normal", + "primary_metric": "slowest-rank latency in milliseconds", + "tokens_per_rank": 4096 + } + }, + "top_k_dimensionless": 8, + "warmup_iterations": 20 + }, + "created_at_utc": "2026-09-01T02:33:40Z", + "git_commit": "df6d2943f0348510772bd6fefe50d0b22d99d70f", + "images": { + "deepep-v1-nvshmem": "159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-deepep-v1-nvshmem@sha256:175a8470e11bc7832024941d1775923d5e8ecae2c6dc06183d5f1653d66b4fac", + "deepep-v2-gin-gda": "159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-deepep-v2-gin-gda@sha256:4d07367ea290c5d6ec3c02b223ac819feed5240a46fd4a6492421e9c0853dbeb", + "uccl": "159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-uccl@sha256:a703a0d35916bbd51b2d34d968626d9617cbceb328d64b920f21ad159d93c91a" + }, + "region": "us-east-1" + }, + "cells": [ + { + "arms": { + "deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_primary_value": [ + 1.0173119902610779, + 1.0266720056533813 + ], + "median_aggregate_input_tokens_per_second": 2008238.105890834, + "median_effective_logical_gigabytes_per_second_per_rank": 21.81749878239802, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 10.90874939119901, + "median_latency_ms": 1.0198000073432922, + "median_primary_value": 1.0198000073432922, + "num_sms_dimensionless": null, + "per_start_median_latency_ms": [ + 1.0266720056533813, + 1.0190080404281616, + 1.0173119902610779, + 1.0205919742584229 + ], + "per_start_primary_values": [ + 1.0266720056533813, + 1.0190080404281616, + 1.0173119902610779, + 1.0205919742584229 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3, + 4 + ], + "run_to_run_cv_percent": 0.39934868082360114, + "starts": 4 + }, + "deepep-v2-gin-gda": { + "bootstrap_95_percent_ci_primary_value": [ + 0.4625280052423477, + 0.4785120040178299 + ], + "median_aggregate_input_tokens_per_second": 4388641.2161425175, + "median_effective_logical_gigabytes_per_second_per_rank": 47.678198172172316, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 23.839099086086158, + "median_latency_ms": 0.46667999774217606, + "median_primary_value": 0.46667999774217606, + "num_sms_dimensionless": 64, + "per_start_median_latency_ms": [ + 0.4697919934988022, + 0.4635680019855499, + 0.4785120040178299, + 0.4625280052423477 + ], + "per_start_primary_values": [ + 0.4697919934988022, + 0.4635680019855499, + 0.4785120040178299, + 0.4625280052423477 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3, + 4 + ], + "run_to_run_cv_percent": 1.5674916669424408, + "starts": 4 + }, + "uccl": { + "bootstrap_95_percent_ci_primary_value": [ + 0.5101599991321564, + 0.5261439979076385 + ], + "median_aggregate_input_tokens_per_second": 3943776.299439854, + "median_effective_logical_gigabytes_per_second_per_rank": 42.845185717114575, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 21.422592858557287, + "median_latency_ms": 0.5193120092153549, + "median_primary_value": 0.5193120092153549, + "num_sms_dimensionless": null, + "per_start_median_latency_ms": [ + 0.5167360007762909, + 0.5101599991321564, + 0.5261439979076385, + 0.521888017654419 + ], + "per_start_primary_values": [ + 0.5167360007762909, + 0.5101599991321564, + 0.5261439979076385, + 0.521888017654419 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3, + 4 + ], + "run_to_run_cv_percent": 1.3279826839964002, + "starts": 4 + } + }, + "comparisons": { + "deepep-v2-gin-gda_vs_deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_improvement_percent": [ + 52.963101919694566, + 54.68041911867596 + ], + "direction_supported": true, + "median_paired_improvement_percent": 54.3745963712967, + "paired_improvement_percent_per_start": [ + 54.24127755389383, + 54.50791518869956, + 52.963101919694566, + 54.68041911867596 + ], + "primary_metric": "slowest-rank latency in milliseconds" + }, + "deepep-v2-gin-gda_vs_uccl": { + "bootstrap_95_percent_ci_improvement_percent": [ + 9.053033785281377, + 11.374089920450698 + ], + "direction_supported": true, + "median_paired_improvement_percent": 9.108769150108744, + "paired_improvement_percent_per_start": [ + 9.084717768253977, + 9.13282053196351, + 9.053033785281377, + 11.374089920450698 + ], + "primary_metric": "slowest-rank latency in milliseconds" + } + }, + "dispatch_dtype": "fp8", + "input_hash_sha256": "829eba61c7ee2f8a7ff38ac4a993ee4508b1d7e6dd1e2182be31943d13e58ed2", + "primary_metric": "slowest-rank latency in milliseconds", + "route_hash_sha256": "2faaa30925b7e296347a15963d52ab5f828cccbced161eac81c84c3134f26884", + "tokens_per_rank": 128, + "workload_profile": "decode", + "world_size_ranks": 16 + }, + { + "arms": { + "deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_primary_value": [ + 1.029919981956482, + 1.0412319898605347 + ], + "median_aggregate_input_tokens_per_second": 1977020.2902959902, + "median_effective_logical_gigabytes_per_second_per_rank": 28.342562881683314, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 14.171281440841657, + "median_latency_ms": 1.0359120070934296, + "median_primary_value": 1.0359120070934296, + "num_sms_dimensionless": null, + "per_start_median_latency_ms": [ + 1.0390719771385193, + 1.0412319898605347, + 1.029919981956482, + 1.0327520370483398 + ], + "per_start_primary_values": [ + 1.0390719771385193, + 1.0412319898605347, + 1.029919981956482, + 1.0327520370483398 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3, + 4 + ], + "run_to_run_cv_percent": 0.5110851319848176, + "starts": 4 + }, + "deepep-v2-gin-gda": { + "bootstrap_95_percent_ci_primary_value": [ + 0.4790079891681671, + 0.5273279845714569 + ], + "median_aggregate_input_tokens_per_second": 4126801.032782062, + "median_effective_logical_gigabytes_per_second_per_rank": 59.16181960596364, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 29.58090980298182, + "median_latency_ms": 0.4963119998574257, + "median_primary_value": 0.4963119998574257, + "num_sms_dimensionless": 64, + "per_start_median_latency_ms": [ + 0.4790079891681671, + 0.5009759962558746, + 0.5273279845714569, + 0.49164800345897675 + ], + "per_start_primary_values": [ + 0.4790079891681671, + 0.5009759962558746, + 0.5273279845714569, + 0.49164800345897675 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3, + 4 + ], + "run_to_run_cv_percent": 4.0975286101088075, + "starts": 4 + }, + "uccl": { + "bootstrap_95_percent_ci_primary_value": [ + 0.5839679837226868, + 0.5956160128116608 + ], + "median_aggregate_input_tokens_per_second": 3445400.0260293, + "median_effective_logical_gigabytes_per_second_per_rank": 49.393254773156045, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 24.696627386578022, + "median_latency_ms": 0.594416007399559, + "median_primary_value": 0.594416007399559, + "num_sms_dimensionless": null, + "per_start_median_latency_ms": [ + 0.5839679837226868, + 0.5940160155296326, + 0.5948159992694855, + 0.5956160128116608 + ], + "per_start_primary_values": [ + 0.5839679837226868, + 0.5940160155296326, + 0.5948159992694855, + 0.5956160128116608 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3, + 4 + ], + "run_to_run_cv_percent": 0.9226761310329404, + "starts": 4 + } + }, + "comparisons": { + "deepep-v2-gin-gda_vs_deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_improvement_percent": [ + 48.7991306305446, + 53.90040346509024 + ], + "direction_supported": true, + "median_paired_improvement_percent": 52.14030381912623, + "paired_improvement_percent_per_start": [ + 53.90040346509024, + 51.88622697589452, + 48.7991306305446, + 52.39438066235794 + ], + "primary_metric": "slowest-rank latency in milliseconds" + }, + "deepep-v2-gin-gda_vs_uccl": { + "bootstrap_95_percent_ci_improvement_percent": [ + 11.34603218153395, + 17.97358716233368 + ], + "direction_supported": true, + "median_paired_improvement_percent": 16.55921181158276, + "paired_improvement_percent_per_start": [ + 17.97358716233368, + 15.662880602773347, + 11.34603218153395, + 17.455543020392174 + ], + "primary_metric": "slowest-rank latency in milliseconds" + } + }, + "dispatch_dtype": "bf16", + "input_hash_sha256": "829eba61c7ee2f8a7ff38ac4a993ee4508b1d7e6dd1e2182be31943d13e58ed2", + "primary_metric": "slowest-rank latency in milliseconds", + "route_hash_sha256": "2faaa30925b7e296347a15963d52ab5f828cccbced161eac81c84c3134f26884", + "tokens_per_rank": 128, + "workload_profile": "decode", + "world_size_ranks": 16 + }, + { + "arms": { + "deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_primary_value": [ + 1.542847990989685, + 1.565551996231079 + ], + "median_aggregate_input_tokens_per_second": 2631129.4593464523, + "median_effective_logical_gigabytes_per_second_per_rank": 14.29229522316993, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 10.719221417377447, + "median_latency_ms": 1.5567520260810852, + "median_primary_value": 1.5567520260810852, + "num_sms_dimensionless": null, + "per_start_median_latency_ms": [ + 1.542847990989685, + 1.565551996231079, + 1.5598400235176086, + 1.5536640286445618 + ], + "per_start_primary_values": [ + 1.542847990989685, + 1.565551996231079, + 1.5598400235176086, + 1.5536640286445618 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3, + 4 + ], + "run_to_run_cv_percent": 0.624762373014811, + "starts": 4 + }, + "deepep-v2-gin-gda": { + "bootstrap_95_percent_ci_primary_value": [ + 0.8295680284500122, + 0.8669599890708923 + ], + "median_aggregate_input_tokens_per_second": 4898703.912922395, + "median_effective_logical_gigabytes_per_second_per_rank": 26.609759654994456, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 19.95731974124584, + "median_latency_ms": 0.8361440151929855, + "median_primary_value": 0.8361440151929855, + "num_sms_dimensionless": 64, + "per_start_median_latency_ms": [ + 0.8669599890708923, + 0.8380800187587738, + 0.8342080116271973, + 0.8295680284500122 + ], + "per_start_primary_values": [ + 0.8669599890708923, + 0.8380800187587738, + 0.8342080116271973, + 0.8295680284500122 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3, + 4 + ], + "run_to_run_cv_percent": 2.0027011709302824, + "starts": 4 + }, + "uccl": { + "bootstrap_95_percent_ci_primary_value": [ + 0.7700800001621246, + 0.7799360156059265 + ], + "median_aggregate_input_tokens_per_second": 5279730.968040694, + "median_effective_logical_gigabytes_per_second_per_rank": 28.679498618397048, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 21.509623963797786, + "median_latency_ms": 0.7757999897003174, + "median_primary_value": 0.7757999897003174, + "num_sms_dimensionless": null, + "per_start_median_latency_ms": [ + 0.7772959768772125, + 0.7700800001621246, + 0.7743040025234222, + 0.7799360156059265 + ], + "per_start_primary_values": [ + 0.7772959768772125, + 0.7700800001621246, + 0.7743040025234222, + 0.7799360156059265 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3, + 4 + ], + "run_to_run_cv_percent": 0.5454967239331312, + "starts": 4 + } + }, + "comparisons": { + "deepep-v2-gin-gda_vs_deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_improvement_percent": [ + 43.80781553763008, + 46.60570025723393 + ], + "direction_supported": true, + "median_paired_improvement_percent": 46.49354292248587, + "paired_improvement_percent_per_start": [ + 43.80781553763008, + 46.46744274374958, + 46.519643101222165, + 46.60570025723393 + ], + "primary_metric": "slowest-rank latency in milliseconds" + }, + "deepep-v2-gin-gda_vs_uccl": { + "bootstrap_95_percent_ci_improvement_percent": [ + -11.53537582349327, + -6.3636005840205945 + ], + "direction_supported": true, + "median_paired_improvement_percent": -8.283375612940784, + "paired_improvement_percent_per_start": [ + -11.53537582349327, + -8.830253815491009, + -7.736497410390561, + -6.3636005840205945 + ], + "primary_metric": "slowest-rank latency in milliseconds" + } + }, + "dispatch_dtype": "fp8", + "input_hash_sha256": "f60cdd82e9a2396f8e4ae7800e19205dd438d4a179b7d4185c479cf895ffacc3", + "primary_metric": "slowest-rank latency in milliseconds", + "route_hash_sha256": "96c8e609e9944e3b151c8ad8d4f5b52d102fbd4b2757f81cf65fe21b1cf7e0e2", + "tokens_per_rank": 128, + "workload_profile": "decode", + "world_size_ranks": 32 + }, + { + "arms": { + "deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_primary_value": [ + 1.5464959740638733, + 1.569599986076355 + ], + "median_aggregate_input_tokens_per_second": 2626007.1376762423, + "median_effective_logical_gigabytes_per_second_per_rank": 18.823219162863303, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 14.11741437214748, + "median_latency_ms": 1.5598079860210419, + "median_primary_value": 1.5598079860210419, + "num_sms_dimensionless": null, + "per_start_median_latency_ms": [ + 1.5464959740638733, + 1.569599986076355, + 1.566111981868744, + 1.5535039901733398 + ], + "per_start_primary_values": [ + 1.5464959740638733, + 1.569599986076355, + 1.566111981868744, + 1.5535039901733398 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3, + 4 + ], + "run_to_run_cv_percent": 0.6923441372697804, + "starts": 4 + }, + "deepep-v2-gin-gda": { + "bootstrap_95_percent_ci_primary_value": [ + 0.9300479888916016, + 0.9611999988555908 + ], + "median_aggregate_input_tokens_per_second": 4347061.519008434, + "median_effective_logical_gigabytes_per_second_per_rank": 31.159736968252457, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 23.36980272618934, + "median_latency_ms": 0.9423200041055679, + "median_primary_value": 0.9423200041055679, + "num_sms_dimensionless": 64, + "per_start_median_latency_ms": [ + 0.9300479888916016, + 0.9339520037174225, + 0.9611999988555908, + 0.9506880044937134 + ], + "per_start_primary_values": [ + 0.9300479888916016, + 0.9339520037174225, + 0.9611999988555908, + 0.9506880044937134 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3, + 4 + ], + "run_to_run_cv_percent": 1.5426687437529003, + "starts": 4 + }, + "uccl": { + "bootstrap_95_percent_ci_primary_value": [ + 0.8695360124111176, + 0.8757120072841644 + ], + "median_aggregate_input_tokens_per_second": 4698111.155141957, + "median_effective_logical_gigabytes_per_second_per_rank": 33.676060760057545, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 25.25704557004316, + "median_latency_ms": 0.8718399852514267, + "median_primary_value": 0.8718399852514267, + "num_sms_dimensionless": null, + "per_start_median_latency_ms": [ + 0.8713759779930115, + 0.8757120072841644, + 0.8723039925098419, + 0.8695360124111176 + ], + "per_start_primary_values": [ + 0.8713759779930115, + 0.8757120072841644, + 0.8723039925098419, + 0.8695360124111176 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3, + 4 + ], + "run_to_run_cv_percent": 0.2968836879442929, + "starts": 4 + } + }, + "comparisons": { + "deepep-v2-gin-gda_vs_deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_improvement_percent": [ + 38.625078539489195, + 40.49745081534491 + ], + "direction_supported": true, + "median_paired_improvement_percent": 39.33229139967528, + "paired_improvement_percent_per_start": [ + 39.86094988352108, + 40.49745081534491, + 38.625078539489195, + 38.803632915829475 + ], + "primary_metric": "slowest-rank latency in milliseconds" + }, + "deepep-v2-gin-gda_vs_uccl": { + "bootstrap_95_percent_ci_improvement_percent": [ + -10.190943422140295, + -6.650587858658817 + ], + "direction_supported": true, + "median_paired_improvement_percent": -8.033026310534792, + "paired_improvement_percent_per_start": [ + -6.733260082946726, + -6.650587858658817, + -10.190943422140295, + -9.332792538122858 + ], + "primary_metric": "slowest-rank latency in milliseconds" + } + }, + "dispatch_dtype": "bf16", + "input_hash_sha256": "f60cdd82e9a2396f8e4ae7800e19205dd438d4a179b7d4185c479cf895ffacc3", + "primary_metric": "slowest-rank latency in milliseconds", + "route_hash_sha256": "96c8e609e9944e3b151c8ad8d4f5b52d102fbd4b2757f81cf65fe21b1cf7e0e2", + "tokens_per_rank": 128, + "workload_profile": "decode", + "world_size_ranks": 32 + }, + { + "arms": { + "deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_primary_value": [ + 2.9566080570220947, + 2.978879928588867 + ], + "median_aggregate_input_tokens_per_second": 22047219.38369958, + "median_effective_logical_gigabytes_per_second_per_rank": 239.52099138451223, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 119.76049569225611, + "median_latency_ms": 2.9725359678268433, + "median_primary_value": 2.9725359678268433, + "num_sms_dimensionless": 20, + "per_start_median_latency_ms": [ + 2.978879928588867, + 2.9679999351501465, + 2.97707200050354, + 2.9566080570220947 + ], + "per_start_primary_values": [ + 2.978879928588867, + 2.9679999351501465, + 2.97707200050354, + 2.9566080570220947 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3, + 4 + ], + "run_to_run_cv_percent": 0.34342494685555125, + "starts": 4 + }, + "deepep-v2-gin-gda": { + "bootstrap_95_percent_ci_primary_value": [ + 2.644816040992737, + 2.663935899734497 + ], + "median_aggregate_input_tokens_per_second": 24747475.940599002, + "median_effective_logical_gigabytes_per_second_per_rank": 268.8565786186675, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 134.42828930933376, + "median_latency_ms": 2.64819198846817, + "median_primary_value": 2.64819198846817, + "num_sms_dimensionless": 64, + "per_start_median_latency_ms": [ + 2.6455039978027344, + 2.644816040992737, + 2.663935899734497, + 2.650879979133606 + ], + "per_start_primary_values": [ + 2.6455039978027344, + 2.644816040992737, + 2.663935899734497, + 2.650879979133606 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3, + 4 + ], + "run_to_run_cv_percent": 0.3341620395217748, + "starts": 4 + }, + "uccl": { + "bootstrap_95_percent_ci_primary_value": [ + 5.261408090591431, + 5.28272008895874 + ], + "median_aggregate_input_tokens_per_second": 12424338.794829214, + "median_effective_logical_gigabytes_per_second_per_rank": 134.97801666702458, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 67.48900833351229, + "median_latency_ms": 5.274808049201965, + "median_primary_value": 5.274808049201965, + "num_sms_dimensionless": 20, + "per_start_median_latency_ms": [ + 5.28272008895874, + 5.275792121887207, + 5.273823976516724, + 5.261408090591431 + ], + "per_start_primary_values": [ + 5.28272008895874, + 5.275792121887207, + 5.273823976516724, + 5.261408090591431 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3, + 4 + ], + "run_to_run_cv_percent": 0.1683924197874614, + "starts": 4 + } + }, + "comparisons": { + "deepep-v2-gin-gda_vs_deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_improvement_percent": [ + 10.34050073571196, + 11.191318172533968 + ], + "direction_supported": true, + "median_paired_improvement_percent": 10.703601509586964, + "paired_improvement_percent_per_start": [ + 11.191318172533968, + 10.88894545885697, + 10.51825756031696, + 10.34050073571196 + ], + "primary_metric": "slowest-rank latency in milliseconds" + }, + "deepep-v2-gin-gda_vs_uccl": { + "bootstrap_95_percent_ci_improvement_percent": [ + 49.48758412119048, + 49.92155644717907 + ], + "direction_supported": true, + "median_paired_improvement_percent": 49.742682503991794, + "paired_improvement_percent_per_start": [ + 49.92155644717907, + 49.86883524048598, + 49.48758412119048, + 49.61652976749761 + ], + "primary_metric": "slowest-rank latency in milliseconds" + } + }, + "dispatch_dtype": "fp8", + "input_hash_sha256": "04a406d03e5e0a9d44119896e94ceed7d3257977c5de9474cf39480200b29084", + "primary_metric": "slowest-rank latency in milliseconds", + "route_hash_sha256": "ee2cd82f82a4f909f261f1d252146085a492a4a74a0e10000a1c48e717ce0b7a", + "tokens_per_rank": 4096, + "workload_profile": "prefill", + "world_size_ranks": 16 + }, + { + "arms": { + "deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_primary_value": [ + 3.519215941429138, + 3.5455039739608765 + ], + "median_aggregate_input_tokens_per_second": 18504139.06613053, + "median_effective_logical_gigabytes_per_second_per_rank": 265.27533765204726, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 132.63766882602363, + "median_latency_ms": 3.541696012020111, + "median_primary_value": 3.541696012020111, + "num_sms_dimensionless": 20, + "per_start_median_latency_ms": [ + 3.5455039739608765, + 3.5443040132522583, + 3.539088010787964, + 3.519215941429138 + ], + "per_start_primary_values": [ + 3.5455039739608765, + 3.5443040132522583, + 3.539088010787964, + 3.519215941429138 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3, + 4 + ], + "run_to_run_cv_percent": 0.3448356585861127, + "starts": 4 + }, + "deepep-v2-gin-gda": { + "bootstrap_95_percent_ci_primary_value": [ + 3.366368055343628, + 3.3743680715560913 + ], + "median_aggregate_input_tokens_per_second": 19456817.42820971, + "median_effective_logical_gigabytes_per_second_per_rank": 278.9329346508144, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 139.4664673254072, + "median_latency_ms": 3.368279993534088, + "median_primary_value": 3.368279993534088, + "num_sms_dimensionless": 64, + "per_start_median_latency_ms": [ + 3.366368055343628, + 3.3695199489593506, + 3.3743680715560913, + 3.3670400381088257 + ], + "per_start_primary_values": [ + 3.366368055343628, + 3.3695199489593506, + 3.3743680715560913, + 3.3670400381088257 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3, + 4 + ], + "run_to_run_cv_percent": 0.1076064520339789, + "starts": 4 + }, + "uccl": { + "bootstrap_95_percent_ci_primary_value": [ + 5.48633599281311, + 5.513792037963867 + ], + "median_aggregate_input_tokens_per_second": 11900024.39118531, + "median_effective_logical_gigabytes_per_second_per_rank": 170.5987496720326, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 85.2993748360163, + "median_latency_ms": 5.507216095924377, + "median_primary_value": 5.507216095924377, + "num_sms_dimensionless": 20, + "per_start_median_latency_ms": [ + 5.505568027496338, + 5.513792037963867, + 5.508864164352417, + 5.48633599281311 + ], + "per_start_primary_values": [ + 5.505568027496338, + 5.513792037963867, + 5.508864164352417, + 5.48633599281311 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3, + 4 + ], + "run_to_run_cv_percent": 0.21841615810779905, + "starts": 4 + } + }, + "comparisons": { + "deepep-v2-gin-gda_vs_deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_improvement_percent": [ + 4.324142247960906, + 5.052481112216213 + ], + "direction_supported": true, + "median_paired_improvement_percent": 4.792855958486179, + "paired_improvement_percent_per_start": [ + 5.052481112216213, + 4.931407228030803, + 4.654304688941554, + 4.324142247960906 + ], + "primary_metric": "slowest-rank latency in milliseconds" + }, + "deepep-v2-gin-gda_vs_uccl": { + "bootstrap_95_percent_ci_improvement_percent": [ + 38.62862131448896, + 38.889244901524314 + ], + "direction_supported": true, + "median_paired_improvement_percent": 38.80089123890183, + "paired_improvement_percent_per_start": [ + 38.85520915315096, + 38.889244901524314, + 38.74657332465271, + 38.62862131448896 + ], + "primary_metric": "slowest-rank latency in milliseconds" + } + }, + "dispatch_dtype": "bf16", + "input_hash_sha256": "04a406d03e5e0a9d44119896e94ceed7d3257977c5de9474cf39480200b29084", + "primary_metric": "slowest-rank latency in milliseconds", + "route_hash_sha256": "ee2cd82f82a4f909f261f1d252146085a492a4a74a0e10000a1c48e717ce0b7a", + "tokens_per_rank": 4096, + "workload_profile": "prefill", + "world_size_ranks": 16 + }, + { + "arms": { + "deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_primary_value": [ + 12.670192241668701, + 12.8504638671875 + ], + "median_aggregate_input_tokens_per_second": 10280762.549769565, + "median_effective_logical_gigabytes_per_second_per_rank": 55.84510217034827, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 41.88382662776121, + "median_latency_ms": 12.749264001846313, + "median_primary_value": 12.749264001846313, + "num_sms_dimensionless": 20, + "per_start_median_latency_ms": [ + 12.735360145568848, + 12.8504638671875, + 12.76316785812378, + 12.670192241668701 + ], + "per_start_primary_values": [ + 12.735360145568848, + 12.8504638671875, + 12.76316785812378, + 12.670192241668701 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3, + 4 + ], + "run_to_run_cv_percent": 0.5859717138222018, + "starts": 4 + }, + "deepep-v2-gin-gda": { + "bootstrap_95_percent_ci_primary_value": [ + 6.618959903717041, + 6.63428807258606 + ], + "median_aggregate_input_tokens_per_second": 19791699.498042516, + "median_effective_logical_gigabytes_per_second_per_rank": 107.50851167336694, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 80.63138375502521, + "median_latency_ms": 6.622575998306274, + "median_primary_value": 6.622575998306274, + "num_sms_dimensionless": 64, + "per_start_median_latency_ms": [ + 6.618959903717041, + 6.6259520053863525, + 6.63428807258606, + 6.619199991226196 + ], + "per_start_primary_values": [ + 6.618959903717041, + 6.6259520053863525, + 6.63428807258606, + 6.619199991226196 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3, + 4 + ], + "run_to_run_cv_percent": 0.10908253927471676, + "starts": 4 + }, + "uccl": { + "bootstrap_95_percent_ci_primary_value": [ + 8.069119930267334, + 8.118288040161133 + ], + "median_aggregate_input_tokens_per_second": 16202922.980013046, + "median_effective_logical_gigabytes_per_second_per_rank": 88.01427762743087, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 66.01070822057315, + "median_latency_ms": 8.089407920837402, + "median_primary_value": 8.089407920837402, + "num_sms_dimensionless": 20, + "per_start_median_latency_ms": [ + 8.118288040161133, + 8.084239959716797, + 8.094575881958008, + 8.069119930267334 + ], + "per_start_primary_values": [ + 8.118288040161133, + 8.084239959716797, + 8.094575881958008, + 8.069119930267334 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3, + 4 + ], + "run_to_run_cv_percent": 0.25533951874322847, + "starts": 4 + } + }, + "comparisons": { + "deepep-v2-gin-gda_vs_deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_improvement_percent": [ + 47.757698818037596, + 48.43803247986149 + ], + "direction_supported": true, + "median_paired_improvement_percent": 48.023481902631296, + "paired_improvement_percent_per_start": [ + 48.026912250141216, + 48.4380324798615, + 48.02005155512138, + 47.757698818037596 + ], + "primary_metric": "slowest-rank latency in milliseconds" + }, + "deepep-v2-gin-gda_vs_uccl": { + "bootstrap_95_percent_ci_improvement_percent": [ + 17.968749399825825, + 18.468525987584115 + ], + "direction_supported": true, + "median_paired_improvement_percent": 18.039488834173856, + "paired_improvement_percent_per_start": [ + 18.468525987584115, + 18.038652509042176, + 18.040325159305535, + 17.968749399825825 + ], + "primary_metric": "slowest-rank latency in milliseconds" + } + }, + "dispatch_dtype": "fp8", + "input_hash_sha256": "f49abec913e14c19c444e1599a536dde42c0ecf5399444a229f780ebbb8b2181", + "primary_metric": "slowest-rank latency in milliseconds", + "route_hash_sha256": "9b0358f379e45666ab47399443462b55fd2be657fe4a0b0de5bc0581acfd8930", + "tokens_per_rank": 4096, + "workload_profile": "prefill", + "world_size_ranks": 32 + }, + { + "arms": { + "deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_primary_value": [ + 13.16048002243042, + 13.215904235839844 + ], + "median_aggregate_input_tokens_per_second": 9947071.174950577, + "median_effective_logical_gigabytes_per_second_per_rank": 71.30060618204574, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 53.47545463653431, + "median_latency_ms": 13.176944017410278, + "median_primary_value": 13.176944017410278, + "num_sms_dimensionless": 20, + "per_start_median_latency_ms": [ + 13.177056312561035, + 13.215904235839844, + 13.176831722259521, + 13.16048002243042 + ], + "per_start_primary_values": [ + 13.177056312561035, + 13.215904235839844, + 13.176831722259521, + 13.16048002243042 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3, + 4 + ], + "run_to_run_cv_percent": 0.17857302707002398, + "starts": 4 + }, + "deepep-v2-gin-gda": { + "bootstrap_95_percent_ci_primary_value": [ + 8.703423976898193, + 8.72648000717163 + ], + "median_aggregate_input_tokens_per_second": 15048865.626301043, + "median_effective_logical_gigabytes_per_second_per_rank": 107.87026880932588, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 80.90270160699441, + "median_latency_ms": 8.709759950637817, + "median_primary_value": 8.709759950637817, + "num_sms_dimensionless": 64, + "per_start_median_latency_ms": [ + 8.7117919921875, + 8.703423976898193, + 8.72648000717163, + 8.707727909088135 + ], + "per_start_primary_values": [ + 8.7117919921875, + 8.703423976898193, + 8.72648000717163, + 8.707727909088135 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3, + 4 + ], + "run_to_run_cv_percent": 0.11497179134425689, + "starts": 4 + }, + "uccl": { + "bootstrap_95_percent_ci_primary_value": [ + 9.878128051757812, + 9.902144432067871 + ], + "median_aggregate_input_tokens_per_second": 13253295.684015695, + "median_effective_logical_gigabytes_per_second_per_rank": 94.99962346302449, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 71.24971759726836, + "median_latency_ms": 9.889768123626709, + "median_primary_value": 9.889768123626709, + "num_sms_dimensionless": 20, + "per_start_median_latency_ms": [ + 9.878128051757812, + 9.885791778564453, + 9.893744468688965, + 9.902144432067871 + ], + "per_start_primary_values": [ + 9.878128051757812, + 9.885791778564453, + 9.893744468688965, + 9.902144432067871 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3, + 4 + ], + "run_to_run_cv_percent": 0.10445347483217275, + "starts": 4 + } + }, + "comparisons": { + "deepep-v2-gin-gda_vs_deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_improvement_percent": [ + 33.774065032415535, + 34.1443171682826 + ], + "direction_supported": true, + "median_paired_improvement_percent": 33.86046431203877, + "paired_improvement_percent_per_start": [ + 33.88666037737897, + 34.1443171682826, + 33.774065032415535, + 33.83426824669857 + ], + "primary_metric": "slowest-rank latency in milliseconds" + }, + "deepep-v2-gin-gda_vs_uccl": { + "bootstrap_95_percent_ci_improvement_percent": [ + 11.798004943541967, + 12.062200578610481 + ], + "direction_supported": true, + "median_paired_improvement_percent": 11.88376606476688, + "paired_improvement_percent_per_start": [ + 11.80725794866329, + 11.960274180870469, + 11.798004943541967, + 12.062200578610481 + ], + "primary_metric": "slowest-rank latency in milliseconds" + } + }, + "dispatch_dtype": "bf16", + "input_hash_sha256": "f49abec913e14c19c444e1599a536dde42c0ecf5399444a229f780ebbb8b2181", + "primary_metric": "slowest-rank latency in milliseconds", + "route_hash_sha256": "9b0358f379e45666ab47399443462b55fd2be657fe4a0b0de5bc0581acfd8930", + "tokens_per_rank": 4096, + "workload_profile": "prefill", + "world_size_ranks": 32 + } + ], + "comparison_scope": "synthetic decode-like and prefill-like dispatch-plus-combine communication workloads; not end-to-end training or serving", + "configuration": { + "dispatch_dtypes": [ + "fp8", + "bf16" + ], + "experts_dimensionless": 256, + "hidden_dimensions": 7168, + "measured_iterations_dimensionless": 100, + "profiles": { + "decode": { + "api_mode": "low-latency", + "primary_metric": "slowest-rank latency in milliseconds", + "tokens_per_rank": 128 + }, + "prefill": { + "api_mode": "normal", + "primary_metric": "slowest-rank latency in milliseconds", + "tokens_per_rank": 4096 + } + }, + "top_k_dimensionless": 8, + "warmup_iterations_dimensionless": 20, + "world_sizes_ranks": [ + 16, + 32 + ] + }, + "images": { + "deepep-v1-nvshmem": "159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-deepep-v1-nvshmem@sha256:175a8470e11bc7832024941d1775923d5e8ecae2c6dc06183d5f1653d66b4fac", + "deepep-v2-gin-gda": "159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-deepep-v2-gin-gda@sha256:4d07367ea290c5d6ec3c02b223ac819feed5240a46fd4a6492421e9c0853dbeb", + "uccl": "159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-uccl@sha256:a703a0d35916bbd51b2d34d968626d9617cbceb328d64b920f21ad159d93c91a" + }, + "independent_starts_per_cell": 4, + "loaded_nccl_versions_per_arm": { + "deepep-v1-nvshmem": [ + 2, + 31, + 2 + ], + "deepep-v2-gin-gda": [ + 2, + 31, + 2 + ], + "uccl": [ + 2, + 31, + 2 + ] + }, + "logical_payload_definition": "per valid expert assignment: dispatch tensor plus FP8 scales when selected plus BF16 combine tensor; backend metadata excluded", + "maximum_run_to_run_cv_percent_for_direction_support": 5.0, + "runtime": { + "cuda_version": "13.0", + "gpu": "NVIDIA B200", + "nccl_version": [ + 2, + 29, + 7 + ], + "torch_version": "2.13.0+cu130" + }, + "schema_version_dimensionless": 2, + "scored_result_records_dimensionless": 96, + "status": "PASS", + "timing_boundaries": { + "decode": "dispatch input ready through dispatch and combine completion; slowest rank CUDA elapsed time; host-side FP8 conversion precedes the boundary", + "prefill": "dispatch input and route ready through required layout, dispatch, and combine completion; slowest rank CUDA elapsed time; host-side FP8 conversion precedes the boundary" + } +} diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/results/b200-us-east-1-2026-09-01-ep16-n16-boxplots.png b/micro-benchmarks/expert-parallelism/ep-backend-comparison/results/b200-us-east-1-2026-09-01-ep16-n16-boxplots.png new file mode 100644 index 000000000..75cb8f72a Binary files /dev/null and b/micro-benchmarks/expert-parallelism/ep-backend-comparison/results/b200-us-east-1-2026-09-01-ep16-n16-boxplots.png differ diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/results/b200-us-east-1-2026-09-01-ep16-n16.json b/micro-benchmarks/expert-parallelism/ep-backend-comparison/results/b200-us-east-1-2026-09-01-ep16-n16.json new file mode 100644 index 000000000..8914d7525 --- /dev/null +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/results/b200-us-east-1-2026-09-01-ep16-n16.json @@ -0,0 +1,1202 @@ +{ + "bootstrap_samples_dimensionless": 20000, + "campaign_provenance": { + "campaign_id": "ep-b200-use1-n16-20260901054057", + "cluster": "ml-clusters-shared-us-east-1", + "comparison": { + "experts": 256, + "hidden_dimensions": 7168, + "independent_starts": 16, + "measured_iterations": 100, + "profiles": { + "decode": { + "api_mode": "low-latency", + "primary_metric": "slowest-rank latency in milliseconds", + "tokens_per_rank": 128 + }, + "prefill": { + "api_mode": "normal", + "primary_metric": "slowest-rank latency in milliseconds", + "tokens_per_rank": 4096 + } + }, + "top_k_dimensionless": 8, + "warmup_iterations": 20 + }, + "created_at_utc": "2026-09-01T05:42:16Z", + "git_commit": "40990df24e49510bc4eea7fde8d4f00f74bf184b", + "images": { + "deepep-v1-nvshmem": "159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-deepep-v1-nvshmem@sha256:175a8470e11bc7832024941d1775923d5e8ecae2c6dc06183d5f1653d66b4fac", + "deepep-v2-gin-gda": "159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-deepep-v2-gin-gda@sha256:4d07367ea290c5d6ec3c02b223ac819feed5240a46fd4a6492421e9c0853dbeb", + "uccl": "159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-uccl@sha256:a703a0d35916bbd51b2d34d968626d9617cbceb328d64b920f21ad159d93c91a" + }, + "region": "us-east-1" + }, + "cells": [ + { + "arms": { + "deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_primary_value": [ + 1.0242639780044556, + 1.0345919728279114 + ], + "median_aggregate_input_tokens_per_second": 1995323.9889498046, + "median_effective_logical_gigabytes_per_second_per_rank": 21.67719981595068, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 10.83859990797534, + "median_latency_ms": 1.0263999998569489, + "median_primary_value": 1.0263999998569489, + "num_sms_dimensionless": null, + "per_start_median_latency_ms": [ + 1.040544033050537, + 1.0382559895515442, + 1.0163040161132812, + 1.0211520195007324, + 1.025871992111206, + 1.0247679948806763, + 1.0244640111923218, + 1.0691519975662231, + 1.0377600193023682, + 1.0345919728279114, + 1.0300480127334595, + 1.0235039591789246, + 1.007856011390686, + 1.0250239968299866, + 1.0269280076026917, + 1.0294399857521057 + ], + "per_start_primary_values": [ + 1.040544033050537, + 1.0382559895515442, + 1.0163040161132812, + 1.0211520195007324, + 1.025871992111206, + 1.0247679948806763, + 1.0244640111923218, + 1.0691519975662231, + 1.0377600193023682, + 1.0345919728279114, + 1.0300480127334595, + 1.0235039591789246, + 1.007856011390686, + 1.0250239968299866, + 1.0269280076026917, + 1.0294399857521057 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16 + ], + "run_to_run_cv_percent": 1.3029893452113868, + "starts": 16 + }, + "deepep-v2-gin-gda": { + "bootstrap_95_percent_ci_primary_value": [ + 0.46638399362564087, + 0.4792480021715164 + ], + "median_aggregate_input_tokens_per_second": 4346582.455675562, + "median_effective_logical_gigabytes_per_second_per_rank": 47.2212717984593, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 23.61063589922965, + "median_latency_ms": 0.47117599844932556, + "median_primary_value": 0.47117599844932556, + "num_sms_dimensionless": 64, + "per_start_median_latency_ms": [ + 0.4719360023736954, + 0.46638399362564087, + 0.4827200025320053, + 0.4623840004205704, + 0.4740000069141388, + 0.46116800606250763, + 0.47841599583625793, + 0.46167999505996704, + 0.4668480008840561, + 0.4635999947786331, + 0.4700160026550293, + 0.4803519994020462, + 0.4792480021715164, + 0.47041599452495575, + 0.5064640045166016, + 0.4843519926071167 + ], + "per_start_primary_values": [ + 0.4719360023736954, + 0.46638399362564087, + 0.4827200025320053, + 0.4623840004205704, + 0.4740000069141388, + 0.46116800606250763, + 0.47841599583625793, + 0.46167999505996704, + 0.4668480008840561, + 0.4635999947786331, + 0.4700160026550293, + 0.4803519994020462, + 0.4792480021715164, + 0.47041599452495575, + 0.5064640045166016, + 0.4843519926071167 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16 + ], + "run_to_run_cv_percent": 2.4526523502177042, + "starts": 16 + }, + "uccl": { + "bootstrap_95_percent_ci_primary_value": [ + 0.5237119793891907, + 0.5283839702606201 + ], + "median_aggregate_input_tokens_per_second": 3895017.0835897475, + "median_effective_logical_gigabytes_per_second_per_rank": 42.31546559611901, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 21.157732798059506, + "median_latency_ms": 0.5258000046014786, + "median_primary_value": 0.5258000046014786, + "num_sms_dimensionless": null, + "per_start_median_latency_ms": [ + 0.5257920026779175, + 0.5237119793891907, + 0.5337440073490143, + 0.5297439992427826, + 0.5250400006771088, + 0.5258080065250397, + 0.519567996263504, + 0.5173919796943665, + 0.5283839702606201, + 0.5225279927253723, + 0.5311999917030334, + 0.5283519923686981, + 0.5171519815921783, + 0.5244799852371216, + 0.5279200077056885, + 0.5316960215568542 + ], + "per_start_primary_values": [ + 0.5257920026779175, + 0.5237119793891907, + 0.5337440073490143, + 0.5297439992427826, + 0.5250400006771088, + 0.5258080065250397, + 0.519567996263504, + 0.5173919796943665, + 0.5283839702606201, + 0.5225279927253723, + 0.5311999917030334, + 0.5283519923686981, + 0.5171519815921783, + 0.5244799852371216, + 0.5279200077056885, + 0.5316960215568542 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16 + ], + "run_to_run_cv_percent": 0.9342984127532674, + "starts": 16 + } + }, + "comparisons": { + "deepep-v2-gin-gda_vs_deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_improvement_percent": [ + 53.06789044691198, + 54.99781332298479 + ], + "direction_supported": true, + "median_paired_improvement_percent": 54.238168239153296, + "paired_improvement_percent_per_start": [ + 54.64526369055884, + 55.080057488800335, + 52.502401360362285, + 54.719376587372196, + 53.79540424544932, + 54.99781332298479, + 53.30084896984778, + 56.81811415861189, + 55.01387679225746, + 55.19006459025119, + 54.36950541676807, + 53.06789044691198, + 52.44876284359032, + 54.10683106153852, + 50.68164459756876, + 52.94995343966063 + ], + "primary_metric": "slowest-rank latency in milliseconds" + }, + "deepep-v2-gin-gda_vs_uccl": { + "bootstrap_95_percent_ci_improvement_percent": [ + 9.08485132259258, + 11.277481544938068 + ], + "direction_supported": true, + "median_paired_improvement_percent": 10.275473448565037, + "paired_improvement_percent_per_start": [ + 10.242833673758344, + 10.946472110569607, + 9.559639848779504, + 12.715575621148473, + 9.721162901330779, + 12.29346066632285, + 7.9204263394189995, + 10.76784852121395, + 11.646070441200559, + 11.277481544938068, + 11.518070407314497, + 9.08485132259258, + 7.329369463878935, + 10.308113223371732, + 4.0642527041802285, + 8.904341396256818 + ], + "primary_metric": "slowest-rank latency in milliseconds" + } + }, + "dispatch_dtype": "fp8", + "input_hash_sha256": "829eba61c7ee2f8a7ff38ac4a993ee4508b1d7e6dd1e2182be31943d13e58ed2", + "primary_metric": "slowest-rank latency in milliseconds", + "route_hash_sha256": "2faaa30925b7e296347a15963d52ab5f828cccbced161eac81c84c3134f26884", + "tokens_per_rank": 128, + "workload_profile": "decode", + "world_size_ranks": 16 + }, + { + "arms": { + "deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_primary_value": [ + 1.0326720476150513, + 1.0391919910907745 + ], + "median_aggregate_input_tokens_per_second": 1977201.7892317297, + "median_effective_logical_gigabytes_per_second_per_rank": 28.34516485042608, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 14.17258242521304, + "median_latency_ms": 1.0358079969882965, + "median_primary_value": 1.0358079969882965, + "num_sms_dimensionless": null, + "per_start_median_latency_ms": [ + 1.0503199696540833, + 1.0468639731407166, + 1.0305759906768799, + 1.0346720218658447, + 1.0416159629821777, + 1.0297279953956604, + 1.0366719961166382, + 1.0349439978599548, + 1.0431360006332397, + 1.0386079549789429, + 1.0367680191993713, + 1.0323039889335632, + 1.0223039984703064, + 1.0345119833946228, + 1.0371520519256592, + 1.0326720476150513 + ], + "per_start_primary_values": [ + 1.0503199696540833, + 1.0468639731407166, + 1.0305759906768799, + 1.0346720218658447, + 1.0416159629821777, + 1.0297279953956604, + 1.0366719961166382, + 1.0349439978599548, + 1.0431360006332397, + 1.0386079549789429, + 1.0367680191993713, + 1.0323039889335632, + 1.0223039984703064, + 1.0345119833946228, + 1.0371520519256592, + 1.0326720476150513 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16 + ], + "run_to_run_cv_percent": 0.6581562338576438, + "starts": 16 + }, + "deepep-v2-gin-gda": { + "bootstrap_95_percent_ci_primary_value": [ + 0.48102399706840515, + 0.5228640139102936 + ], + "median_aggregate_input_tokens_per_second": 4231973.475250162, + "median_effective_logical_gigabytes_per_second_per_rank": 60.66957174118632, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 30.33478587059316, + "median_latency_ms": 0.48393600434064865, + "median_primary_value": 0.48393600434064865, + "num_sms_dimensionless": 64, + "per_start_median_latency_ms": [ + 0.5306079983711243, + 0.4832320064306259, + 0.48294399678707123, + 0.530239999294281, + 0.48552000522613525, + 0.48019200563430786, + 0.4766879975795746, + 0.48236799240112305, + 0.47815999388694763, + 0.4769279956817627, + 0.48102399706840515, + 0.5263200104236603, + 0.5228640139102936, + 0.49035198986530304, + 0.5284479856491089, + 0.4846400022506714 + ], + "per_start_primary_values": [ + 0.5306079983711243, + 0.4832320064306259, + 0.48294399678707123, + 0.530239999294281, + 0.48552000522613525, + 0.48019200563430786, + 0.4766879975795746, + 0.48236799240112305, + 0.47815999388694763, + 0.4769279956817627, + 0.48102399706840515, + 0.5263200104236603, + 0.5228640139102936, + 0.49035198986530304, + 0.5284479856491089, + 0.4846400022506714 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16 + ], + "run_to_run_cv_percent": 4.47001150341075, + "starts": 16 + }, + "uccl": { + "bootstrap_95_percent_ci_primary_value": [ + 0.5943200141191483, + 0.6011680066585541 + ], + "median_aggregate_input_tokens_per_second": 3421595.89930885, + "median_effective_logical_gigabytes_per_second_per_rank": 49.051998812491675, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 24.525999406245838, + "median_latency_ms": 0.5985519886016846, + "median_primary_value": 0.5985519886016846, + "num_sms_dimensionless": null, + "per_start_median_latency_ms": [ + 0.6024959981441498, + 0.5957279801368713, + 0.6045440137386322, + 0.6013279855251312, + 0.597823977470398, + 0.5992799997329712, + 0.5919199883937836, + 0.5923359990119934, + 0.6010240018367767, + 0.59620800614357, + 0.6011680066585541, + 0.5943839848041534, + 0.5924320220947266, + 0.6037279963493347, + 0.6006560027599335, + 0.5923999845981598 + ], + "per_start_primary_values": [ + 0.6024959981441498, + 0.5957279801368713, + 0.6045440137386322, + 0.6013279855251312, + 0.597823977470398, + 0.5992799997329712, + 0.5919199883937836, + 0.5923359990119934, + 0.6010240018367767, + 0.59620800614357, + 0.6011680066585541, + 0.5943839848041534, + 0.5924320220947266, + 0.6037279963493347, + 0.6006560027599335, + 0.5923999845981598 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16 + ], + "run_to_run_cv_percent": 0.7323590396181588, + "starts": 16 + } + }, + "comparisons": { + "deepep-v2-gin-gda_vs_deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_improvement_percent": [ + 49.48129963235138, + 53.61595676564933 + ], + "direction_supported": true, + "median_paired_improvement_percent": 53.252770493762526, + "paired_improvement_percent_per_start": [ + 49.48129963235138, + 53.84003855048404, + 53.138438974318156, + 48.75284263141777, + 53.38781062493734, + 53.3671020132069, + 54.01747135398251, + 53.39187498081461, + 54.161298853008745, + 54.080074835221914, + 53.60350742301361, + 49.01501727534901, + 48.85435108415253, + 52.600646707225785, + 49.04816659544276, + 53.069321148960704 + ], + "primary_metric": "slowest-rank latency in milliseconds" + }, + "deepep-v2-gin-gda_vs_uccl": { + "bootstrap_95_percent_ci_improvement_percent": [ + 12.021525928158292, + 19.871845239575304 + ], + "direction_supported": true, + "median_paired_improvement_percent": 18.782388516591446, + "paired_improvement_percent_per_start": [ + 11.931697470930919, + 18.883782104778582, + 20.114336456589804, + 11.821832334773191, + 18.785458007131133, + 19.871845239575304, + 19.467494437364596, + 18.565139852093264, + 20.44244615428785, + 20.0064422538271, + 19.985097054306024, + 11.451179056064221, + 11.74278323755252, + 18.77931902605176, + 12.021525928158292, + 18.19040937696593 + ], + "primary_metric": "slowest-rank latency in milliseconds" + } + }, + "dispatch_dtype": "bf16", + "input_hash_sha256": "829eba61c7ee2f8a7ff38ac4a993ee4508b1d7e6dd1e2182be31943d13e58ed2", + "primary_metric": "slowest-rank latency in milliseconds", + "route_hash_sha256": "2faaa30925b7e296347a15963d52ab5f828cccbced161eac81c84c3134f26884", + "tokens_per_rank": 128, + "workload_profile": "decode", + "world_size_ranks": 16 + }, + { + "arms": { + "deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_primary_value": [ + 2.9700000286102295, + 2.9862719774246216 + ], + "median_aggregate_input_tokens_per_second": 22034892.86585099, + "median_effective_logical_gigabytes_per_second_per_rank": 239.38707609460516, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 119.69353804730258, + "median_latency_ms": 2.974191963672638, + "median_primary_value": 2.974191963672638, + "num_sms_dimensionless": 20, + "per_start_median_latency_ms": [ + 2.9862719774246216, + 2.973039984703064, + 3.0060319900512695, + 2.97268807888031, + 2.9737759828567505, + 2.968208074569702, + 2.9801440238952637, + 2.9301120042800903, + 2.9700000286102295, + 2.9865280389785767, + 2.9746079444885254, + 3.0014560222625732, + 2.961632013320923, + 2.9690719842910767, + 2.9903680086135864, + 2.9760000705718994 + ], + "per_start_primary_values": [ + 2.9862719774246216, + 2.973039984703064, + 3.0060319900512695, + 2.97268807888031, + 2.9737759828567505, + 2.968208074569702, + 2.9801440238952637, + 2.9301120042800903, + 2.9700000286102295, + 2.9865280389785767, + 2.9746079444885254, + 3.0014560222625732, + 2.961632013320923, + 2.9690719842910767, + 2.9903680086135864, + 2.9760000705718994 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16 + ], + "run_to_run_cv_percent": 0.5801707814040735, + "starts": 16 + }, + "deepep-v2-gin-gda": { + "bootstrap_95_percent_ci_primary_value": [ + 2.6516640186309814, + 2.6614160537719727 + ], + "median_aggregate_input_tokens_per_second": 24657617.211329274, + "median_effective_logical_gigabytes_per_second_per_rank": 267.88035338388124, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 133.94017669194062, + "median_latency_ms": 2.6578399538993835, + "median_primary_value": 2.6578399538993835, + "num_sms_dimensionless": 64, + "per_start_median_latency_ms": [ + 2.654911994934082, + 2.6537439823150635, + 2.6510239839553833, + 2.6516640186309814, + 2.657983899116516, + 2.6636160612106323, + 2.651471972465515, + 2.666848063468933, + 2.6473119258880615, + 2.6474239826202393, + 2.6600000858306885, + 2.657696008682251, + 2.664896011352539, + 2.6684958934783936, + 2.660144090652466, + 2.659216046333313 + ], + "per_start_primary_values": [ + 2.654911994934082, + 2.6537439823150635, + 2.6510239839553833, + 2.6516640186309814, + 2.657983899116516, + 2.6636160612106323, + 2.651471972465515, + 2.666848063468933, + 2.6473119258880615, + 2.6474239826202393, + 2.6600000858306885, + 2.657696008682251, + 2.664896011352539, + 2.6684958934783936, + 2.660144090652466, + 2.659216046333313 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16 + ], + "run_to_run_cv_percent": 0.248590406198716, + "starts": 16 + }, + "uccl": { + "bootstrap_95_percent_ci_primary_value": [ + 5.266440033912659, + 5.27892804145813 + ], + "median_aggregate_input_tokens_per_second": 12437731.749601986, + "median_effective_logical_gigabytes_per_second_per_rank": 135.123517727676, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 67.561758863838, + "median_latency_ms": 5.269128084182739, + "median_primary_value": 5.269128084182739, + "num_sms_dimensionless": 20, + "per_start_median_latency_ms": [ + 5.264800071716309, + 5.269968032836914, + 5.28982400894165, + 5.2534401416778564, + 5.27892804145813, + 5.267823934555054, + 5.267439842224121, + 5.268079996109009, + 5.300639867782593, + 5.2849280834198, + 5.277519941329956, + 5.2682881355285645, + 5.252239942550659, + 5.271360158920288, + 5.25816011428833, + 5.278991937637329 + ], + "per_start_primary_values": [ + 5.264800071716309, + 5.269968032836914, + 5.28982400894165, + 5.2534401416778564, + 5.27892804145813, + 5.267823934555054, + 5.267439842224121, + 5.268079996109009, + 5.300639867782593, + 5.2849280834198, + 5.277519941329956, + 5.2682881355285645, + 5.252239942550659, + 5.271360158920288, + 5.25816011428833, + 5.278991937637329 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16 + ], + "run_to_run_cv_percent": 0.24402472835165115, + "starts": 16 + } + }, + "comparisons": { + "deepep-v2-gin-gda_vs_deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_improvement_percent": [ + 10.453219473067161, + 11.042919032370907 + ], + "direction_supported": true, + "median_paired_improvement_percent": 10.769415625979349, + "paired_improvement_percent_per_start": [ + 11.096108626258024, + 10.73971436747732, + 11.809854561455662, + 10.799116884481377, + 10.61922907309479, + 10.261814728174883, + 11.028730450421333, + 8.984773975418031, + 10.864919178912109, + 11.354457481481221, + 10.57644787242484, + 11.45310845904673, + 10.019340709234474, + 10.12357034127118, + 11.042919032370907, + 10.644624217959441 + ], + "primary_metric": "slowest-rank latency in milliseconds" + }, + "deepep-v2-gin-gda_vs_uccl": { + "bootstrap_95_percent_ci_improvement_percent": [ + 49.43612211983287, + 49.649173501854825 + ], + "direction_supported": true, + "median_paired_improvement_percent": 49.58497112182044, + "paired_improvement_percent_per_start": [ + 49.572406192651705, + 49.644021258199025, + 49.88445779152148, + 49.52518831243242, + 49.649173501854825, + 49.43612211983287, + 49.66298520941515, + 49.377229171943846, + 50.05674801681052, + 49.90614932063314, + 49.59753605098917, + 49.5529488837344, + 49.26172375022192, + 49.377469703664275, + 49.409222373736995, + 49.626442363473764 + ], + "primary_metric": "slowest-rank latency in milliseconds" + } + }, + "dispatch_dtype": "fp8", + "input_hash_sha256": "04a406d03e5e0a9d44119896e94ceed7d3257977c5de9474cf39480200b29084", + "primary_metric": "slowest-rank latency in milliseconds", + "route_hash_sha256": "ee2cd82f82a4f909f261f1d252146085a492a4a74a0e10000a1c48e717ce0b7a", + "tokens_per_rank": 4096, + "workload_profile": "prefill", + "world_size_ranks": 16 + }, + { + "arms": { + "deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_primary_value": [ + 3.5396639108657837, + 3.5512959957122803 + ], + "median_aggregate_input_tokens_per_second": 18487721.7398634, + "median_effective_logical_gigabytes_per_second_per_rank": 265.0399788626817, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 132.51998943134086, + "median_latency_ms": 3.544839918613434, + "median_primary_value": 3.544839918613434, + "num_sms_dimensionless": 20, + "per_start_median_latency_ms": [ + 3.558351993560791, + 3.5400320291519165, + 3.543199896812439, + 3.5464799404144287, + 3.543168067932129, + 3.5387840270996094, + 3.5504961013793945, + 3.5054879188537598, + 3.538383960723877, + 3.551136016845703, + 3.5488799810409546, + 3.566208004951477, + 3.5309600830078125, + 3.556112051010132, + 3.5565600395202637, + 3.5396639108657837 + ], + "per_start_primary_values": [ + 3.558351993560791, + 3.5400320291519165, + 3.543199896812439, + 3.5464799404144287, + 3.543168067932129, + 3.5387840270996094, + 3.5504961013793945, + 3.5054879188537598, + 3.538383960723877, + 3.551136016845703, + 3.5488799810409546, + 3.566208004951477, + 3.5309600830078125, + 3.556112051010132, + 3.5565600395202637, + 3.5396639108657837 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16 + ], + "run_to_run_cv_percent": 0.390127984054333, + "starts": 16 + }, + "deepep-v2-gin-gda": { + "bootstrap_95_percent_ci_primary_value": [ + 3.365935981273651, + 3.374000072479248 + ], + "median_aggregate_input_tokens_per_second": 19441899.666362844, + "median_effective_logical_gigabytes_per_second_per_rank": 278.7190736169777, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 139.35953680848885, + "median_latency_ms": 3.3708640336990356, + "median_primary_value": 3.3708640336990356, + "num_sms_dimensionless": 64, + "per_start_median_latency_ms": [ + 3.3604960441589355, + 3.3787840604782104, + 3.371840000152588, + 3.37062406539917, + 3.3606879711151123, + 3.367311954498291, + 3.3587359189987183, + 3.3612959384918213, + 3.398368000984192, + 3.374000072479248, + 3.3732799291610718, + 3.3711040019989014, + 3.3763999938964844, + 3.370576024055481, + 3.3748159408569336, + 3.369808077812195 + ], + "per_start_primary_values": [ + 3.3604960441589355, + 3.3787840604782104, + 3.371840000152588, + 3.37062406539917, + 3.3606879711151123, + 3.367311954498291, + 3.3587359189987183, + 3.3612959384918213, + 3.398368000984192, + 3.374000072479248, + 3.3732799291610718, + 3.3711040019989014, + 3.3763999938964844, + 3.370576024055481, + 3.3748159408569336, + 3.369808077812195 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16 + ], + "run_to_run_cv_percent": 0.2808444701672359, + "starts": 16 + }, + "uccl": { + "bootstrap_95_percent_ci_primary_value": [ + 5.50383996963501, + 5.513599872589111 + ], + "median_aggregate_input_tokens_per_second": 11896550.742065294, + "median_effective_logical_gigabytes_per_second_per_rank": 170.54895143824803, + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": 85.27447571912401, + "median_latency_ms": 5.508823990821838, + "median_primary_value": 5.508823990821838, + "num_sms_dimensionless": 20, + "per_start_median_latency_ms": [ + 5.505807876586914, + 5.520784139633179, + 5.52507209777832, + 5.496752023696899, + 5.510672092437744, + 5.511647939682007, + 5.505743980407715, + 5.5033278465271, + 5.522495985031128, + 5.514671802520752, + 5.507440090179443, + 5.513599872589111, + 5.488175868988037, + 5.50383996963501, + 5.510207891464233, + 5.502848148345947 + ], + "per_start_primary_values": [ + 5.505807876586914, + 5.520784139633179, + 5.52507209777832, + 5.496752023696899, + 5.510672092437744, + 5.511647939682007, + 5.505743980407715, + 5.5033278465271, + 5.522495985031128, + 5.514671802520752, + 5.507440090179443, + 5.513599872589111, + 5.488175868988037, + 5.50383996963501, + 5.510207891464233, + 5.502848148345947 + ], + "primary_metric": "slowest-rank latency in milliseconds", + "run_indices_dimensionless": [ + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16 + ], + "run_to_run_cv_percent": 0.1720412522563558, + "starts": 16 + } + }, + "comparisons": { + "deepep-v2-gin-gda_vs_deepep-v1-nvshmem": { + "bootstrap_95_percent_ci_improvement_percent": [ + 4.756794777113315, + 5.150195907119811 + ], + "direction_supported": true, + "median_paired_improvement_percent": 4.953322418192913, + "paired_improvement_percent_per_start": [ + 5.560325391077005, + 4.5549861511376255, + 4.836303388189054, + 4.958603403089006, + 5.150195907119811, + 4.845508267478449, + 5.400940513810901, + 4.113321275090487, + 3.957059530391971, + 4.9881486804832695, + 4.9480414332968214, + 5.470909231365217, + 4.37728225405674, + 5.2173841626263835, + 5.1101091122826965, + 4.7986429596374585 + ], + "primary_metric": "slowest-rank latency in milliseconds" + }, + "deepep-v2-gin-gda_vs_uccl": { + "bootstrap_95_percent_ci_improvement_percent": [ + 38.75338267935767, + 38.922484136339754 + ], + "direction_supported": true, + "median_paired_improvement_percent": 38.808295821728315, + "paired_improvement_percent_per_start": [ + 38.9645240174612, + 38.79883771904349, + 38.97201809351168, + 38.67971393164248, + 39.01491660650685, + 38.9055325857303, + 38.995784566975175, + 38.922484136339754, + 38.46318747545388, + 38.81775392441314, + 38.7504925350688, + 38.85838508596967, + 38.47864801535491, + 38.75955618893108, + 38.75338267935767, + 38.76247377778186 + ], + "primary_metric": "slowest-rank latency in milliseconds" + } + }, + "dispatch_dtype": "bf16", + "input_hash_sha256": "04a406d03e5e0a9d44119896e94ceed7d3257977c5de9474cf39480200b29084", + "primary_metric": "slowest-rank latency in milliseconds", + "route_hash_sha256": "ee2cd82f82a4f909f261f1d252146085a492a4a74a0e10000a1c48e717ce0b7a", + "tokens_per_rank": 4096, + "workload_profile": "prefill", + "world_size_ranks": 16 + } + ], + "comparison_scope": "synthetic decode-like and prefill-like dispatch-plus-combine communication workloads; not end-to-end training or serving", + "configuration": { + "dispatch_dtypes": [ + "fp8", + "bf16" + ], + "experts_dimensionless": 256, + "hidden_dimensions": 7168, + "measured_iterations_dimensionless": 100, + "profiles": { + "decode": { + "api_mode": "low-latency", + "primary_metric": "slowest-rank latency in milliseconds", + "tokens_per_rank": 128 + }, + "prefill": { + "api_mode": "normal", + "primary_metric": "slowest-rank latency in milliseconds", + "tokens_per_rank": 4096 + } + }, + "top_k_dimensionless": 8, + "warmup_iterations_dimensionless": 20, + "world_sizes_ranks": [ + 16 + ] + }, + "images": { + "deepep-v1-nvshmem": "159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-deepep-v1-nvshmem@sha256:175a8470e11bc7832024941d1775923d5e8ecae2c6dc06183d5f1653d66b4fac", + "deepep-v2-gin-gda": "159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-deepep-v2-gin-gda@sha256:4d07367ea290c5d6ec3c02b223ac819feed5240a46fd4a6492421e9c0853dbeb", + "uccl": "159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-uccl@sha256:a703a0d35916bbd51b2d34d968626d9617cbceb328d64b920f21ad159d93c91a" + }, + "independent_starts_per_cell": 16, + "loaded_nccl_versions_per_arm": { + "deepep-v1-nvshmem": [ + 2, + 31, + 2 + ], + "deepep-v2-gin-gda": [ + 2, + 31, + 2 + ], + "uccl": [ + 2, + 31, + 2 + ] + }, + "logical_payload_definition": "per valid expert assignment: dispatch tensor plus FP8 scales when selected plus BF16 combine tensor; backend metadata excluded", + "maximum_run_to_run_cv_percent_for_direction_support": 5.0, + "runtime": { + "cuda_version": "13.0", + "gpu": "NVIDIA B200", + "nccl_version": [ + 2, + 29, + 7 + ], + "torch_version": "2.13.0+cu130" + }, + "schema_version_dimensionless": 2, + "scored_result_records_dimensionless": 192, + "status": "PASS", + "timing_boundaries": { + "decode": "dispatch input ready through dispatch and combine completion; slowest rank CUDA elapsed time; host-side FP8 conversion precedes the boundary", + "prefill": "dispatch input and route ready through required layout, dispatch, and combine completion; slowest rank CUDA elapsed time; host-side FP8 conversion precedes the boundary" + } +} diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_ep_comparison.sh b/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_ep_comparison.sh new file mode 100755 index 000000000..0038ebce8 --- /dev/null +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_ep_comparison.sh @@ -0,0 +1,645 @@ +#!/usr/bin/env bash +set -euo pipefail + +: "${CAMPAIGN_ID:?Set a unique CAMPAIGN_ID}" +: "${EP_BENCHMARK_NODES:?Set comma-separated B200 node names}" +: "${PROTECTED_NODES_CSV:=}" +: "${ARTIFACT_ROOT:?Set the durable artifact directory}" +: "${KUBECTL_CONTEXT:?Set the target kubectl context explicitly}" +: "${CAMPAIGN_NAMESPACE:=${CAMPAIGN_ID}}" +: "${SHARED_LOCK_NAME:=adai-ap-south-1-gpu-campaign-lock}" +: "${SHARED_LOCK_NAMESPACE:=default}" +: "${LOCK_MODE:=exclusive}" +: "${EXPECTED_LOCK_HOLDER:=}" +: "${LOCK_DURATION_SECONDS:=172800}" +: "${INDEPENDENT_STARTS:=20}" +: "${EP_WORLD_SIZES:=16 32}" +: "${EP_NUM_SMS:=0}" +: "${EP_REGION:=ap-south-1}" +: "${EP_CLUSTER_NAME:=ml-clusters-shared-ap-south-1}" +: "${EP_INSTANCE_TYPE:=p6-b200.48xlarge}" +: "${WARMUP_ITERATIONS:=20}" +: "${MEASURED_ITERATIONS:=100}" +: "${CASE_TIMEOUT_SECONDS:=1800}" +: "${EFA_PER_NODE:=8}" +: "${UCCL_IMAGE:=159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-uccl@sha256:a703a0d35916bbd51b2d34d968626d9617cbceb328d64b920f21ad159d93c91a}" +: "${DEEPEP_V1_IMAGE:=159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-deepep-v1-nvshmem@sha256:175a8470e11bc7832024941d1775923d5e8ecae2c6dc06183d5f1653d66b4fac}" +: "${DEEPEP_V2_IMAGE:=159553542841.dkr.ecr.ap-south-1.amazonaws.com/adai/dsv3-ep-backend-comparison-deepep-v2-gin-gda@sha256:4d07367ea290c5d6ec3c02b223ac819feed5240a46fd4a6492421e9c0853dbeb}" + +[[ "${CAMPAIGN_ID}" =~ ^[a-z0-9][a-z0-9-]{0,62}$ ]] +[[ "${CAMPAIGN_NAMESPACE}" =~ ^[a-z0-9][a-z0-9-]{0,62}$ ]] +[[ "${LOCK_MODE}" == exclusive || "${LOCK_MODE}" == observe ]] +for value in "${LOCK_DURATION_SECONDS}" "${INDEPENDENT_STARTS}" \ + "${WARMUP_ITERATIONS}" "${MEASURED_ITERATIONS}" \ + "${CASE_TIMEOUT_SECONDS}" "${EFA_PER_NODE}"; do + [[ "${value}" =~ ^[1-9][0-9]*$ ]] || { + printf 'Iteration, timeout, Lease, and EFA values must be positive integers\n' >&2 + exit 2 + } +done +[[ "${EP_NUM_SMS}" =~ ^[0-9]+$ ]] || { + printf 'EP_NUM_SMS must be a nonnegative integer (0 keeps automatic SM counts)\n' >&2 + exit 2 +} +if [[ "${LOCK_MODE}" == observe && -z "${EXPECTED_LOCK_HOLDER}" ]]; then + printf 'LOCK_MODE=observe requires EXPECTED_LOCK_HOLDER\n' >&2 + exit 2 +fi +if [[ "${LOCK_MODE}" == observe && -z "${PROTECTED_NODES_CSV}" ]]; then + printf 'LOCK_MODE=observe requires PROTECTED_NODES_CSV\n' >&2 + exit 2 +fi +[[ "${INDEPENDENT_STARTS}" -ge 2 ]] || { + printf 'This scored matrix requires at least 2 independent starts\n' >&2 + exit 2 +} +read -r -a world_sizes <<<"${EP_WORLD_SIZES}" +((${#world_sizes[@]} > 0)) || { + printf 'EP_WORLD_SIZES must contain at least one EP size\n' >&2 + exit 2 +} +max_world_size=0 +min_world_size=0 +for world_size in "${world_sizes[@]}"; do + if ! [[ "${world_size}" =~ ^[1-9][0-9]*$ ]] || ((world_size % 8 != 0)); then + printf 'Each EP_WORLD_SIZES entry must be a positive multiple of 8: %s\n' \ + "${world_size}" >&2 + exit 2 + fi + ((world_size > max_world_size)) && max_world_size="${world_size}" + ((min_world_size == 0 || world_size < min_world_size)) && \ + min_world_size="${world_size}" +done +[[ "${WARMUP_ITERATIONS}" -eq 20 && "${MEASURED_ITERATIONS}" -eq 100 ]] || { + printf 'This scored matrix requires 20 warmup and 100 measured iterations\n' >&2 + exit 2 +} + +case_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +if [[ -e "${ARTIFACT_ROOT}" && ! -d "${ARTIFACT_ROOT}" ]]; then + printf 'ARTIFACT_ROOT exists and is not a directory: %s\n' "${ARTIFACT_ROOT}" >&2 + exit 2 +fi +if [[ -d "${ARTIFACT_ROOT}" && -n "$(find "${ARTIFACT_ROOT}" -mindepth 1 -print -quit)" ]]; then + printf 'Refusing to reuse nonempty ARTIFACT_ROOT: %s\n' "${ARTIFACT_ROOT}" >&2 + exit 2 +fi +mkdir -p "${ARTIFACT_ROOT}/control" "${ARTIFACT_ROOT}/runs" \ + "${ARTIFACT_ROOT}/summary" "${ARTIFACT_ROOT}/teardown" +K=(kubectl --context "${KUBECTL_CONTEXT}") + +IFS=, read -r -a selected_nodes <<<"${EP_BENCHMARK_NODES}" +protected_nodes=() +if [[ -n "${PROTECTED_NODES_CSV}" ]]; then + IFS=, read -r -a protected_nodes <<<"${PROTECTED_NODES_CSV}" +fi +((${#selected_nodes[@]} >= max_world_size / 8)) || { + printf 'EP_BENCHMARK_NODES must contain at least %s nodes for EP%s\n' \ + "$((max_world_size / 8))" "${max_world_size}" >&2 + exit 2 +} +[[ "$(printf '%s\n' "${selected_nodes[@]}" | sort -u | wc -l)" -eq \ + "${#selected_nodes[@]}" ]] + +declare -A protected=() +for node in "${protected_nodes[@]}"; do + protected["${node}"]=1 +done +for node in "${selected_nodes[@]}"; do + [[ -z "${protected[${node}]:-}" ]] || { + printf 'Selected node is protected by the concurrent campaign: %s\n' "${node}" >&2 + exit 1 + } +done + +declare -A images=( + [uccl]="${UCCL_IMAGE}" + [deepep-v1-nvshmem]="${DEEPEP_V1_IMAGE}" + [deepep-v2-gin-gda]="${DEEPEP_V2_IMAGE}" +) +for arm in "${!images[@]}"; do + [[ "${images[${arm}]}" =~ @sha256:[0-9a-f]{64}$ ]] || { + printf 'Image for %s is not pinned by digest: %s\n' \ + "${arm}" "${images[${arm}]}" >&2 + exit 2 + } +done + +current_case="" +namespace_created=0 +lock_claimed=0 + +check_shared_lock() { + local holder + holder="$("${K[@]}" -n "${SHARED_LOCK_NAMESPACE}" get lease \ + "${SHARED_LOCK_NAME}" -o jsonpath='{.spec.holderIdentity}' 2>/dev/null || true)" + if [[ "${LOCK_MODE}" == exclusive ]]; then + [[ "${holder}" == "${CAMPAIGN_ID}" ]] || { + printf 'Exclusive shared Lease is no longer held by %s: holder=%s\n' \ + "${CAMPAIGN_ID}" "${holder}" >&2 + return 1 + } + elif [[ "${holder}" != "${EXPECTED_LOCK_HOLDER}" ]]; then + printf 'Shared Lease holder changed from protected campaign %s to %s\n' \ + "${EXPECTED_LOCK_HOLDER}" "${holder}" >&2 + return 1 + fi +} + +claim_shared_lock() { + local attempt current holder now candidate + [[ "${LOCK_MODE}" == exclusive ]] || return 0 + for attempt in 1 2 3; do + current="$("${K[@]}" -n "${SHARED_LOCK_NAMESPACE}" get lease \ + "${SHARED_LOCK_NAME}" -o json)" + holder="$(jq -r '.spec.holderIdentity // ""' <<<"${current}")" + if [[ "${holder}" == "${CAMPAIGN_ID}" ]]; then + lock_claimed=1 + printf '%s\n' "${current}" >"${ARTIFACT_ROOT}/control/shared-lease-claimed.json" + return 0 + fi + [[ -z "${holder}" ]] || { + printf 'Shared Lease is held by another campaign: %s\n' "${holder}" >&2 + return 1 + } + now="$(date -u +%FT%T.000000Z)" + candidate="$(jq \ + --arg holder "${CAMPAIGN_ID}" --arg now "${now}" \ + --argjson duration "${LOCK_DURATION_SECONDS}" ' + .spec.holderIdentity=$holder | + .spec.acquireTime=$now | + .spec.renewTime=$now | + .spec.leaseDurationSeconds=$duration | + .metadata.labels["adai.aws/campaign"]=$holder | + .metadata.labels["adai.aws/owner"]="ep-backend-comparison"' \ + <<<"${current}")" + if printf '%s\n' "${candidate}" | "${K[@]}" replace -f - \ + >"${ARTIFACT_ROOT}/control/shared-lease-claim-attempt-${attempt}.json" 2>&1; then + lock_claimed=1 + "${K[@]}" -n "${SHARED_LOCK_NAMESPACE}" get lease "${SHARED_LOCK_NAME}" -o json \ + >"${ARTIFACT_ROOT}/control/shared-lease-claimed.json" + return 0 + fi + done + printf 'Failed to claim shared Lease after 3 optimistic attempts\n' >&2 + return 1 +} + +release_shared_lock() { + local current holder now + ((lock_claimed == 1)) || return 0 + current="$("${K[@]}" -n "${SHARED_LOCK_NAMESPACE}" get lease \ + "${SHARED_LOCK_NAME}" -o json 2>/dev/null || true)" + holder="$(jq -r '.spec.holderIdentity // ""' <<<"${current}")" + [[ "${holder}" == "${CAMPAIGN_ID}" ]] || { + printf 'Refusing to release shared Lease held by %s\n' "${holder}" >&2 + return 1 + } + now="$(date -u +%FT%T.000000Z)" + jq --arg now "${now}" ' + .spec.holderIdentity="" | + .spec.renewTime=$now | + .spec.leaseDurationSeconds=1 | + del(.metadata.labels["adai.aws/campaign"], .metadata.labels["adai.aws/owner"])' \ + <<<"${current}" | "${K[@]}" replace -f - \ + >"${ARTIFACT_ROOT}/teardown/shared-lease-release.json" + lock_claimed=0 +} + +gpu_requests_on_node() { + local node="$1" + "${K[@]}" get pods -A --field-selector "spec.nodeName=${node}" -o json | jq ' + [.items[] + | select(.status.phase != "Succeeded" and .status.phase != "Failed") + | ([.spec.containers[]?.resources.requests["nvidia.com/gpu"] // 0 | tonumber] + | add // 0) as $app + | ([.spec.initContainers[]?.resources.requests["nvidia.com/gpu"] // 0 | tonumber] + | max // 0) as $init + | [$app, $init] | max] + | add // 0' +} + +verify_node_free() { + # A pod from the previous case can still be Terminating (phase Running) + # when the next case is admitted, so retry before failing the campaign. + local node="$1" attempt ready instance_type gpu efa requests + for attempt in $(seq 1 18); do + ready="$("${K[@]}" get node "${node}" -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}')" + instance_type="$("${K[@]}" get node "${node}" -o jsonpath='{.metadata.labels.node\.kubernetes\.io/instance-type}')" + gpu="$("${K[@]}" get node "${node}" -o jsonpath='{.status.allocatable.nvidia\.com/gpu}')" + efa="$("${K[@]}" get node "${node}" -o jsonpath='{.status.allocatable.vpc\.amazonaws\.com/efa}')" + requests="$(gpu_requests_on_node "${node}")" + if [[ "${ready}" == True && "${instance_type}" == "${EP_INSTANCE_TYPE}" && \ + "${gpu}" == 8 && "${efa}" == 8 && "${requests}" -eq 0 ]]; then + return 0 + fi + printf 'Node not admittable yet (attempt %s/18): node=%s ready=%s type=%s gpu=%s efa=%s requested_gpu=%s\n' \ + "${attempt}" "${node}" "${ready}" "${instance_type}" "${gpu}" "${efa}" "${requests}" >&2 + sleep 10 + done + printf 'Node admission failed: node=%s ready=%s type=%s gpu=%s efa=%s requested_gpu=%s\n' \ + "${node}" "${ready}" "${instance_type}" "${gpu}" "${efa}" "${requests}" >&2 + return 1 +} + +cleanup_case() { + local case_to_delete cleanup_status=0 + [[ -n "${current_case}" ]] || return 0 + case_to_delete="${current_case}" + "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" delete statefulset "${case_to_delete}" \ + --ignore-not-found --wait=true --timeout=5m >/dev/null 2>&1 || cleanup_status=1 + # StatefulSet deletion can return before its cascading Pod deletions finish. + # Wait for the GPU requests to disappear before admitting the next arm. + "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" wait --for=delete pod \ + -l "app=${case_to_delete}" --timeout=5m >/dev/null 2>&1 || cleanup_status=1 + "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" delete service "${case_to_delete}" \ + --ignore-not-found >/dev/null 2>&1 || cleanup_status=1 + current_case="" + return "${cleanup_status}" +} + +finish() { + local command_status=$? teardown_status=0 owned="" remaining=0 namespace_remaining=0 + trap - EXIT INT TERM + set +e + cleanup_case || teardown_status=1 + if ((namespace_created == 1)); then + owned="$("${K[@]}" get namespace "${CAMPAIGN_NAMESPACE}" \ + -o jsonpath='{.metadata.labels.adai\.aws/campaign}' 2>/dev/null || true)" + if [[ "${owned}" == "${CAMPAIGN_ID}" ]]; then + "${K[@]}" delete namespace "${CAMPAIGN_NAMESPACE}" \ + --wait=true --timeout=10m >"${ARTIFACT_ROOT}/teardown/namespace-delete.log" 2>&1 || \ + teardown_status=1 + else + printf 'Refusing to delete namespace without owned campaign label: %s\n' \ + "${CAMPAIGN_NAMESPACE}" >"${ARTIFACT_ROOT}/teardown/refused.txt" + teardown_status=1 + fi + fi + if "${K[@]}" get namespace "${CAMPAIGN_NAMESPACE}" >/dev/null 2>&1; then + namespace_remaining=1 + teardown_status=1 + fi + if ((namespace_created == 1 || lock_claimed == 1)); then + check_shared_lock || teardown_status=1 + fi + release_shared_lock || teardown_status=1 + "${K[@]}" get all -A -l "adai.aws/campaign=${CAMPAIGN_ID}" -o json \ + >"${ARTIFACT_ROOT}/teardown/remaining-resources.json" 2>&1 || teardown_status=1 + remaining="$(jq '.items | length' "${ARTIFACT_ROOT}/teardown/remaining-resources.json" 2>/dev/null || printf '1')" + [[ "${remaining}" -eq 0 ]] || teardown_status=1 + "${K[@]}" -n "${SHARED_LOCK_NAMESPACE}" get lease "${SHARED_LOCK_NAME}" -o json \ + >"${ARTIFACT_ROOT}/teardown/shared-lease-after.json" 2>&1 || true + "${K[@]}" get nodes -o json >"${ARTIFACT_ROOT}/control/fleet-nodes-after.json" 2>&1 || \ + teardown_status=1 + "${K[@]}" get pods -A -o json >"${ARTIFACT_ROOT}/control/fleet-pods-after.json" 2>&1 || \ + teardown_status=1 + if ((command_status == 0 && teardown_status == 0)); then + printf 'PASS teardown_verified=1_dimensionless remaining_resources=0_resources\n' \ + >"${ARTIFACT_ROOT}/STATUS" + printf 'PASS completed_at=%s\n' "$(date -u +%FT%TZ)" \ + >"${ARTIFACT_ROOT}/CAMPAIGN_COMPLETE" + else + printf 'FAIL command_status=%s_dimensionless teardown_status=%s_dimensionless remaining_resources=%s_resources namespace_remaining=%s_namespaces\n' \ + "${command_status}" "${teardown_status}" "${remaining}" \ + "${namespace_remaining}" >"${ARTIFACT_ROOT}/STATUS" + rm -f "${ARTIFACT_ROOT}/CAMPAIGN_COMPLETE" + fi + find "${ARTIFACT_ROOT}" -type f ! -name SHA256SUMS -print0 | sort -z | \ + xargs -0 sha256sum >"${ARTIFACT_ROOT}/SHA256SUMS" + if ((command_status == 0 && teardown_status != 0)); then + command_status=1 + fi + exit "${command_status}" +} +trap finish EXIT +trap 'exit 130' INT +trap 'exit 143' TERM + +"${K[@]}" get nodes -o json >"${ARTIFACT_ROOT}/control/fleet-nodes-before.json" +"${K[@]}" get pods -A -o json >"${ARTIFACT_ROOT}/control/fleet-pods-before.json" +"${K[@]}" get namespaces -o json >"${ARTIFACT_ROOT}/control/namespaces-before.json" +"${K[@]}" -n "${SHARED_LOCK_NAMESPACE}" get lease "${SHARED_LOCK_NAME}" -o json \ + >"${ARTIFACT_ROOT}/control/shared-lease-before.json" +if "${K[@]}" get namespace "${CAMPAIGN_NAMESPACE}" >/dev/null 2>&1; then + printf 'Refusing to reuse existing namespace: %s\n' "${CAMPAIGN_NAMESPACE}" >&2 + exit 1 +fi +claim_shared_lock +check_shared_lock +aws sts get-caller-identity --output json >"${ARTIFACT_ROOT}/control/aws-caller-identity.json" +printf '%s\n' "${selected_nodes[@]}" >"${ARTIFACT_ROOT}/control/selected-nodes.txt" +: >"${ARTIFACT_ROOT}/control/protected-nodes.txt" +if ((${#protected_nodes[@]} > 0)); then + printf '%s\n' "${protected_nodes[@]}" | sort -u \ + >"${ARTIFACT_ROOT}/control/protected-nodes.txt" +fi + +for node in "${selected_nodes[@]}"; do + verify_node_free "${node}" +done + +"${K[@]}" create namespace "${CAMPAIGN_NAMESPACE}" --dry-run=client -o json | \ + jq --arg campaign "${CAMPAIGN_ID}" ' + .metadata.labels["adai.aws/campaign"]=$campaign | + .metadata.labels["adai.aws/owner"]="ep-backend-comparison"' | \ + "${K[@]}" create -f - >/dev/null +namespace_created=1 + +# Read the live host mitigation before invoking DeepEP V2. The EFA 3.3.0g +# revalidation exposed a UVM HMM kernel panic, so a non-mitigated node is a hard +# admission failure rather than a benchmark attempt. +for ((index = 0; index < ${#selected_nodes[@]}; index++)); do + node="${selected_nodes[${index}]}" + "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" apply -f - >/dev/null <&2 + exit 1 + } + rg -q '^GDRDRV=character-device$' "${ARTIFACT_ROOT}/control/host-audit-${index}.log" || { + printf 'Benchmark admission blocked by missing /dev/gdrdrv on node %s\n' \ + "${selected_nodes[${index}]}" >&2 + exit 1 + } +done +"${K[@]}" -n "${CAMPAIGN_NAMESPACE}" delete pod -l \ + "adai.aws/campaign=${CAMPAIGN_ID}" --wait=true --timeout=5m >/dev/null + +"${K[@]}" -n "${CAMPAIGN_NAMESPACE}" create configmap ep-benchmark-scripts \ + --from-file=ep_benchmark.py="${case_dir}/ep_benchmark.py" \ + --from-file=run_ep_rank.sh="${case_dir}/run_ep_rank.sh" \ + --dry-run=client -o yaml | "${K[@]}" apply -f - >/dev/null + +jq -n \ + --arg campaign_id "${CAMPAIGN_ID}" \ + --arg created_at_utc "$(date -u +%FT%TZ)" \ + --arg region "${EP_REGION}" \ + --arg cluster "${EP_CLUSTER_NAME}" \ + --arg git_commit "$(git -C "${case_dir}" rev-parse HEAD)" \ + --arg uccl "${UCCL_IMAGE}" \ + --arg v1 "${DEEPEP_V1_IMAGE}" \ + --arg v2 "${DEEPEP_V2_IMAGE}" \ + --argjson warmups "${WARMUP_ITERATIONS}" \ + --argjson iterations "${MEASURED_ITERATIONS}" \ + --argjson starts "${INDEPENDENT_STARTS}" \ + '{campaign_id:$campaign_id,created_at_utc:$created_at_utc,region:$region, + cluster:$cluster,git_commit:$git_commit, + images:{uccl:$uccl,"deepep-v1-nvshmem":$v1,"deepep-v2-gin-gda":$v2}, + comparison:{profiles:{decode:{tokens_per_rank:128,api_mode:"low-latency", + primary_metric:"slowest-rank latency in milliseconds"}, + prefill:{tokens_per_rank:4096,api_mode:"normal", + primary_metric:"slowest-rank latency in milliseconds"}}, + hidden_dimensions:7168,experts:256,top_k_dimensionless:8, + warmup_iterations:$warmups,measured_iterations:$iterations, + independent_starts:$starts}}' \ + >"${ARTIFACT_ROOT}/control/provenance.json" + +run_case() { + local profile="$1" arm="$2" world_size="$3" run_index="$4" dtype_order="$5" + local warmups="${6:-${WARMUP_ITERATIONS}}" iterations="${7:-${MEASURED_ITERATIONS}}" + local nccl_debug="${8:-WARN}" label="${9:-measurement}" + local nodes=$((world_size / 8)) safe_arm="${arm//-}" node_values="" out="" + current_case="ep-${profile:0:1}${world_size}-r${run_index}-${safe_arm:0:20}-${label}" + current_case="${current_case:0:63}" + out="${ARTIFACT_ROOT}/runs/${profile}/ep${world_size}/${label}-repeat-${run_index}/${arm}" + mkdir -p "${out}" + + check_shared_lock + for ((index = 0; index < nodes; index++)); do + verify_node_free "${selected_nodes[${index}]}" + node_values+=$'\n - '"${selected_nodes[${index}]}" + done + "${K[@]}" get pods -A -o json >"${out}/pods-before.json" + "${K[@]}" -n "${SHARED_LOCK_NAMESPACE}" get lease "${SHARED_LOCK_NAME}" -o json \ + >"${out}/shared-lease-before.json" + + "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" apply -f - >/dev/null <"${out}/statefulset.yaml" + + local deadline=$((SECONDS + CASE_TIMEOUT_SECONDS)) complete=0 pod="" + while ((SECONDS < deadline)); do + complete=0 + for ((index = 0; index < nodes; index++)); do + pod="${current_case}-${index}" + if "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" logs "${pod}" --tail=30 2>/dev/null | \ + rg -q '^ADAI_EP_COMPLETE$'; then + complete=$((complete + 1)) + elif "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" logs "${pod}" --tail=80 2>/dev/null | \ + rg -q '^ADAI_EP_FAILED$'; then + complete=-1 + break + fi + done + ((complete == nodes || complete == -1)) && break + sleep 10 + done + + for ((index = 0; index < nodes; index++)); do + pod="${current_case}-${index}" + "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" logs "${pod}" >"${out}/${pod}.log" 2>&1 || true + "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" get pod "${pod}" -o yaml \ + >"${out}/${pod}.yaml" 2>&1 || true + "${K[@]}" -n "${CAMPAIGN_NAMESPACE}" describe pod "${pod}" \ + >"${out}/${pod}-describe.txt" 2>&1 || true + done + ((complete == nodes)) || { + printf 'Case failed or timed out: profile=%s arm=%s EP%s repeat=%s label=%s complete=%s/%s\n' \ + "${profile}" "${arm}" "${world_size}" "${run_index}" "${label}" "${complete}" "${nodes}" >&2 + return 1 + } + local rank_zero_log="${out}/${current_case}-0.log" result_count + python3 "${case_dir}/extract_results.py" \ + "${rank_zero_log}" "${out}/results.jsonl" >/dev/null + result_count="$(wc -l <"${out}/results.jsonl")" + [[ "${result_count}" -eq 2 ]] || { + printf 'Expected 2 benchmark results in %s, found %s\n' "${rank_zero_log}" "${result_count}" >&2 + return 1 + } + printf 'PASS profile=%s arm=%s EP%s repeat=%s label=%s at %s UTC\n' \ + "${profile}" "${arm}" "${world_size}" "${run_index}" "${label}" "$(date -u +%FT%TZ)" \ + >"${out}/STATUS" + cleanup_case +} + +# A short V2-only admission run proves the HMM mitigation, GIN/GDAKI path, and +# common-harness correctness before any scored matrix work. +run_case decode deepep-v2-gin-gda "${min_world_size}" 0 bf16,fp8 2 5 INFO admission +v2_admission_dir="${ARTIFACT_ROOT}/runs/decode/ep${min_world_size}/admission-repeat-0/deepep-v2-gin-gda" +rg -q 'GDAKI.*createContext|gin GDAKI: createContext done' \ + "${v2_admission_dir}"/*.log || { + printf 'DeepEP V2 admission completed without a GDAKI context proof\n' >&2 + exit 1 +} +printf 'PASS\n' >"${v2_admission_dir}/GIN_ADMISSION_STATUS" + +# Exercise every backend's normal high-throughput API before entering the +# doubled scored matrix. These short runs are admission evidence only. +for arm in uccl deepep-v1-nvshmem deepep-v2-gin-gda; do + run_case prefill "${arm}" "${min_world_size}" 0 fp8,bf16 2 5 WARN admission +done + +for world_size in "${world_sizes[@]}"; do + for ((run_index = 1; run_index <= INDEPENDENT_STARTS; run_index++)); do + # The 3 rotation patterns cycle across however many starts run. + case "$(((run_index - 1) % 3))" in + 0) + order=(uccl deepep-v1-nvshmem deepep-v2-gin-gda) + profiles=(decode prefill) + dtypes=fp8,bf16 + ;; + 1) + order=(deepep-v2-gin-gda uccl deepep-v1-nvshmem) + profiles=(prefill decode) + dtypes=bf16,fp8 + ;; + 2) + order=(deepep-v1-nvshmem deepep-v2-gin-gda uccl) + profiles=(decode prefill) + dtypes=fp8,bf16 + ;; + esac + for profile in "${profiles[@]}"; do + for arm in "${order[@]}"; do + run_case "${profile}" "${arm}" "${world_size}" "${run_index}" "${dtypes}" + done + done + done +done + +check_shared_lock +for node in "${selected_nodes[@]}"; do + verify_node_free "${node}" +done +python3 "${case_dir}/summarize_results.py" "${ARTIFACT_ROOT}/runs" \ + --starts="${INDEPENDENT_STARTS}" \ + --world-sizes="${EP_WORLD_SIZES// /,}" \ + --provenance="${ARTIFACT_ROOT}/control/provenance.json" \ + --json="${ARTIFACT_ROOT}/summary/summary.json" \ + --markdown="${ARTIFACT_ROOT}/summary/summary.md" +printf 'PASS EP backend comparison completed at %s UTC\n' "$(date -u +%FT%TZ)" diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_ep_rank.sh b/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_ep_rank.sh new file mode 100755 index 000000000..d5b865076 --- /dev/null +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/run_ep_rank.sh @@ -0,0 +1,89 @@ +#!/usr/bin/env bash +set -euo pipefail + +: "${EP_ARM:?Set EP_ARM}" +: "${EP_NODES:?Set EP_NODES}" +: "${EP_SERVICE:?Set EP_SERVICE}" +: "${EP_MASTER_ADDR:?Set EP_MASTER_ADDR}" +: "${POD_NAME:?Set POD_NAME}" +: "${EP_RUN_INDEX:?Set EP_RUN_INDEX}" +: "${EP_WORKLOAD_PROFILE:?Set EP_WORKLOAD_PROFILE}" +: "${EP_DISPATCH_DTYPES:?Set EP_DISPATCH_DTYPES}" +: "${EP_WARMUPS:=20}" +: "${EP_ITERATIONS:=100}" +: "${EP_NUM_SMS:=0}" +: "${EP_SEED:=20260824}" +: "${EP_NCCL_DEBUG:=WARN}" + +ordinal="${POD_NAME##*-}" +export FI_PROVIDER=efa +export FI_EFA_USE_DEVICE_RDMA=1 +export NCCL_SOCKET_IFNAME='^lo,docker,veth' +export NCCL_NET_PLUGIN=ofi +export NCCL_DEBUG="${EP_NCCL_DEBUG}" +export CUDA_DEVICE_MAX_CONNECTIONS=1 + +case "${EP_ARM}" in + uccl) + export PER_EXPERT_BATCHING=1 + export UCCL_SOCKET_IFNAME='^lo,docker,veth' + ;; + deepep-v1-nvshmem) + export NVSHMEM_REMOTE_TRANSPORT=libfabric + export NVSHMEM_LIBFABRIC_PROVIDER=efa + export NVSHMEM_BOOTSTRAP_UID_SOCK_IFNAME='^lo,docker,veth' + export NVSHMEM_NETDEVS_POLICY=EXTERNAL_SHARING_PCIE_SWITCH_NIC_EXCLUSIVE + ;; + deepep-v2-gin-gda) + export FI_EFA_USE_HW_CNTR=1 + export NCCL_GIN_TYPE=5 + export NCCL_SYM_GIN_KERNELS_ENABLE=0 + # EP_BUFFER_DEBUG printfs inside the CPU-wait path cost ~3% on decode, + # so scored measurement runs (WARN) leave it off. + if [[ "${NCCL_DEBUG}" == INFO ]]; then + export EP_BUFFER_DEBUG=1 + export NCCL_DEBUG_SUBSYS=INIT,ENV,NET + export FI_LOG_LEVEL=info + export FI_LOG_PROV=efa + export FI_LOG_SUBSYS=cntr + fi + ;; + *) + printf 'Unsupported EP_ARM=%s\n' "${EP_ARM}" >&2 + exit 2 + ;; +esac + +printf 'ADAI_EP_LAUNCH profile=%s arm=%s nodes=%s ranks=%s run_index=%s_dimensionless dtype_order=%s warmups=%s_iterations measured=%s_iterations\n' \ + "${EP_WORKLOAD_PROFILE}" "${EP_ARM}" "${EP_NODES}" "$((EP_NODES * 8))" "${EP_RUN_INDEX}" \ + "${EP_DISPATCH_DTYPES}" "${EP_WARMUPS}" "${EP_ITERATIONS}" + +set +e +torchrun \ + --nnodes="${EP_NODES}" \ + --nproc-per-node=8 \ + --node-rank="${ordinal}" \ + --master-addr="${EP_MASTER_ADDR}" \ + --master-port=29400 \ + /opt/benchmark/ep_benchmark.py \ + --arm="${EP_ARM}" \ + --profile="${EP_WORKLOAD_PROFILE}" \ + --hidden=7168 \ + --top-k=8 \ + --experts=256 \ + --seed="${EP_SEED}" \ + --warmups="${EP_WARMUPS}" \ + --iterations="${EP_ITERATIONS}" \ + --num-sms="${EP_NUM_SMS}" \ + --run-index="${EP_RUN_INDEX}" \ + --dispatch-dtypes="${EP_DISPATCH_DTYPES}" +status=$? +set -e + +printf 'ADAI_EP_EXIT_STATUS=%s_dimensionless\n' "${status}" +if ((status == 0)); then + printf 'ADAI_EP_COMPLETE\n' +else + printf 'ADAI_EP_FAILED\n' +fi +sleep infinity diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/summarize_results.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/summarize_results.py new file mode 100755 index 000000000..717ec720c --- /dev/null +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/summarize_results.py @@ -0,0 +1,703 @@ +#!/usr/bin/env python3 +"""Validate and summarize common-boundary EP benchmark logs.""" + +from __future__ import annotations + +import argparse +import json +import math +import random +import re +import statistics +from collections import defaultdict +from pathlib import Path +from typing import Any, Iterable + +from result_io import load_result_log + + +ARMS = ("uccl", "deepep-v1-nvshmem", "deepep-v2-gin-gda") +DTYPES = ("fp8", "bf16") +PROFILES = ("decode", "prefill") +PROFILE_CONFIG = { + "decode": { + "tokens_per_rank": 128, + "api_mode": "low-latency", + "primary_metric": "slowest-rank latency in milliseconds", + "timing_boundary": ( + "dispatch input ready through dispatch and combine completion; " + "slowest rank CUDA elapsed time; host-side FP8 conversion precedes " + "the boundary" + ), + }, + "prefill": { + "tokens_per_rank": 4_096, + "api_mode": "normal", + "primary_metric": "slowest-rank latency in milliseconds", + "timing_boundary": ( + "dispatch input and route ready through required layout, dispatch, " + "and combine completion; slowest rank CUDA elapsed time; host-side " + "FP8 conversion precedes the boundary" + ), + }, +} +ARM_LABELS = { + "uccl": "UCCL", + "deepep-v1-nvshmem": "DeepEP V1 NVSHMEM", + "deepep-v2-gin-gda": "DeepEP V2 NCCL GIN", +} +BOOTSTRAP_SAMPLES = 20_000 +MAX_RUN_TO_RUN_CV_PERCENT = 5.0 +DEFAULT_STARTS = 20 +MINIMUM_STARTS = 2 +EXPECTED_WARMUPS = 20 +EXPECTED_ITERATIONS = 100 +LOGICAL_PAYLOAD_DEFINITION = ( + "per valid expert assignment: dispatch tensor plus FP8 scales when selected " + "plus BF16 combine tensor; backend metadata excluded" +) +SHA256_PATTERN = re.compile(r"[0-9a-f]{64}") + + +def percentile(values: list[float], quantile: float) -> float: + ordered = sorted(values) + if not ordered: + raise ValueError("percentile requires at least one value") + position = (len(ordered) - 1) * quantile + lower = int(position) + upper = min(lower + 1, len(ordered) - 1) + fraction = position - lower + return ordered[lower] * (1 - fraction) + ordered[upper] * fraction + + +def bootstrap_median_ci( + values: list[float], seed: int, samples: int = BOOTSTRAP_SAMPLES +) -> tuple[float, float]: + rng = random.Random(seed) + medians = [ + statistics.median(rng.choices(values, k=len(values))) for _ in range(samples) + ] + return percentile(medians, 0.025), percentile(medians, 0.975) + + +def load_results(root: Path) -> list[dict[str, Any]]: + results: dict[tuple[str, str, int, int, str], dict[str, Any]] = {} + sources: dict[tuple[str, str, int, int, str], Path] = {} + for path in sorted(root.rglob("*.log")): + for result in load_result_log(path): + key = ( + result["workload_profile"], + result["arm"], + result["world_size_ranks"], + result["run_index_dimensionless"], + result["dispatch_dtype"], + ) + if key in results and results[key] != result: + raise ValueError( + f"conflicting result for {key}: {sources[key]} and {path}" + ) + results[key] = result + sources[key] = path + return list(results.values()) + + +def observed_world_sizes(results: list[dict[str, Any]]) -> tuple[int, ...]: + """Derive the scored EP sizes present in the loaded results.""" + + worlds = sorted( + { + result["world_size_ranks"] + for result in results + if result["run_index_dimensionless"] > 0 + } + ) + if not worlds: + raise ValueError("no scored results found to derive world sizes from") + return tuple(worlds) + + +def validate( + results: list[dict[str, Any]], + starts: int, + world_sizes: tuple[int, ...] | None = None, +) -> None: + if starts < MINIMUM_STARTS: + raise ValueError( + f"scored matrix requires at least {MINIMUM_STARTS} independent starts" + ) + if world_sizes is None: + world_sizes = observed_world_sizes(results) + expected = { + (profile, arm, world, run, dtype) + for profile in PROFILES + for arm in ARMS + for world in world_sizes + for run in range(1, starts + 1) + for dtype in DTYPES + } + observed = { + ( + result["workload_profile"], + result["arm"], + result["world_size_ranks"], + result["run_index_dimensionless"], + result["dispatch_dtype"], + ) + for result in results + if result["run_index_dimensionless"] > 0 + } + missing = expected - observed + extra = observed - expected + if missing or extra: + raise ValueError( + f"result matrix mismatch; missing={sorted(missing)}, extra={sorted(extra)}" + ) + + measured = [result for result in results if result["run_index_dimensionless"] > 0] + if len(measured) != len(expected): + raise ValueError( + f"result matrix has {len(measured)} scored records; expected {len(expected)}" + ) + common_shape = { + "benchmark": "common-boundary-dispatch-combine", + "hidden_dimensions": 7168, + "experts": 256, + "top_k_dimensionless": 8, + "gpus_per_node": 8, + "warmup_iterations": EXPECTED_WARMUPS, + "measured_iterations": EXPECTED_ITERATIONS, + "logical_payload_definition": LOGICAL_PAYLOAD_DEFINITION, + } + for result in measured: + profile = result.get("workload_profile") + if profile not in PROFILE_CONFIG: + raise ValueError(f"unexpected workload profile: {profile}") + profile_config = PROFILE_CONFIG[profile] + expected_shape = { + **common_shape, + "tokens_per_rank": profile_config["tokens_per_rank"], + "timing_boundary": profile_config["timing_boundary"], + "primary_metric": profile_config["primary_metric"], + "layout_in_timed_region": profile == "prefill", + "backend_api_mode": ( + "elastic" + if result.get("arm") == "deepep-v2-gin-gda" + else profile_config["api_mode"] + ), + } + for field, expected_value in expected_shape.items(): + if result.get(field) != expected_value: + raise ValueError( + f"unexpected {field}: expected {expected_value}, got {result.get(field)}" + ) + world_size = result["world_size_ranks"] + if result.get("nodes") != world_size // 8: + raise ValueError(f"invalid node count for EP{world_size}: {result}") + tokens_per_rank = profile_config["tokens_per_rank"] + if result.get("global_input_tokens") != tokens_per_rank * world_size: + raise ValueError(f"invalid global input token count: {result}") + if result["correctness"]["status"] != "PASS": + raise ValueError(f"correctness did not pass: {result}") + tolerance = 9e-4 if result["dispatch_dtype"] == "fp8" else 1e-5 + correctness = result["correctness"] + if correctness.get("tolerance_dimensionless") != tolerance: + raise ValueError(f"unexpected correctness tolerance: {result}") + if not math.isfinite( + correctness.get("normalized_diff_dimensionless", math.inf) + ): + raise ValueError(f"non-finite correctness result: {result}") + if correctness["normalized_diff_dimensionless"] > tolerance: + raise ValueError(f"correctness exceeds tolerance: {result}") + image_reference = result["runtime"]["image_reference"] + digest = image_reference.rsplit("@sha256:", 1)[-1] + if "@sha256:" not in image_reference or not SHA256_PATTERN.fullmatch(digest): + raise ValueError( + f"image is not digest pinned: {result['runtime']['image_reference']}" + ) + for field in ("gpu", "torch_version", "cuda_version", "nccl_version"): + if not result["runtime"].get(field): + raise ValueError(f"runtime is missing {field}: {result}") + for field in ("route_hash_sha256", "input_hash_sha256"): + if not SHA256_PATTERN.fullmatch(result.get(field, "")): + raise ValueError(f"invalid {field}: {result}") + expected_selections = result["global_input_tokens"] * 8 + if result.get("global_valid_expert_selections") != expected_selections: + raise ValueError(f"invalid valid-expert selection count: {result}") + dispatch_bytes = 7_168 * 2 + if result["dispatch_dtype"] == "fp8": + dispatch_bytes = 7_168 + math.ceil(7_168 / 128) * 4 + expected_logical_bytes = tokens_per_rank * 8 * (dispatch_bytes + 7_168 * 2) + logical_bytes = result.get("avg_logical_payload_bytes_per_rank") + scaleout_bytes = result.get("avg_scaleout_logical_payload_bytes_per_rank") + if logical_bytes != expected_logical_bytes: + raise ValueError(f"invalid logical payload: {result}") + if not isinstance(scaleout_bytes, (int, float)) or not ( + 0 < scaleout_bytes <= logical_bytes + ): + raise ValueError(f"invalid scale-out logical payload: {result}") + latency = result.get("latency_ms", {}).get("median", 0) + if ( + not isinstance(latency, (int, float)) + or not math.isfinite(latency) + or latency <= 0 + ): + raise ValueError(f"invalid median latency: {result}") + elapsed_seconds = latency / 1e3 + expected_metrics = { + "aggregate_input_tokens_per_second": result["global_input_tokens"] + / elapsed_seconds, + "effective_logical_gigabytes_per_second_per_rank": logical_bytes + / elapsed_seconds + / 1e9, + "effective_scaleout_logical_gigabytes_per_second_per_rank": scaleout_bytes + / elapsed_seconds + / 1e9, + } + for field in ( + "aggregate_input_tokens_per_second", + "effective_logical_gigabytes_per_second_per_rank", + "effective_scaleout_logical_gigabytes_per_second_per_rank", + ): + value = result.get(field, 0) + if ( + not isinstance(value, (int, float)) + or not math.isfinite(value) + or value <= 0 + ): + raise ValueError(f"invalid positive metric {field}: {result}") + if not math.isclose( + value, expected_metrics[field], rel_tol=1e-12, abs_tol=1e-9 + ): + raise ValueError( + f"metric {field} does not match common accounting: {result}" + ) + + common_fields = ( + "warmup_iterations", + "measured_iterations", + "logical_payload_definition", + ) + for field in common_fields: + values = {json.dumps(result[field], sort_keys=True) for result in measured} + if len(values) != 1: + raise ValueError(f"scored results disagree on {field}: {sorted(values)}") + + runtime_signatures = { + json.dumps( + { + key: result["runtime"].get(key) + for key in ("gpu", "torch_version", "cuda_version", "nccl_version") + }, + sort_keys=True, + ) + for result in measured + } + if len(runtime_signatures) != 1: + raise ValueError(f"runtime stack mismatch: {sorted(runtime_signatures)}") + + for arm in ARMS: + image_references = { + result["runtime"]["image_reference"] + for result in measured + if result["arm"] == arm + } + if len(image_references) != 1: + raise ValueError( + f"{arm} did not use one immutable image: {image_references}" + ) + # The loaded NCCL differs per arm image by design; require agreement + # only within one arm. + loaded_versions = { + json.dumps(result["runtime"].get("nccl_version_loaded")) + for result in measured + if result["arm"] == arm + } + if len(loaded_versions) != 1: + raise ValueError( + f"{arm} results disagree on the loaded NCCL version: " + f"{sorted(loaded_versions)}" + ) + + for profile in PROFILES: + for world in world_sizes: + same_world = [ + result + for result in measured + if result["workload_profile"] == profile + and result["world_size_ranks"] == world + ] + route_hashes = {result["route_hash_sha256"] for result in same_world} + input_hashes = {result["input_hash_sha256"] for result in same_world} + if len(route_hashes) != 1 or len(input_hashes) != 1: + raise ValueError( + f"{profile} EP{world} did not replay one route/input: " + f"routes={route_hashes}, inputs={input_hashes}" + ) + for dtype in DTYPES: + same_cell = [ + result for result in same_world if result["dispatch_dtype"] == dtype + ] + for field in ( + "avg_logical_payload_bytes_per_rank", + "avg_scaleout_logical_payload_bytes_per_rank", + "global_valid_expert_selections", + ): + values = {result[field] for result in same_cell} + if len(values) != 1: + raise ValueError( + f"{profile} EP{world} {dtype} disagrees on {field}: " + f"{sorted(values)}" + ) + # The SM count is a per-backend control, so it must only be + # stable within one arm of a cell. + for arm in ARMS: + num_sms_values = { + result.get("num_sms_dimensionless") + for result in same_cell + if result["arm"] == arm + } + if len(num_sms_values) != 1: + raise ValueError( + f"{profile} EP{world} {dtype} {arm} disagrees on " + f"num_sms: {sorted(num_sms_values, key=str)}" + ) + + +def validate_provenance( + provenance: dict[str, Any], results: list[dict[str, Any]], starts: int +) -> None: + measured = [result for result in results if result["run_index_dimensionless"] > 0] + expected_images = { + arm: next( + result["runtime"]["image_reference"] + for result in measured + if result["arm"] == arm + ) + for arm in ARMS + } + if provenance.get("images") != expected_images: + raise ValueError( + f"provenance images do not match scored results: {provenance.get('images')}" + ) + expected_comparison = { + "profiles": { + profile: { + "tokens_per_rank": PROFILE_CONFIG[profile]["tokens_per_rank"], + "api_mode": PROFILE_CONFIG[profile]["api_mode"], + "primary_metric": PROFILE_CONFIG[profile]["primary_metric"], + } + for profile in PROFILES + }, + "hidden_dimensions": 7_168, + "experts": 256, + "top_k_dimensionless": 8, + "warmup_iterations": EXPECTED_WARMUPS, + "measured_iterations": EXPECTED_ITERATIONS, + "independent_starts": starts, + } + if provenance.get("comparison") != expected_comparison: + raise ValueError( + "provenance comparison does not match the scored matrix: " + f"{provenance.get('comparison')}" + ) + for field in ("campaign_id", "created_at_utc", "region", "cluster", "git_commit"): + if not provenance.get(field): + raise ValueError(f"provenance is missing {field}") + + +def primary_metric_value(result: dict[str, Any]) -> float: + return result["latency_ms"]["median"] + + +def arm_summary( + results: Iterable[dict[str, Any]], seed: int, profile: str +) -> dict[str, Any]: + ordered = sorted(results, key=lambda item: item["run_index_dimensionless"]) + latencies = [item["latency_ms"]["median"] for item in ordered] + token_rates = [item["aggregate_input_tokens_per_second"] for item in ordered] + logical_rates = [ + item["effective_logical_gigabytes_per_second_per_rank"] for item in ordered + ] + scaleout_rates = [ + item["effective_scaleout_logical_gigabytes_per_second_per_rank"] + for item in ordered + ] + primary_values = [primary_metric_value(item) for item in ordered] + primary_mean = statistics.fmean(primary_values) + primary_stdev = statistics.stdev(primary_values) if len(primary_values) > 1 else 0.0 + ci_low, ci_high = bootstrap_median_ci(primary_values, seed) + return { + "starts": len(ordered), + "run_indices_dimensionless": [ + item["run_index_dimensionless"] for item in ordered + ], + "num_sms_dimensionless": ordered[0].get("num_sms_dimensionless"), + "per_start_median_latency_ms": latencies, + "median_latency_ms": statistics.median(latencies), + "primary_metric": PROFILE_CONFIG[profile]["primary_metric"], + "per_start_primary_values": primary_values, + "median_primary_value": statistics.median(primary_values), + "bootstrap_95_percent_ci_primary_value": [ci_low, ci_high], + "run_to_run_cv_percent": ( + primary_stdev / primary_mean * 100 if primary_mean else 0.0 + ), + "median_aggregate_input_tokens_per_second": statistics.median(token_rates), + "median_effective_logical_gigabytes_per_second_per_rank": statistics.median( + logical_rates + ), + "median_effective_scaleout_logical_gigabytes_per_second_per_rank": statistics.median( + scaleout_rates + ), + } + + +def summarize( + results: list[dict[str, Any]], + starts: int, + provenance: dict[str, Any] | None = None, + world_sizes: tuple[int, ...] | None = None, +) -> dict[str, Any]: + if world_sizes is None: + world_sizes = observed_world_sizes(results) + measured = [result for result in results if result["run_index_dimensionless"] > 0] + by_cell_arm: dict[tuple[str, int, str, str], list[dict[str, Any]]] = defaultdict( + list + ) + for result in measured: + by_cell_arm[ + ( + result["workload_profile"], + result["world_size_ranks"], + result["dispatch_dtype"], + result["arm"], + ) + ].append(result) + + cells = [] + for profile_index, profile in enumerate(PROFILES): + for world in world_sizes: + for dtype in DTYPES: + arms = { + arm: arm_summary( + by_cell_arm[(profile, world, dtype, arm)], + seed=( + 20260824 + + profile_index * 1_000 + + world + + len(dtype) + + index + ), + profile=profile, + ) + for index, arm in enumerate(ARMS) + } + comparisons = {} + v2_by_run = { + result["run_index_dimensionless"]: primary_metric_value(result) + for result in by_cell_arm[ + (profile, world, dtype, "deepep-v2-gin-gda") + ] + } + for index, baseline in enumerate(("uccl", "deepep-v1-nvshmem")): + baseline_by_run = { + result["run_index_dimensionless"]: primary_metric_value(result) + for result in by_cell_arm[(profile, world, dtype, baseline)] + } + paired = [ + (baseline_by_run[run] - v2_by_run[run]) + / baseline_by_run[run] + * 100 + for run in range(1, starts + 1) + ] + ci_low, ci_high = bootstrap_median_ci( + paired, + 20260900 + profile_index * 1_000 + world + index, + ) + stable = ( + arms[baseline]["run_to_run_cv_percent"] + <= MAX_RUN_TO_RUN_CV_PERCENT + and arms["deepep-v2-gin-gda"]["run_to_run_cv_percent"] + <= MAX_RUN_TO_RUN_CV_PERCENT + ) + comparisons[f"deepep-v2-gin-gda_vs_{baseline}"] = { + "primary_metric": PROFILE_CONFIG[profile]["primary_metric"], + "paired_improvement_percent_per_start": paired, + "median_paired_improvement_percent": statistics.median(paired), + "bootstrap_95_percent_ci_improvement_percent": [ + ci_low, + ci_high, + ], + "direction_supported": stable and (ci_low > 0 or ci_high < 0), + } + same_cell = [ + result + for result in measured + if result["workload_profile"] == profile + and result["world_size_ranks"] == world + and result["dispatch_dtype"] == dtype + ] + cells.append( + { + "workload_profile": profile, + "world_size_ranks": world, + "dispatch_dtype": dtype, + "tokens_per_rank": PROFILE_CONFIG[profile]["tokens_per_rank"], + "primary_metric": PROFILE_CONFIG[profile]["primary_metric"], + "route_hash_sha256": same_cell[0]["route_hash_sha256"], + "input_hash_sha256": same_cell[0]["input_hash_sha256"], + "arms": arms, + "comparisons": comparisons, + } + ) + images = { + arm: next( + result["runtime"]["image_reference"] + for result in measured + if result["arm"] == arm + ) + for arm in ARMS + } + loaded_nccl_versions = { + arm: next( + result["runtime"].get("nccl_version_loaded") + for result in measured + if result["arm"] == arm + ) + for arm in ARMS + } + runtime = { + field: measured[0]["runtime"][field] + for field in ("gpu", "torch_version", "cuda_version", "nccl_version") + } + summary = { + "schema_version_dimensionless": 2, + "status": "PASS", + "scored_result_records_dimensionless": len(measured), + "independent_starts_per_cell": starts, + "bootstrap_samples_dimensionless": BOOTSTRAP_SAMPLES, + "maximum_run_to_run_cv_percent_for_direction_support": MAX_RUN_TO_RUN_CV_PERCENT, + "timing_boundaries": { + profile: PROFILE_CONFIG[profile]["timing_boundary"] for profile in PROFILES + }, + "logical_payload_definition": measured[0]["logical_payload_definition"], + "comparison_scope": ( + "synthetic decode-like and prefill-like dispatch-plus-combine " + "communication workloads; not end-to-end training or serving" + ), + "configuration": { + "world_sizes_ranks": list(world_sizes), + "dispatch_dtypes": list(DTYPES), + "profiles": { + profile: { + "tokens_per_rank": PROFILE_CONFIG[profile]["tokens_per_rank"], + "api_mode": PROFILE_CONFIG[profile]["api_mode"], + "primary_metric": PROFILE_CONFIG[profile]["primary_metric"], + } + for profile in PROFILES + }, + "hidden_dimensions": 7_168, + "experts_dimensionless": 256, + "top_k_dimensionless": 8, + "warmup_iterations_dimensionless": EXPECTED_WARMUPS, + "measured_iterations_dimensionless": EXPECTED_ITERATIONS, + }, + "runtime": runtime, + "images": images, + "loaded_nccl_versions_per_arm": loaded_nccl_versions, + "cells": cells, + } + if provenance is not None: + summary["campaign_provenance"] = provenance + return summary + + +def markdown(summary: dict[str, Any]) -> str: + lines = [ + "# Common-Boundary EP Results", + "", + f"Each cell has {summary['independent_starts_per_cell']} independent process starts. Values are medians across independent starts. Both profiles use slowest-rank latency as the primary metric; compare across EP sizes with aggregate input tokens/s. Logical throughput columns are secondary, and the scale-out column is comparable only within one EP size. These are synthetic communication workloads, not end-to-end training or serving results.", + ] + for profile in PROFILES: + title = "Decode-like latency" if profile == "decode" else "Prefill-like latency" + tokens = PROFILE_CONFIG[profile]["tokens_per_rank"] + lines.extend( + [ + "", + f"## {title}, {tokens:,} tokens/rank", + "", + "| EP size | Dispatch dtype | Backend | Latency (ms) | 95% bootstrap CI (ms) | Run-to-run CV (%) | Input throughput (tokens/s) | Logical throughput (GB/s/rank) | Scale-out logical throughput (GB/s/rank) |", + "|---:|:---:|:---|---:|:---:|---:|---:|---:|---:|", + ] + ) + for cell in summary["cells"]: + if cell["workload_profile"] != profile: + continue + for arm in ARMS: + value = cell["arms"][arm] + ci = value["bootstrap_95_percent_ci_primary_value"] + lines.append( + f"| {cell['world_size_ranks']} ranks | {cell['dispatch_dtype'].upper()} | {ARM_LABELS[arm]} | " + f"{value['median_latency_ms']:.4f} ms | " + f"[{ci[0]:.4f}, {ci[1]:.4f}] ms | " + f"{value['run_to_run_cv_percent']:.2f}% | " + f"{value['median_aggregate_input_tokens_per_second']:,.2f} tokens/s | " + f"{value['median_effective_logical_gigabytes_per_second_per_rank']:.2f} GB/s/rank | " + f"{value['median_effective_scaleout_logical_gigabytes_per_second_per_rank']:.2f} GB/s/rank |" + ) + lines.extend( + [ + "", + "## Paired DeepEP V2 improvements", + "", + "Positive values mean DeepEP V2 had lower slowest-rank latency. A direction is supported only when the paired bootstrap interval excludes 0% and both arms have at most 5% run-to-run CV.", + "", + "| Profile | EP size | Dispatch dtype | Baseline | Primary metric | Median improvement (%) | 95% bootstrap CI (%) | Direction supported |", + "|:---|---:|:---:|:---|:---|---:|:---:|:---:|", + ] + ) + for cell in summary["cells"]: + for baseline in ("uccl", "deepep-v1-nvshmem"): + comparison = cell["comparisons"][f"deepep-v2-gin-gda_vs_{baseline}"] + ci = comparison["bootstrap_95_percent_ci_improvement_percent"] + lines.append( + f"| {cell['workload_profile']} | {cell['world_size_ranks']} ranks | {cell['dispatch_dtype'].upper()} | {ARM_LABELS[baseline]} | {comparison['primary_metric']} | " + f"{comparison['median_paired_improvement_percent']:.2f}% | " + f"[{ci[0]:.2f}, {ci[1]:.2f}]% | " + f"{'yes' if comparison['direction_supported'] else 'no'} |" + ) + lines.append("") + return "\n".join(lines) + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("root", type=Path) + parser.add_argument("--starts", type=int, default=DEFAULT_STARTS) + parser.add_argument( + "--world-sizes", + type=lambda text: tuple(int(item) for item in text.split(",") if item), + default=None, + help=( + "Comma-separated scored EP sizes; derived from the loaded results " + "when omitted" + ), + ) + parser.add_argument("--provenance", type=Path, required=True) + parser.add_argument("--json", type=Path, required=True) + parser.add_argument("--markdown", type=Path, required=True) + args = parser.parse_args() + results = load_results(args.root) + validate(results, args.starts, args.world_sizes) + provenance = json.loads(args.provenance.read_text()) + validate_provenance(provenance, results, args.starts) + summary = summarize(results, args.starts, provenance, args.world_sizes) + args.json.write_text(json.dumps(summary, indent=2, sort_keys=True) + "\n") + args.markdown.write_text(markdown(summary)) + print( + f"PASS EP matrix: {len(summary['cells'])} cells, " + f"{args.starts} independent starts per arm/cell" + ) + + +if __name__ == "__main__": + main() diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_ep_benchmark.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_ep_benchmark.py new file mode 100644 index 000000000..b1fc9c522 --- /dev/null +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_ep_benchmark.py @@ -0,0 +1,132 @@ +import importlib.util +from pathlib import Path +import sys +import tempfile +import unittest + +try: + import torch +except ModuleNotFoundError: + torch = None + + +MODULE = None +if torch is not None: + module_path = Path(__file__).with_name("ep_benchmark.py") + spec = importlib.util.spec_from_file_location("ep_benchmark", module_path) + assert spec and spec.loader + MODULE = importlib.util.module_from_spec(spec) + sys.modules[spec.name] = MODULE + spec.loader.exec_module(MODULE) + + +@unittest.skipIf(torch is None, "PyTorch is required for benchmark helper tests") +class EpBenchmarkTest(unittest.TestCase): + def test_route_is_balanced_and_unique(self): + routes = [ + MODULE.make_route(rank, 128, 256, 8, 20260824, torch.device("cpu")) + for rank in range(32) + ] + route = torch.cat(routes) + histogram = torch.bincount(route.flatten(), minlength=256) + self.assertEqual(histogram.min().item(), 128) + self.assertEqual(histogram.max().item(), 128) + for row in route: + self.assertEqual(torch.unique(row).numel(), 8) + + def test_common_payload_formula(self): + route = MODULE.make_route(0, 128, 256, 8, 20260824, torch.device("cpu")) + bf16_all, bf16_remote, selections = MODULE.logical_payload_bytes_per_rank( + route, 0, 16, 8, 7168, "bf16", 256 + ) + fp8_all, fp8_remote, _ = MODULE.logical_payload_bytes_per_rank( + route, 0, 16, 8, 7168, "fp8", 256 + ) + self.assertEqual(selections, 1024) + self.assertEqual(bf16_all, 1024 * (7168 * 2 + 7168 * 2)) + self.assertEqual(fp8_all, 1024 * (7168 + 56 * 4 + 7168 * 2)) + self.assertGreater(bf16_remote, 0) + self.assertGreater(fp8_remote, 0) + self.assertLess(bf16_remote, bf16_all) + self.assertLess(fp8_remote, fp8_all) + + def test_profile_shapes_are_fixed(self): + self.assertEqual(MODULE.WORKLOAD_PROFILES["decode"].tokens_per_rank, 128) + self.assertEqual(MODULE.WORKLOAD_PROFILES["decode"].api_mode, "low-latency") + self.assertEqual(MODULE.WORKLOAD_PROFILES["prefill"].tokens_per_rank, 4096) + self.assertEqual(MODULE.WORKLOAD_PROFILES["prefill"].api_mode, "normal") + + def test_prefill_payload_scales_with_tokens(self): + decode_route = MODULE.make_route(0, 128, 256, 8, 20260824, torch.device("cpu")) + prefill_route = MODULE.make_route( + 0, 4096, 256, 8, 20260824, torch.device("cpu") + ) + decode_bytes, _, _ = MODULE.logical_payload_bytes_per_rank( + decode_route, 0, 16, 8, 7168, "bf16", 256 + ) + prefill_bytes, _, _ = MODULE.logical_payload_bytes_per_rank( + prefill_route, 0, 16, 8, 7168, "bf16", 256 + ) + self.assertEqual(prefill_bytes, decode_bytes * 32) + + def test_percentile_interpolates(self): + self.assertEqual(MODULE.percentile([1.0, 2.0, 3.0], 0.5), 2.0) + self.assertAlmostEqual(MODULE.percentile([1.0, 2.0], 0.95), 1.95) + + def test_deepep_v2_build_lib_requires_one_extension_package(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + package = root / "build" / "lib.linux-x86_64-cpython-312" / "deep_ep" + package.mkdir(parents=True) + (package / "_C.cpython-312-x86_64-linux-gnu.so").touch() + self.assertEqual(MODULE.deepep_v2_build_lib(root), package.parent) + + second = root / "build" / "lib.second" / "deep_ep" + second.mkdir(parents=True) + (second / "_C.so").touch() + with self.assertRaisesRegex(RuntimeError, "exactly one"): + MODULE.deepep_v2_build_lib(root) + + def test_received_fp8_accepts_noncontiguous_scales(self): + adapter = object.__new__(MODULE.BackendAdapter) + adapter.arm = "uccl" + adapter.profile = MODULE.WORKLOAD_PROFILES["decode"] + adapter.hidden = 256 + observed = {} + + def cast_back(fp8, scales): + observed["fp8_shape"] = tuple(fp8.shape) + observed["scales_shape"] = tuple(scales.shape) + return torch.zeros((fp8.shape[0], 256), dtype=torch.bfloat16) + + adapter._cast_back = cast_back + fp8 = torch.zeros((2, 3, 256), dtype=torch.float8_e4m3fn) + scales = torch.arange(12, dtype=torch.float32).reshape(2, 6).t() + self.assertFalse(scales.is_contiguous()) + + received = adapter.received_as_bf16((fp8, scales), "fp8") + + self.assertEqual(observed["fp8_shape"], (6, 256)) + self.assertEqual(observed["scales_shape"], (6, 2)) + self.assertEqual(tuple(received.shape), (2, 3, 256)) + + def test_identity_expert_output_applies_local_gates(self): + adapter = object.__new__(MODULE.BackendAdapter) + adapter.arm = "uccl" + adapter.hidden = 2 + adapter.profile = MODULE.WORKLOAD_PROFILES["prefill"] + state = MODULE.DispatchState( + recv_x=torch.tensor([[2.0, 4.0]], dtype=torch.bfloat16), + recv_topk_idx=torch.tensor([[0, -1, 1]]), + recv_topk_weights=torch.tensor([[0.25, 0.5, 0.125]]), + handle=None, + ) + output = adapter.identity_expert_output(state, "bf16") + torch.testing.assert_close( + output, + torch.tensor([[0.75, 1.5]], dtype=torch.bfloat16), + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_summarize_results.py b/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_summarize_results.py new file mode 100644 index 000000000..42b1bfe21 --- /dev/null +++ b/micro-benchmarks/expert-parallelism/ep-backend-comparison/test_summarize_results.py @@ -0,0 +1,253 @@ +import importlib.util +from pathlib import Path +import sys +import tempfile +import unittest + + +module_path = Path(__file__).with_name("summarize_results.py") +sys.path.insert(0, str(module_path.parent)) +spec = importlib.util.spec_from_file_location("summarize_results", module_path) +assert spec and spec.loader +summary_module = importlib.util.module_from_spec(spec) +sys.modules[spec.name] = summary_module +spec.loader.exec_module(summary_module) + + +def fake_result(profile, arm, world_size, run_index, dtype, latency_ms): + digest = { + "uccl": "a", + "deepep-v1-nvshmem": "b", + "deepep-v2-gin-gda": "c", + }[arm] + tokens_per_rank = summary_module.PROFILE_CONFIG[profile]["tokens_per_rank"] + decode_logical_bytes = 22_249_472 if dtype == "fp8" else 29_360_128 + logical_bytes = decode_logical_bytes * tokens_per_rank // 128 + scaleout_bytes = logical_bytes * (world_size - 8) // world_size + tolerance = 9e-4 if dtype == "fp8" else 1e-5 + global_input_tokens = tokens_per_rank * world_size + result = { + "benchmark": "common-boundary-dispatch-combine", + "workload_profile": profile, + "backend_api_mode": ( + "elastic" + if arm == "deepep-v2-gin-gda" + else summary_module.PROFILE_CONFIG[profile]["api_mode"] + ), + "primary_metric": summary_module.PROFILE_CONFIG[profile]["primary_metric"], + "layout_in_timed_region": profile == "prefill", + "arm": arm, + "world_size_ranks": world_size, + "nodes": world_size // 8, + "gpus_per_node": 8, + "run_index_dimensionless": run_index, + "dispatch_dtype": dtype, + "tokens_per_rank": tokens_per_rank, + "global_input_tokens": global_input_tokens, + "hidden_dimensions": 7168, + "experts": 256, + "top_k_dimensionless": 8, + "warmup_iterations": 20, + "measured_iterations": 100, + "num_sms_dimensionless": 64 if arm == "deepep-v2-gin-gda" else None, + "detected_rdma_gigabytes_per_second": ( + 50.0 if arm == "deepep-v2-gin-gda" else None + ), + "route_hash_sha256": ("d" if world_size == 16 else "e") * 64, + "input_hash_sha256": ("f" if world_size == 16 else "0") * 64, + "global_valid_expert_selections": global_input_tokens * 8, + "avg_logical_payload_bytes_per_rank": logical_bytes, + "avg_scaleout_logical_payload_bytes_per_rank": scaleout_bytes, + "correctness": { + "status": "PASS", + "tolerance_dimensionless": tolerance, + "normalized_diff_dimensionless": 0.0, + }, + "runtime": { + "image_reference": f"example.invalid/{arm}@sha256:{digest * 64}", + "gpu": "NVIDIA B200", + "torch_version": "2.13.0+cu130", + "cuda_version": "13.0", + "nccl_version": [2, 29, 7], + "nccl_version_loaded": ( + [2, 31, 2] if arm == "deepep-v2-gin-gda" else [2, 29, 7] + ), + }, + "latency_ms": {"median": latency_ms}, + "aggregate_input_tokens_per_second": global_input_tokens / (latency_ms / 1e3), + "effective_logical_gigabytes_per_second_per_rank": logical_bytes + / (latency_ms / 1e3) + / 1e9, + "effective_scaleout_logical_gigabytes_per_second_per_rank": scaleout_bytes + / (latency_ms / 1e3) + / 1e9, + "timing_boundary": summary_module.PROFILE_CONFIG[profile]["timing_boundary"], + "logical_payload_definition": summary_module.LOGICAL_PAYLOAD_DEFINITION, + } + return result + + +class SummarizeResultsTest(unittest.TestCase): + def setUp(self): + arm_latency = { + "uccl": 1.0, + "deepep-v1-nvshmem": 1.2, + "deepep-v2-gin-gda": 0.8, + } + self.results = [ + fake_result( + profile, + arm, + world, + run, + dtype, + arm_latency[arm] * (1 + (run - 2) * 0.01), + ) + for profile in summary_module.PROFILES + for arm in summary_module.ARMS + for world in (16, 32) + for run in range(1, 4) + for dtype in summary_module.DTYPES + ] + + def test_valid_matrix_and_paired_delta(self): + summary_module.validate(self.results, 3) + summary = summary_module.summarize(self.results, 3) + self.assertEqual(summary["status"], "PASS") + self.assertEqual(len(summary["cells"]), 8) + comparison = summary["cells"][0]["comparisons"]["deepep-v2-gin-gda_vs_uccl"] + self.assertAlmostEqual(comparison["median_paired_improvement_percent"], 20.0) + self.assertTrue(comparison["direction_supported"]) + + prefill = next( + cell + for cell in summary["cells"] + if cell["workload_profile"] == "prefill" + and cell["world_size_ranks"] == 16 + and cell["dispatch_dtype"] == "fp8" + ) + prefill_comparison = prefill["comparisons"]["deepep-v2-gin-gda_vs_uccl"] + self.assertAlmostEqual( + prefill_comparison["median_paired_improvement_percent"], 20.0 + ) + + def test_missing_start_is_rejected(self): + with self.assertRaises(ValueError): + summary_module.validate(self.results[:-1], 3) + + def test_single_world_size_matrix_is_summarized(self): + ep16_only = [ + result for result in self.results if result["world_size_ranks"] == 16 + ] + summary_module.validate(ep16_only, 3) + summary = summary_module.summarize(ep16_only, 3) + self.assertEqual(summary["status"], "PASS") + self.assertEqual(len(summary["cells"]), 4) + self.assertEqual(summary["configuration"]["world_sizes_ranks"], [16]) + + def test_explicit_world_sizes_reject_a_missing_matrix(self): + ep16_only = [ + result for result in self.results if result["world_size_ranks"] == 16 + ] + with self.assertRaises(ValueError): + summary_module.validate(ep16_only, 3, (16, 32)) + + def test_too_few_starts_are_rejected(self): + with self.assertRaises(ValueError): + summary_module.validate(self.results, 1) + + def test_route_mismatch_is_rejected(self): + self.results[0]["route_hash_sha256"] = "different" + with self.assertRaises(ValueError): + summary_module.validate(self.results, 3) + + def test_runtime_mismatch_is_rejected(self): + self.results[0]["runtime"]["nccl_version"] = [9, 9, 9] + with self.assertRaises(ValueError): + summary_module.validate(self.results, 3) + + def test_loaded_nccl_may_differ_across_arms_but_not_within_one(self): + summary_module.validate(self.results, 3) + summary = summary_module.summarize(self.results, 3) + self.assertEqual( + summary["loaded_nccl_versions_per_arm"]["deepep-v2-gin-gda"], [2, 31, 2] + ) + self.results[0]["runtime"]["nccl_version_loaded"] = [9, 9, 9] + with self.assertRaises(ValueError): + summary_module.validate(self.results, 3) + + def test_num_sms_mismatch_within_one_arm_cell_is_rejected(self): + target = next( + result + for result in self.results + if result["arm"] == "deepep-v2-gin-gda" + and result["run_index_dimensionless"] == 1 + ) + target["num_sms_dimensionless"] = 24 + with self.assertRaises(ValueError): + summary_module.validate(self.results, 3) + + def test_logical_payload_mismatch_is_rejected(self): + self.results[0]["avg_logical_payload_bytes_per_rank"] += 1 + with self.assertRaises(ValueError): + summary_module.validate(self.results, 3) + + def test_derived_metric_mismatch_is_rejected(self): + self.results[0]["aggregate_input_tokens_per_second"] += 1 + with self.assertRaises(ValueError): + summary_module.validate(self.results, 3) + + def test_provenance_image_mismatch_is_rejected(self): + provenance = { + "campaign_id": "test-campaign", + "created_at_utc": "2026-08-24T00:00:00Z", + "region": "ap-south-1", + "cluster": "test-cluster", + "git_commit": "a" * 40, + "images": { + arm: next( + result["runtime"]["image_reference"] + for result in self.results + if result["arm"] == arm + ) + for arm in summary_module.ARMS + }, + "comparison": { + "profiles": { + profile: { + "tokens_per_rank": summary_module.PROFILE_CONFIG[profile][ + "tokens_per_rank" + ], + "api_mode": summary_module.PROFILE_CONFIG[profile]["api_mode"], + "primary_metric": summary_module.PROFILE_CONFIG[profile][ + "primary_metric" + ], + } + for profile in summary_module.PROFILES + }, + "hidden_dimensions": 7168, + "experts": 256, + "top_k_dimensionless": 8, + "warmup_iterations": 20, + "measured_iterations": 100, + "independent_starts": 3, + }, + } + provenance["images"]["uccl"] = "example.invalid/changed@sha256:" + "9" * 64 + with self.assertRaises(ValueError): + summary_module.validate_provenance(provenance, self.results, 3) + + def test_load_results_accepts_native_diagnostic_after_json(self): + result = fake_result("decode", "deepep-v2-gin-gda", 32, 1, "fp8", 0.9) + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / "rank-zero.log" + path.write_text( + "ADAI_EP_RESULT " + + summary_module.json.dumps(result) + + "Elastic buffer uses 3 channels per SM\n" + ) + self.assertEqual(summary_module.load_results(Path(directory)), [result]) + + +if __name__ == "__main__": + unittest.main()