Replace EP backend comparison with Decode-like and Prefill-like B200 benchmarks

[KeitaW]

Purpose

Relates to #1234.

Replace the previous backend-native EP result tables with a common-boundary B200 microbenchmark. The retired B300, B200, H100, and Decode-only result files are removed because their latency and bandwidth fields did not share one timing boundary or byte numerator.

This microbenchmark covers synthetic Decode-like and Prefill-like expert-parallel communication.

What changed

• Add separate Decode-like and Prefill-like profiles. Decode-like uses 128 tokens/rank and the low-latency API. Prefill-like uses 4,096 tokens/rank and the normal high-throughput API, including required layout work.
• Run UCCL, DeepEP V1 NVSHMEM, and DeepEP V2 NCCL GIN with the same deterministic BF16 input, exact top-k route and weights, model shape, named nodes, and runtime stack.
• Measure one external CUDA event boundary from dispatch-input readiness through required layout, dispatch, and combine completion. Host-side FP8 conversion runs once before the boundary; low-latency kernels that quantize internally do so inside it. Each iteration reports the slowest rank.
• Use slowest-rank latency as the primary metric for both profiles. Aggregate input tokens/s is the cross-EP-size metric; logical throughput columns are secondary, and the scale-out column is comparable only within one EP size.
• Rotate backend, profile, and dtype order across independent process starts per cell (default 20). Each start contains 20 warmup iterations and 100 measured iterations per dtype.
• Record per-arm runtime provenance: the loaded NCCL library version, the resolved communication-kernel SM count, and the detected RDMA GB/s. EP_NUM_SMS pins the SM count.
• Support variable node counts and EP sizes (EP_WORLD_SIZES; the summarizer takes --world-sizes or derives them from the logs), so a 2-node EP16 campaign summarizes and plots.
• Reject incomplete matrices, correctness failures, mutable image tags, route or input mismatches, runtime mismatches, payload mismatches, and derived metrics that disagree with latency and logical-byte accounting.
• Coordinate through the shared Lease, reject selected/protected node overlap, retain every log and rendered object, and require verified teardown before writing CAMPAIGN_COMPLETE.
• Replace the retired result set with a human-readable report and machine-readable summaries. Update the SGLang documentation that referenced the retired tables.

Review response

df6d2943 addresses whn09's review: the FP8 cast is hoisted out of the timed region, the start count is raised so the bootstrap interval carries information, slowest-rank latency is the primary metric for both profiles, EP_BUFFER_DEBUG is off on scored runs, the summarizer and launcher accept variable node counts and EP sizes, and every result records the loaded NCCL version and the resolved SM count. 40990df2 retries node admission while a prior case's pod is still terminating. d35c861e replaces the tables with measurements taken on that harness.

B200 results

Two campaigns on 4 p6-b200.48xlarge nodes in us-east-1c. The primary campaign is EP16 with 16 independent starts; the companion campaign adds EP32 at 4 starts for the EP-size axis, where the intervals are weak.

Slowest-rank dispatch-plus-combine latency, medians across starts:

EP size	Dispatch dtype	Starts	UCCL	DeepEP V1 NVSHMEM	DeepEP V2 NCCL GIN
16 ranks decode	FP8	16	0.5258 ms	1.0264 ms	0.4712 ms
16 ranks decode	BF16	16	0.5986 ms	1.0358 ms	0.4839 ms
16 ranks prefill	FP8	16	5.2691 ms	2.9742 ms	2.6578 ms
16 ranks prefill	BF16	16	5.5088 ms	3.5448 ms	3.3709 ms
32 ranks decode	FP8	4	0.7758 ms	1.5568 ms	0.8361 ms
32 ranks decode	BF16	4	0.8718 ms	1.5598 ms	0.9423 ms
32 ranks prefill	FP8	4	8.0894 ms	12.7493 ms	6.6226 ms
32 ranks prefill	BF16	4	9.8898 ms	13.1769 ms	8.7098 ms

DeepEP V2 has the lowest latency in all four EP16 cells, by about 10% (FP8) and 19% (BF16) over UCCL for decode and 39% to 50% for prefill. All eight paired directions are supported. At EP32 the decode ranking differs: UCCL leads both decode cells while DeepEP V2 leads both prefill cells, so an EP16 decode result does not carry to EP32. Every arm is faster with FP8 than with BF16; the FP8-slower-than-BF16 decode pattern in the retired tables does not appear here.

The recorded SM counts show the prefill arms are not SM-matched: DeepEP V2 resolves 64 SMs from its bandwidth heuristic while the normal-mode UCCL and DeepEP V1 buffers report 20. Part of the prefill margin may follow from that rather than from the transport. EP_NUM_SMS now pins the count for a matched comparison.

Per-start values, bootstrap intervals, paired improvements, provenance, and scope limits are in RESULTS.md. Machine-readable summaries: ep16-n16 SHA-256 60d2917ad4fb1f4662b4c522c740927d1403ba6a6f23c762937562e524a6b7c1 and ep16-ep32-n4 SHA-256 f4edd9a172d9b93e0903fbb3d43b499bf17aaee62e4dd3db3d212bab440173ff. The ap-south-1 summary is kept as a dated historical artifact.

Qualification and custody

• EKS cluster ml-clusters-shared-us-east-1 in us-east-1c, 4 named p6-b200.48xlarge nodes, each with 8 NVIDIA B200 GPUs and 8 allocatable EFA devices.
• EP16 used 2 nodes and 16 GPU ranks. EP32 used 4 nodes and 32 GPU ranks.
• PyTorch 2.13.0+cu130 and CUDA 13.0 across all scored records. Every arm loaded NCCL 2.31.2; the PyTorch build constant reports 2.29.7, and both are recorded.
• 3 backend images pinned by SHA-256 digest, the same images used for the retired ap-south-1 campaign. The DeepEP and UCCL source commits inside those images are not recorded.
• Primary campaign: 4 admission cases, 96 scored distributed starts, 192 scored records, 0 correctness failures, 1,124 checksum entries. Companion campaign: 4 admission cases, 48 scored distributed starts, 96 scored records, 0 correctness failures, 740 checksum entries.
• Exclusive shared-Lease ownership for each campaign. Every owned Pod, StatefulSet, and Service was removed and the Lease released.
• Both campaigns report a teardown failure: the owned namespace stays Terminating because two stale visibility.kueue.x-k8s.io APIServices in this cluster fail namespace deletion discovery. That is a cluster property, not a campaign property, and no scored record depends on it.
• The pinned UCCL normal-mode worker uses synchronized process-lifetime cleanup after result output because explicit proxy destruction invalidates the CUDA context. This happens after all timed iterations and does not change the timing boundary.

Validation

• Real B200 EKS campaigns: all admission cases and all 144 scored distributed starts passed across the two campaigns.
• The two campaigns' shared EP16 cells agree to within 1.3% in every arm and cell except BF16 decode for DeepEP V2 (2.5%), which is the cell with the highest run-to-run variability in both.
• Local unit suite: 13 tests passed and 8 CUDA-dependent tests skipped because local PyTorch is unavailable.
• ruff check, ruff format --check, py_compile, bash -n, shellcheck, and git diff --check: passed.

Final directory

micro-benchmarks/expert-parallelism/ep-backend-comparison/
├── README.md
├── RESULTS.md
├── ep_benchmark.py
├── extract_results.py
├── plot_results.py
├── result_io.py
├── results/
│   ├── b200-ap-south-1-2026-08-25-boxplots.png
│   ├── b200-ap-south-1-2026-08-25.json
│   ├── b200-us-east-1-2026-09-01-ep16-ep32-n4-boxplots.png
│   ├── b200-us-east-1-2026-09-01-ep16-ep32-n4.json
│   ├── b200-us-east-1-2026-09-01-ep16-n16-boxplots.png
│   └── b200-us-east-1-2026-09-01-ep16-n16.json
├── run_ep_comparison.sh
├── run_ep_rank.sh
├── summarize_results.py
├── test_ep_benchmark.py
└── test_summarize_results.py

[Xuan-1998]

What does latency here mean? why the latency here is lower but the bandwidth below is also lower? In DeepEPv2's benchmark bandwidth = (number of bytes) / latency

[KeitaW]

Update: I removed the previous backend-native tables and reran the comparison with separate Decode-like and Prefill-like profiles.

Latency now means the slowest-rank CUDA elapsed time for one dispatch followed by one combine. The boundary starts with the BF16 input and route ready, includes required FP8 conversion, and ends after combine completion. Prefill-like also includes the dispatch layout required by the normal UCCL and DeepEP V1 APIs.

Cross-backend throughput is now calculated as common logical payload bytes divided by that same elapsed time. For a fixed profile, EP size, and dtype, every backend has the same byte numerator, so lower latency necessarily corresponds to higher logical GB/s/rank. FP8 and BF16 have different useful-byte numerators and remain separate cells. Backend-native SO, SU, and aggregate bandwidth values are no longer presented as interchangeable metrics.

The replacement B200 run contains 3 independent process starts per cell, 20 warmup iterations and 100 measured iterations per dtype/start, and 72 scored records. All records passed correctness and matrix validation; the maximum run-to-run CV is 3.31 percent. The updated method, Decode-like and Prefill-like tables, provenance, and scope limits are in the replacement report.

[whn09]

Independent run on 2× p6-b300, plus a methodology review

Thanks for putting a single common timing boundary and a SHA-256-verified input/route on this — that part is a genuine improvement over what we do internally, and we're adopting the input+route hash check for our own comparisons.

We rebuilt this harness on 2× p6-b300.48xlarge (B300, sm_103, 16 EFA devices/node, EFA installer 1.50.0) and ran your ep_benchmark.py with one addition: a --num-sms / --tokens-per-rank flag pair and a num_sms_dimensionless record field. Arms: deepep-v2-gin-gda (DeepEP at 8e7b42e, deployed with the recipe merged in #1234) and uccl. deepep-v1-nvshmem was not built — DeepEP V1 cannot go cross-node on EFA. Your 16 unit tests pass against the edited file inside our V2 image.

2 nodes is exactly EP16, so we cannot check your EP32 column or any 16→32 scaling conclusion. Everything below is 3 starts × (20 warmup + 100 measured), median of per-start medians, dtype order rotated between starts, a fresh process group and port per run.

Reviewed at head b7a5e107.

---

1. The FP8-slower-than-BF16 decode result is the harness, not DeepEP

Your EP16 decode row has V2 at 0.6151 ms FP8 vs 0.4921 ms BF16 (1.250×) while FP8 puts fewer bytes on the wire. It reproduces on our hardware at every SM count and node count:

cell (24 SM)	FP8	BF16	FP8/BF16
V2 decode, 2N / EP16	667.1 µs	517.5 µs	1.289
V2 decode, 1N / EP8	333.9 µs	171.4 µs	1.949
UCCL decode, 2N / EP16	433.8 µs	465.5 µs	0.932

(logical payload 22.25 MB/rank FP8 vs 29.36 MB/rank BF16; UCCL is the control — it shows the expected FP8 win.)

We then re-ran the same cells through DeepEP's own tests/elastic/test_ep.py — same nodes, same image, same env, same SM count, same shapes, with the dispatch mode pinned to the one ep_benchmark.py uses (expert_alignment=1, async_with_compute_stream=1, do_handle_copy=1, allocate_on_comm_stream=0, no bias). That reports per-op kernel time from the profiler instead of one wall-clock event pair. Comparable pair: dispatch_impl + dispatch_copy_epilogue_impl, and combine_impl + combine_reduce_epilogue_impl.

At kernel level the sign flips in every cell:

cell (24 SM, dispatch+combine, slowest rank)	kernel µs	your wall µs	wall/kernel	overhead
decode 1N bf16	102.4	171.4	1.67×	+69.0
decode 1N fp8	94.8	333.9	3.52×	+239.1
decode 2N bf16	466.1	517.5	1.11×	+51.5
decode 2N fp8	453.4	667.1	1.47×	+213.7
prefill 1N bf16	1206.5	1677.2	1.39×	+470.7
prefill 1N fp8	921.6	1451.6	1.57×	+529.9
prefill 2N bf16	2141.9	2451.0	1.14×	+309.1
prefill 2N fp8	1766.8	2218.7	1.26×	+451.8

Kernel FP8/BF16 is 0.926 at 1 node and 0.973 at 2 nodes. FP8 dispatch is the faster kernel (272.8 vs 286.1 µs at 2N) and combine is identical to 0.1% (182.94 vs 182.78 µs) because combine is BF16 either way. There is no FP8 penalty in the kernels.

Where it comes from. iteration() (ep_benchmark.py:536-539) times prepare_dispatch_input + dispatch + combine under one event pair. prepare_dispatch_input (:332-339) returns x unchanged for BF16, but returns self._cast_to_fp8(x) when self.is_elastic or self.profile.api_mode == "normal". For the V2 arm that resolves to deep_ep.utils.math.per_token_cast_to_fp8, which is @torch.compile(dynamic=True) (deep_ep/utils/math.py:30 in the image). Timed alone on b300 at the decode shape it costs 89.6 µs device / 77.1 µs CPU (empty-event-pair floor 7.3 µs) — CPU ≈ device means it is launch/guard-bound with the GPU idle on Python, not arithmetic. In situ it costs about twice that: the FP8-only excess is 239.1 − 69.0 ≈ 170 µs at 1 node.

Two asymmetries, both against V2:

• Decode: only the V2 arm casts. The non-elastic low-latency path returns x unquantized and lets low_latency_dispatch(use_fp8=True) quantize inside the kernel, so UCCL and V1 never pay this at all.
• Prefill: all arms cast (api_mode == "normal"), but not with the same function. V2 uses the torch.compiled deep_ep one; UCCL uses /opt/uccl/ep/bench/utils.py:720, which has no @torch.compile decorator. We verified both in the running images.

A supporting hint in your own data: the FP8 penalty is +123.0 µs at EP16 but only +16.5 µs at EP32 (0.9642 vs 0.9477 ms). A property of the FP8 dispatch kernels would not shrink 7× when you add two nodes.

Fix (one line). x is fixed across the timed loop, so the cast can be hoisted out of iteration() entirely and done once before timing. If you'd rather keep conversion inside the boundary — a defensible choice, since it's real work in a real serving path — then use the same implementation for every arm and say so, because right now the cell measures three different quantizers.

Consequence for the headline: V2's FP8 dispatch+combine kernels are 453.4 µs against UCCL's 433.8 µs wall-clock on the same box, so the 1.54× FP8 decode gap is mostly harness. We can't close this properly — test_ep.py is DeepEP's bench and yields no kernel-only UCCL number — but the current figure is not a backend result.

---

2. The "95% bootstrap CI" is identically [min, max] of the 3 starts

bootstrap_median_ci (summarize_results.py:71-78) resamples 3 values with replacement and returns percentile(medians, 0.025), percentile(medians, 0.975). For n=3 the bootstrap median can only take the 3 sample values, with

P(median = min) = P(median = max) = 7/27 = 0.259

Since 0.259 > 0.025, the 2.5th percentile is always the sample min and the 97.5th always the sample max — for any data, any seed, and any number of BOOTSTRAP_SAMPLES. The interval is [min, max] by construction and its width carries no confidence information.

The published tables show exactly this signature: in all 40 rows (24 primary + 16 paired) the point estimate is strictly inside its interval and never equal to a bound.

Two knock-ons:

• direction_supported = stable and (ci_low > 0 or ci_high < 0) (:486) reduces to "all 3 paired deltas had the same sign." Under a null of independent per-start sign that is p = 0.25, not 0.05. All 16 directions are currently marked supported.
• paired (:455-468) matches arm A's start k with arm B's start k. But starts are independent process launches, and backend/dtype order rotates across starts, so "start 1" is not the same condition for both arms. The pairing is arbitrary and buys no variance reduction.

Fix: with n=3, report min / median / max explicitly and drop the CI and "direction supported" language; or raise the start count to where a percentile bootstrap means something (≥20); or use a test appropriate for n=3 and state the exact p.

---

3. The per-selection logical numerator over-counts DeepEP by a node-count-dependent factor — it inverts the 16→32 scaling sign

README.md:45: "Scale-out logical bytes include only assignments whose destination expert is on another node," and each assignment contributes a full tensor. But DeepEP's internode path sends one copy per destination node and NVLink-forwards to the local experts. So the numerator over-counts by (remote assignments) / (distinct remote nodes) — and that factor changes with node count.

Your own table pins it exactly:

	scale-out / total	remote assignments	remote nodes	over-count
EP16 (2 nodes)	126.42 / 252.84 = 0.500	4 of 8	1	4×
EP32 (4 nodes)	78.91 / 105.21 = 0.750	6 of 8	3	2×

Correcting each cell by its own factor:

	reported scale-out	corrected
EP16 FP8 prefill	126.42 GB/s/rank	31.6 GB/s/rank
EP32 FP8 prefill	78.91 GB/s/rank	39.5 GB/s/rank	+25%

So real per-rank scale-out bandwidth rises 25% from EP16 to EP32, while the reported metric falls 58% (252.84 → 105.21 GB/s/rank). The conclusion flips sign. Because the distortion is node-count dependent, this metric is not comparable along the EP-size axis at all — which is the only axis the prefill table has.

Your own Input throughput (tokens/s) column already tells a much more honest story across the same two cells: 23,272,859 → 19,368,582 tokens/s, a 16.8% drop, not 58%.

Fix: make latency (or the aggregate tokens/s you already compute) the prefill primary metric, or use a denominator that accounts for per-node deduplication. The "these are logical efficiency metrics, not observed wire bandwidth" disclaimer covers the absolute level but not the cross-EP-size comparison, which is where it actually misleads.

---

4. num_sms is uncontrolled, unrecorded, and derived from one NIC's bandwidth

You never pass num_sms, so ElasticBuffer auto-sizes. On b300 that does work at 2 nodes on EFA (at pin 8e7b42e, get_rdma_gbs reads sysfs before the ibstat fallback — which matters, because ibstat -l on p6-b300 lists only the two DOWN ConnectX-7 devices and never the 16 EFA ones). It prints:

rdma_gbs=50.0 … num_sms=64
[WARN] DeepEP GIN signal budget reduced the number of channels per SM from 4 to 3

50 GB/s is one EFA device's 400 Gb/s. A b300 GPU has two NICs, so the real per-GPU figure is 100 GB/s; the heuristic is fed half the bandwidth. Separately, prefer_overlap_with_compute=False floors the result at 64, so on this hardware the wrong input happens not to change the outcome — but the value is neither recorded in the results nor listed in the controls table, and it is the single largest knob we found.

Measured SM sensitivity on b300, 2N/EP16:

profile	dtype	12 SM	24 SM	48 SM	64 SM
decode	bf16	569.3	517.5	499.8	502.4
decode	fp8	710.9	667.1	659.6	642.5
prefill	bf16	2843.9	2451.0	2196.7	2134.2
prefill	fp8	2569.1	2218.7	1910.0	1894.9

(µs, medians of 3 starts.) Decode is essentially SM-flat above 24; prefill is not — 24 SM is 14.9% (bf16) / 17.1% (fp8) off the best point. The knee is shape-dependent, so there is no single value that could be hard-coded instead. Suggest recording num_sms in every result and adding it to the controls table.

---

5. EP_BUFFER_DEBUG=1 is set on the V2 arm only, and it is not free

run_ep_rank.sh:40 enables it for V2 and nowhere else; FI_EFA_USE_HW_CNTR=1 (:37) likewise. EP_BUFFER_DEBUG makes csrc/elastic/buffer.hpp printf ~24 numbers per dispatch inside the CPU-wait path, and you run do_cpu_sync=True.

A/B on b300, 2N/EP16/24 SM:

dtype	debug on	debug off	cost
fp8	667.1 µs	648.2 µs	+2.9%
bf16	517.5 µs	504.3 µs	+2.6%

Small in absolute terms, but ~5× the run-to-run CV of those cells (0.36% / 0.56%) and charged to one arm only. Likely worse on your side: ours goes to a file, yours to a Kubernetes pod log pipe. (FI_EFA_USE_HW_CNTR=1 we measured as neutral.)

---

6. runtime.nccl_version reports torch's constant, not the loaded library

ep_benchmark.py:641 records list(torch.cuda.nccl.version()), which is torch's compile-time constant. In our running V2 container /proc/self/maps shows /opt/nccl/build/lib/libnccl.so.2.31.2 actually loaded, and DeepEP itself prints DeepEP initialized with NCCL version: 2.31.2 (loaded library) while the record says (2, 28, 9). The field is misleading precisely in the source-built-NCCL image that GIN requires. Reading it from the loaded object (as DeepEP does) fixes it.

---

7. summarize_results.py cannot summarize anything smaller than a 4-node campaign

WORLD_SIZES = (16, 32) (:20) and validate() (:102-110) hard-require the full 3 arms × 2 world sizes × 2 dtypes × 2 profiles × 3 starts matrix, so a 2-node campaign is rejected outright rather than partially summarized. Anyone with 2 nodes — which is most people who will want to re-run this — has to write their own aggregator. We did, and we reused your result_io.load_result_log, which is genuinely necessary: a plain json.loads on a result line with native diagnostics appended raises Extra data. Nice catch, worth keeping.

---

8. Deleted material the "口径 doesn't match" rationale doesn't cover

RESULTS-p5.md and nccl-alltoall.yaml are deleted outright. The new harness maxes at 4 nodes, so several results in there can never be re-derived from it:

• the 32-node / 256-rank runnability result, and the source-level rank caps behind it (deep_ep.cpp:158, the 64 < cap ≤ 128 PE low-latency limit, the nvshmemi_process_multisend_rma -11 (EAGAIN) fan-out limit at ≥128 PEs on EFA);
• the NCCL all-to-all reference baseline — the only backend-independent absolute calibration in the whole comparison, whose loss compounds finding 3;
• your own note that DeepEP's default config uses only 20 SMs, i.e. that any default-config Blackwell number is a floor — which is finding 4.

benchmarks/README.md still keeps "An EP16 result cannot decide an EP32 or EP64 deployment" while now pointing at a matrix that is only EP16/EP32. (No dangling links, though — we checked.) Suggestion: keep the old file as a clearly-dated historical document rather than deleting it.

---

9. Smaller notes

• The route is perfectly balanced. make_route (:94) is (t*17 + k*31 + seed) % 256 with a stride coprime to num_experts, so every token hits exactly 8 distinct experts and every expert carries identical load. V2's own get_theoretical_num_sms docstring warns it assumes a balanced gate and must not be used for V3's group-limited gate. test_ep.py has --unbalanced-ratio / --masked-ratio; rankings may not survive skew. We haven't tested this axis yet either.
• API asymmetry in the decode cell. The PR body says decode uses "the low-latency API" for all arms; the README's API table is the correct one. V2 has only ElasticBuffer on EFA, so that cell pits UCCL/V1 purpose-built LL kernels against V2's general kernels. It penalises V2 and deserves a stated caveat.
• Every iteration is a cold start. The per-iteration end.synchronize() + all-reduce + .item() (:554-559) measures single-shot latency, not pipelined steady state. Defensible, but it should be labelled — it is a large part of the wall-vs-kernel gap in finding 1, and it means these absolutes cannot be compared against kernel-time figures from any other benchmark.
• No backend source commits in provenance. Only image digests, torch/cuda/nccl, and your repo's git_commit. We have measured ~1.8× swings in 2-node decode dispatch between two DeepEP V2 commits a few weeks apart on the same nodes, so a digest alone doesn't let a reader know which DeepEP produced a number. Recording the DeepEP/UCCL SHAs and the EFA installer version would be cheap.
• combine_input is all zeros in the timed loop (:534). Harmless as configured (BF16 combine, no logfmt), but it would matter the moment a compressed combine path is enabled.
• CUDA_DEVICE_MAX_CONNECTIONS=1 (run_ep_rank.sh:23) is set for all three arms and may serialise comm/compute given async_with_compute_stream=True.
• Controls table (README.md:23-38) omits SM/channel count, QP count, route distribution character, and the per-arm env vars from finding 5.
• Replicate outliers. Three of our single-node decode cells contain one replicate ~1.6× the others (277.5 vs ~170 µs; 526.6 vs ~331 µs). With n=3 that single replicate drives the run-to-run CV to 23-26%, so your 5% gate would reject the cell rather than report it. Medians across starts are unaffected, which is why we report those — but it's worth knowing the gate can silently drop cells rather than flagging a single bad start.

---

What reproduced

Your direction holds on different hardware with your own harness: UCCL wins decode, DeepEP V2 wins prefill. b300, 2N/EP16/24 SM, V2/UCCL ratio (>1 = V2 slower):

profile	BF16	FP8
decode	1.112	1.538
prefill	0.431	0.388

So "UCCL wins decode" survives — the BF16 figure is a real kernel result. The FP8 decode margin is mostly finding 1, and the prefill margin is larger than you report (+96.02% FP8 / +62.45% BF16 in your EP16 table vs 2.58×/2.32× here). One more note on prefill: FP8's advantage is larger at kernel level than a wall-clock boundary shows. Our wall-clock FP8/BF16 ratios are 0.87-0.91, while the kernels give 0.76 (1N) / 0.83 (2N) — the cast sits on the FP8 side in both arms here, so it compresses the measured gap.

Happy to share the full kit (launchers, table generator, raw logs) if that's useful, or to re-run any cell you'd like on this hardware.

[whn09]

Follow-up to my earlier review, on one point only: the decode column depends on which DeepEP tree the V2 arm is built from, and with two small kernel-side patches the ranking reverses. Same 2 × p6-b300, same harness, same nodes.

Disclosure: the two DeepEP PRs below are mine, so please weight this accordingly — everything here is reproducible from the recipe at the bottom, including the negative control.

The pin

This is not a stale-pin report. Our V2 arm is built at 8e7b42e, which was tip of amazon-contributing/DeepEP main when we built it and is still only 1 commit behind today's main (9c1f251, an unrelated QP-range clamp) — so "track main" does not fix decode; the fix is in two still-open PRs. Which commit your images contain we cannot tell, since the PR records no backend SHA (section 9 of the earlier review) — 8e7b42e is what the #1234 recipe resolved to for us, and the mechanism below holds for every mainline commit to date.

On that tree kNumParts is chosen from the GIN indexed-signal budget alone and is never lowered because a channel holds too few tokens — so at 128 tokens/rank a decode channel gets described as more parts than it has tokens for, and those tokens leave as separate single-token puts. That budget is loosest exactly at decode shapes, so decode lands on the worst end of the axis with no way to opt out.

amazon-contributing/DeepEP #2 adds the missing clamp; #1 is the JIT plumbing it stacks on. compare/8e7b42e...5a594a5 is ahead 2, behind 0 and touches two files, so the two images below differ by exactly those PRs and nothing else.

Effect on this PR's own numbers

Your ep_benchmark.py, 2 nodes = EP16, 24 SM, EP_BUFFER_DEBUG off on both arms, 3 starts × (20 + 100), medians. The UCCL denominator was re-measured under these same conditions rather than reused:

V2 arm	fp8 µs	bf16 µs	V2/UCCL fp8	V2/UCCL bf16
DeepEP 8e7b42e (mainline, unpatched)	647.9	504.4	1.489	1.064
DeepEP 5a594a5 (#1 + #2)	554.2	392.9	1.274	0.829
5a594a5 + EP_MIN_TOKENS_PER_PART=1 (clamp off, same image)	649.0	504.9	1.492	1.065
UCCL	435.1	474.2	—	—

So BF16 decode goes from V2 losing by 6.4% to V2 winning by 17%. The clamp-off control lands on the old image to within 0.1%, in the same binary, so this is the clamp rather than a build or environment difference.

Through DeepEP's own test_ep.py on the same cell, the effect is confined to dispatch: 283.35 → 176.84 µs (−37.6%), with combine flat to +0.7%. Prefill is unchanged (within 0.23%, run-to-run CV ≤ 0.9%), so nothing is being traded away. The clamp also flattens the SM curve — patched, 12 SM equals 24 SM; unpatched, dropping to 12 SM costs 11%.

And the FP8 term from section 1, now measured exactly

In this same cell the wall-clock-minus-kernel term is 33.0 µs for BF16 against 201.8 µs for FP8 (slowest rank, dispatch + combine, same code, same image). That 168.8 µs is prepare_dispatch_input's @torch.compiled per_token_cast_to_fp8, which only the elastic arm pays inside the timed region. 554.2 − 168.8 = 385.4, right next to BF16's 392.9 — the two dtypes agree once it is removed, which is what you would expect since FP8 moves fewer bytes.

With the clamp in and the cast hoisted out of the timed region, V2 wins decode in both dtypes on this hardware.

Suggestion

No change is needed to your harness for the first part — only the DeepEP tree the V2 arm is built from. #1 and #2 are both still open, so the low-friction order is probably:

1. Independently of the PRs, hoist the FP8 cast out of the timed region (or quantize once outside the loop). That is orthogonal and fixes the fp8 column regardless of any pin.
2. Once Fix numpy version in ddp test case requirements #2 lands, bump the V2 arm's DeepEP ref and re-run decode. If you want to check it before then, you do not need a second image: build once at 5a594a5 and A/B EP_MIN_TOKENS_PER_PART=1 against unset, which is the exact pre-patch geometry in the same binary.

One reproduction trap, if you do build two images. DeepEP's JIT cache key is name$$compiler$$flags$$code where code is the generated wrapper — the impl headers are only #included, so their content is not in the key, and with the env knobs unset flags is byte-identical between the two trees. Sharing one EP_JIT_CACHE_DIR (or one host mount for ~/.deep_ep) between the images makes the patched one load the old cubin, and the clamp measures as a clean no-op. One cache directory per image.

Caveats

• 2 nodes = EP16 only. Do not extrapolate to your EP32 column: on H200 with NCCL_GIN_TYPE=5 we measured the same clamp at 4 nodes and the decode-dispatch win drops to about −8%, because the indexed-signal budget tightens with rank count.
• On sm_103 the win is entirely Fix numpy version in ddp test case requirements #2. Bump numpy from 1.21.5 to 1.22.0 in /3.test_cases/4.DDP #1's EP_NUM_SUB_PARTS=1 is neutral to slightly negative here (397.0 vs 392.9 µs), unlike on H200 where it stacks further.
• Arms were run in blocks rather than interleaved within a rep; the clamp-off control ran at the end of each block and still matched the base image, which bounds drift over a block.

[KeitaW]

Thanks, this found real problems. Fixes are in df6d2943, and d35c861e replaces the tables with a re-measurement.

Fixed, keyed to your numbering:

1. The FP8 cast runs once before the timed region. The decode cell is now asymmetric in the other direction, since the low-latency UCCL and V1 kernels still quantize inside the boundary, so the README states that each arm is measured from its own API's dispatch-ready entry point.
2. Start count is configurable with a default of 20, so the percentile bootstrap no longer collapses to [min, max].
3. Slowest-rank latency is the primary metric for both profiles, aggregate input tokens/s is the cross-EP-size metric, and the scale-out column is documented as comparable only inside one EP size. We dropped the cross-size claim rather than correcting the numerator: the over-count factor depends on each backend's forwarding strategy, so a dedup-corrected denominator cannot be backend-common.
4. num_sms and the detected RDMA bandwidth are recorded per result, and EP_NUM_SMS pins the count.
5. EP_BUFFER_DEBUG is off for scored runs and stays on only for the INFO admission run.
6. The loaded NCCL version is recorded as a per-arm field. Torch's constant stays in the cross-arm signature, since replacing it there would fail every campaign once the arms load different builds.
7. EP_WORLD_SIZES and --world-sizes take any node count, and the summarizer derives the EP sizes from the logs when the flag is omitted, so a 2-node EP16 campaign aggregates and plots.
8. Single-shot iteration timing and the balanced route are now stated as interpretation limits.

Re-measured on 4 p6-b200 in us-east-1: EP16 at 16 starts (192 scored records), plus EP16 and EP32 at 4 starts for the size axis. 16 rather than 20 to fit the capacity window. At EP16 DeepEP V2 is fastest in all four cells, ahead of UCCL by 10.28% [9.08, 11.28] in FP8 decode and 18.78% [12.02, 19.87] in BF16 decode, and every arm is faster with FP8 than with BF16. Your EP32 caution holds: UCCL leads both decode cells there, 0.7758 ms against 0.8361 ms in FP8, so the EP16 decode result does not carry to EP32. These ran on a different cluster as well as a different harness from the earlier tables, so we are not attributing the decode change to any single fix.

[whn09]

LGTM

[KeitaW]

Thank you @whn09 !