nccl-tuner-amd

NCCL tuner plugin that optimizes AllReduce protocol selection (LL / LL128 / Simple) on AMD EPYC systems with PCIe-connected GPUs.

NCCL's built-in cost model is tuned for NVLink topologies and makes suboptimal protocol choices on PCIe-based AMD systems, leaving up to 38% latency on the table in the message size range critical for LLM inference (512K–2M).

The problem

NCCL selects communication protocols using an internal cost model (time = latency + nBytes / bandwidth). On AMD EPYC systems with PCIe GPUs:

1. LL128 is completely disabled for PHB/SYS paths (typeIntra > PATH_NVB) due to CPU cache-line atomicity concerns, even though it works correctly on modern AMD platforms
2. LL → Simple crossover is too late — the cost model overestimates Simple's latency, keeping LL active ~2x longer than optimal
3. No LL128 zone exists — without LL128, there's a significant latency gap between where LL saturates and Simple becomes efficient

Measured impact (8 GPU AllReduce)

Message size	NCCL default	With tuner plugin	Latency reduction
512K	57.6 µs (LL)	54.5 µs (LL128)	5%
768K	77.3 µs (LL)	63.8 µs (LL128)	17%
1M	98.1 µs (LL)	75.6 µs (LL128)	23%
1.5M	152.6 µs (LL)	96.7 µs (LL128)	37%
2M	193.8 µs (LL)	119.8 µs (LL128)	38%

LLM inference impact

While the plugin significantly improves raw NCCL AllReduce performance, it has zero measurable impact on LLM decode throughput in typical MoE inference configurations. Tested with:

• Qwen3.5-397B NVFP4 (TP=4, cutlass MoE, --disable-custom-all-reduce, SGLang)
• GLM-5 NVFP4 (TP=8, b12x MoE, SGLang)
• Full decode benchmark: concurrency 1–64, context 0–128K tokens

Root cause: AllReduce message sizes are too small. NCCL call tracing during Qwen3.5 TP=4 inference reveals:

AllReduce size distribution (CUDA graph capture, ~363 calls per forward pass):

Size       Calls    %      Protocol Zone
  8 KB       363    8.3%   LL          (bs=1)
 16 KB       363    8.3%   LL          (bs=2)
 32 KB       363    8.3%   LL          (bs=4)
 64 KB       363    8.3%   LL          (bs=8)
 96 KB       363    8.3%   LL          (bs=12)
128 KB       363    8.3%   LL          (bs=16)
192 KB       363    8.3%   LL          (bs=24)
256 KB       363    8.3%   LL          (bs=32)
320 KB       363    8.3%   LL          (bs=40)
384 KB       363    8.3%   LL          (bs=48)
448 KB       363    8.3%   LL          (bs=56)
512 KB       363    8.3%   LL128 ←     (bs=64)

LL (≤448K):    91.7%  — plugin changes nothing here
LL128 (>448K):  8.3%  — only max batch size reaches this zone

The per-layer AllReduce payload is batch_size × hidden_size_per_shard × 2B (bf16). With hidden=4096 and TP=4, even at max batch (64), the payload is only 512 KB — barely into LL128 territory. At typical serving concurrency (1–16), all calls stay in the LL zone.

When the plugin would help more:

• TP=2 (doubles per-GPU hidden → 2× payload)
• Dense models with larger hidden states (>8K)
• Expert Parallelism with larger all-to-all payloads
• Any NCCL-heavy workload with payloads in 512K–2M range

Additional finding: SGLang's --enable-flashinfer-allreduce-fusion does not work on SM120 (RTX PRO 6000). The code requires is_sm90_supported() or is_sm100_supported(), neither of which is true for SM120 (capability 12.0). All allreduce goes through NCCL regardless.

How it works

The plugin implements NCCL's tuner plugin API (v5). On each AllReduce call, it receives the cost table computed by NCCL core and penalizes non-optimal protocols for the Ring algorithm based on message size:

≤4 GPUs (same NUMA node):       >4 GPUs (cross-socket):
  0 – 576K  → LL                  0 – 448K  → LL
  576K+     → Simple               448K – 3M → LL128
                                   3M+       → Simple

These thresholds were determined by exhaustive latency measurements (500 iterations per data point) on a dual-socket AMD EPYC 9575F (Turin) system with 8x NVIDIA RTX PRO 6000 (Blackwell).

Build

git clone https://github.com/voipmonitor/nccl-tuner-amd.git
cd nccl-tuner-amd
make

Produces libnccl-tuner.so. No NCCL source tree or CUDA SDK required — the plugin is a standalone shared library with zero dependencies.

Usage

export NCCL_TUNER_PLUGIN=/path/to/libnccl-tuner.so
export NCCL_PROTO=LL,LL128,Simple    # Required: enables all 3 protocols (NCCL disables LL128 by default on PHB/SYS)
export NCCL_P2P_LEVEL=SYS            # Recommended: enables P2P transport on AMD (see below)

Then run your application normally. The plugin is loaded automatically by NCCL at communicator init.

For 8-GPU cross-socket systems

If NCCL detects your cross-socket topology poorly (common on dual-socket AMD), you may also need:

export NCCL_GRAPH_FILE=/path/to/graph.xml    # topology override (see examples/ directory)

Verify it's working

NCCL_DEBUG=INFO your_application 2>&1 | grep TUNER
# Should show: TUNER/Plugin: Using AMD-Turin-Optimal (v5)

Tuning for your system

The default thresholds are optimal for AMD EPYC 9575F + RTX PRO 6000. For other AMD systems, you can override them via environment variables:

Variable	Default	Description
NCCL_TUNER_LL_MAX_4GPU	576K	Max message size for LL protocol (≤4 GPUs)
NCCL_TUNER_LL_MAX_8GPU	448K	Max message size for LL protocol (>4 GPUs)
NCCL_TUNER_LL128_MAX_8GPU	3M	Max message size for LL128 protocol (>4 GPUs)

Values accept K and M suffixes (e.g., 512K, 2M).

Finding optimal thresholds for your hardware

Use nccl-tests to measure latency at each protocol and message size:

# Install nccl-tests if needed
git clone https://github.com/NVIDIA/nccl-tests.git
cd nccl-tests && make MPI=0

# Test each protocol individually (8 GPUs, 500 iterations, key sizes)
for proto in LL LL128 Simple; do
  echo "=== $proto ==="
  for size in 256K 384K 448K 512K 576K 640K 768K 1M 1536K 2M 3M 4M; do
    NCCL_PROTO=$proto NCCL_ALGO=Ring NCCL_P2P_LEVEL=SYS \
      ./build/all_reduce_perf -b $size -e $size -g 8 -n 500 -w 50 2>&1 \
      | grep -E "^\s+[0-9]" | awk -v s=$size -v p=$proto '{printf "%s %s latency=%.2f us  busbw=%.2f GB/s\n", s, p, $6, $9}'
  done
done

Look for the crossover points where one protocol's latency drops below another. The optimal thresholds are:

• LL_MAX: the largest size where LL latency < LL128 latency (or < Simple if LL128 doesn't help)
• LL128_MAX: the largest size where LL128 latency < Simple latency

Reproducing our benchmark results

System configuration

• CPU: 2x AMD EPYC 9575F (Turin, Zen 5), 64 cores per socket
• GPU: 8x NVIDIA RTX PRO 6000 (Blackwell, sm_120)
• Topology: GPU 0-3 on NUMA 0, GPU 4-7 on NUMA 1, PCIe Gen5
• NCCL: 2.29.7+cuda13.2 (built from source, master branch commit 3619159)
• OS: Linux 6.17.0

Reproducing the measurements

# Build NCCL from source
git clone https://github.com/NVIDIA/nccl.git && cd nccl
make -j$(nproc) src.build
export LD_LIBRARY_PATH=$(pwd)/build/lib

# Build nccl-tests
git clone https://github.com/NVIDIA/nccl-tests.git && cd nccl-tests
make MPI=0 NCCL_HOME=../nccl/build

# Build the tuner plugin
git clone https://github.com/voipmonitor/nccl-tuner-amd.git && cd nccl-tuner-amd
make

# --- 4 GPU same NUMA ---
# Stock NCCL (no plugin):
CUDA_VISIBLE_DEVICES=0,1,2,3 NCCL_P2P_LEVEL=SYS \
  ../nccl-tests/build/all_reduce_perf -b 256K -e 4M -f 2 -g 4 -n 500 -w 50

# With tuner plugin:
CUDA_VISIBLE_DEVICES=0,1,2,3 NCCL_P2P_LEVEL=SYS \
  NCCL_TUNER_PLUGIN=$(pwd)/libnccl-tuner.so NCCL_PROTO=LL,LL128,Simple \
  ../nccl-tests/build/all_reduce_perf -b 256K -e 4M -f 2 -g 4 -n 500 -w 50

# --- 8 GPU cross-socket ---
# Stock NCCL:
NCCL_P2P_LEVEL=SYS NCCL_GRAPH_FILE=$(pwd)/examples/graph_8gpu_epyc9575f.xml \
  ../nccl-tests/build/all_reduce_perf -b 256K -e 16M -f 2 -g 8 -n 500 -w 50

# With tuner plugin:
NCCL_P2P_LEVEL=SYS NCCL_GRAPH_FILE=$(pwd)/examples/graph_8gpu_epyc9575f.xml \
  NCCL_TUNER_PLUGIN=$(pwd)/libnccl-tuner.so NCCL_PROTO=LL,LL128,Simple \
  ../nccl-tests/build/all_reduce_perf -b 256K -e 16M -f 2 -g 8 -n 500 -w 50

Companion NCCL patches

This plugin addresses protocol selection, but there are two additional AMD-related issues in NCCL itself:

1. PR #2036 — AMD inter-CPU bandwidth detection: NCCL assigns a flat 16 GB/s for all AMD CPUs (vs 4 model-specific tiers for Intel). This PR adds per-generation bandwidth values (Zen 1/2: 16, Zen 3/4: 24, Zen 5: 32 GB/s).

2. PR #2080 — P2P transport on AMD with >2 GPUs: NCCL's default P2P level (PATH_PXB) forces shared memory transport for GPUs connected through the PCIe Host Bridge, losing 24-44% bandwidth. This PR enables PHB-level P2P for AMD systems.

Both PRs are independent of this plugin and address different layers of the stack.

Why not patch NCCL directly?

NCCL provides the tuner plugin API specifically for hardware-specific tuning. The protocol thresholds are highly dependent on the exact CPU, GPU, PCIe topology, and BIOS settings — values optimal for one system may not be optimal for another. A plugin is the right mechanism for this kind of per-system optimization.

The safety check that disables LL128 on PHB/SYS paths exists because CPUs can split 128-byte stores into two 64-byte cache lines, and with PCIe Relaxed Ordering the second half (containing the flag) could arrive before the first half (containing data). On modern AMD EPYC (Zen 3+) with current GPU drivers, this has not been observed in practice, but NCCL_PROTO=LL,LL128,Simple is needed to override the check. Use at your own risk on untested platforms.

LL128 safety validation

NCCL disables LL128 on PHB/SYS paths due to a theoretical concern: CPUs may split 128-byte PCIe writes into two 64-byte cache lines, and with Relaxed Ordering enabled, the second half (containing the LL128 flag) could arrive before the first half (containing data), causing silent corruption.

We validated LL128 safety on AMD EPYC 9575F (Turin, Zen 5) with PCIe Relaxed Ordering enabled (RlxdOrd+) across the inter-socket Infinity Fabric link. ~1.5 million LL128 operations with zero data corruption, including under full Infinity Fabric saturation.

Test results

Test	Operations	Sizes	Result
AllReduce Ring, 10K iters x 6 sizes	60,000	512K-3M	0 errors
AllReduce Ring, 100K iters x 5 sizes	500,000	256K-4M	0 errors
AllReduce Tree, 100K iters x 5 sizes	500,000	256K-4M	0 errors
AllReduce half/float/double, 20K iters	180,000	512K-2M	0 errors
ReduceScatter, 50K iters x 4 sizes	200,000	512K-4M	0 errors
AllReduce under Infinity Fabric saturation	100,000	256K-4M	0 errors

How to reproduce

# Prerequisites: nccl-tests built against NCCL 2.29.7+

# 1. Basic validation — 100K iterations, 8 GPUs cross-socket
NCCL_PROTO=LL128 NCCL_ALGO=Ring NCCL_P2P_LEVEL=SYS \
  NCCL_GRAPH_FILE=examples/graph_8gpu_epyc9575f.xml \
  all_reduce_perf -b 256K -e 4M -f 2 -g 8 -n 100000 -w 500
# Check #wrong columns — must all be 0

# 2. Stress test under Infinity Fabric saturation
#    Terminal 1 — saturate the inter-socket link:
numactl --cpunodebind=1 --membind=0 stress-ng --vm 16 --vm-bytes 256M &
numactl --cpunodebind=0 --membind=1 stress-ng --vm 16 --vm-bytes 256M &

#    Terminal 2 — run LL128 validation under load:
NCCL_PROTO=LL128 NCCL_ALGO=Ring NCCL_P2P_LEVEL=SYS \
  NCCL_GRAPH_FILE=examples/graph_8gpu_epyc9575f.xml \
  all_reduce_perf -b 256K -e 4M -f 2 -g 8 -n 20000 -w 200
# Latency will be ~100x higher due to contention, but #wrong must be 0

#    Clean up:
killall stress-ng

Note

This validation covers AMD EPYC 9575F (Turin, Zen 5) only. Other AMD generations or non-AMD platforms may behave differently. Always validate on your specific hardware before enabling LL128 in production.

License

Apache-2.0