A practical guide for selecting the right parallelism strategy for Large Language Model (LLM) training. This project provides hands-on experiments, performance analysis, and decision guidelines to help ML engineers understand when to use Data Parallelism (DDP), Pipeline Parallelism (PP), Tensor Parallelism (TP), or Hybrid approaches based on model size, GPU memory, and infrastructure constraints.
What You'll Learn:
- How
ncclAllReduce,ncclSend, andncclRecvoperations behave across different strategies - Trade-offs between throughput, memory efficiency, and communication overhead
- When to choose DDP vs Pipeline vs Tensor vs Hybrid parallelism
- How to profile and analyze NCCL communication bottlenecks
Last Updated: January 2026 Infrastructure: Oracle Cloud Infrastructure (OCI) OKE with NVIDIA A10 GPUs
Important Note on Hardware & Expected Results
This guide was developed using NVIDIA A10 GPUs (24GB VRAM, PCIe-connected, no NVLink). The A10 is a workstation/inference-class GPU with limited memory bandwidth and no high-speed GPU-to-GPU interconnect. As a result:
Hardware Aspect A10 (This Guide) Datacenter GPUs (A100/H100/H200/GB200) GPU Memory 24 GB 40-192 GB GPU Interconnect PCIe Gen4 NVLink (600-900 GB/s) Network 10-25 GbE InfiniBand (200-400 Gb/s) Tensor Parallelism Limited benefit Highly efficient Pipeline Bubbles More pronounced Better hidden with faster comm What This Means:
- Throughput numbers will be significantly higher on production GPU clusters (A100, H100, H200, GB200) due to NVLink enabling near-instantaneous GPU-to-GPU communication
- Tensor Parallelism scales much better with NVLink—our A10 results underestimate TP benefits on datacenter hardware
- Pipeline bubble overhead decreases when using InfiniBand and NVLink, as activation transfers complete faster
- Hybrid parallelism (PP×TP) becomes essential for 100B+ parameter models on high-end clusters
Purpose of This Guide: The goal is not to provide absolute performance numbers, but to demonstrate how to evaluate and compare parallelism strategies before committing to expensive large-scale training runs. Use this methodology to:
- Understand NCCL communication patterns for each strategy
- Profile and identify bottlenecks in your specific infrastructure
- Make informed decisions about parallelism configuration
- Validate your distributed training setup before scaling up
Always run similar benchmarks on your target hardware to establish baseline metrics before production training.
| Track | Strategy | GPUs | Tokens/sec | NCCL Operations | Scaling Efficiency |
|---|---|---|---|---|---|
| A | Single-Node DDP | 2 | 49,290 | ncclAllReduce |
100% (baseline) |
| B | Multi-Node DDP | 4 | 52,847 | ncclAllReduce |
54% |
| C | Pipeline PP=2 | 2 | 9,665 | ncclSend/ncclRecv |
20% |
| D | Hybrid PP=2×TP=2 | 4 | 10,069 | ncclAllReduce + ncclSend/ncclRecv |
10% |
| E | Pipeline PP=4 | 4 | 8,343 | ncclSend/ncclRecv |
8% |
- Understanding NCCL Communication Patterns
- Parallelism Strategies Explained
- Infrastructure Requirements
- Quick Start
- Running on OCI Infrastructure
- Running On-Premises / Bare Metal
- Detailed Benchmark Results
- Nsight Systems Profiling
- Troubleshooting
NVIDIA Collective Communications Library (NCCL) is the backbone of distributed GPU training. Understanding its operations is critical for optimizing LLM training performance.
Purpose: Combine gradients from all GPUs and distribute the result back
Operation Flow:
GPU0: [g0] ─┐
GPU1: [g1] ─┼──► AllReduce ──► All GPUs get: [g0+g1+g2+g3]
GPU2: [g2] ─┤
GPU3: [g3] ─┘
NCCL Log Pattern:
NCCL INFO Ring 00 : 0 -> 1 -> 2 -> 3 -> 0
NCCL INFO comm 0x7f... rank 0 nranks 4 cudaDev 0 ... allReduce
When Used:
- Data Parallelism (DDP): Synchronize gradients after backward pass
- Tensor Parallelism (TP): Aggregate partial results within layers
Performance Characteristics:
- Bandwidth-bound: Scales with total gradient size
- Ring Algorithm: O(2(n-1)/n × data_size) - near-optimal bandwidth utilization
- Tree Algorithm: Better latency for small messages
- Best for: High-bandwidth interconnects (NVLink, InfiniBand, high-speed Ethernet)
Purpose: Transfer activations between pipeline stages
Operation Flow (Forward Pass):
Stage 0 (GPU0) ──ncclSend──► Stage 1 (GPU1)
│ │
[activations] [activations]
│ │
Stage 0 (GPU0) ◄──ncclRecv── Stage 1 (GPU1)
(Backward Pass)
NCCL Log Pattern:
NCCL INFO comm 0x7f... rank 0 nranks 4 ... send to 1
NCCL INFO comm 0x7f... rank 1 nranks 4 ... recv from 0
When Used:
- Pipeline Parallelism: Transfer activations between stages
- Async operations (isend/irecv) for overlap with compute
Performance Characteristics:
- Latency-bound: Each transfer adds latency
- Sequential dependency: Creates pipeline bubbles
- Best for: Large models that don't fit in single GPU memory
Purpose: Distribute/collect tensor shards across GPUs
AllGather (Column Parallel → Row Parallel transition):
GPU0: [A0] ─┐
GPU1: [A1] ─┼──► AllGather ──► All GPUs get: [A0, A1, A2, A3]
GPU2: [A2] ─┤
GPU3: [A3] ─┘
ReduceScatter (Gradient aggregation with distribution):
GPU0: [g0,g1,g2,g3] ─┐
GPU1: [g0,g1,g2,g3] ─┼──► ReduceScatter ──► GPU0: [Σg0], GPU1: [Σg1], ...
GPU2: [g0,g1,g2,g3] ─┤
GPU3: [g0,g1,g2,g3] ─┘
| Operation | Use Case | Latency | Bandwidth | Scalability |
|---|---|---|---|---|
ncclAllReduce |
DDP gradients, TP | Medium | High | Excellent |
ncclSend/Recv |
PP activations | High | Medium | Limited by stages |
ncclAllGather |
TP (fwd pass) | Medium | High | Good |
ncclReduceScatter |
TP (bwd pass) | Medium | High | Good |
Concept: Replicate the entire model on each GPU, partition the data batch.
┌─────────────────────────────────────────────────────────────┐
│ DATA PARALLELISM (DDP) │
├─────────────────────────────────────────────────────────────┤
│ │
│ GPU 0 GPU 1 GPU 2 │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │Full Model│ │Full Model│ │Full Model│ │
│ │ (copy) │ │ (copy) │ │ (copy) │ │
│ └────┬─────┘ └────┬─────┘ └────┬─────┘ │
│ │ │ │ │
│ Data Batch 0 Data Batch 1 Data Batch 2 │
│ │ │ │ │
│ └──────────────────┼──────────────────┘ │
│ │ │
│ ncclAllReduce │
│ (Gradient Synchronization) │
│ │
└─────────────────────────────────────────────────────────────┘
NCCL Communication:
- Forward: None (independent computation)
- Backward: ncclAllReduce after each layer's gradient computation
- Frequency: O(num_parameters) per step
Benchmark Results (Our Tests):
- Track A (2 GPU): 49,290 tokens/sec - Baseline performance
- Track B (4 GPU): 52,847 tokens/sec - 54% scaling efficiency
When to Use DDP:
- Model fits entirely in GPU memory
- High-bandwidth interconnect available
- Want maximum throughput for medium-sized models
Concept: Split model layers across GPUs, process micro-batches in pipeline fashion.
┌─────────────────────────────────────────────────────────────┐
│ PIPELINE PARALLELISM (PP=4) │
├─────────────────────────────────────────────────────────────┤
│ │
│ Stage 0 Stage 1 Stage 2 Stage 3 │
│ (GPU 0) (GPU 1) (GPU 2) (GPU 3) │
│ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ │
│ │Layers │──►│Layers │──►│Layers │──►│Layers │ │
│ │ 0-2 │ │ 3-5 │ │ 6-8 │ │ 9-11 │ │
│ └────────┘ └────────┘ └────────┘ └────────┘ │
│ │ │ │ │ │
│ ncclSend ─► ncclRecv ncclSend ─► ncclRecv │
│ (activations) (activations) │
│ │
│ Timeline (1F1B Schedule with 4 micro-batches): │
│ │
│ GPU0: [F0][F1][F2][F3][ ][B3][B2][B1][B0] │
│ GPU1: [ ][F0][F1][F2][F3][B3][B2][B1][B0] │
│ GPU2: [ ][ ][F0][F1][F2][F3][B3][B2][B1][B0] │
│ GPU3: [ ][ ][ ][F0][F1][F2][F3][B3][B2][B1][B0]│
│ ↑ ↑ │
│ Bubble Bubble │
│ │
└─────────────────────────────────────────────────────────────┘
NCCL Communication:
- Forward: ncclSend (stage N) → ncclRecv (stage N+1)
- Backward: ncclSend (stage N+1) → ncclRecv (stage N)
- Point-to-point, sequential dependency
Benchmark Results (Our Tests):
- Track C (PP=2): 9,665 tokens/sec - 70-75% bubble overhead
- Track E (PP=4): 8,343 tokens/sec - Higher bubble due to more stages
Pipeline Bubble Analysis:
Bubble Fraction = (Idle Time) / (Total Time)
For PP=4 with 4 micro-batches:
Bubble ≈ (PP_size - 1) / (PP_size + num_microbatches - 1)
Bubble ≈ (4 - 1) / (4 + 4 - 1) = 3/7 ≈ 43% (theoretical minimum)
Our measured: ~70-75% (includes communication overhead)
When to Use PP:
- Model too large for single GPU memory
- Acceptable latency (not real-time inference)
- Memory-constrained environments
Concept: Split individual layers (weight matrices) across GPUs.
┌─────────────────────────────────────────────────────────────┐
│ TENSOR PARALLELISM (TP=2) │
├─────────────────────────────────────────────────────────────┤
│ │
│ For a Linear Layer: Y = XW + b │
│ │
│ Column Parallel (Split W by columns): │
│ ┌────────────────────────────────────────────────────┐ │
│ │ X × [W0 | W1] = [Y0|Y1]│ │
│ │ (replicated) (split) (partial) │ │
│ │ │ │
│ │ GPU 0: Y0 = X × W0 │ │
│ │ GPU 1: Y1 = X × W1 │ │
│ │ ↓ │ │
│ │ ncclAllGather │ │
│ │ Y = [Y0, Y1] (complete) │ │
│ └────────────────────────────────────────────────────┘ │
│ │
│ Row Parallel (Split W by rows): │
│ ┌────────────────────────────────────────────────────┐ │
│ │ [X0] × W0 = Y0 │ │
│ │ [X1] W1 Y1 │ │
│ │ (split) (split) (partial) │ │
│ │ ↓ │ │
│ │ ncclAllReduce │ │
│ │ Y = Y0 + Y1 (complete) │ │
│ └────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
NCCL Communication:
- Within each transformer block: 2× ncclAllReduce
- High frequency, small messages
- Requires high-bandwidth interconnect (NVLink preferred)
When to Use TP:
- Individual layers too large for GPU memory
- High-bandwidth intra-node connection (NVLink)
- Combined with PP for very large models
Concept: Combine Pipeline and Tensor parallelism for optimal scaling.
┌─────────────────────────────────────────────────────────────┐
│ HYBRID PARALLELISM (PP=2 × TP=2) │
├─────────────────────────────────────────────────────────────┤
│ │
│ Pipeline Stage 0 Pipeline Stage 1 │
│ ┌─────────────────────┐ ┌─────────────────────┐ │
│ │ GPU0 ◄─AllReduce─► GPU1│ │ GPU2 ◄─AllReduce─► GPU3│
│ │ (TP=0) (TP) (TP=1)│ │ (TP=0) (TP) (TP=1)│
│ │ │ │ │ │
│ │ Layers 0-5 split │ │ Layers 6-11 split │ │
│ │ across TP dim │ │ across TP dim │ │
│ └──────────┬──────────┘ └──────────┬──────────┘ │
│ │ │ │
│ └────── Send/Recv ────────┘ │
│ (Pipeline Activations) │
│ │
│ NCCL Communication Pattern: │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ 1. Forward through Stage 0: │ │
│ │ - GPU0/GPU1: ncclAllReduce (TP sync) │ │
│ │ │ │
│ │ 2. Stage 0 → Stage 1: │ │
│ │ - GPU0 → GPU2: ncclSend (activations) │ │
│ │ - GPU1 → GPU3: ncclSend (activations) │ │
│ │ │ │
│ │ 3. Forward through Stage 1: │ │
│ │ - GPU2/GPU3: ncclAllReduce (TP sync) │ │
│ │ │ │
│ │ 4. Backward: Reverse pattern │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
Benchmark Results (Our Tests):
- Track D (PP=2×TP=2): 10,069 tokens/sec
- Communication: 45% AllReduce + 55% Send/Recv (from nsys profiling)
When to Use Hybrid:
- Very large models (100B+ parameters)
- Need to balance memory across many GPUs
- Production LLM training (GPT-3, LLaMA, etc.)
| Component | Our Setup | Minimum Recommended |
|---|---|---|
| GPU Nodes | 2× VM.GPU.A10.2 | 2+ nodes |
| GPUs/Node | 2× NVIDIA A10 (24GB) | 2+ GPUs |
| Total GPUs | 4 | 4+ |
| Interconnect | OCI VCN (Ethernet) | 10Gbps+ |
| Storage | OCI FSS (shared) | NFS/shared storage |
# Verified versions from our benchmark
PyTorch: 2.2.0
CUDA: 12.1
NCCL: 2.19.3
Python: 3.10
Container: pytorch/pytorch:2.2.0-cuda12.1-cudnn8-devel# Clone repository
git clone <repo-url>
cd training-parallelism
# Deploy to Kubernetes
kubectl apply -f k8s/namespace.yaml
kubectl apply -f k8s/configmap.yaml
kubectl apply -f k8s/service.yaml
kubectl apply -f k8s/workers.yaml
# Wait for pods
kubectl -n llm-benchmark wait --for=condition=Ready pod/worker-0 --timeout=300s
kubectl -n llm-benchmark wait --for=condition=Ready pod/worker-1 --timeout=300s# Track A: Single-Node DDP (2 GPU)
kubectl -n llm-benchmark exec worker-0 -- bash -c '
cd /mnt/fss/scripts
torchrun --nproc_per_node=2 train_ddp.py \
--max_steps 50 --batch_size 8 --experiment track_a
'
# Track B: Multi-Node DDP (4 GPU)
# Run on both nodes simultaneously
kubectl -n llm-benchmark exec worker-0 -- bash -c '
cd /mnt/fss/scripts
torchrun --nnodes=2 --nproc_per_node=2 --node_rank=0 \
--master_addr=worker-0.llm-benchmark-svc --master_port=29500 \
train_ddp.py --max_steps 50 --batch_size 8 --experiment track_b
' &
kubectl -n llm-benchmark exec worker-1 -- bash -c '
cd /mnt/fss/scripts
torchrun --nnodes=2 --nproc_per_node=2 --node_rank=1 \
--master_addr=worker-0.llm-benchmark-svc --master_port=29500 \
train_ddp.py --max_steps 50 --batch_size 8 --experiment track_b
'
# Track C: Pipeline PP=2
kubectl -n llm-benchmark exec worker-0 -- bash -c '
torchrun --nproc_per_node=2 /mnt/fss/scripts/train_pipeline.py \
--pipeline_parallel_size 2 --max_steps 50 --experiment track_c
'
# Track D: Hybrid PP=2×TP=2 (4 GPU)
kubectl -n llm-benchmark exec worker-0 -- bash -c '
torchrun --nproc_per_node=4 /mnt/fss/scripts/train_hybrid.py \
--pp_size 2 --tp_size 2 --max_steps 50 --experiment track_d
'
# Track E: Pipeline PP=4
kubectl -n llm-benchmark exec worker-0 -- bash -c '
torchrun --nproc_per_node=4 /mnt/fss/scripts/train_pipeline.py \
--pipeline_parallel_size 4 --max_steps 50 --experiment track_e
'# Check results
ls results/
# BENCHMARK_SUMMARY.md
# track_a_single_node_ddp/
# track_b_multi_node_ddp/
# track_c_pipeline_pp2/
# track_d_pp2_tp2/
# track_e_pp4/
# nsys_profiles/
# LLM_Training_Parallelism_Benchmark.png
# LLM_Training_Convergence.png
# LLM_Parallelism_Architecture.png| Shape | GPUs | GPU Memory | Interconnect | Best For |
|---|---|---|---|---|
| VM.GPU.A10.1 | 1× A10 | 24 GB | - | Development |
| VM.GPU.A10.2 | 2× A10 | 48 GB | PCIe | Small models |
| BM.GPU.A10.4 | 4× A10 | 96 GB | PCIe | Medium models |
| BM.GPU4.8 | 8× A100 | 320 GB | NVLink | Large models |
| BM.GPU.H100.8 | 8× H100 | 640 GB | NVLink + NVSwitch | Very large models |
# 1. Create OKE Cluster with GPU Node Pool
oci ce cluster create \
--compartment-id <compartment-ocid> \
--name llm-benchmark-cluster \
--kubernetes-version v1.28.2 \
--vcn-id <vcn-ocid> \
--service-lb-subnet-ids '["<subnet-ocid>"]'
# 2. Create GPU Node Pool
oci ce node-pool create \
--cluster-id <cluster-ocid> \
--compartment-id <compartment-ocid> \
--name gpu-workers \
--node-shape VM.GPU.A10.2 \
--size 2 \
--kubernetes-version v1.28.2 \
--node-image-id <gpu-image-ocid> \
--placement-configs '[{"availabilityDomain":"AD-1","subnetId":"<subnet-ocid>"}]'
# 3. Configure kubectl
oci ce cluster create-kubeconfig \
--cluster-id <cluster-ocid> \
--file ~/.kube/config \
--region <region>
# 4. Install NVIDIA Device Plugin
kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.14.0/nvidia-device-plugin.yml
# 5. Verify GPU availability
kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\\.com/gpu# Create FSS for shared storage across nodes
# 1. Create File System in OCI Console or CLI
oci fs file-system create \
--compartment-id <compartment-ocid> \
--availability-domain <AD> \
--display-name llm-training-fss
# 2. Create Mount Target
oci fs mount-target create \
--compartment-id <compartment-ocid> \
--availability-domain <AD> \
--subnet-id <subnet-ocid> \
--display-name llm-mount-target
# 3. Mount on worker nodes (in pod spec or node setup)
# Mount path: /mnt/coecommonfss/llmcore/training-parallelism# ═══════════════════════════════════════════════════════════════
# OCI NCCL SETTINGS
# ═══════════════════════════════════════════════════════════════
# Network Interface (check with: ip addr show)
export NCCL_SOCKET_IFNAME=ens3 # Primary NIC on OCI
# Disable InfiniBand for standard shapes
export NCCL_IB_DISABLE=1
# For RDMA-enabled shapes (BM.GPU4.8, BM.GPU.H100.8)
# export NCCL_IB_DISABLE=0
# export NCCL_IB_HCA=mlx5_0
# export NCCL_NET_GDR_LEVEL=2
# Debugging and logging
export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=INIT,COLL,P2P
export NCCL_DEBUG_FILE=/mnt/fss/logs/nccl_%h_%p.log
# Timeout for large models
export NCCL_TIMEOUT=1800
export TORCH_NCCL_ASYNC_ERROR_HANDLING=1
# Buffer size optimization
export NCCL_BUFFSIZE=16777216 # 16MB# 1. Use dedicated subnet for GPU nodes
# - Isolate training traffic
# - Configure security lists for NCCL ports (29500-29600)
# 2. Security List Rules for NCCL
# Ingress: Allow TCP/UDP 29500-29600 from GPU subnet CIDR
# Egress: Allow all to GPU subnet CIDR
# 3. Check connectivity between nodes
kubectl exec -n llm-benchmark worker-0 -- ping -c 3 worker-1.llm-benchmark-svc
# 4. Verify NCCL can establish connections
kubectl exec -n llm-benchmark worker-0 -- bash -c '
export NCCL_DEBUG=INFO
python -c "import torch.distributed as dist; dist.init_process_group(backend=\"nccl\")"
'# ═══════════════════════════════════════════════════════════════
# UNIVERSAL NCCL SETTINGS (Copy to all environments)
# ═══════════════════════════════════════════════════════════════
# Debugging (enable during benchmarks)
export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=INIT,COLL,P2P
export NCCL_DEBUG_FILE=/logs/nccl_%h_%p.log
# Timeout (increase for large models)
export NCCL_TIMEOUT=1800
export TORCH_NCCL_ASYNC_ERROR_HANDLING=1
# Algorithm selection
export NCCL_ALGO=Ring # Ring, Tree, CollnetDirect
export NCCL_PROTO=Simple # Simple, LL, LL128
# Buffer size (default 4MB, increase for large tensors)
export NCCL_BUFFSIZE=16777216 # 16MB
# ═══════════════════════════════════════════════════════════════
# NETWORK-SPECIFIC SETTINGS
# ═══════════════════════════════════════════════════════════════
# For Ethernet (most cloud providers)
export NCCL_SOCKET_IFNAME=eth0 # or ens3, ens5
export NCCL_IB_DISABLE=1
# For InfiniBand/RoCE
export NCCL_IB_DISABLE=0
export NCCL_IB_HCA=mlx5_0
export NCCL_IB_GID_INDEX=3
export NCCL_NET_GDR_LEVEL=2
# For NVLink (intra-node)
export NCCL_P2P_LEVEL=NVL
export NCCL_SHM_DISABLE=0| Component | Minimum | Recommended |
|---|---|---|
| GPU Servers | 2 nodes | 4+ nodes |
| GPUs per Node | 2× NVIDIA (16GB+) | 8× A100/H100 |
| Network | 10GbE Ethernet | 100GbE or InfiniBand |
| Storage | NFS shared mount | High-speed NFS or Lustre |
| RAM | 128 GB | 256+ GB |
# ═══════════════════════════════════════════════════════════════
# ON-PREMISES MULTI-NODE TRAINING SETUP
# ═══════════════════════════════════════════════════════════════
# 1. Set up shared storage (NFS example)
# On NFS server:
sudo mkdir -p /export/llm-training
sudo chown -R nobody:nogroup /export/llm-training
echo "/export/llm-training *(rw,sync,no_subtree_check)" | sudo tee -a /etc/exports
sudo exportfs -ra
# On all GPU nodes:
sudo mkdir -p /mnt/shared
sudo mount -t nfs nfs-server:/export/llm-training /mnt/shared
# 2. Copy training scripts to shared storage
scp -r scripts/ /mnt/shared/
scp -r configs/ /mnt/shared/
# 3. Set environment variables on ALL nodes
export MASTER_ADDR=192.168.1.100 # IP of master node
export MASTER_PORT=29500
export WORLD_SIZE=4 # Total GPUs across all nodes
export NCCL_SOCKET_IFNAME=eth0 # Network interface
export NCCL_DEBUG=INFO
# 4. Run training on each node
# Node 0 (master):
export NODE_RANK=0
torchrun --nnodes=2 --nproc_per_node=2 \
--node_rank=$NODE_RANK \
--master_addr=$MASTER_ADDR \
--master_port=$MASTER_PORT \
/mnt/shared/scripts/train_ddp.py --max_steps 50
# Node 1:
export NODE_RANK=1
torchrun --nnodes=2 --nproc_per_node=2 \
--node_rank=$NODE_RANK \
--master_addr=$MASTER_ADDR \
--master_port=$MASTER_PORT \
/mnt/shared/scripts/train_ddp.py --max_steps 50# ═══════════════════════════════════════════════════════════════
# ON-PREMISES NCCL SETTINGS
# ═══════════════════════════════════════════════════════════════
# For Ethernet networks
export NCCL_SOCKET_IFNAME=eth0 # Check: ip addr show
export NCCL_IB_DISABLE=1
# For InfiniBand networks
export NCCL_IB_DISABLE=0
export NCCL_IB_HCA=mlx5_0 # Check: ibstat
export NCCL_IB_GID_INDEX=3
export NCCL_NET_GDR_LEVEL=2 # GPUDirect RDMA level
# For NVLink-connected GPUs (intra-node)
export NCCL_P2P_LEVEL=NVL
export NCCL_SHM_DISABLE=0
# Debugging and logging
export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=INIT,COLL,P2P
export NCCL_DEBUG_FILE=/mnt/shared/logs/nccl_%h_%p.log
# Timeout for large models
export NCCL_TIMEOUT=1800
export TORCH_NCCL_ASYNC_ERROR_HANDLING=1# ═══════════════════════════════════════════════════════════════
# FIREWALL CONFIGURATION
# ═══════════════════════════════════════════════════════════════
# Open ports for NCCL and PyTorch distributed
sudo firewall-cmd --permanent --add-port=29500-29600/tcp
sudo firewall-cmd --permanent --add-port=29500-29600/udp
sudo firewall-cmd --reload
# Or using iptables
sudo iptables -A INPUT -p tcp --dport 29500:29600 -j ACCEPT
sudo iptables -A INPUT -p udp --dport 29500:29600 -j ACCEPT
# ═══════════════════════════════════════════════════════════════
# VERIFY CONNECTIVITY
# ═══════════════════════════════════════════════════════════════
# Test network connectivity
ping -c 3 <other-node-ip>
# Test NCCL port
nc -zv <other-node-ip> 29500
# Test NCCL initialization
python -c "
import os
import torch.distributed as dist
os.environ['MASTER_ADDR'] = '192.168.1.100'
os.environ['MASTER_PORT'] = '29500'
os.environ['RANK'] = '0'
os.environ['WORLD_SIZE'] = '1'
dist.init_process_group(backend='nccl')
print('NCCL initialized successfully')
dist.destroy_process_group()
"# ═══════════════════════════════════════════════════════════════
# TRACK A: Single-Node DDP (2 GPU)
# ═══════════════════════════════════════════════════════════════
torchrun --nproc_per_node=2 \
/mnt/shared/scripts/train_ddp.py \
--max_steps 50 --batch_size 8 --experiment track_a
# ═══════════════════════════════════════════════════════════════
# TRACK B: Multi-Node DDP (4 GPU across 2 nodes)
# ═══════════════════════════════════════════════════════════════
# Run on Node 0:
torchrun --nnodes=2 --nproc_per_node=2 --node_rank=0 \
--master_addr=$MASTER_ADDR --master_port=29500 \
/mnt/shared/scripts/train_ddp.py --max_steps 50 --experiment track_b
# Run on Node 1 (simultaneously):
torchrun --nnodes=2 --nproc_per_node=2 --node_rank=1 \
--master_addr=$MASTER_ADDR --master_port=29500 \
/mnt/shared/scripts/train_ddp.py --max_steps 50 --experiment track_b
# ═══════════════════════════════════════════════════════════════
# TRACK C: Pipeline Parallelism (PP=2)
# ═══════════════════════════════════════════════════════════════
torchrun --nproc_per_node=2 \
/mnt/shared/scripts/train_pipeline.py \
--pipeline_parallel_size 2 --max_steps 50 --experiment track_c
# ═══════════════════════════════════════════════════════════════
# TRACK D: Hybrid PP=2 × TP=2 (requires 4 GPUs on single node)
# ═══════════════════════════════════════════════════════════════
torchrun --nproc_per_node=4 \
/mnt/shared/scripts/train_hybrid.py \
--pp_size 2 --tp_size 2 --max_steps 50 --experiment track_d
# ═══════════════════════════════════════════════════════════════
# TRACK E: Full Pipeline (PP=4)
# ═══════════════════════════════════════════════════════════════
torchrun --nproc_per_node=4 \
/mnt/shared/scripts/train_pipeline.py \
--pipeline_parallel_size 4 --max_steps 50 --experiment track_e# Install nsys if not available
# Download from NVIDIA: https://developer.nvidia.com/nsight-systems
# Profile training run
nsys profile -o /mnt/shared/nsys_profiles/track_a_ddp \
--trace=cuda,nvtx,osrt,cudnn,cublas \
--cuda-memory-usage=true \
torchrun --nproc_per_node=2 /mnt/shared/scripts/train_ddp.py --max_steps 20
# Generate statistics
nsys stats /mnt/shared/nsys_profiles/track_a_ddp.nsys-rep \
--report cuda_gpu_kern_sum
# Open in GUI
nsys-ui /mnt/shared/nsys_profiles/track_a_ddp.nsys-repConfiguration:
- GPUs: 2× NVIDIA A10
- Model: GPT-style, 162M parameters
- Batch size: 8, Sequence length: 128
Results:
- Throughput: 49,290 tokens/sec
- Memory: 0.74 GB/GPU
- NCCL: 100% ncclAllReduce
NCCL Profile (from nsys):
ncclAllReduce operations: Gradient synchronization
Ring algorithm used for optimal bandwidth
Configuration:
- GPUs: 4× NVIDIA A10 (2 nodes × 2 GPUs)
- Model: GPT-style, 162M parameters
Results:
- Throughput: 52,847 tokens/sec
- Scaling efficiency: 54% (vs ideal 100%)
- Network overhead: ~7% (inter-node AllReduce)
NCCL Profile:
Intra-node: PCIe (GPU0 ↔ GPU1)
Inter-node: Ethernet (Node A ↔ Node B)
Configuration:
- GPUs: 2× NVIDIA A10
- Pipeline stages: 2
- Micro-batch size: 8
Results:
- Throughput: 9,665 tokens/sec
- Bubble fraction: ~70%
- Memory: ~0.57 GB/GPU (split model)
NCCL Profile:
ncclSend/ncclRecv: Activation transfer between stages
Sequential dependency creates pipeline bubbles
Configuration:
- GPUs: 4× NVIDIA A10
- Pipeline stages: 2, Tensor parallel: 2
Results:
- Throughput: 10,069 tokens/sec
- Communication split: 45% AllReduce, 55% Send/Recv
NCCL Profile (from nsys):
ncclDevKernel_SendRecv: 44.8% of GPU time
ncclDevKernel_AllReduce: 10.2% of GPU time
ampere_sgemm kernels: Matrix operations
Configuration:
- GPUs: 4× NVIDIA A10
- Pipeline stages: 4 (1 GPU per stage)
Results:
- Throughput: 8,343 tokens/sec
- Bubble fraction: ~75%
- Memory: ~2.3 GB/GPU (minimal per stage)
NCCL Profile:
Chain of ncclSend/ncclRecv operations
Maximum memory efficiency, lowest throughput
| Profile | Size | Key Observations |
|---|---|---|
track_a_ddp.nsys-rep |
8.5 MB | ncclAllReduce dominates |
track_b_ddp.nsys-rep |
8.8 MB | Inter-node AllReduce visible |
track_c_pipeline.nsys-rep |
8.4 MB | ncclSend/Recv pattern |
track_d_hybrid_pp2tp1.nsys-rep |
15 MB | Mixed AllReduce + Send/Recv |
track_e_pipeline_pp2.nsys-rep |
30 MB | Sequential Send/Recv chain |
# Open in Nsight Systems GUI
nsys-ui results/nsys_profiles/track_d_hybrid_pp2tp1.nsys-rep
# Generate statistics
nsys stats results/nsys_profiles/track_d_hybrid_pp2tp1.nsys-rep \
--report cuda_gpu_kern_sum
# Look for:
# 1. NCCL kernels: ncclDevKernel_SendRecv, ncclDevKernel_AllReduce
# 2. Compute kernels: ampere_sgemm, cutlass_sgemm
# 3. Memory operations: cudaMemcpyTrack D (Hybrid) GPU Kernel Summary:
ncclDevKernel_SendRecv: 44.8% of GPU time (1.2B ns)
ncclDevKernel_AllReduce_Sum: 10.2% of GPU time (275M ns)
ampere_sgemm_128x64_tn: 6.9% of GPU time (185M ns)
Track E (Pipeline) GPU Kernel Summary:
ampere_sgemm_128x64_tn: 17.7% of GPU time
ncclDevKernel_SendRecv: 17.6% of GPU time
cutlass_sgemm_256x128: 11.6% of GPU time
# Error: NCCL timeout after 1800000ms
# Solution 1: Increase timeout
export NCCL_TIMEOUT=3600
# Solution 2: Check network
ping <other-node-ip>
nc -zv <other-node-ip> 29500
# Solution 3: Verify interface
export NCCL_DEBUG=INFO
export NCCL_SOCKET_IFNAME=eth0# Error: CUDA out of memory
# Solution: Reduce batch size or increase pipeline stages
--micro_batch_size 4
--pipeline_parallel_size 4
--gradient_checkpointing# Symptom: Training hangs at send/recv
# Solution: Use async operations
# In code:
send_op = dist.isend(tensor, dst_rank)
recv_op = dist.irecv(buffer, src_rank)
send_op.wait()
recv_op.wait()# Check NCCL algorithm
export NCCL_DEBUG=INFO
# Look for: "Ring" or "Tree" in logs
# Try different algorithms
export NCCL_ALGO=Tree # Better for latency
export NCCL_ALGO=Ring # Better for bandwidth| Scenario | Recommended Strategy | Reason |
|---|---|---|
| Model fits in 1 GPU | DDP | Maximum throughput |
| Model fits in node (multi-GPU) | DDP + TP | Balance memory/throughput |
| Model > node memory | PP or Hybrid | Split across nodes |
| Very large model (100B+) | Hybrid (PP×TP×DP) | Full utilization |
| Memory-constrained | PP (more stages) | Minimize per-GPU memory |
| Latency-sensitive | DDP | No pipeline bubbles |
| Strategy | Primary NCCL Op | Secondary NCCL Op | Communication % |
|---|---|---|---|
| DDP | AllReduce | - | 5-15% |
| PP | Send/Recv | - | 20-40% |
| TP | AllReduce | AllGather | 10-30% |
| Hybrid | AllReduce (TP) | Send/Recv (PP) | 30-50% |
- OCI GPU Instances
- OCI Container Engine for Kubernetes (OKE)
- OCI File Storage Service
- OCI GPU Scheduling with Kubernetes
- PyTorch Distributed Documentation
- NCCL Documentation
- Megatron-LM Paper - Pipeline and Tensor Parallelism
- GPipe Paper - Pipeline Parallelism
- Nsight Systems User Guide
- ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
- Efficient Large-Scale Language Model Training on GPU Clusters
Apache 2.0
Infrastructure: Oracle Cloud Infrastructure (OCI) Last Updated: January 2026