From 4e05434898d366f3d8d5e5c9ee9d3af6c65eb026 Mon Sep 17 00:00:00 2001 From: dzmitry-huba <123517587+dzmitry-huba@users.noreply.github.com> Date: Wed, 29 Apr 2026 14:42:14 -0700 Subject: [PATCH 1/2] Expand white paper with per-subsection engine analyses --- papers/parallelisms_whitepaper.md | 3316 +++++++++++++++++++++++++++++ 1 file changed, 3316 insertions(+) create mode 100644 papers/parallelisms_whitepaper.md diff --git a/papers/parallelisms_whitepaper.md b/papers/parallelisms_whitepaper.md new file mode 100644 index 0000000..b4eeb34 --- /dev/null +++ b/papers/parallelisms_whitepaper.md @@ -0,0 +1,3316 @@ +# Parallelisms White Paper (Expanded, Multi-Engine Draft) + +## Authoring protocol +For each subsection, this paper uses an independent "engine" block to emulate separate specialist analysis. All engines share a common notation contract, then produce self-contained explanations. + +## Global notation and assumptions +- Model parameters: $|\theta|$. Hidden size: $H$. Layers: $L$. Global batch: $B$. Sequence length: $S$. +- Mesh axes: $P_{dp},P_{tp},P_{pp},P_{cp},P_{ep}$ and $P=\prod P_i$. +- Attention heads $N_h$, head dim $d_h$, so $H=N_h d_h$. +- Collectives: $\operatorname{AR}$, $\operatorname{AG}$, $\operatorname{RS}$, $\operatorname{A2A}$. +- Communication cost uses $T=\alpha\cdot f(p)+\beta\cdot n\cdot g(p)$ with topology-aware $f,g$. + +### Common equations used throughout +$$M_{rank}=M_{params}+M_{grads}+M_{opt}+M_{acts}+M_{kv}+M_{tmp}.$$ +$$\eta_{pipe}\approx 1-\frac{P_{pp}-1}{m+P_{pp}-1},\quad\eta_{bubble}=1-\eta_{pipe}.$$ +$$\text{step\_time}\approx\max(T_{compute},T_{comm}-T_{overlap})+T_{sync}. $$ + +## Abstract +**Engine 1: scope and objective.** Provide a self-contained treatment of **Abstract** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - Abstract is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +## How To Read This Paper +**Engine 2: scope and objective.** Provide a self-contained treatment of **How To Read This Paper** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - How To Read This Paper is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +## 1. Shared System Model, Notation, Placement, and Evidence Policy +**Engine 3: scope and objective.** Provide a self-contained treatment of **1. Shared System Model, Notation, Placement, and Evidence Policy** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 1. Shared System Model, Notation, Placement, and Evidence Policy is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 1.1 Evidence Grades and Source Interpretation +**Engine 4: scope and objective.** Provide a self-contained treatment of **1.1 Evidence Grades and Source Interpretation** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 1.1 Evidence Grades and Source Interpretation is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 1.2 Shared Symbols and Transformer Shapes +**Engine 5: scope and objective.** Provide a self-contained treatment of **1.2 Shared Symbols and Transformer Shapes** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 1.2 Shared Symbols and Transformer Shapes is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 1.3 Device Meshes, Rank Coordinates, and Process Groups +**Engine 6: scope and objective.** Provide a self-contained treatment of **1.3 Device Meshes, Rank Coordinates, and Process Groups** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 1.3 Device Meshes, Rank Coordinates, and Process Groups is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 1.4 Placement States and Tensor Layout Transitions +**Engine 7: scope and objective.** Provide a self-contained treatment of **1.4 Placement States and Tensor Layout Transitions** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 1.4 Placement States and Tensor Layout Transitions is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 1.5 Communication Primitives and Cost Models +**Engine 8: scope and objective.** Provide a self-contained treatment of **1.5 Communication Primitives and Cost Models** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 1.5 Communication Primitives and Cost Models is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 1.6 Memory, Bandwidth, and Overlap Accounting +**Engine 9: scope and objective.** Provide a self-contained treatment of **1.6 Memory, Bandwidth, and Overlap Accounting** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 1.6 Memory, Bandwidth, and Overlap Accounting is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 1.7 How Later Sections Use This Shared Layer +**Engine 10: scope and objective.** Provide a self-contained treatment of **1.7 How Later Sections Use This Shared Layer** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 1.7 How Later Sections Use This Shared Layer is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +## 2. Data Parallelism, FSDP, ZeRO, HSDP, and State Sharding +**Engine 11: scope and objective.** Provide a self-contained treatment of **2. Data Parallelism, FSDP, ZeRO, HSDP, and State Sharding** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 2. Data Parallelism, FSDP, ZeRO, HSDP, and State Sharding is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 2.1 DDP: Replicated State, Batch Shards, and Gradient All-Reduce +**Engine 12: scope and objective.** Provide a self-contained treatment of **2.1 DDP: Replicated State, Batch Shards, and Gradient All-Reduce** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 2.1 DDP: Replicated State, Batch Shards, and Gradient All-Reduce is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 2.2 The Primitive Mechanisms Behind Sharded Data Parallelism +**Engine 13: scope and objective.** Provide a self-contained treatment of **2.2 The Primitive Mechanisms Behind Sharded Data Parallelism** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 2.2 The Primitive Mechanisms Behind Sharded Data Parallelism is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 2.3 ZeRO, ZeRO++, and Distributed Optimizers +**Engine 14: scope and objective.** Provide a self-contained treatment of **2.3 ZeRO, ZeRO++, and Distributed Optimizers** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 2.3 ZeRO, ZeRO++, and Distributed Optimizers is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 2.4 FSDP and FSDP2 Execution +**Engine 15: scope and objective.** Provide a self-contained treatment of **2.4 FSDP and FSDP2 Execution** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 2.4 FSDP and FSDP2 Execution is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 2.5 HSDP: Hybrid Sharded Data Parallelism +**Engine 16: scope and objective.** Provide a self-contained treatment of **2.5 HSDP: Hybrid Sharded Data Parallelism** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 2.5 HSDP: Hybrid Sharded Data Parallelism is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 2.6 Offload: CPU and NVMe as Additional State Tiers +**Engine 17: scope and objective.** Provide a self-contained treatment of **2.6 Offload: CPU and NVMe as Additional State Tiers** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 2.6 Offload: CPU and NVMe as Additional State Tiers is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 2.7 Low Precision, Quantized Communication, and Platform Support +**Engine 18: scope and objective.** Provide a self-contained treatment of **2.7 Low Precision, Quantized Communication, and Platform Support** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 2.7 Low Precision, Quantized Communication, and Platform Support is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 2.8 Choosing Among DDP, FSDP2, ZeRO, ZeRO++, HSDP, and Offload +**Engine 19: scope and objective.** Provide a self-contained treatment of **2.8 Choosing Among DDP, FSDP2, ZeRO, ZeRO++, HSDP, and Offload** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 2.8 Choosing Among DDP, FSDP2, ZeRO, ZeRO++, HSDP, and Offload is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 2.9 Sources and Lineage +**Engine 20: scope and objective.** Provide a self-contained treatment of **2.9 Sources and Lineage** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 2.9 Sources and Lineage is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 2.10 Caveats +**Engine 21: scope and objective.** Provide a self-contained treatment of **2.10 Caveats** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 2.10 Caveats is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +## 3. Tensor Parallelism, Sequence-Parallel Activations, and Vocabulary Parallelism +**Engine 22: scope and objective.** Provide a self-contained treatment of **3. Tensor Parallelism, Sequence-Parallel Activations, and Vocabulary Parallelism** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 3. Tensor Parallelism, Sequence-Parallel Activations, and Vocabulary Parallelism is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 3.1 TP-Specific Notation and Layout Contract +**Engine 23: scope and objective.** Provide a self-contained treatment of **3.1 TP-Specific Notation and Layout Contract** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 3.1 TP-Specific Notation and Layout Contract is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 3.2 Column-Parallel Linear Layers +**Engine 24: scope and objective.** Provide a self-contained treatment of **3.2 Column-Parallel Linear Layers** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 3.2 Column-Parallel Linear Layers is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 3.3 Row-Parallel Linear Layers +**Engine 25: scope and objective.** Provide a self-contained treatment of **3.3 Row-Parallel Linear Layers** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 3.3 Row-Parallel Linear Layers is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 3.4 Transformer Block Recipes: MLP and Attention +**Engine 26: scope and objective.** Provide a self-contained treatment of **3.4 Transformer Block Recipes: MLP and Attention** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 3.4 Transformer Block Recipes: MLP and Attention is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +#### MLP / SwiGLU +**Engine 27: scope and objective.** Provide a self-contained treatment of **MLP / SwiGLU** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - MLP / SwiGLU is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +#### Attention +**Engine 28: scope and objective.** Provide a self-contained treatment of **Attention** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - Attention is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 3.5 Sequence-Parallel Activations as a TP Companion +**Engine 29: scope and objective.** Provide a self-contained treatment of **3.5 Sequence-Parallel Activations as a TP Companion** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 3.5 Sequence-Parallel Activations as a TP Companion is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 3.6 Vocabulary Parallelism +**Engine 30: scope and objective.** Provide a self-contained treatment of **3.6 Vocabulary Parallelism** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 3.6 Vocabulary Parallelism is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +#### Input Embedding +**Engine 31: scope and objective.** Provide a self-contained treatment of **Input Embedding** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - Input Embedding is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +#### LM Head and Cross-Entropy +**Engine 32: scope and objective.** Provide a self-contained treatment of **LM Head and Cross-Entropy** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - LM Head and Cross-Entropy is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 3.7 Async Tensor Parallelism +**Engine 33: scope and objective.** Provide a self-contained treatment of **3.7 Async Tensor Parallelism** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 3.7 Async Tensor Parallelism is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 3.8 TP-Aware CUDA and Kernel Techniques +**Engine 34: scope and objective.** Provide a self-contained treatment of **3.8 TP-Aware CUDA and Kernel Techniques** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 3.8 TP-Aware CUDA and Kernel Techniques is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +#### 3.8.1 Local GEMM Shape Quality +**Engine 35: scope and objective.** Provide a self-contained treatment of **3.8.1 Local GEMM Shape Quality** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 3.8.1 Local GEMM Shape Quality is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +#### 3.8.2 Fused Epilogues and Layout-Stable Collectives +**Engine 36: scope and objective.** Provide a self-contained treatment of **3.8.2 Fused Epilogues and Layout-Stable Collectives** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 3.8.2 Fused Epilogues and Layout-Stable Collectives is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +#### 3.8.3 Low-Precision Communication and Scale Layout +**Engine 37: scope and objective.** Provide a self-contained treatment of **3.8.3 Low-Precision Communication and Scale Layout** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 3.8.3 Low-Precision Communication and Scale Layout is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +#### 3.8.4 TP-Aware Megakernels +**Engine 38: scope and objective.** Provide a self-contained treatment of **3.8.4 TP-Aware Megakernels** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 3.8.4 TP-Aware Megakernels is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 3.9 Choosing a TP Degree +**Engine 39: scope and objective.** Provide a self-contained treatment of **3.9 Choosing a TP Degree** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 3.9 Choosing a TP Degree is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 3.10 Sources and Lineage +**Engine 40: scope and objective.** Provide a self-contained treatment of **3.10 Sources and Lineage** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 3.10 Sources and Lineage is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 3.11 Caveats +**Engine 41: scope and objective.** Provide a self-contained treatment of **3.11 Caveats** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 3.11 Caveats is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +## 4. Pipeline Parallelism: Depth-Axis Parallelism and Modern Schedules +**Engine 42: scope and objective.** Provide a self-contained treatment of **4. Pipeline Parallelism: Depth-Axis Parallelism and Modern Schedules** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 4. Pipeline Parallelism: Depth-Axis Parallelism and Modern Schedules is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 4.1 System Model, Variables, and Shapes +**Engine 43: scope and objective.** Provide a self-contained treatment of **4.1 System Model, Variables, and Shapes** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 4.1 System Model, Variables, and Shapes is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 4.2 Primitive: Stage Partitioning and Boundary Contracts +**Engine 44: scope and objective.** Provide a self-contained treatment of **4.2 Primitive: Stage Partitioning and Boundary Contracts** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 4.2 Primitive: Stage Partitioning and Boundary Contracts is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 4.3 Primitive: Microbatching, Warmup, Steady State, and Drain +**Engine 45: scope and objective.** Provide a self-contained treatment of **4.3 Primitive: Microbatching, Warmup, Steady State, and Drain** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 4.3 Primitive: Microbatching, Warmup, Steady State, and Drain is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 4.4 Primitive: Point-to-Point Send/Recv and Collective Composition +**Engine 46: scope and objective.** Provide a self-contained treatment of **4.4 Primitive: Point-to-Point Send/Recv and Collective Composition** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 4.4 Primitive: Point-to-Point Send/Recv and Collective Composition is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 4.5 Primitive: The Schedule DAG and the F/B/W Split +**Engine 47: scope and objective.** Provide a self-contained treatment of **4.5 Primitive: The Schedule DAG and the F/B/W Split** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 4.5 Primitive: The Schedule DAG and the F/B/W Split is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 4.6 Zero Bubble Pipeline Parallelism +**Engine 48: scope and objective.** Provide a self-contained treatment of **4.6 Zero Bubble Pipeline Parallelism** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 4.6 Zero Bubble Pipeline Parallelism is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 4.7 Controllable-Memory Pipeline Schedules +**Engine 49: scope and objective.** Provide a self-contained treatment of **4.7 Controllable-Memory Pipeline Schedules** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 4.7 Controllable-Memory Pipeline Schedules is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 4.8 PipeOffload: Activation Offload and Prefetch as Schedule Primitives +**Engine 50: scope and objective.** Provide a self-contained treatment of **4.8 PipeOffload: Activation Offload and Prefetch as Schedule Primitives** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 4.8 PipeOffload: Activation Offload and Prefetch as Schedule Primitives is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 4.9 Vocabulary-Balanced Pipeline Parallelism +**Engine 51: scope and objective.** Provide a self-contained treatment of **4.9 Vocabulary-Balanced Pipeline Parallelism** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 4.9 Vocabulary-Balanced Pipeline Parallelism is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 4.10 DualPipe and DualPipeV: Pipeline Scheduling for MoE Communication +**Engine 52: scope and objective.** Provide a self-contained treatment of **4.10 DualPipe and DualPipeV: Pipeline Scheduling for MoE Communication** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 4.10 DualPipe and DualPipeV: Pipeline Scheduling for MoE Communication is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 4.11 PyTorch Pipelining and TorchTitan Runtime Support +**Engine 53: scope and objective.** Provide a self-contained treatment of **4.11 PyTorch Pipelining and TorchTitan Runtime Support** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 4.11 PyTorch Pipelining and TorchTitan Runtime Support is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 4.12 Optimizer Barriers, Gradient Synchronization, and Correctness +**Engine 54: scope and objective.** Provide a self-contained treatment of **4.12 Optimizer Barriers, Gradient Synchronization, and Correctness** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 4.12 Optimizer Barriers, Gradient Synchronization, and Correctness is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 4.13 Composing PP With Tensor, Context, Expert, and Data Parallelism +**Engine 55: scope and objective.** Provide a self-contained treatment of **4.13 Composing PP With Tensor, Context, Expert, and Data Parallelism** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 4.13 Composing PP With Tensor, Context, Expert, and Data Parallelism is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 4.14 Choosing a Modern PP Schedule +**Engine 56: scope and objective.** Provide a self-contained treatment of **4.14 Choosing a Modern PP Schedule** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 4.14 Choosing a Modern PP Schedule is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 4.15 Lineage: Older PP Work as Context +**Engine 57: scope and objective.** Provide a self-contained treatment of **4.15 Lineage: Older PP Work as Context** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 4.15 Lineage: Older PP Work as Context is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 4.16 Practical Caveats +**Engine 58: scope and objective.** Provide a self-contained treatment of **4.16 Practical Caveats** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 4.16 Practical Caveats is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +## 5. Sequence Parallelism and Context Parallelism +**Engine 59: scope and objective.** Provide a self-contained treatment of **5. Sequence Parallelism and Context Parallelism** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 5. Sequence Parallelism and Context Parallelism is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 5.1 Notation, Tensor Shapes, and the Attention Operator +**Engine 60: scope and objective.** Provide a self-contained treatment of **5.1 Notation, Tensor Shapes, and the Attention Operator** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 5.1 Notation, Tensor Shapes, and the Attention Operator is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 5.2 The Primitive Operations and Their Collectives +**Engine 61: scope and objective.** Provide a self-contained treatment of **5.2 The Primitive Operations and Their Collectives** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 5.2 The Primitive Operations and Their Collectives is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 5.3 Megatron Sequence Parallelism: Activation Sharding, Not Full Long-Context Attention +**Engine 62: scope and objective.** Provide a self-contained treatment of **5.3 Megatron Sequence Parallelism: Activation Sharding, Not Full Long-Context Attention** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 5.3 Megatron Sequence Parallelism: Activation Sharding, Not Full Long-Context Attention is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 5.4 Ulysses: Sequence Shards Become Head Shards Through All-to-All +**Engine 63: scope and objective.** Provide a self-contained treatment of **5.4 Ulysses: Sequence Shards Become Head Shards Through All-to-All** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 5.4 Ulysses: Sequence Shards Become Head Shards Through All-to-All is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 5.5 Ring Attention: KV Streaming With Point-to-Point Send/Recv +**Engine 64: scope and objective.** Provide a self-contained treatment of **5.5 Ring Attention: KV Streaming With Point-to-Point Send/Recv** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 5.5 Ring Attention: KV Streaming With Point-to-Point Send/Recv is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 5.6 The Numerical Primitive: Blockwise Online Softmax +**Engine 65: scope and objective.** Provide a self-contained treatment of **5.6 The Numerical Primitive: Blockwise Online Softmax** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 5.6 The Numerical Primitive: Blockwise Online Softmax is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 5.7 Megatron Context Parallelism: Sequence Sharding for the Whole Network +**Engine 66: scope and objective.** Provide a self-contained treatment of **5.7 Megatron Context Parallelism: Sequence Sharding for the Whole Network** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 5.7 Megatron Context Parallelism: Sequence Sharding for the Whole Network is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 5.8 Dynamic Context Parallelism: Choose CP Size Per Microbatch +**Engine 67: scope and objective.** Provide a self-contained treatment of **5.8 Dynamic Context Parallelism: Choose CP Size Per Microbatch** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 5.8 Dynamic Context Parallelism: Choose CP Size Per Microbatch is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 5.9 THD and Ragged Packing: Token-Head-Dimension Layout +**Engine 68: scope and objective.** Provide a self-contained treatment of **5.9 THD and Ragged Packing: Token-Head-Dimension Layout** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 5.9 THD and Ragged Packing: Token-Head-Dimension Layout is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 5.10 Sparse and Hybrid Attention: Communicate Selected KV, Not Necessarily All KV +**Engine 69: scope and objective.** Provide a self-contained treatment of **5.10 Sparse and Hybrid Attention: Communicate Selected KV, Not Necessarily All KV** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 5.10 Sparse and Hybrid Attention: Communicate Selected KV, Not Necessarily All KV is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 5.11 Inference KV and Context Placement +**Engine 70: scope and objective.** Provide a self-contained treatment of **5.11 Inference KV and Context Placement** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 5.11 Inference KV and Context Placement is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 5.12 How to Choose Among SP, Ulysses, Ring, CP, Dynamic-CP, and Sparse Context Placement +**Engine 71: scope and objective.** Provide a self-contained treatment of **5.12 How to Choose Among SP, Ulysses, Ring, CP, Dynamic-CP, and Sparse Context Placement** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 5.12 How to Choose Among SP, Ulysses, Ring, CP, Dynamic-CP, and Sparse Context Placement is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 5.13 Evidence Boundaries and Terminology Caveats +**Engine 72: scope and objective.** Provide a self-contained treatment of **5.13 Evidence Boundaries and Terminology Caveats** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 5.13 Evidence Boundaries and Terminology Caveats is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +## 6. Expert and MoE Parallelism +**Engine 73: scope and objective.** Provide a self-contained treatment of **6. Expert and MoE Parallelism** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 6. Expert and MoE Parallelism is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 6.1 Symbols, Shapes, and the MoE Layer +**Engine 74: scope and objective.** Provide a self-contained treatment of **6.1 Symbols, Shapes, and the MoE Layer** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 6.1 Symbols, Shapes, and the MoE Layer is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 6.2 Primitive Technique: Routing and Top-k Assignment +**Engine 75: scope and objective.** Provide a self-contained treatment of **6.2 Primitive Technique: Routing and Top-k Assignment** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 6.2 Primitive Technique: Routing and Top-k Assignment is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 6.3 Primitive Technique: Expert Ownership and Local Shard Shapes +**Engine 76: scope and objective.** Provide a self-contained treatment of **6.3 Primitive Technique: Expert Ownership and Local Shard Shapes** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 6.3 Primitive Technique: Expert Ownership and Local Shard Shapes is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 6.4 Primitive Technique: Permutation, Packing, and Inverse Permutation +**Engine 77: scope and objective.** Provide a self-contained treatment of **6.4 Primitive Technique: Permutation, Packing, and Inverse Permutation** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 6.4 Primitive Technique: Permutation, Packing, and Inverse Permutation is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 6.5 Primitive Technique: All-to-All Dispatch, Notify, and Combine +**Engine 78: scope and objective.** Provide a self-contained treatment of **6.5 Primitive Technique: All-to-All Dispatch, Notify, and Combine** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 6.5 Primitive Technique: All-to-All Dispatch, Notify, and Combine is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 6.6 Primitive Technique: Capacity, Dropless Execution, and Padding +**Engine 79: scope and objective.** Provide a self-contained treatment of **6.6 Primitive Technique: Capacity, Dropless Execution, and Padding** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 6.6 Primitive Technique: Capacity, Dropless Execution, and Padding is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 6.7 Primitive Technique: Grouped GEMM, MegaBlocks, DeepGEMM, and Mega MoE +**Engine 80: scope and objective.** Provide a self-contained treatment of **6.7 Primitive Technique: Grouped GEMM, MegaBlocks, DeepGEMM, and Mega MoE** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 6.7 Primitive Technique: Grouped GEMM, MegaBlocks, DeepGEMM, and Mega MoE is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 6.8 Primitive Technique: Backward Pass, Weight Gradients, and Overlap +**Engine 81: scope and objective.** Provide a self-contained treatment of **6.8 Primitive Technique: Backward Pass, Weight Gradients, and Overlap** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 6.8 Primitive Technique: Backward Pass, Weight Gradients, and Overlap is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 6.9 Primitive Technique: DeepEP Modes and Network-Aware EP +**Engine 82: scope and objective.** Provide a self-contained treatment of **6.9 Primitive Technique: DeepEP Modes and Network-Aware EP** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 6.9 Primitive Technique: DeepEP Modes and Network-Aware EP is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 6.10 Primitive Technique: Wide-EP Serving and Hot-Expert Replication +**Engine 83: scope and objective.** Provide a self-contained treatment of **6.10 Primitive Technique: Wide-EP Serving and Hot-Expert Replication** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 6.10 Primitive Technique: Wide-EP Serving and Hot-Expert Replication is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 6.11 Composition With TP, SP, PP, CP, DP, and Precision +**Engine 84: scope and objective.** Provide a self-contained treatment of **6.11 Composition With TP, SP, PP, CP, DP, and Precision** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 6.11 Composition With TP, SP, PP, CP, DP, and Precision is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 6.12 End-to-End MoE Execution Recipe +**Engine 85: scope and objective.** Provide a self-contained treatment of **6.12 End-to-End MoE Execution Recipe** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 6.12 End-to-End MoE Execution Recipe is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 6.13 Evidence Boundaries and Caveats +**Engine 86: scope and objective.** Provide a self-contained treatment of **6.13 Evidence Boundaries and Caveats** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 6.13 Evidence Boundaries and Caveats is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +## 7. Hybrid Mesh Parallelism +**Engine 87: scope and objective.** Provide a self-contained treatment of **7. Hybrid Mesh Parallelism** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 7. Hybrid Mesh Parallelism is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 7.1 Rank Mapping and Topology-Aware Placement +**Engine 88: scope and objective.** Provide a self-contained treatment of **7.1 Rank Mapping and Topology-Aware Placement** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 7.1 Rank Mapping and Topology-Aware Placement is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 7.2 Checkpoint Resharding and Restart Semantics +**Engine 89: scope and objective.** Provide a self-contained treatment of **7.2 Checkpoint Resharding and Restart Semantics** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 7.2 Checkpoint Resharding and Restart Semantics is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 7.3 Composition Rules +**Engine 90: scope and objective.** Provide a self-contained treatment of **7.3 Composition Rules** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 7.3 Composition Rules is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 7.4 System Perspectives +**Engine 91: scope and objective.** Provide a self-contained treatment of **7.4 System Perspectives** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 7.4 System Perspectives is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 7.5 Caveats and Evidence Boundaries +**Engine 92: scope and objective.** Provide a self-contained treatment of **7.5 Caveats and Evidence Boundaries** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 7.5 Caveats and Evidence Boundaries is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +## 8. Inference Parallelism and Serving Systems +**Engine 93: scope and objective.** Provide a self-contained treatment of **8. Inference Parallelism and Serving Systems** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 8. Inference Parallelism and Serving Systems is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 8.1 Primitive Technique: Replicas and Request Routing +**Engine 94: scope and objective.** Provide a self-contained treatment of **8.1 Primitive Technique: Replicas and Request Routing** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 8.1 Primitive Technique: Replicas and Request Routing is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 8.2 Primitive Technique: Continuous Batching +**Engine 95: scope and objective.** Provide a self-contained treatment of **8.2 Primitive Technique: Continuous Batching** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 8.2 Primitive Technique: Continuous Batching is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 8.3 Primitive Technique: Paged KV Cache +**Engine 96: scope and objective.** Provide a self-contained treatment of **8.3 Primitive Technique: Paged KV Cache** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 8.3 Primitive Technique: Paged KV Cache is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 8.4 Primitive Technique: Prefix and Radix Caching +**Engine 97: scope and objective.** Provide a self-contained treatment of **8.4 Primitive Technique: Prefix and Radix Caching** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 8.4 Primitive Technique: Prefix and Radix Caching is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 8.5 Primitive Technique: Prefill/Decode Disaggregation +**Engine 98: scope and objective.** Provide a self-contained treatment of **8.5 Primitive Technique: Prefill/Decode Disaggregation** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 8.5 Primitive Technique: Prefill/Decode Disaggregation is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 8.6 Primitive Technique: Chunked Prefill and SplitFuse +**Engine 99: scope and objective.** Provide a self-contained treatment of **8.6 Primitive Technique: Chunked Prefill and SplitFuse** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 8.6 Primitive Technique: Chunked Prefill and SplitFuse is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 8.7 Primitive Technique: Speculative and [MTP](https://arxiv.org/abs/2412.19437) (multi-token prediction) Decoding +**Engine 100: scope and objective.** Provide a self-contained treatment of **8.7 Primitive Technique: Speculative and [MTP](https://arxiv.org/abs/2412.19437) (multi-token prediction) Decoding** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 8.7 Primitive Technique: Speculative and [MTP](https://arxiv.org/abs/2412.19437) (multi-token prediction) Decoding is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 8.8 Primitive Technique: TP, PP, EP, and CP for Serving +**Engine 101: scope and objective.** Provide a self-contained treatment of **8.8 Primitive Technique: TP, PP, EP, and CP for Serving** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 8.8 Primitive Technique: TP, PP, EP, and CP for Serving is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 8.9 End-to-End Serving Map +**Engine 102: scope and objective.** Provide a self-contained treatment of **8.9 End-to-End Serving Map** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 8.9 End-to-End Serving Map is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 8.10 Sources and Caveats +**Engine 103: scope and objective.** Provide a self-contained treatment of **8.10 Sources and Caveats** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 8.10 Sources and Caveats is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +## 9. Cross-Cutting CUDA and Platform Techniques +**Engine 104: scope and objective.** Provide a self-contained treatment of **9. Cross-Cutting CUDA and Platform Techniques** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 9. Cross-Cutting CUDA and Platform Techniques is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 9.1 Overlap: Streams, Events, Buckets, and Progress +**Engine 105: scope and objective.** Provide a self-contained treatment of **9.1 Overlap: Streams, Events, Buckets, and Progress** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 9.1 Overlap: Streams, Events, Buckets, and Progress is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 9.2 CUDA Graphs and Static Metadata +**Engine 106: scope and objective.** Provide a self-contained treatment of **9.2 CUDA Graphs and Static Metadata** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 9.2 CUDA Graphs and Static Metadata is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 9.3 Low Precision as Payload Plus Scale Layout +**Engine 107: scope and objective.** Provide a self-contained treatment of **9.3 Low Precision as Payload Plus Scale Layout** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 9.3 Low Precision as Payload Plus Scale Layout is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 9.4 Attention Kernels: FlashAttention, cuDNN SDPA, NSA, and FlashMLA +**Engine 108: scope and objective.** Provide a self-contained treatment of **9.4 Attention Kernels: FlashAttention, cuDNN SDPA, NSA, and FlashMLA** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 9.4 Attention Kernels: FlashAttention, cuDNN SDPA, NSA, and FlashMLA is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 9.5 CUTLASS, CuTe, GEMM, and Grouped Expert Compute +**Engine 109: scope and objective.** Provide a self-contained treatment of **9.5 CUTLASS, CuTe, GEMM, and Grouped Expert Compute** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 9.5 CUTLASS, CuTe, GEMM, and Grouped Expert Compute is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 9.6 NVSHMEM, GPU-Initiated Communication, and Persistent Kernels +**Engine 110: scope and objective.** Provide a self-contained treatment of **9.6 NVSHMEM, GPU-Initiated Communication, and Persistent Kernels** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 9.6 NVSHMEM, GPU-Initiated Communication, and Persistent Kernels is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 9.7 Kernel Authoring DSLs: Triton, Pallas, TileLang, and ThunderKittens +**Engine 111: scope and objective.** Provide a self-contained treatment of **9.7 Kernel Authoring DSLs: Triton, Pallas, TileLang, and ThunderKittens** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 9.7 Kernel Authoring DSLs: Triton, Pallas, TileLang, and ThunderKittens is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +## 10. Frontier Model and Runtime Case Studies +**Engine 112: scope and objective.** Provide a self-contained treatment of **10. Frontier Model and Runtime Case Studies** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 10. Frontier Model and Runtime Case Studies is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 10.1 DeepSeek-V3: MoE, FP8, DualPipe, and Expert Overlap +**Engine 113: scope and objective.** Provide a self-contained treatment of **10.1 DeepSeek-V3: MoE, FP8, DualPipe, and Expert Overlap** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 10.1 DeepSeek-V3: MoE, FP8, DualPipe, and Expert Overlap is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 10.2 DeepSeek-V4: Hybrid Attention, mHC, FP4 Experts, and Serving Layout +**Engine 114: scope and objective.** Provide a self-contained treatment of **10.2 DeepSeek-V4: Hybrid Attention, mHC, FP4 Experts, and Serving Layout** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 10.2 DeepSeek-V4: Hybrid Attention, mHC, FP4 Experts, and Serving Layout is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 10.3 Meta Llama 3 and TorchTitan: Dense Hybrid Parallelism +**Engine 115: scope and objective.** Provide a self-contained treatment of **10.3 Meta Llama 3 and TorchTitan: Dense Hybrid Parallelism** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 10.3 Meta Llama 3 and TorchTitan: Dense Hybrid Parallelism is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 10.4 Qwen, Kimi, and Open MoE Directions +**Engine 116: scope and objective.** Provide a self-contained treatment of **10.4 Qwen, Kimi, and Open MoE Directions** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 10.4 Qwen, Kimi, and Open MoE Directions is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 10.5 Google Pathways, GSPMD, and MaxText: Compiler-Mesh Parallelism +**Engine 117: scope and objective.** Provide a self-contained treatment of **10.5 Google Pathways, GSPMD, and MaxText: Compiler-Mesh Parallelism** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 10.5 Google Pathways, GSPMD, and MaxText: Compiler-Mesh Parallelism is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +## 11. Source Carryover From The GPU-Kernel Paper +**Engine 118: scope and objective.** Provide a self-contained treatment of **11. Source Carryover From The GPU-Kernel Paper** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 11. Source Carryover From The GPU-Kernel Paper is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +## 12. Open Problems and Evidence Boundaries +**Engine 119: scope and objective.** Provide a self-contained treatment of **12. Open Problems and Evidence Boundaries** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 12. Open Problems and Evidence Boundaries is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 12.1 Unresolved Technical Questions +**Engine 120: scope and objective.** Provide a self-contained treatment of **12.1 Unresolved Technical Questions** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 12.1 Unresolved Technical Questions is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### 12.2 The Main Boundary +**Engine 121: scope and objective.** Provide a self-contained treatment of **12.2 The Main Boundary** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 12.2 The Main Boundary is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +## 13. Conclusion +**Engine 122: scope and objective.** Provide a self-contained treatment of **13. Conclusion** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - 13. Conclusion is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +## Bibliography and Source Notes +**Engine 123: scope and objective.** Provide a self-contained treatment of **Bibliography and Source Notes** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - Bibliography and Source Notes is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### Recent Training, Parallelism, and Runtime Systems +**Engine 124: scope and objective.** Provide a self-contained treatment of **Recent Training, Parallelism, and Runtime Systems** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - Recent Training, Parallelism, and Runtime Systems is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### Pipeline Parallelism +**Engine 125: scope and objective.** Provide a self-contained treatment of **Pipeline Parallelism** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - Pipeline Parallelism is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### Sequence, Context, and Attention Parallelism +**Engine 126: scope and objective.** Provide a self-contained treatment of **Sequence, Context, and Attention Parallelism** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - Sequence, Context, and Attention Parallelism is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### MoE and Expert Parallelism +**Engine 127: scope and objective.** Provide a self-contained treatment of **MoE and Expert Parallelism** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - MoE and Expert Parallelism is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### Inference and Serving +**Engine 128: scope and objective.** Provide a self-contained treatment of **Inference and Serving** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - Inference and Serving is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### Low Precision, Kernels, and Platform +**Engine 129: scope and objective.** Provide a self-contained treatment of **Low Precision, Kernels, and Platform** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - Low Precision, Kernels, and Platform is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### Compiler and Mesh Systems +**Engine 130: scope and objective.** Provide a self-contained treatment of **Compiler and Mesh Systems** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - Compiler and Mesh Systems is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +### Model Reports Used As Parallelism Context +**Engine 131: scope and objective.** Provide a self-contained treatment of **Model Reports Used As Parallelism Context** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - Model Reports Used As Parallelism Context is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. + +## Verification Notes +**Engine 132: scope and objective.** Provide a self-contained treatment of **Verification Notes** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. + +1. **Conceptual model.** + - Verification Notes is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. + - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. +2. **Mathematical formulation.** + - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. + - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. +3. **Execution recipe.** + - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. + - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. +4. **Performance engineering.** + - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. + - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. + - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. +5. **Failure modes and diagnostics.** + - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. + - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. +6. **Decision rubric.** + - Prefer the smallest parallel axis set meeting memory and throughput targets. + - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. + +**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. From 88f75a145f5ab606e76a74fa70dea34dc9944876 Mon Sep 17 00:00:00 2001 From: dzmitry-huba <123517587+dzmitry-huba@users.noreply.github.com> Date: Wed, 29 Apr 2026 14:55:14 -0700 Subject: [PATCH 2/2] Rewrite white paper into detailed end-to-end source-driven explainer --- papers/parallelisms_whitepaper.md | 3685 +++-------------------------- 1 file changed, 373 insertions(+), 3312 deletions(-) diff --git a/papers/parallelisms_whitepaper.md b/papers/parallelisms_whitepaper.md index b4eeb34..121a8f0 100644 --- a/papers/parallelisms_whitepaper.md +++ b/papers/parallelisms_whitepaper.md @@ -1,3316 +1,377 @@ -# Parallelisms White Paper (Expanded, Multi-Engine Draft) +# End-to-End Parallelisms for Large Language Models -## Authoring protocol -For each subsection, this paper uses an independent "engine" block to emulate separate specialist analysis. All engines share a common notation contract, then produce self-contained explanations. +## Abstract +This white paper explains, from first principles to production practice, how modern LLM training and serving systems compose parallelism strategies. It synthesizes the source list in `papers/parallelism_reference.md` into one coherent narrative: what each strategy does, which collectives it uses, where the memory and bandwidth go, how to reason about correctness, and how to choose trade-offs on real clusters. + +--- + +## 1) System model and shared notation +We assume a transformer with: +- batch size $B$ +- sequence length $S$ +- hidden size $H$ +- attention heads $N_h$, head width $d_h$ so $H=N_h d_h$ +- layers $L$ +- vocabulary size $V$ + +A global mesh is factored into axes: +$$ +(P_{dp},P_{tp},P_{pp},P_{cp},P_{ep}),\qquad P=\prod_i P_i. +$$ -## Global notation and assumptions -- Model parameters: $|\theta|$. Hidden size: $H$. Layers: $L$. Global batch: $B$. Sequence length: $S$. -- Mesh axes: $P_{dp},P_{tp},P_{pp},P_{cp},P_{ep}$ and $P=\prod P_i$. -- Attention heads $N_h$, head dim $d_h$, so $H=N_h d_h$. -- Collectives: $\operatorname{AR}$, $\operatorname{AG}$, $\operatorname{RS}$, $\operatorname{A2A}$. -- Communication cost uses $T=\alpha\cdot f(p)+\beta\cdot n\cdot g(p)$ with topology-aware $f,g$. +- **DP** data-parallel group +- **TP** tensor-parallel group +- **PP** pipeline stage group +- **CP** context/sequence-parallel group +- **EP** expert-parallel group (MoE) -### Common equations used throughout -$$M_{rank}=M_{params}+M_{grads}+M_{opt}+M_{acts}+M_{kv}+M_{tmp}.$$ -$$\eta_{pipe}\approx 1-\frac{P_{pp}-1}{m+P_{pp}-1},\quad\eta_{bubble}=1-\eta_{pipe}.$$ -$$\text{step\_time}\approx\max(T_{compute},T_{comm}-T_{overlap})+T_{sync}. $$ +Collectives: +- $\operatorname{AR}$ all-reduce +- $\operatorname{AG}$ all-gather +- $\operatorname{RS}$ reduce-scatter +- $\operatorname{A2A}$ all-to-all -## Abstract -**Engine 1: scope and objective.** Provide a self-contained treatment of **Abstract** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - Abstract is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -## How To Read This Paper -**Engine 2: scope and objective.** Provide a self-contained treatment of **How To Read This Paper** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - How To Read This Paper is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -## 1. Shared System Model, Notation, Placement, and Evidence Policy -**Engine 3: scope and objective.** Provide a self-contained treatment of **1. Shared System Model, Notation, Placement, and Evidence Policy** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 1. Shared System Model, Notation, Placement, and Evidence Policy is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 1.1 Evidence Grades and Source Interpretation -**Engine 4: scope and objective.** Provide a self-contained treatment of **1.1 Evidence Grades and Source Interpretation** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 1.1 Evidence Grades and Source Interpretation is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 1.2 Shared Symbols and Transformer Shapes -**Engine 5: scope and objective.** Provide a self-contained treatment of **1.2 Shared Symbols and Transformer Shapes** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 1.2 Shared Symbols and Transformer Shapes is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 1.3 Device Meshes, Rank Coordinates, and Process Groups -**Engine 6: scope and objective.** Provide a self-contained treatment of **1.3 Device Meshes, Rank Coordinates, and Process Groups** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 1.3 Device Meshes, Rank Coordinates, and Process Groups is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 1.4 Placement States and Tensor Layout Transitions -**Engine 7: scope and objective.** Provide a self-contained treatment of **1.4 Placement States and Tensor Layout Transitions** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 1.4 Placement States and Tensor Layout Transitions is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 1.5 Communication Primitives and Cost Models -**Engine 8: scope and objective.** Provide a self-contained treatment of **1.5 Communication Primitives and Cost Models** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 1.5 Communication Primitives and Cost Models is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 1.6 Memory, Bandwidth, and Overlap Accounting -**Engine 9: scope and objective.** Provide a self-contained treatment of **1.6 Memory, Bandwidth, and Overlap Accounting** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 1.6 Memory, Bandwidth, and Overlap Accounting is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 1.7 How Later Sections Use This Shared Layer -**Engine 10: scope and objective.** Provide a self-contained treatment of **1.7 How Later Sections Use This Shared Layer** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 1.7 How Later Sections Use This Shared Layer is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -## 2. Data Parallelism, FSDP, ZeRO, HSDP, and State Sharding -**Engine 11: scope and objective.** Provide a self-contained treatment of **2. Data Parallelism, FSDP, ZeRO, HSDP, and State Sharding** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 2. Data Parallelism, FSDP, ZeRO, HSDP, and State Sharding is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 2.1 DDP: Replicated State, Batch Shards, and Gradient All-Reduce -**Engine 12: scope and objective.** Provide a self-contained treatment of **2.1 DDP: Replicated State, Batch Shards, and Gradient All-Reduce** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 2.1 DDP: Replicated State, Batch Shards, and Gradient All-Reduce is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 2.2 The Primitive Mechanisms Behind Sharded Data Parallelism -**Engine 13: scope and objective.** Provide a self-contained treatment of **2.2 The Primitive Mechanisms Behind Sharded Data Parallelism** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 2.2 The Primitive Mechanisms Behind Sharded Data Parallelism is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 2.3 ZeRO, ZeRO++, and Distributed Optimizers -**Engine 14: scope and objective.** Provide a self-contained treatment of **2.3 ZeRO, ZeRO++, and Distributed Optimizers** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 2.3 ZeRO, ZeRO++, and Distributed Optimizers is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 2.4 FSDP and FSDP2 Execution -**Engine 15: scope and objective.** Provide a self-contained treatment of **2.4 FSDP and FSDP2 Execution** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 2.4 FSDP and FSDP2 Execution is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 2.5 HSDP: Hybrid Sharded Data Parallelism -**Engine 16: scope and objective.** Provide a self-contained treatment of **2.5 HSDP: Hybrid Sharded Data Parallelism** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 2.5 HSDP: Hybrid Sharded Data Parallelism is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 2.6 Offload: CPU and NVMe as Additional State Tiers -**Engine 17: scope and objective.** Provide a self-contained treatment of **2.6 Offload: CPU and NVMe as Additional State Tiers** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 2.6 Offload: CPU and NVMe as Additional State Tiers is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 2.7 Low Precision, Quantized Communication, and Platform Support -**Engine 18: scope and objective.** Provide a self-contained treatment of **2.7 Low Precision, Quantized Communication, and Platform Support** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 2.7 Low Precision, Quantized Communication, and Platform Support is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 2.8 Choosing Among DDP, FSDP2, ZeRO, ZeRO++, HSDP, and Offload -**Engine 19: scope and objective.** Provide a self-contained treatment of **2.8 Choosing Among DDP, FSDP2, ZeRO, ZeRO++, HSDP, and Offload** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 2.8 Choosing Among DDP, FSDP2, ZeRO, ZeRO++, HSDP, and Offload is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 2.9 Sources and Lineage -**Engine 20: scope and objective.** Provide a self-contained treatment of **2.9 Sources and Lineage** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 2.9 Sources and Lineage is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 2.10 Caveats -**Engine 21: scope and objective.** Provide a self-contained treatment of **2.10 Caveats** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 2.10 Caveats is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -## 3. Tensor Parallelism, Sequence-Parallel Activations, and Vocabulary Parallelism -**Engine 22: scope and objective.** Provide a self-contained treatment of **3. Tensor Parallelism, Sequence-Parallel Activations, and Vocabulary Parallelism** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 3. Tensor Parallelism, Sequence-Parallel Activations, and Vocabulary Parallelism is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 3.1 TP-Specific Notation and Layout Contract -**Engine 23: scope and objective.** Provide a self-contained treatment of **3.1 TP-Specific Notation and Layout Contract** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 3.1 TP-Specific Notation and Layout Contract is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 3.2 Column-Parallel Linear Layers -**Engine 24: scope and objective.** Provide a self-contained treatment of **3.2 Column-Parallel Linear Layers** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 3.2 Column-Parallel Linear Layers is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 3.3 Row-Parallel Linear Layers -**Engine 25: scope and objective.** Provide a self-contained treatment of **3.3 Row-Parallel Linear Layers** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 3.3 Row-Parallel Linear Layers is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 3.4 Transformer Block Recipes: MLP and Attention -**Engine 26: scope and objective.** Provide a self-contained treatment of **3.4 Transformer Block Recipes: MLP and Attention** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 3.4 Transformer Block Recipes: MLP and Attention is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -#### MLP / SwiGLU -**Engine 27: scope and objective.** Provide a self-contained treatment of **MLP / SwiGLU** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - MLP / SwiGLU is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -#### Attention -**Engine 28: scope and objective.** Provide a self-contained treatment of **Attention** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - Attention is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 3.5 Sequence-Parallel Activations as a TP Companion -**Engine 29: scope and objective.** Provide a self-contained treatment of **3.5 Sequence-Parallel Activations as a TP Companion** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 3.5 Sequence-Parallel Activations as a TP Companion is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 3.6 Vocabulary Parallelism -**Engine 30: scope and objective.** Provide a self-contained treatment of **3.6 Vocabulary Parallelism** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 3.6 Vocabulary Parallelism is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -#### Input Embedding -**Engine 31: scope and objective.** Provide a self-contained treatment of **Input Embedding** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - Input Embedding is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -#### LM Head and Cross-Entropy -**Engine 32: scope and objective.** Provide a self-contained treatment of **LM Head and Cross-Entropy** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - LM Head and Cross-Entropy is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 3.7 Async Tensor Parallelism -**Engine 33: scope and objective.** Provide a self-contained treatment of **3.7 Async Tensor Parallelism** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 3.7 Async Tensor Parallelism is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 3.8 TP-Aware CUDA and Kernel Techniques -**Engine 34: scope and objective.** Provide a self-contained treatment of **3.8 TP-Aware CUDA and Kernel Techniques** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 3.8 TP-Aware CUDA and Kernel Techniques is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -#### 3.8.1 Local GEMM Shape Quality -**Engine 35: scope and objective.** Provide a self-contained treatment of **3.8.1 Local GEMM Shape Quality** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 3.8.1 Local GEMM Shape Quality is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -#### 3.8.2 Fused Epilogues and Layout-Stable Collectives -**Engine 36: scope and objective.** Provide a self-contained treatment of **3.8.2 Fused Epilogues and Layout-Stable Collectives** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 3.8.2 Fused Epilogues and Layout-Stable Collectives is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -#### 3.8.3 Low-Precision Communication and Scale Layout -**Engine 37: scope and objective.** Provide a self-contained treatment of **3.8.3 Low-Precision Communication and Scale Layout** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 3.8.3 Low-Precision Communication and Scale Layout is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -#### 3.8.4 TP-Aware Megakernels -**Engine 38: scope and objective.** Provide a self-contained treatment of **3.8.4 TP-Aware Megakernels** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 3.8.4 TP-Aware Megakernels is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 3.9 Choosing a TP Degree -**Engine 39: scope and objective.** Provide a self-contained treatment of **3.9 Choosing a TP Degree** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 3.9 Choosing a TP Degree is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 3.10 Sources and Lineage -**Engine 40: scope and objective.** Provide a self-contained treatment of **3.10 Sources and Lineage** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 3.10 Sources and Lineage is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 3.11 Caveats -**Engine 41: scope and objective.** Provide a self-contained treatment of **3.11 Caveats** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 3.11 Caveats is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -## 4. Pipeline Parallelism: Depth-Axis Parallelism and Modern Schedules -**Engine 42: scope and objective.** Provide a self-contained treatment of **4. Pipeline Parallelism: Depth-Axis Parallelism and Modern Schedules** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 4. Pipeline Parallelism: Depth-Axis Parallelism and Modern Schedules is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 4.1 System Model, Variables, and Shapes -**Engine 43: scope and objective.** Provide a self-contained treatment of **4.1 System Model, Variables, and Shapes** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 4.1 System Model, Variables, and Shapes is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 4.2 Primitive: Stage Partitioning and Boundary Contracts -**Engine 44: scope and objective.** Provide a self-contained treatment of **4.2 Primitive: Stage Partitioning and Boundary Contracts** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 4.2 Primitive: Stage Partitioning and Boundary Contracts is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 4.3 Primitive: Microbatching, Warmup, Steady State, and Drain -**Engine 45: scope and objective.** Provide a self-contained treatment of **4.3 Primitive: Microbatching, Warmup, Steady State, and Drain** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 4.3 Primitive: Microbatching, Warmup, Steady State, and Drain is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 4.4 Primitive: Point-to-Point Send/Recv and Collective Composition -**Engine 46: scope and objective.** Provide a self-contained treatment of **4.4 Primitive: Point-to-Point Send/Recv and Collective Composition** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 4.4 Primitive: Point-to-Point Send/Recv and Collective Composition is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 4.5 Primitive: The Schedule DAG and the F/B/W Split -**Engine 47: scope and objective.** Provide a self-contained treatment of **4.5 Primitive: The Schedule DAG and the F/B/W Split** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 4.5 Primitive: The Schedule DAG and the F/B/W Split is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 4.6 Zero Bubble Pipeline Parallelism -**Engine 48: scope and objective.** Provide a self-contained treatment of **4.6 Zero Bubble Pipeline Parallelism** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 4.6 Zero Bubble Pipeline Parallelism is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 4.7 Controllable-Memory Pipeline Schedules -**Engine 49: scope and objective.** Provide a self-contained treatment of **4.7 Controllable-Memory Pipeline Schedules** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 4.7 Controllable-Memory Pipeline Schedules is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 4.8 PipeOffload: Activation Offload and Prefetch as Schedule Primitives -**Engine 50: scope and objective.** Provide a self-contained treatment of **4.8 PipeOffload: Activation Offload and Prefetch as Schedule Primitives** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 4.8 PipeOffload: Activation Offload and Prefetch as Schedule Primitives is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 4.9 Vocabulary-Balanced Pipeline Parallelism -**Engine 51: scope and objective.** Provide a self-contained treatment of **4.9 Vocabulary-Balanced Pipeline Parallelism** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 4.9 Vocabulary-Balanced Pipeline Parallelism is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 4.10 DualPipe and DualPipeV: Pipeline Scheduling for MoE Communication -**Engine 52: scope and objective.** Provide a self-contained treatment of **4.10 DualPipe and DualPipeV: Pipeline Scheduling for MoE Communication** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 4.10 DualPipe and DualPipeV: Pipeline Scheduling for MoE Communication is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 4.11 PyTorch Pipelining and TorchTitan Runtime Support -**Engine 53: scope and objective.** Provide a self-contained treatment of **4.11 PyTorch Pipelining and TorchTitan Runtime Support** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 4.11 PyTorch Pipelining and TorchTitan Runtime Support is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 4.12 Optimizer Barriers, Gradient Synchronization, and Correctness -**Engine 54: scope and objective.** Provide a self-contained treatment of **4.12 Optimizer Barriers, Gradient Synchronization, and Correctness** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 4.12 Optimizer Barriers, Gradient Synchronization, and Correctness is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 4.13 Composing PP With Tensor, Context, Expert, and Data Parallelism -**Engine 55: scope and objective.** Provide a self-contained treatment of **4.13 Composing PP With Tensor, Context, Expert, and Data Parallelism** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 4.13 Composing PP With Tensor, Context, Expert, and Data Parallelism is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 4.14 Choosing a Modern PP Schedule -**Engine 56: scope and objective.** Provide a self-contained treatment of **4.14 Choosing a Modern PP Schedule** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 4.14 Choosing a Modern PP Schedule is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 4.15 Lineage: Older PP Work as Context -**Engine 57: scope and objective.** Provide a self-contained treatment of **4.15 Lineage: Older PP Work as Context** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 4.15 Lineage: Older PP Work as Context is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 4.16 Practical Caveats -**Engine 58: scope and objective.** Provide a self-contained treatment of **4.16 Practical Caveats** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 4.16 Practical Caveats is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -## 5. Sequence Parallelism and Context Parallelism -**Engine 59: scope and objective.** Provide a self-contained treatment of **5. Sequence Parallelism and Context Parallelism** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 5. Sequence Parallelism and Context Parallelism is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 5.1 Notation, Tensor Shapes, and the Attention Operator -**Engine 60: scope and objective.** Provide a self-contained treatment of **5.1 Notation, Tensor Shapes, and the Attention Operator** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 5.1 Notation, Tensor Shapes, and the Attention Operator is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 5.2 The Primitive Operations and Their Collectives -**Engine 61: scope and objective.** Provide a self-contained treatment of **5.2 The Primitive Operations and Their Collectives** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 5.2 The Primitive Operations and Their Collectives is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 5.3 Megatron Sequence Parallelism: Activation Sharding, Not Full Long-Context Attention -**Engine 62: scope and objective.** Provide a self-contained treatment of **5.3 Megatron Sequence Parallelism: Activation Sharding, Not Full Long-Context Attention** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 5.3 Megatron Sequence Parallelism: Activation Sharding, Not Full Long-Context Attention is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 5.4 Ulysses: Sequence Shards Become Head Shards Through All-to-All -**Engine 63: scope and objective.** Provide a self-contained treatment of **5.4 Ulysses: Sequence Shards Become Head Shards Through All-to-All** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 5.4 Ulysses: Sequence Shards Become Head Shards Through All-to-All is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 5.5 Ring Attention: KV Streaming With Point-to-Point Send/Recv -**Engine 64: scope and objective.** Provide a self-contained treatment of **5.5 Ring Attention: KV Streaming With Point-to-Point Send/Recv** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 5.5 Ring Attention: KV Streaming With Point-to-Point Send/Recv is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 5.6 The Numerical Primitive: Blockwise Online Softmax -**Engine 65: scope and objective.** Provide a self-contained treatment of **5.6 The Numerical Primitive: Blockwise Online Softmax** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 5.6 The Numerical Primitive: Blockwise Online Softmax is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 5.7 Megatron Context Parallelism: Sequence Sharding for the Whole Network -**Engine 66: scope and objective.** Provide a self-contained treatment of **5.7 Megatron Context Parallelism: Sequence Sharding for the Whole Network** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 5.7 Megatron Context Parallelism: Sequence Sharding for the Whole Network is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 5.8 Dynamic Context Parallelism: Choose CP Size Per Microbatch -**Engine 67: scope and objective.** Provide a self-contained treatment of **5.8 Dynamic Context Parallelism: Choose CP Size Per Microbatch** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 5.8 Dynamic Context Parallelism: Choose CP Size Per Microbatch is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 5.9 THD and Ragged Packing: Token-Head-Dimension Layout -**Engine 68: scope and objective.** Provide a self-contained treatment of **5.9 THD and Ragged Packing: Token-Head-Dimension Layout** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 5.9 THD and Ragged Packing: Token-Head-Dimension Layout is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 5.10 Sparse and Hybrid Attention: Communicate Selected KV, Not Necessarily All KV -**Engine 69: scope and objective.** Provide a self-contained treatment of **5.10 Sparse and Hybrid Attention: Communicate Selected KV, Not Necessarily All KV** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 5.10 Sparse and Hybrid Attention: Communicate Selected KV, Not Necessarily All KV is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 5.11 Inference KV and Context Placement -**Engine 70: scope and objective.** Provide a self-contained treatment of **5.11 Inference KV and Context Placement** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 5.11 Inference KV and Context Placement is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 5.12 How to Choose Among SP, Ulysses, Ring, CP, Dynamic-CP, and Sparse Context Placement -**Engine 71: scope and objective.** Provide a self-contained treatment of **5.12 How to Choose Among SP, Ulysses, Ring, CP, Dynamic-CP, and Sparse Context Placement** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 5.12 How to Choose Among SP, Ulysses, Ring, CP, Dynamic-CP, and Sparse Context Placement is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 5.13 Evidence Boundaries and Terminology Caveats -**Engine 72: scope and objective.** Provide a self-contained treatment of **5.13 Evidence Boundaries and Terminology Caveats** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 5.13 Evidence Boundaries and Terminology Caveats is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -## 6. Expert and MoE Parallelism -**Engine 73: scope and objective.** Provide a self-contained treatment of **6. Expert and MoE Parallelism** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 6. Expert and MoE Parallelism is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 6.1 Symbols, Shapes, and the MoE Layer -**Engine 74: scope and objective.** Provide a self-contained treatment of **6.1 Symbols, Shapes, and the MoE Layer** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 6.1 Symbols, Shapes, and the MoE Layer is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 6.2 Primitive Technique: Routing and Top-k Assignment -**Engine 75: scope and objective.** Provide a self-contained treatment of **6.2 Primitive Technique: Routing and Top-k Assignment** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 6.2 Primitive Technique: Routing and Top-k Assignment is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 6.3 Primitive Technique: Expert Ownership and Local Shard Shapes -**Engine 76: scope and objective.** Provide a self-contained treatment of **6.3 Primitive Technique: Expert Ownership and Local Shard Shapes** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 6.3 Primitive Technique: Expert Ownership and Local Shard Shapes is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 6.4 Primitive Technique: Permutation, Packing, and Inverse Permutation -**Engine 77: scope and objective.** Provide a self-contained treatment of **6.4 Primitive Technique: Permutation, Packing, and Inverse Permutation** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 6.4 Primitive Technique: Permutation, Packing, and Inverse Permutation is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 6.5 Primitive Technique: All-to-All Dispatch, Notify, and Combine -**Engine 78: scope and objective.** Provide a self-contained treatment of **6.5 Primitive Technique: All-to-All Dispatch, Notify, and Combine** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 6.5 Primitive Technique: All-to-All Dispatch, Notify, and Combine is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 6.6 Primitive Technique: Capacity, Dropless Execution, and Padding -**Engine 79: scope and objective.** Provide a self-contained treatment of **6.6 Primitive Technique: Capacity, Dropless Execution, and Padding** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 6.6 Primitive Technique: Capacity, Dropless Execution, and Padding is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 6.7 Primitive Technique: Grouped GEMM, MegaBlocks, DeepGEMM, and Mega MoE -**Engine 80: scope and objective.** Provide a self-contained treatment of **6.7 Primitive Technique: Grouped GEMM, MegaBlocks, DeepGEMM, and Mega MoE** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 6.7 Primitive Technique: Grouped GEMM, MegaBlocks, DeepGEMM, and Mega MoE is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 6.8 Primitive Technique: Backward Pass, Weight Gradients, and Overlap -**Engine 81: scope and objective.** Provide a self-contained treatment of **6.8 Primitive Technique: Backward Pass, Weight Gradients, and Overlap** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 6.8 Primitive Technique: Backward Pass, Weight Gradients, and Overlap is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 6.9 Primitive Technique: DeepEP Modes and Network-Aware EP -**Engine 82: scope and objective.** Provide a self-contained treatment of **6.9 Primitive Technique: DeepEP Modes and Network-Aware EP** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 6.9 Primitive Technique: DeepEP Modes and Network-Aware EP is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 6.10 Primitive Technique: Wide-EP Serving and Hot-Expert Replication -**Engine 83: scope and objective.** Provide a self-contained treatment of **6.10 Primitive Technique: Wide-EP Serving and Hot-Expert Replication** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 6.10 Primitive Technique: Wide-EP Serving and Hot-Expert Replication is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 6.11 Composition With TP, SP, PP, CP, DP, and Precision -**Engine 84: scope and objective.** Provide a self-contained treatment of **6.11 Composition With TP, SP, PP, CP, DP, and Precision** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 6.11 Composition With TP, SP, PP, CP, DP, and Precision is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 6.12 End-to-End MoE Execution Recipe -**Engine 85: scope and objective.** Provide a self-contained treatment of **6.12 End-to-End MoE Execution Recipe** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 6.12 End-to-End MoE Execution Recipe is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 6.13 Evidence Boundaries and Caveats -**Engine 86: scope and objective.** Provide a self-contained treatment of **6.13 Evidence Boundaries and Caveats** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 6.13 Evidence Boundaries and Caveats is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -## 7. Hybrid Mesh Parallelism -**Engine 87: scope and objective.** Provide a self-contained treatment of **7. Hybrid Mesh Parallelism** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 7. Hybrid Mesh Parallelism is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 7.1 Rank Mapping and Topology-Aware Placement -**Engine 88: scope and objective.** Provide a self-contained treatment of **7.1 Rank Mapping and Topology-Aware Placement** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 7.1 Rank Mapping and Topology-Aware Placement is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 7.2 Checkpoint Resharding and Restart Semantics -**Engine 89: scope and objective.** Provide a self-contained treatment of **7.2 Checkpoint Resharding and Restart Semantics** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 7.2 Checkpoint Resharding and Restart Semantics is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 7.3 Composition Rules -**Engine 90: scope and objective.** Provide a self-contained treatment of **7.3 Composition Rules** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 7.3 Composition Rules is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 7.4 System Perspectives -**Engine 91: scope and objective.** Provide a self-contained treatment of **7.4 System Perspectives** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 7.4 System Perspectives is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 7.5 Caveats and Evidence Boundaries -**Engine 92: scope and objective.** Provide a self-contained treatment of **7.5 Caveats and Evidence Boundaries** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 7.5 Caveats and Evidence Boundaries is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -## 8. Inference Parallelism and Serving Systems -**Engine 93: scope and objective.** Provide a self-contained treatment of **8. Inference Parallelism and Serving Systems** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 8. Inference Parallelism and Serving Systems is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 8.1 Primitive Technique: Replicas and Request Routing -**Engine 94: scope and objective.** Provide a self-contained treatment of **8.1 Primitive Technique: Replicas and Request Routing** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 8.1 Primitive Technique: Replicas and Request Routing is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 8.2 Primitive Technique: Continuous Batching -**Engine 95: scope and objective.** Provide a self-contained treatment of **8.2 Primitive Technique: Continuous Batching** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 8.2 Primitive Technique: Continuous Batching is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 8.3 Primitive Technique: Paged KV Cache -**Engine 96: scope and objective.** Provide a self-contained treatment of **8.3 Primitive Technique: Paged KV Cache** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 8.3 Primitive Technique: Paged KV Cache is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 8.4 Primitive Technique: Prefix and Radix Caching -**Engine 97: scope and objective.** Provide a self-contained treatment of **8.4 Primitive Technique: Prefix and Radix Caching** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 8.4 Primitive Technique: Prefix and Radix Caching is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 8.5 Primitive Technique: Prefill/Decode Disaggregation -**Engine 98: scope and objective.** Provide a self-contained treatment of **8.5 Primitive Technique: Prefill/Decode Disaggregation** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 8.5 Primitive Technique: Prefill/Decode Disaggregation is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 8.6 Primitive Technique: Chunked Prefill and SplitFuse -**Engine 99: scope and objective.** Provide a self-contained treatment of **8.6 Primitive Technique: Chunked Prefill and SplitFuse** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 8.6 Primitive Technique: Chunked Prefill and SplitFuse is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 8.7 Primitive Technique: Speculative and [MTP](https://arxiv.org/abs/2412.19437) (multi-token prediction) Decoding -**Engine 100: scope and objective.** Provide a self-contained treatment of **8.7 Primitive Technique: Speculative and [MTP](https://arxiv.org/abs/2412.19437) (multi-token prediction) Decoding** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 8.7 Primitive Technique: Speculative and [MTP](https://arxiv.org/abs/2412.19437) (multi-token prediction) Decoding is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 8.8 Primitive Technique: TP, PP, EP, and CP for Serving -**Engine 101: scope and objective.** Provide a self-contained treatment of **8.8 Primitive Technique: TP, PP, EP, and CP for Serving** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 8.8 Primitive Technique: TP, PP, EP, and CP for Serving is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 8.9 End-to-End Serving Map -**Engine 102: scope and objective.** Provide a self-contained treatment of **8.9 End-to-End Serving Map** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 8.9 End-to-End Serving Map is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 8.10 Sources and Caveats -**Engine 103: scope and objective.** Provide a self-contained treatment of **8.10 Sources and Caveats** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 8.10 Sources and Caveats is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -## 9. Cross-Cutting CUDA and Platform Techniques -**Engine 104: scope and objective.** Provide a self-contained treatment of **9. Cross-Cutting CUDA and Platform Techniques** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 9. Cross-Cutting CUDA and Platform Techniques is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 9.1 Overlap: Streams, Events, Buckets, and Progress -**Engine 105: scope and objective.** Provide a self-contained treatment of **9.1 Overlap: Streams, Events, Buckets, and Progress** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 9.1 Overlap: Streams, Events, Buckets, and Progress is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 9.2 CUDA Graphs and Static Metadata -**Engine 106: scope and objective.** Provide a self-contained treatment of **9.2 CUDA Graphs and Static Metadata** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 9.2 CUDA Graphs and Static Metadata is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 9.3 Low Precision as Payload Plus Scale Layout -**Engine 107: scope and objective.** Provide a self-contained treatment of **9.3 Low Precision as Payload Plus Scale Layout** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 9.3 Low Precision as Payload Plus Scale Layout is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 9.4 Attention Kernels: FlashAttention, cuDNN SDPA, NSA, and FlashMLA -**Engine 108: scope and objective.** Provide a self-contained treatment of **9.4 Attention Kernels: FlashAttention, cuDNN SDPA, NSA, and FlashMLA** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 9.4 Attention Kernels: FlashAttention, cuDNN SDPA, NSA, and FlashMLA is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 9.5 CUTLASS, CuTe, GEMM, and Grouped Expert Compute -**Engine 109: scope and objective.** Provide a self-contained treatment of **9.5 CUTLASS, CuTe, GEMM, and Grouped Expert Compute** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 9.5 CUTLASS, CuTe, GEMM, and Grouped Expert Compute is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 9.6 NVSHMEM, GPU-Initiated Communication, and Persistent Kernels -**Engine 110: scope and objective.** Provide a self-contained treatment of **9.6 NVSHMEM, GPU-Initiated Communication, and Persistent Kernels** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 9.6 NVSHMEM, GPU-Initiated Communication, and Persistent Kernels is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 9.7 Kernel Authoring DSLs: Triton, Pallas, TileLang, and ThunderKittens -**Engine 111: scope and objective.** Provide a self-contained treatment of **9.7 Kernel Authoring DSLs: Triton, Pallas, TileLang, and ThunderKittens** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 9.7 Kernel Authoring DSLs: Triton, Pallas, TileLang, and ThunderKittens is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -## 10. Frontier Model and Runtime Case Studies -**Engine 112: scope and objective.** Provide a self-contained treatment of **10. Frontier Model and Runtime Case Studies** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 10. Frontier Model and Runtime Case Studies is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 10.1 DeepSeek-V3: MoE, FP8, DualPipe, and Expert Overlap -**Engine 113: scope and objective.** Provide a self-contained treatment of **10.1 DeepSeek-V3: MoE, FP8, DualPipe, and Expert Overlap** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 10.1 DeepSeek-V3: MoE, FP8, DualPipe, and Expert Overlap is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 10.2 DeepSeek-V4: Hybrid Attention, mHC, FP4 Experts, and Serving Layout -**Engine 114: scope and objective.** Provide a self-contained treatment of **10.2 DeepSeek-V4: Hybrid Attention, mHC, FP4 Experts, and Serving Layout** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 10.2 DeepSeek-V4: Hybrid Attention, mHC, FP4 Experts, and Serving Layout is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 10.3 Meta Llama 3 and TorchTitan: Dense Hybrid Parallelism -**Engine 115: scope and objective.** Provide a self-contained treatment of **10.3 Meta Llama 3 and TorchTitan: Dense Hybrid Parallelism** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 10.3 Meta Llama 3 and TorchTitan: Dense Hybrid Parallelism is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 10.4 Qwen, Kimi, and Open MoE Directions -**Engine 116: scope and objective.** Provide a self-contained treatment of **10.4 Qwen, Kimi, and Open MoE Directions** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 10.4 Qwen, Kimi, and Open MoE Directions is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 10.5 Google Pathways, GSPMD, and MaxText: Compiler-Mesh Parallelism -**Engine 117: scope and objective.** Provide a self-contained treatment of **10.5 Google Pathways, GSPMD, and MaxText: Compiler-Mesh Parallelism** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 10.5 Google Pathways, GSPMD, and MaxText: Compiler-Mesh Parallelism is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -## 11. Source Carryover From The GPU-Kernel Paper -**Engine 118: scope and objective.** Provide a self-contained treatment of **11. Source Carryover From The GPU-Kernel Paper** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 11. Source Carryover From The GPU-Kernel Paper is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -## 12. Open Problems and Evidence Boundaries -**Engine 119: scope and objective.** Provide a self-contained treatment of **12. Open Problems and Evidence Boundaries** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 12. Open Problems and Evidence Boundaries is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 12.1 Unresolved Technical Questions -**Engine 120: scope and objective.** Provide a self-contained treatment of **12.1 Unresolved Technical Questions** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 12.1 Unresolved Technical Questions is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### 12.2 The Main Boundary -**Engine 121: scope and objective.** Provide a self-contained treatment of **12.2 The Main Boundary** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 12.2 The Main Boundary is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -## 13. Conclusion -**Engine 122: scope and objective.** Provide a self-contained treatment of **13. Conclusion** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - 13. Conclusion is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -## Bibliography and Source Notes -**Engine 123: scope and objective.** Provide a self-contained treatment of **Bibliography and Source Notes** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - Bibliography and Source Notes is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### Recent Training, Parallelism, and Runtime Systems -**Engine 124: scope and objective.** Provide a self-contained treatment of **Recent Training, Parallelism, and Runtime Systems** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - Recent Training, Parallelism, and Runtime Systems is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### Pipeline Parallelism -**Engine 125: scope and objective.** Provide a self-contained treatment of **Pipeline Parallelism** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - Pipeline Parallelism is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### Sequence, Context, and Attention Parallelism -**Engine 126: scope and objective.** Provide a self-contained treatment of **Sequence, Context, and Attention Parallelism** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - Sequence, Context, and Attention Parallelism is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### MoE and Expert Parallelism -**Engine 127: scope and objective.** Provide a self-contained treatment of **MoE and Expert Parallelism** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - MoE and Expert Parallelism is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### Inference and Serving -**Engine 128: scope and objective.** Provide a self-contained treatment of **Inference and Serving** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - Inference and Serving is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### Low Precision, Kernels, and Platform -**Engine 129: scope and objective.** Provide a self-contained treatment of **Low Precision, Kernels, and Platform** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - Low Precision, Kernels, and Platform is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### Compiler and Mesh Systems -**Engine 130: scope and objective.** Provide a self-contained treatment of **Compiler and Mesh Systems** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - Compiler and Mesh Systems is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -### Model Reports Used As Parallelism Context -**Engine 131: scope and objective.** Provide a self-contained treatment of **Model Reports Used As Parallelism Context** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - Model Reports Used As Parallelism Context is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. - -## Verification Notes -**Engine 132: scope and objective.** Provide a self-contained treatment of **Verification Notes** with definitions, mechanism, cost model, implementation choices, failure modes, and decision rules. - -1. **Conceptual model.** - - Verification Notes is framed as a tensor-placement transformation over the device mesh. Inputs, outputs, and dependency edges are explicitly identified. - - Correctness condition: transformed execution must preserve numerical equivalence up to floating-point non-associativity and stochastic layers. -2. **Mathematical formulation.** - - Let local shard size be $n_{local}=n/p$ along the chosen axis. Memory scales as $O(n_{local})$ for sharded states and $O(n)$ for replicated states. - - Communication lower bound (single phase): $\Omega(n\,\beta)$ payload plus startup term $\Omega(\alpha)$. -3. **Execution recipe.** - - Precondition checks: mesh groups, dtype policy, seed policy, checkpoint layout metadata. - - Runtime sequence: schedule compute kernels, issue collectives, place synchronization barriers only on true dependencies, and opportunistically overlap. -4. **Performance engineering.** - - Increase arithmetic intensity using fused kernels and shape-friendly GEMMs. - - Reduce bytes with precision controls (FP8/quantized links) while managing scales and error feedback. - - Place high-frequency collectives on fast links (e.g., intra-node) whenever possible. -5. **Failure modes and diagnostics.** - - Symptoms: stragglers, tail latency spikes, allocator fragmentation, imbalance, and collective stalls. - - Diagnostics: per-rank timeline traces, bucket size sweeps, microbatch sweep, and reshard-count audits. -6. **Decision rubric.** - - Prefer the smallest parallel axis set meeting memory and throughput targets. - - Add axes incrementally; after each addition, re-measure MFU, link utilization, and p95/p99 latency. - -**Practical takeaway.** This subsection should be implemented as a topology-aware schedule with explicit tensor-layout contracts; otherwise theoretical gains are lost to reshards and synchronization overhead. +Communication model: +$$ +T_{comm}(n,p)\approx \alpha\,f(p)+\beta\,n\,g(p), +$$ +where $n$ is payload bytes, $p$ is group size, $\alpha$ captures startup/latency, $\beta$ inverse bandwidth. + +Per-rank memory budget: +$$ +M_{rank}=M_{params}+M_{grads}+M_{opt}+M_{acts}+M_{kv}+M_{tmp}. +$$ + +This single model is enough to explain all major LLM parallelisms as **layout transformations + collectives + schedule constraints**. + +--- + +## 2) Data parallelism family: DDP, ZeRO, FSDP, HSDP + +## 2.1 DDP baseline +In DDP, every rank has a full copy of model parameters and optimizer states. Each rank processes a different mini-batch shard and computes gradients locally. + +- Backward pass produces per-rank gradient tensor $g^{(r)}$. +- Synchronization uses all-reduce: +$$ +\bar g = \frac{1}{P_{dp}}\sum_{r=1}^{P_{dp}} g^{(r)}. +$$ +- All ranks then apply the same optimizer step. + +**Strengths:** simple, stable, excellent for small-to-mid models. +**Bottlenecks:** memory scales poorly because params+grads+optimizer are replicated. + +## 2.2 Why ZeRO/FSDP exist +Memory replication becomes dominant at scale. ZeRO and FSDP replace replication with sharding: +- shard optimizer states +- shard gradients +- shard parameters (full sharding) + +A useful identity: +$$ +\operatorname{AR}(x)=\operatorname{AG}(\operatorname{RS}(x)). +$$ +This is why many sharded systems use RS+AG rather than monolithic AR. + +## 2.3 ZeRO stages (conceptual) +- **Stage 1:** optimizer states sharded. +- **Stage 2:** optimizer + gradients sharded. +- **Stage 3:** optimizer + gradients + parameters sharded. + +At stage 3, parameter materialization is on-demand around each layer compute. The trade-off is memory reduction vs added communication frequency. + +## 2.4 FSDP/FSDP2 execution pattern +For each wrapped module: +1. **Pre-forward**: all-gather parameter shards to assemble full weights. +2. Compute forward. +3. Optionally reshard after forward. +4. Backward compute. +5. Reduce-scatter gradients. + +This turns “always-resident full parameters” into a **just-in-time weight residency model**. + +Practical tuning levers: +- wrap granularity (too fine => overhead; too coarse => memory spikes) +- prefetch strategy +- bucket sizes +- mixed precision + accumulation strategy + +## 2.5 HSDP: hierarchical sharding +HSDP combines sharding and replication across hierarchy levels (e.g., shard within node, replicate across nodes, or vice versa). + +Goal: align expensive collectives with fast links (NVLink/NVSwitch) and reduce slow-link traffic (IB/Ethernet). It is topology-aware ZeRO/FSDP behavior. + +## 2.6 Offload tiers (CPU/NVMe) +When GPU memory is insufficient, offload states to CPU RAM or NVMe. + +Benefit: +- enables larger models / batches than VRAM allows. + +Cost: +- introduces transfer latency and bandwidth constraints. +- requires strict overlap of transfer with compute to avoid throughput collapse. + +Rule of thumb: offload is a **capacity unlock**, not a free speed win. + +--- + +## 3) Tensor parallelism (TP), sequence-parallel companions, vocabulary parallelism + +## 3.1 TP core idea +Split matrix multiplications inside a layer across devices. + +For $Y=XW$: +- **Column parallel**: split columns of $W$ across ranks. +- **Row parallel**: split rows of $W$ across ranks. + +This lowers per-rank compute and memory for very wide layers. + +## 3.2 Column-parallel linear +If $W=[W_1,\dots,W_{P_{tp}}]$ by columns, each rank computes +$$ +Y_i = XW_i. +$$ +Output may stay sharded or be gathered depending on next op. + +## 3.3 Row-parallel linear +If $W^T=[W_1^T,\dots,W_{P_{tp}}^T]$ by rows, each rank computes a partial output and results are summed (AR-like semantic) to form full output. + +## 3.4 TP in MLP and attention +Typical transformer block pattern: +- MLP up-projection: column parallel +- activation/gating local +- down-projection: row parallel + +Attention often uses TP for QKV projections and output projection; placement choices determine where collectives happen. + +## 3.5 Sequence parallel (SP) as TP companion +SP shards activations along sequence dimension for selected ops (norm/dropout/residual), reducing activation memory pressure that TP alone doesn’t fix. + +## 3.6 Vocabulary parallelism +Embedding and LM head are costly for large $V$. Partition vocabulary dimension so each rank stores a slice of embedding/output weights; softmax/logits then use distributed reduction/gather patterns. + +## 3.7 Choosing TP degree +Increasing $P_{tp}$: +- helps when layers are very wide and single-GPU GEMMs are too large +- hurts if interconnect is weak or collective frequency becomes dominant + +You want the point where local GEMMs stay efficient while communication does not dominate the roofline. + +--- + +## 4) Pipeline parallelism (PP) + +PP partitions layers across depth-axis stages. + +With $P_{pp}$ stages and microbatch count $m$, bubble fraction is approximately: +$$ +\eta_{bubble}\approx\frac{P_{pp}-1}{m+P_{pp}-1}. +$$ +So higher $m$ reduces bubbles but raises activation memory. + +## 4.1 Schedule anatomy +A full iteration has: +- warmup +- steady state +- drain + +Classic schedules include GPipe-like and 1F1B variants. Modern schedules reduce bubbles and manage memory explicitly. + +## 4.2 Correctness constraints +Because gradients are delayed across stages, update ordering and barriers matter: +- stage-consistent microbatch ordering +- synchronized optimizer step boundary +- deterministic accumulation semantics + +## 4.3 PP interactions +PP composes naturally with TP and DP but introduces additional point-to-point traffic and schedule complexity. + +--- + +## 5) Context/sequence parallelism for long context + +Long-context attention is often the dominant memory and bandwidth bottleneck. + +## 5.1 Ring and blockwise attention ideas +Many CP/ring methods avoid materializing full $S\times S$ attention globally by streaming KV blocks and using numerically stable online softmax. + +Given running stats $(m,l)$ and block stats $(m_b,l_b)$: +$$ +m' = \max(m,m_b),\qquad l' = l\,e^{m-m'} + l_b\,e^{m_b-m'}. +$$ +This preserves softmax normalization while processing blocks incrementally. + +## 5.2 Ulysses-style remapping +Ulysses-like approaches transform sequence sharding into head-local work via all-to-all, then reverse map. This changes communication shape and can improve compute locality depending on topology. + +## 5.3 Megatron context parallelism +CP shards context across ranks for broader network coverage than classic SP. It can unlock larger sequence lengths while balancing communication and memory. + +## 5.4 Dynamic CP +Adaptive CP degree by microbatch/context length can reduce wasted work on heterogeneous request lengths. + +--- + +## 6) Expert parallelism (MoE) + +MoE replaces one dense FFN with many experts and a router. + +For token representation $x$, router produces top-$k$ experts $\mathcal{E}_k(x)$ and weights $w_e(x)$. +Output: +$$ +y = \sum_{e\in\mathcal{E}_k(x)} w_e(x)\,\mathrm{Expert}_e(x). +$$ + +## 6.1 Distributed MoE flow +1. Router picks experts. +2. Tokens are permuted/packed by destination expert. +3. A2A dispatch sends token batches to owning ranks. +4. Local expert GEMMs run (often grouped GEMM). +5. A2A combine returns outputs. +6. Inverse permutation restores token order. + +## 6.2 Capacity and load balancing +Capacity factor limits tokens per expert; overflow may be dropped or rerouted depending on policy. Load imbalance is the central systems challenge. + +## 6.3 Performance drivers +- efficient token packing +- A2A bandwidth and overlap +- grouped/fused expert kernels +- routing stability and auxiliary balancing losses + +MoE wins when sparse activation savings exceed dispatch overhead. + +--- + +## 7) Hybrid mesh composition: how real systems are built + +Production training usually combines multiple axes: +- DP for batch scaling +- TP for width scaling +- PP for depth scaling +- CP/SP for long context and activation control +- EP for sparse capacity + +### 7.1 Composition heuristic +1. Fit model memory target first (choose sharding/offload). +2. Reach per-step throughput target (TP/PP tuning). +3. Add long-context strategy (SP/CP/ring). +4. Add MoE/EP if model architecture requires it. +5. Re-map axes to physical topology. + +### 7.2 Topology-aware rank mapping +Keep high-frequency, low-payload-sensitive collectives on fastest links. +- intra-node: TP, many EP patterns +- inter-node: DP replicas or less frequent sync paths + +### 7.3 Checkpointing and resharding +Checkpoint format must encode layout metadata to support restart on different meshes. Without robust reshard tooling, operational flexibility suffers. + +--- + +## 8) Inference parallelism and serving systems + +Training-optimal choices are not always serving-optimal. + +## 8.1 Continuous batching +Scheduler merges arriving requests into rolling decode batches. Gains throughput but needs careful fairness and tail-latency control. + +## 8.2 Paged KV cache +KV stored in paged blocks to reduce fragmentation and enable reuse/eviction policies for long-running services. + +## 8.3 Prefix/radix caching +Shared prompt prefixes are reused across requests to amortize prefill. + +## 8.4 Prefill/decode disaggregation +Separate hardware pools for prefill and decode can improve utilization because these phases have different compute/memory profiles. + +## 8.5 Speculative / multi-token decoding +Use draft proposals and verification to reduce average latency per accepted token. + +## 8.6 Serving-time TP/PP/EP/CP +Serving parallelism must optimize p50 and p99, not just raw throughput. KV movement and scheduler behavior often dominate. + +--- + +## 9) Cross-cutting kernel/runtime techniques + +These techniques amplify all parallel axes: +- CUDA graphs reduce launch overhead and jitter. +- Fused kernels raise arithmetic intensity. +- Low precision (FP8 and related) reduces bandwidth and memory, if scale handling is robust. +- Advanced attention kernels reduce IO and improve long-context efficiency. +- Overlap orchestration (streams/events) determines realized vs theoretical gains. + +A practical model: +$$ +T_{step}=\max(T_{critical\_compute},T_{critical\_comm})+T_{runtime\_overheads}. +$$ +Most engineering work reduces the critical path and overhead terms. + +--- + +## 10) Failure modes, diagnostics, and validation workflow + +Common failure modes: +- collective stalls from stragglers +- MoE hot experts +- PP bubbles larger than expected +- activation memory spikes from poor wrap/schedule +- long-tail latency in serving due to batching policy + +Diagnostics checklist: +1. per-rank timeline trace +2. link utilization and collective time breakdown +3. microbatch/bucket sweep +4. precision sweep with convergence checks +5. rank-map A/B test (topology sensitivity) + +Validation principles: +- verify numerical parity against simpler baseline (small scale) +- scale gradually and re-measure convergence + throughput + stability +- distinguish algorithmic improvements from kernel/runtime artifacts + +--- + +## 11) End-to-end decision framework + +Given model + hardware + objective (train speed, cost, serving latency), choose as follows: + +1. **Memory fit** + - If model fits comfortably: DDP baseline. + - If not: FSDP/ZeRO, then offload if still necessary. + +2. **Compute scaling** + - If layers too wide/slow per device: add TP. + - If depth too large for single rank efficiency: add PP. + +3. **Context scaling** + - For long contexts: add SP/CP/ring depending on attention strategy and interconnect. + +4. **Architecture scaling** + - For MoE models: add EP and prioritize A2A efficiency + load balance. + +5. **Topology optimization** + - Map axes to physical hierarchy, then retune bucket sizes, microbatch count, overlap. + +6. **Serving-specific rewrite** + - Re-optimize around KV cache, batching, and latency SLOs. + +--- + +## 12) What “good” looks like in production + +A strong production parallel stack has: +- explicit layout contracts per layer +- minimal unnecessary reshards +- stable overlap schedule +- predictable checkpoint/resume semantics +- robust observability on collectives, memory, and latency percentiles +- tunable policies for workload heterogeneity + +In practice, the best system is not one axis; it is a **co-designed hybrid** where algorithms, collectives, kernels, scheduler, and topology are optimized together. + +--- + +## 13) Source mapping +This paper is derived from the sectioned source map in `papers/parallelism_reference.md`, including PyTorch distributed/FSDP docs, Megatron-Core parallelism guidance, DeepSpeed ZeRO/ZeRO++ material, NCCL collective definitions, context/ring/Ulysses references, MoE/EP sources, and serving/runtime references listed there.