Add comprehensive whitepaper: End-to-End Parallelisms for LLMs

[dzmitry-huba]

Motivation

• Provide a single, self-contained study guide that consolidates notation, cost/memory models, primitives, and practical recipes for data/tensor/pipeline/context/expert parallelism and their composition.
• Give engineers and researchers a canonical reference mapping concepts to collectives, topologies, failure modes, diagnostics, and real-world case patterns for LLM training and serving.

Description

• Add new file papers/parallelisms_whitepaper.md containing an exhaustive guide (~6000 lines) that unifies symbols and notation, introduces a shared system model, and documents primitives (AR, AG, RS, A2A, Send/Recv) and cost models.
• Cover axis-specific mechanics for Data Parallelism (DDP/FSDP/ZeRO/HSDP), Tensor Parallelism (column/row TP, TP-aware kernels), Pipeline Parallelism (schedules, microbatching, PipeOffload), Sequence/Context Parallelism, and Expert/MoE parallelism with routing, dispatch, and capacity considerations.
• Include cross-cutting CUDA and platform techniques (streams, CUDA graphs, attention kernels, NVSHMEM), inference-serving primitives (KV cache, prefill/paged caching, speculative decoding), decision matrices, pseudocode, diagnostics, failure modes, and multiple end-to-end case studies and lineage pointers to papers/parallelism_reference.md.

Testing

• No automated tests were added or executed as part of this documentation-only addition; downstream CI/documentation build should validate rendering and link integrity.

---

Codex Task