Decentralized LLMs @ ICML 2026

A curated list of ICML 2026 (main conference) papers on training and serving large language models in a decentralized / collaborative dfdf way.

Full author lists and abstracts are in papers.md.

In scope: decentralized/distributed methods for training across the full pipeline (pretraining, mid-training, SFT, RL) and for serving/inference. Out of scope: downstream task-specific fine-tuning / adaptation, single-node-only efficiency tricks, and datacenter tensor/pipeline parallelism that assumes high-bandwidth interconnects.

This list was compiled with the help of Claude: candidate papers were surfaced from the ICML 2026 accepted-paper metadata, summarized, and curated to the scope above. Summaries and categorization may contain errors — corrections and additions welcome (see Contributing).

Legend: 🎤 oral · 📍 poster session & board number. Title links go to OpenReview; [abstract] links to the abstract + full author list in papers.md. Sections ordered roughly by closeness to the topic's core.

Contents

• Low-communication & decentralized training
• Asynchronous, fault-tolerant & distributed post-training
• Decentralized & distributed optimization (methods & theory)
• Communication-efficient & low-precision training
• Distributed & disaggregated serving / inference
• Position & vision
• Contributing
• License

Low-communication & decentralized training

• Decentralized Instruction Tuning: Conflict-Aware Splitting and Weight Merging — Splits instruction-tuning data by gradient conflict, fine-tunes each partition communication-free, then merges once by weighted averaging.
  _{Minsik Choi et al. · 📍 Poster Session 1, #2107 · abstract}
• Factored Gossip DiLoCo: Reducing Blocking Communication within DiLoCo — Factorizes DiLoCo's outer sync into a non-blocking gossip step plus a blocking mixing step; degrades gracefully under stragglers and comm failures.
  _{Chamin Hewa Koneputugodage et al. · 📍 Poster Session 1, #3811 · abstract}
• LoRDO: Distributed Low-Rank Optimization with Infrequent Communication — Unifies low-rank optimization with infrequent communication for distributed foundation-model training, cutting communication ~10×.
  _{Andrej Jovanović et al. · 📍 Poster Session 3, #3705 · abstract}
• MuLoCo: Muon is a Practical Inner Optimizer for DiLoCo — Uses Muon as DiLoCo's inner optimizer, yielding more directionally-correct pseudogradients and beating DiLoCo across 150M–3.1B models.
  _{Benjamin Thérien et al. · 📍 Poster Session 5, #3600 · abstract}

Asynchronous, fault-tolerant & distributed post-training

Asynchronous pipeline parallelism

• Mitigating Staleness in Asynchronous Pipeline Parallelism via Basis Rotation — Rectifies stale gradients in asynchronous pipeline parallelism via basis rotation, restoring scalable async training (1B LLM).
  _{Hyunji Jung et al. · 📍 Poster Session 1, #3713 · abstract}
• One-Step Gradient Delay is Not a Barrier for Large-Scale Asynchronous Pipeline Parallel LLM Pretraining — Shows one-step-delay async pipeline (PipeDream-2BW) trains well with Muon + an error-feedback fix, up to 10B parameters.
  _{Philip Zmushko et al. · 📍 Poster Session 1, #3610 · abstract}

Fault tolerance at scale

• Identifying and Mitigating Errors in Gradient Aggregation of Distributed Data Parallel Training — Detects and mitigates silent data corruption in distributed data-parallel gradient aggregation via dynamic asynchronous parameter sync.
  _{Zhenheng Tang et al. · 📍 Poster Session 4, #605 · abstract}
• SPARe: Stacked Parallelism with Adaptive Reordering for Fault-Tolerant LLM Pretraining Systems with 100k+ GPUs — Fault tolerance for 100k+-GPU LLM pretraining: stacked-parallelism redundancy and adaptive reordering mask node failures at near-constant overhead.
  _{Jin Lee et al. · 📍 Poster Session 5, #4101 · abstract}

Distributed RL post-training

• D-ARL: A Distribution-Matched Asynchronous Reinforcement Learning Framework for Language Reasoning — Distribution-matched asynchronous RL for LLM post-training: selects stale rollout samples well-aligned with the current policy.
  _{白 寅岐 et al. · 📍 Poster Session 1, #2111 · abstract}
• DistFlow: A Fully Distributed RL Framework for Scalable and Efficient LLM Post-Training — Fully distributed, multi-controller RL framework for LLM post-training; decouples data from control, scaling near-linearly to 512 GPUs.
  _{zhixin wang et al. · 📍 Poster Session 1, #3809 · abstract}

Decentralized & distributed optimization (methods & theory)

Decentralized SGD: convergence & topology

• Accelerated Dual Method for Distributed Optimization: An Inexact-Gradient View of Local Updates — Accelerated dual method with multi-step local SGD; attains optimal communication complexity when local computation suffices.
  _{Junchi Yang et al. · 📍 Poster Session 4, #3800 · abstract}
• High-Probability Convergence Guarantees of Decentralized SGD — First high-probability convergence guarantees (and linear speed-up) for Decentralized SGD under light-tailed noise.
  _{Aleksandar Armacki et al. · 📍 Poster Session 1, #3815 · abstract}
• Improved Convergence Analysis of Topology Dependence in Decentralized SGD — Tighter analysis showing all mixing-matrix eigenvalues — not just the spectral gap — govern Decentralized SGD convergence.
  _{Yuki Takezawa et al. · 📍 Poster Session 7, #3713 · abstract}

Communication compression & error-feedback

• On the Convergence of Decentralized Stochastic Minimax Optimization Algorithm with Compressed Communication — Communication-efficient decentralized stochastic minimax (SGDA) with an error-feedback compression mechanism and convergence guarantees.
  _{Yihan Zhang et al. · 📍 poster TBD · abstract}
• On the Interaction of Batch Noise, Adaptivity, and Compression, under $(L_0,L_1)$-Smoothness: An SDE Approach — An SDE framework for distributed compressed / sign SGD under (L0,L1)-smoothness and heavy-tailed noise.
  _{Enea Monzio Compagnoni et al. · 📍 Poster Session 4, #3600 · abstract}
• Temporal Difference Learning with Compressed Updates: Error-Feedback meets Reinforcement Learning — Brings error-feedback gradient compression to TD/Q-learning; multi-agent linear speed-up while communicating ~O(1) bits per iteration.
  _{Aritra Mitra et al. · 📍 Poster Session 3, #118 · abstract}

Distributed bilevel / minimax / multi-level

• Convergence Analysis of Decentralized Hessian-/Jacobian-Free Algorithm for Nonconvex Stochastic Bilevel Optimization — First-order decentralized algorithm for nonconvex stochastic bilevel optimization under the PL condition (no Hessians/Jacobians).
  _{Yihan Zhang et al. · 📍 poster TBD · abstract}
• Distributed Stochastic $K$-Level Optimization Over Networks — Decentralized variance-reduced algorithm for stochastic K-level (K>2) optimization over networks.
  _{Xinwen Zhang et al. · 📍 poster TBD · abstract}
• FAB: A First-Order AB-based Gradient Algorithm for Distributed Bilevel Optimization over Time-Varying Directed Graphs — First-order Push–Pull (AB) algorithm for distributed bilevel optimization over time-varying directed graphs.
  _{Yaoshuai Ma et al. · 📍 Poster Session 1, #3810 · abstract}

Decentralized unlearning

• Trajectory-Aware Certified Decentralized Unlearning via SGD Stability — Certified unlearning of a client's influence from a decentralized-trained model via SGD-stability sensitivity analysis.
  _{Hengliang Wu et al. · 📍 Poster Session 7, #3100 · abstract}

Communication-efficient & low-precision training

Low-precision / fully-quantized training

• AGoQ: Activation and Gradient Quantization for Memory-Efficient Distributed Training of LLMs — 4-bit activation + 8-bit gradient quantization with precision-preserving 8-bit all-reduce for memory-efficient distributed LLM training.
  _{WenXiang Lin et al. · 📍 Poster Session 2, #2402 · abstract}
• Clover: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimation — Fully-NVFP4 LLM pretraining with an unbiased micro-scaled quantizer (>2× lower error than stochastic rounding).
  _{Andrei Panferov et al. · 📍 Poster Session 3, #2901 · abstract}
• ECO: Quantized Training without Full-Precision Master Weights — Quantized training without high-precision master weights: injects quantization error into optimizer momentum (error-feedback).
  _{Mahdi Nikdan et al. · 📍 Poster Session 6, #3410 · abstract}
• TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Control — End-to-end NVFP4 LLM training with oscillation suppression and outlier control, halving the accuracy gap to BF16.
  _{Yuxiang Chen et al. · 📍 Poster Session 3, #2809 · abstract}
• WinQ: Accelerating Quantization-Aware Training of Large Language Models around Saddle Points — Accelerates quantization-aware LLM training by escaping saddle points via weight interpolation and noise injection.
  _{Dongyue Li et al. · 📍 Poster Session 5, #1703 · abstract}

Communication-efficient optimizers

• AdaGC: Enhancing LLM Pretraining Stability via Adaptive Gradient Clipping — Adaptive per-tensor gradient clipping that stabilizes LLM pretraining and lowers communication vs global clipping.
  _{Guoxia Wang et al. · 📍 Poster Session 3, #2311 · abstract}
• BAS: Bridging Adam and SignSGD for Memory-Efficient LLM Training — Block-Adaptive Signum bridging Adam and SignSGD: low optimizer-state memory plus a communication-efficient variant.
  _{Yijie Zhou et al. · 📍 Poster Session 7, #3910 · abstract}
• Softsignum: Smooth Your Signum For Better Heterogeneity Handling — A smooth sign→SGD transition that handles parameter heterogeneity, with convergence guarantees incl. LLM pretraining.
  _{Dmitrii Feoktistov et al. · 📍 Poster Session 4, #3708 · abstract}

Communication-efficient MoE parallelism

• Multi-Head LatentMoE and Head Parallel: Communication-Efficient and Deterministic MoE Parallelism — MoE parallelism with O(1), fully-balanced, deterministic communication regardless of the number of activated experts.
  _{Chenwei Cui et al. · 📍 Poster Session 4, #2010 · abstract}

Distributed & disaggregated serving / inference

Disaggregated & distributed serving

• EasyBalance: Cross-Layer Load Balancing in Distributed MoE Inference — Cross-layer load balancing for expert-parallel distributed MoE inference, with no expert-device remapping.
  _{Yize Wu et al. · 📍 Poster Session 7, #1912 · abstract}
• Efficient Multi-round LLM Inference over Disaggregated Serving — Disaggregated serving tuned for multi-round / agentic workloads via adaptive prefill placement and scheduling.
  _{Wenhao He et al. · 📍 Poster Session 3, #3706 · abstract}
• HexGen-3: A Fully Disaggregated LLM Serving Framework with Fine-Grained Heterogeneous Resource Autoscaling — Fully disaggregated LLM serving with fine-grained heterogeneous-resource autoscaling.
  _{Youhe Jiang et al. · 📍 Poster Session 2, #3707 · abstract}
• Not All Prefills Are Equal: PPD Disaggregation for Multi-turn LLM Serving — Prefill-capable decode nodes reuse cached KV for multi-turn serving, cutting KV-transfer congestion.
  _{Zongze Li et al. · 📍 Poster Session 5, #4003 · abstract}
• Semantic Cache Distillation: Efficient State Transfer via Reuse and Selective Patching — Replaces raw KV-cache transfer in disaggregated serving with compact semantic codes (up to 2.6× less transfer).
  _{Qianli Ma et al. · 📍 Poster Session 2, #1906 · abstract}

Edge–cloud collaborative inference

• Bridging On-Device and Cloud LLMs for Collaborative Reasoning: A Unified Methodology for Local Routing and Post-Training — An on-device LLM learns via RL when to offload a query to a cloud LLM for collaborative reasoning.
  _{Wenzhi Fang et al. · 📍 Poster Session 5, #2613 · abstract}
• HybridFlow: Resource-Adaptive Subtask Routing for Efficient Edge-Cloud LLM Inference — Edge–cloud LLM inference with dependency-aware subtask routing for parallel, budget-adaptive execution.
  _{Jiangwen Dong et al. · 📍 Poster Session 1, #3816 · abstract}

Low-bandwidth multi-device inference

• ASTRA: Communication-Efficient Acceleration for Multi-Device Transformer Inference — Communication-efficient multi-device transformer inference at bandwidths as low as 10 Mbps via quantized token transfer.
  _{Xiao Liu et al. · 📍 poster TBD · abstract}
• CoCoQuant: Breaking the Bandwidth Wall via Co-Optimized Communication and Computation Quantization — Co-optimizes communication and computation quantization to break the bandwidth wall in distributed LLM inference.
  _{Haojie Duanmu et al. · 📍 Poster Session 8, #2701 · abstract}

Position & vision

• Position: Enabling Fair Revenue Sharing for Data Providers in GenAI Systems — Calls for decentralized, contribution-aware revenue sharing with data providers in GenAI systems.
  _{Gengrui (Edward) Zhang · 📍 Poster Session 6, #3308 · abstract}
• Position: Federated Learning is a Lens towards a Democratized Future for the Scaling Law Era — Argues federated learning still points toward a decentralized, democratized ML future in the scaling-law era.
  _{Harry Jiang et al. · 📍 Poster Session 1, #3308 · abstract}
• Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpowered — Argues zeroth-order optimization is underexplored; its forward-only updates suit communication-efficient, pipeline-friendly training.
  _{Sijia Liu et al. · 📍 Poster Session 6, #3705 · abstract}

Contributing

Contributions welcome!

• Is your ICML 2026 paper missing? If it fits the scope above but isn't listed, please open a PR to add it — this list was curated semi-automatically (with Claude) from the accepted-paper metadata and may well have missed papers. No need to be shy: if it belongs here, we'd like it here.
• Corrections to descriptions, categories, poster sessions, or board numbers are equally welcome.
• Format: a one-line entry in README.md (- [Title](OpenReview link) — description. <br><sub>First Author et al. · 📍 poster location · [abstract](papers.md#anchor)</sub>) plus a matching section in papers.md with the full author list and abstract.

License

— to the extent possible under law, contributors have waived all copyright to this list. Paper titles, abstracts, and metadata belong to their respective authors; conference metadata is sourced from ICML 2026. Compiled with assistance from Claude (Anthropic).