[Paper Suggestion]: Large Language Diffusion Models

[NicklasOsterbacka]

Paper Title

Large Language Diffusion Models

Paper Link

https://arxiv.org/abs/2502.09992v2

Brief Summary

Autoregressive models (ARMs) are widely regarded as the cornerstone of large language models (LLMs). We challenge this notion by introducing LLaDA, a diffusion model trained from scratch under the pre-training and supervised fine-tuning (SFT) paradigm. LLaDA models distributions through a forward data masking process and a reverse process, parameterized by a vanilla Transformer to predict masked tokens. By optimizing a likelihood bound, it provides a principled generative approach for probabilistic inference. Across extensive benchmarks, LLaDA demonstrates strong scalability, outperforming our self-constructed ARM baselines. Remarkably, LLaDA 8B is competitive with strong LLMs like LLaMA3 8B in in-context learning and, after SFT, exhibits impressive instruction-following abilities in case studies such as multi-turn dialogue. Moreover, LLaDA addresses the reversal curse, surpassing GPT-4o in a reversal poem completion task. Our findings establish diffusion models as a viable and promising alternative to ARMs, challenging the assumption that key LLM capabilities discussed above are inherently tied to ARMs.

Relevance to Group

Text diffusion is getting more attention lately (see DeepMind's Gemini Diffusion and Nvidia's GenMol). This paper demonstrates that it is possible to build a strong LLM based on the technique.

Voting Instructions (do not edit)

---
## 🗳️ Voting Instructions
To vote for this paper, **click the ↑ upvote button in the top-left corner of this discussion post**
We'll prioritize papers with the most votes!  

↓ Right here