[Reading Group]: Large Language Diffusion Models

[nicaudinet]

📆 Date & Time

Friday 29th of August at 13:00

📍 Location or Link

CSE EDIT 5128, https://chalmers.zoom.us/j/68241194766

📄 Paper Link

https://arxiv.org/abs/2502.09992v2

📖 Curator

Nicklas Osterbacka

🗒️ Agenda

Paper Summary

Autoregressive models (ARMs) are widely regarded as the cornerstone of large language models (LLMs). We challenge this notion by introducing LLaDA, a diffusion model trained from scratch under the pre-training and supervised fine-tuning (SFT) paradigm. LLaDA models distributions through a forward data masking process and a reverse process, parameterized by a vanilla Transformer to predict masked tokens. By optimizing a likelihood bound, it provides a principled generative approach for probabilistic inference. Across extensive benchmarks, LLaDA demonstrates strong scalability, outperforming our self-constructed ARM baselines. Remarkably, LLaDA 8B is competitive with strong LLMs like LLaMA3 8B in in-context learning and, after SFT, exhibits impressive instruction-following abilities in case studies such as multi-turn dialogue. Moreover, LLaDA addresses the reversal curse, surpassing GPT-4o in a reversal poem completion task. Our findings establish diffusion models as a viable and promising alternative to ARMs, challenging the assumption that key LLM capabilities discussed above are inherently tied to ARMs.

Relevance to Group

Text diffusion is getting more attention lately (see DeepMind's Gemini Diffusion and Nvidia's GenMol). This paper demonstrates that it is possible to build a strong LLM based on the technique.

[nicaudinet]

Just finished reading the paper ... really interesting and well-written! It felt like there was a lot of careful effort put behind these results.

Some mild concerns:

• How computationally expensive is inference compared to ARM models?
• How well does LLaDA scale with context size?
• How much of a difference does increasing $t$ make on the quality of the generated text?

Some more open-ended questions:

• Does reasoning work (e.g. with their proposed semi-autoregressive decoding)?
• Re-masking tokens in intermediate steps feels like a bit of a waste ... we generated a probability then throw it away. Could it be better to have a weighted sum of the probabilities from each step favouring the ones closer to $t=0$?
• Intuitively I would think that humans write word-by-word from start to finish and then edit the text, with maybe some planning done before / while writing. I would also think that a model that is more similar in structure to the DGP will be better at learning the DGP. So it's quite surprising to me that LLaDA works as well as it does! Does it mean that the first or second statements are incorrect (or both)?
• What would we expect LLaDA to be better or worse at in comparison to ARM models?

[ricj]

humans write word-by-word from start to finish and then edit the text

I'm not particularly well-read in psycholinguistics, but it would be interesting to know what the literature says about this! My own hunch would be that the human speech process is rather different from the writing process in this respect. It seems to me that the speech process is at least somewhat autoregressive (i.e. we may start a sentence without knowing exactly where we'll end up). Maybe the semi-autoregressive decoding algorithm would be a good model of the speech process? For writing, I agree that the underlying process should take editing into account.

a model that is more similar in structure to the DGP will be better at learning the DGP

Yes that is also interesting to discuss. The authors make some arguments (building on earlier papers) that in the end, the MDM minimizes the same loss as the ARM, so maybe asymptotically they can be considered equivalent. But then on the other hand they also argue that the MDM are better at some tasks, so for realistically sized datasets there will be some priors/biases that make the models behave differently in practice.

Philosophical point: will models that are more human-like learn language more easily?

[m3hrdadfi]

I did my best to read this paper at a surface level, so my concerns might not be entirely accurate.

Concerns:

... LLaDA currently exhibits sensitivity to inference hyperparameters ...

As they mention in the study, so if remasking is poorly tuned, that could mean we might loop nonsense forever. In that case, would the ARMs be more predictable?

"...no specialized attention mechanisms or position embeddings were designed for LLaDA..."

ARMs have greatly benefited from those architectures. Without them, I'm not sure LLaDA would be effective or efficient with long contexts or, in particular, for reasoning. The reason I'm somewhat speculating about the task is that, to some extent, models must maintain dependencies across many tokens so they don't face forgetting, repeating, or clues being spread far apart.

[ricj]

ARMs have greatly benefited from those architectures

It's always a bit tricky to compare a completely new technology to one where the field has had several years to refine the best practices.

About ARMs: what are the specific architectural innovations you are thinking of? It seems to me that most innovations try to deal with scalability issues. (Such as Group Query Attention being more effient than standard MHA.)

Would be interesting to think of what efficiency tricks could be applied to the MDM. I can't think of any caching approach similary to KV caching for ARMs.

[NicklasOsterbacka]

Super interesting stuff!

1. The authors state that sequential token-by-token generation incurs high computational costs, and then go on to benchmark LLaDA by setting the number of sampling steps to be equal to the answer length. That sounds more expensive than just generating the sequence autoregressively.
2. To follow up on that, it would be interesting to see how LLaDA holds up in an inference compute matched setting.
3. In Table 1, the number of shots differ from the experiments taken from literature. Why?
4. Table 3 shows that LLaDA is marginally better at the reverse poem task, but significantly worse at the forward poem task. Are text diffusion models simply worse at the latter task?

[nicaudinet]

Thank you for the great discussion!