Questions on variable length input data construction

[bowenxieoregon]

Hi all,

I noticed that the book says, “We are training the model with training data presented in similarly sized chunks for simplicity and efficiency. However, in practice, it can also be beneficial to train an LLM with variable-length inputs to help the LLM better generalize across different types of inputs when it is being used.” in the Note section on page 142 of Chapter 5.

I cannot fully understand this, especially how to implement it. To utilize the GPU in the best way, shouldn't we always use batched inputs? And in this way, shouldn't every input in a batch be of the same length? One thing I can think of now is using [PAD] tokens.

[casinca]

Hi,

I believe you mostly answered your question. In CH5 for simplicity we use a fixed sliding window to have the same length within a batch but nothing stops you to have batches of different sequence lengths. For instance, you could change the size of the window for each batch.

Your intuition about padding is right and this is done in the next chapters CH6-CH7 iirc with batching variable length sequences.

[rasbt]

Thanks @casinca . Adding to that, modern training pipelines it's common to do a multi-stage pre-training. I.e., training with a fixed chunk size like 4k or 8k for a while. And then add the end switching to long context training to train on a subset of very long documents with a chunk size of like 128k tokens, for example