The root of this is that if you instantiate your model with, say, a context length of 128, and then you have a VQA task where images get embedded as a 16x16 block of patches, then you've got more image tokens than you have in your context length. And since we don't predict image tokens, they all get masked as 0, which eventually leads to dividing by 0 in the loss calculation which leads to NaNs in your losses.
[rank0]: File "/home/eihli/src/mugato/mugato/utils.py", line 105, in generic_collate_fn
[rank0]: raise ValueError(
[rank0]: ValueError: No samples in batch could fit within sequence_length=2048. Consider increasing block_size or using datasets with smaller episodes.
No reason to have a training run entirely stop when this happens. Let's just turn this into a noop.
The root of this is that if you instantiate your model with, say, a context length of 128, and then you have a VQA task where images get embedded as a 16x16 block of patches, then you've got more image tokens than you have in your context length. And since we don't predict image tokens, they all get masked as 0, which eventually leads to dividing by 0 in the loss calculation which leads to NaNs in your losses.
No reason to have a training run entirely stop when this happens. Let's just turn this into a noop.