Hi MMaDA team,
Thank you very much for your excellent work.
I noticed that you currently use the EOS token (rather than the PAD token) for sequence padding (i.e., following the LLaDA behavior) in MMU part of design and that the loss is computed over these padded masked positions during training. For example, when a padded EOS token is randomly masked in the MMU stage (such as in your VQA data processing pipeline).
Could you please confirm whether this is the intended behavior? In addition, do you have any experimental evidence or ablation studies supporting this design choice?
Thank you in advance for any clarification.
Hi MMaDA team,
Thank you very much for your excellent work.
I noticed that you currently use the EOS token (rather than the PAD token) for sequence padding (i.e., following the LLaDA behavior) in MMU part of design and that the loss is computed over these padded masked positions during training. For example, when a padded EOS token is randomly masked in the MMU stage (such as in your VQA data processing pipeline).
Could you please confirm whether this is the intended behavior? In addition, do you have any experimental evidence or ablation studies supporting this design choice?
Thank you in advance for any clarification.