Does attention mask reduce computation cost？

[brotherb]

Hey, there.
After I read the code, I am confused that the computation cost can be reduced by mask more tokens. Did I miss anything?

PS. I see the FLOPS is calculated by the length of tokens retented at each layer which is counted during inference.

Do you have inference latency metrics?

[yaoyao926]

Did you solve it in the end?

[nachtsky1077]

I believe so, since FFN is computed in token level and its cost, hence masking the tokens in some of the layers could, at least, save the computation for FFN.