Question about mask in CausalAttention chapter 3

[lehoanglong95]

First of all, thank you for wonderful book!

For section 3.5 - hiding future words with causal attention, I understand that the mask should be lower triangular matrix. So the code for mask should be:

self.register_buffer("mask", torch.tril(torch.ones(context_length, context_length), diagonal=1))

ATM, the code for mask is:

self.register_buffer("mask", torch.triu(torch.ones(context_length, context_length), diagonal=1)) # upper triangular matrix

Do I understand it correctly?

Thanks so much

[lehoanglong95]

Sorry my bad. Author use self.mask.bool().

Everything is correct.