https://github.com/sooftware/attentions/blob/b1dd65be8f12fbe19525bc4ae0dbbc14975778a7/attentions.py#L286 It seems the mask is not correct. Since there is a permute of query, key, and value. The mask should also has a permute.
attentions/attentions.py
Line 286 in b1dd65b
It seems the mask is not correct. Since there is a permute of query, key, and value. The mask should also has a permute.