Add & Norm

[xuwenshen]

normalization seems different from the paper #attention is all you need#

in paper, normalization layer stays after mha and feed forward layer, in torchnlp, it stays before them

    x = inputs
    
    # Layer Normalization
    x_norm = self.layer_norm_mha(x)
    
    # Multi-head attention
    y = self.multi_head_attention(x_norm, x_norm, x_norm)
    
    # Dropout and residual
    x = self.dropout(x + y)
    
    # Layer Normalization
    x_norm = self.layer_norm_ffn(x)
    
    # Positionwise Feedforward
    y = self.positionwise_feed_forward(x_norm)
    
    # Dropout and residual
    y = self.dropout(x + y)

[kolloldas]

Yes it's from the updated Transformer model. You can find the Tensorflow version maintained by the Authors here