Skip to content

Wrong number of parameters for the normalization layers in the transformer block #37

Description

@pierremarchal

" layer_norm_parameter_count = 2 * embedding_dim\n",

Since there are two layers, shouldn't the number of parameters be:

layer_norm_parameter_count = 4 * embedding_dim

(layer_1 gamma + beta; layer_2 gamma + beta)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions