feat: add MiRU1Cell, MiRU1, MiRU2Cell and MiRU2 implementations - #45
Conversation
Signed-off-by: 🄂ʏᴇᴅ 🄰ʙᴅᴜʟ 🄰ᴍᴀ🄝 ✧ <amanbaba9404522@gmail.com>
| self.lambda_ = nn.Parameter( | ||
| torch.full((hidden_size,), 0.5, device=device, dtype=dtype) | ||
| ) |
There was a problem hiding this comment.
lambda is not a learnable parameter. also call it just lambda
| candidate = self.nonlinearity(ch) | ||
|
|
||
| # Update: h(t) = lambda * h + (1 - lambda) * h~ | ||
| lam = torch.sigmoid(self.lambda_) |
There was a problem hiding this comment.
why is the parameter passed through a nonlinearity?
| self.lambda_ = nn.Parameter( | ||
| torch.full((hidden_size,), 0.5, device=device, dtype=dtype) | ||
| ) | ||
| self.theta = nn.Parameter( | ||
| torch.full((hidden_size,), 0.5, device=device, dtype=dtype) | ||
| ) |
There was a problem hiding this comment.
not learnable parameters, they are hyperparameters of the cell
| inp, state, is_batched = self._preprocess_input_and_state(inp, state) | ||
|
|
||
| # Candidate: h~(t) = tanh(W_h x + b_h + U_h (theta * h)) | ||
| theta = torch.sigmoid(self.theta) |
There was a problem hiding this comment.
theta is not passed through an activation function in the paper definition
| candidate = self.nonlinearity(ch) | ||
|
|
||
| # Update: h(t) = lambda * h + (1 - lambda) * h~ | ||
| lam = torch.sigmoid(self.lambda_) |
There was a problem hiding this comment.
lambda is not passed through an activation function in the paper definition
|
There are a couple of points to address before merging. also make sure to add the cell and the model to the documentation |
Signed-off-by: 🄂ʏᴇᴅ 🄰ʙᴅᴜʟ 🄰ᴍᴀ🄝 ✧ <amanbaba9404522@gmail.com>
Signed-off-by: 🄂ʏᴇᴅ 🄰ʙᴅᴜʟ 🄰ᴍᴀ🄝 ✧ <amanbaba9404522@gmail.com>
|
"Hi @MartinuzziFrancesco, I've addressed all the review comments: lambda_ and theta are now plain float hyperparameters (not nn.Parameter) Could you also let me know where to add the documentation? Happy to add it!" |
| Default: True | ||
| recurrent_bias: If ``False``, the layer does not use recurrent biases. | ||
| Default: True | ||
| lambda_: Mixing hyperparameter controlling the blend between the |
| Default: ``True``. | ||
| recurrent_bias: If ``False``, the layer does not use recurrent biases. | ||
| Default: ``True``. | ||
| lambda_: Mixing hyperparameter controlling the blend between the |
| "hidden_size", | ||
| "bias", | ||
| "recurrent_bias", | ||
| "lambda_", |
| hidden_size: int, | ||
| bias: bool = True, | ||
| recurrent_bias: bool = True, | ||
| lambda_: float = 0.5, |
| super(MiRU1Cell, self).__init__( | ||
| input_size, hidden_size, bias, recurrent_bias, device=device, dtype=dtype | ||
| ) | ||
| self.lambda_ = lambda_ |
| candidate = self.nonlinearity(ch) | ||
|
|
||
| # Update: h(t) = lambda * h + (1 - lambda) * h~ | ||
| new_state = self.lambda_ * state + (1.0 - self.lambda_) * candidate |
|
"Hi @MartinuzziFrancesco, lambda is a reserved keyword in Python and cannot be used as a variable or parameter name. lambda_ is the standard convention (used by PyTorch itself, e.g. lr_lambda). Could we keep lambda_ for this reason?" |
|
"Also, could you point me to an existing cell's doc page so I can follow the same format for MiRU?" |
Signed-off-by: 🄂ʏᴇᴅ 🄰ʙᴅᴜʟ 🄰ᴍᴀ🄝 ✧ <amanbaba9404522@gmail.com>
|
"Hi @MartinuzziFrancesco, I've added the documentation: MiRU1 and MiRU2 added to docs/models.rst (alphabetically after MGU) Also regarding lambda_ — since lambda is a reserved keyword in Python, I've kept it as lambda_ following standard convention. Please let me know if you'd prefer a different name!" |
yeha good point I completely forgot about that. I also think that we could use more expressive names, so following the paper I would use |
Signed-off-by: 🄂ʏᴇᴅ 🄰ʙᴅᴜʟ 🄰ᴍᴀ🄝 ✧ <amanbaba9404522@gmail.com>
|
"Hi @MartinuzziFrancesco, I've renamed lambda_ to update_coefficient and theta to reset_coefficient following the paper's naming convention. The math equations in the docstrings still use \lambda and \theta as per the paper." |
0efbdea
into
MartinuzziFrancesco:main
2 participants
Closes #44
Summary
Implements the Minion Gated Unit (MiRU) from:
https://doi.org/10.1016/j.neucom.2026.132847
Changes
MiRU1CellandMiRU1— variant with reset gater(t)MiRU2CellandMiRU2— variant with learnedθparametercells/__init__.pyandtorchrecurrent/__init__.pyEquations
MiRU1:
r(t) = σ(Wr x(t) + br + Ur h(t-1))
h~(t) = tanh(Wh x(t) + bh + Uh(r(t) ⊙ h(t-1)))
h(t) = λ ⊙ h(t-1) + (1-λ) ⊙ h~(t)
MiRU2:
h~(t) = tanh(Wh x(t) + bh + Uh(θ ⊙ h(t-1)))
h(t) = λ ⊙ h(t-1) + (1-λ) ⊙ h~(t)