Hi,
Awesome work, thank you very much!
While skimming through VPT injections paths, I noticed that the VPT implementation for MoGe in moge/model/dinov2/models/vision_transformer.py allocates a separate set of prompt tokens per transformer block (deep VPT):
line 187-197: self.prompt_tokens = nn.Parameter(torch.empty(len(self.blocks)...)
but the forward pass hardcodes index 0:
line 333: x = torch.cat([x[:, :1], self.prompt_tokens[0].expand(x.shape[0], -1, -1), x[:, 1:]], dim=1)
so only the first layer's prompts are used.
The remaining per-layer prompt parameters never participate in the forward pass and therefore receive no gradients.
Since, in the paper you do not mention the type of VPT (deep/shallow) you are aiming for, I decided to notify you of this inconsistency/bug and maybe get your comment if this was intentional.
Cheers,
Ivan
Hi,
Awesome work, thank you very much!
While skimming through VPT injections paths, I noticed that the VPT implementation for MoGe in
moge/model/dinov2/models/vision_transformer.pyallocates a separate set of prompt tokens per transformer block (deep VPT):but the forward pass hardcodes index 0:
so only the first layer's prompts are used.
The remaining per-layer prompt parameters never participate in the forward pass and therefore receive no gradients.
Since, in the paper you do not mention the type of VPT (deep/shallow) you are aiming for, I decided to notify you of this inconsistency/bug and maybe get your comment if this was intentional.
Cheers,
Ivan