Can we say that it is a merger of Prompt tuning and LORA ?

[ambuje]

I am looking for something Prompt tuning + LORA, can we say that this is exactly what I am looking?

[ShiZhengyan]

Hi @ambuje, thanks for starting a discussion. I agree that our proposed method is a merger of Prompt tuning and LORA, but there are still some differences:

• Unlike LoRA, DePT avoids introducing trainable parameters at every layer.
• DePT applies a single low-rank adaptation directly to the token representations matrix, while LoRA focuses on projection matrices (e.g., query projection matrices).
• DePT uses a fewer number of trainable parameters compared to LoRA. For tasks that LMs already have learned a good understanding from the pretraining, the main challenge lies in crafting effective prompts, and it may not be necessary to use a large number of trainable parameters.

[ambuje]

Thank you for the reply.

To make things more clear (it might help others as well), we use low-rank metrics on Soft Prompt and then update it to the base model

[ShiZhengyan]

Thanks for your discussion. We employ low-rank matrices over word embeddings because word embeddings are fixed and non-trainable, and we aim to introduce some level of adaptability within these embeddings. Adding low-rank matrices may not be necessary for soft prompts, as the soft prompt is already trainable.

BTW, DePT can support not only generation tasks but also sequence classification tasks ☺️.