I am trying to reproduce the unlearning algorithms from scratch, currently fixated on reproducing the results in the RWKU paper.
I have three questions:
- For single-target training, after you train on a subject (say, Stephen King), do you train on the next subject using the trained model for the previous subject, or use a new model?
- For batch-target unlearning, I suppose you randomly select BS number of samples from S (say, 50) targets, and then in the next stage, do you randomly sample the targets with or without replacement?
- Did you try doing unlearning using GA on all the targets at once? I was trying to do that with a BS of 32 (BS 4 x gradient accumulation 8 times) and AdamW LR 1e-6, but the results are pretty bad. The forget set performance seems to be good, but not for the other subsets. If you have any insight into this, kindly share.
Also, I am trying to do it for Llama-3.2-1B-Instruct -- Is using the train_forget_llama3 subset valid for this case?
Also, for the forget and neighbour set evaluation, it is understandable that we take only the rows corresponding to the target subject or all the rows in the forget_level1, forget_level2, forget_level3, neighbour_level1 and neighbour_level3 subsets? Why, even for the utility_general subset, are there subject-wise samples?
I am trying to reproduce the unlearning algorithms from scratch, currently fixated on reproducing the results in the RWKU paper.
I have three questions:
Also, I am trying to do it for Llama-3.2-1B-Instruct -- Is using the train_forget_llama3 subset valid for this case?
Also, for the forget and neighbour set evaluation, it is understandable that we take only the rows corresponding to the target subject or all the rows in the forget_level1, forget_level2, forget_level3, neighbour_level1 and neighbour_level3 subsets? Why, even for the utility_general subset, are there subject-wise samples?