Hello, thanks for the great work.
Reading Table 1 and 2 I noticed overall good performance for all methods, it's particularly surprising to see GA performing so well in forgetting while maintaining all the utility accuracy. Being unbounded GA is notoriously unstable, but skimming thru the paper and code you don't seem to be using any regularization (retain/general sft or KL), I would expect GA to diverge very quickly when left unchecked, did I miss something here?
BR!
Hello, thanks for the great work.
Reading Table 1 and 2 I noticed overall good performance for all methods, it's particularly surprising to see GA performing so well in forgetting while maintaining all the utility accuracy. Being unbounded GA is notoriously unstable, but skimming thru the paper and code you don't seem to be using any regularization (retain/general sft or KL), I would expect GA to diverge very quickly when left unchecked, did I miss something here?
BR!