Stop scheduler before saving weight

[Paulo-21]

Hello,

Il would like to know if it's possible to "disable" the scheduler when we reach a number of epoch and then stay with a low decay to fine tune our NN before saving the weight ?

Because i assume that if we are in a phase where the optimizer try to not stay in a local optimium by incressing the decay maybe we will not save the best weight for our Neural Network when saving nn weights.

ty

[adefazio]

This can best be implemented by switching the optimizer mid-run to regular AdamW (or SGD) with a small LR, and calling .eval() before switching so it starts with the smoothed parameters. There isn't really a way to freeze the averaging mechanism in schedulefree, you could change the weighting to slow it down, but it's unclear if that would achieve what you want.