Posting this as a new issue in case anybody else has the problem.
There might be hacky workarounds for this but it seems to be a pytorch issue: pytorch/pytorch#40497
So I guess the answer is to train with fp16_run: false for now. Might want to note this in the README to save people time. Also the config file default is set to true, could probably set that to false.
Posting this as a new issue in case anybody else has the problem.
There might be hacky workarounds for this but it seems to be a pytorch issue: pytorch/pytorch#40497
So I guess the answer is to train with fp16_run: false for now. Might want to note this in the README to save people time. Also the config file default is set to true, could probably set that to false.