loss becomes NaN

[ohppy]

Hello, I recently read your paper and think your work is excellent. But I keep getting NaN loss when reproducing your project. Is this a normal issue? I asked GPT for fixes, and it suggested changing the original normalization approach, which I worry will hurt the final performance. Have you run into this NaN loss problem before? If yes, how did you fix it?

Traceback (most recent call last):
File "/home/ohppy/Downloads/CCL-FSOD-main/DeViT/tools/train_net.py", line 202, in
launch(
File "/home/ohppy/Downloads/CCL-FSOD-main/DeViT/tools/../detectron2/engine/launch.py", line 82, in launch
main_func(*args)
File "/home/ohppy/Downloads/CCL-FSOD-main/DeViT/tools/train_net.py", line 196, in main
return trainer.train()
File "/home/ohppy/Downloads/CCL-FSOD-main/DeViT/tools/../detectron2/engine/defaults.py", line 496, in train
super().train(self.start_iter, self.max_iter)
File "/home/ohppy/Downloads/CCL-FSOD-main/DeViT/tools/../detectron2/engine/train_loop.py", line 149, in train
self.run_step()
File "/home/ohppy/Downloads/CCL-FSOD-main/DeViT/tools/../detectron2/engine/defaults.py", line 506, in run_step
self._trainer.run_step()
File "/home/ohppy/Downloads/CCL-FSOD-main/DeViT/tools/../detectron2/engine/train_loop.py", line 287, in run_step
self._write_metrics(loss_dict, data_time)
File "/home/ohppy/Downloads/CCL-FSOD-main/DeViT/tools/../detectron2/engine/train_loop.py", line 329, in _write_metrics
raise FloatingPointError(
FloatingPointError: Loss became infinite or NaN at iteration=3969!
loss_dict = {'aux_bce_loss_0': np.float64(0.5806289315223694), 'aux_dice_loss_0': np.float64(0.26168298721313477), 'rg_l1_loss_0': np.float64(0.19794051576251567), 'aux_bce_loss_1': np.float64(0.2694101333618164), 'aux_dice_loss_1': np.float64(0.17516636848449707), 'rg_l1_loss_1': np.float64(0.09996911302159193), 'aux_bce_loss_2': np.float64(0.19801048934459686), 'aux_dice_loss_2': np.float64(0.14150327444076538), 'rg_l1_loss_2': np.float64(0.09891143087010972), 'aux_bce_loss_3': np.float64(0.19926844537258148), 'aux_dice_loss_3': np.float64(0.15015611052513123), 'rg_l1_loss_3': np.float64(0.08126689008605865), 'aux_bce_loss_4': np.float64(0.21137547492980957), 'aux_dice_loss_4': np.float64(0.1491101086139679), 'rg_l1_loss_4': np.float64(0.07804708551255915), 'focal_loss_0': np.float64(nan), 'focal_loss_1': np.float64(0.08443963527679443), 'focal_loss_2': np.float64(0.08257567882537842), 'bbox_loss': np.float64(0.7516565071981935)}

[RuoyuChen10]

Hello, I recently read your paper and think your work is excellent. But I keep getting NaN loss when reproducing your project. Is this a normal issue? I asked GPT for fixes, and it suggested changing the original normalization approach, which I worry will hurt the final performance. Have you run into this NaN loss problem before? If yes, how did you fix it?

Traceback (most recent call last): File "/home/ohppy/Downloads/CCL-FSOD-main/DeViT/tools/train_net.py", line 202, in launch( File "/home/ohppy/Downloads/CCL-FSOD-main/DeViT/tools/../detectron2/engine/launch.py", line 82, in launch main_func(*args) File "/home/ohppy/Downloads/CCL-FSOD-main/DeViT/tools/train_net.py", line 196, in main return trainer.train() File "/home/ohppy/Downloads/CCL-FSOD-main/DeViT/tools/../detectron2/engine/defaults.py", line 496, in train super().train(self.start_iter, self.max_iter) File "/home/ohppy/Downloads/CCL-FSOD-main/DeViT/tools/../detectron2/engine/train_loop.py", line 149, in train self.run_step() File "/home/ohppy/Downloads/CCL-FSOD-main/DeViT/tools/../detectron2/engine/defaults.py", line 506, in run_step self._trainer.run_step() File "/home/ohppy/Downloads/CCL-FSOD-main/DeViT/tools/../detectron2/engine/train_loop.py", line 287, in run_step self._write_metrics(loss_dict, data_time) File "/home/ohppy/Downloads/CCL-FSOD-main/DeViT/tools/../detectron2/engine/train_loop.py", line 329, in _write_metrics raise FloatingPointError( FloatingPointError: Loss became infinite or NaN at iteration=3969! loss_dict = {'aux_bce_loss_0': np.float64(0.5806289315223694), 'aux_dice_loss_0': np.float64(0.26168298721313477), 'rg_l1_loss_0': np.float64(0.19794051576251567), 'aux_bce_loss_1': np.float64(0.2694101333618164), 'aux_dice_loss_1': np.float64(0.17516636848449707), 'rg_l1_loss_1': np.float64(0.09996911302159193), 'aux_bce_loss_2': np.float64(0.19801048934459686), 'aux_dice_loss_2': np.float64(0.14150327444076538), 'rg_l1_loss_2': np.float64(0.09891143087010972), 'aux_bce_loss_3': np.float64(0.19926844537258148), 'aux_dice_loss_3': np.float64(0.15015611052513123), 'rg_l1_loss_3': np.float64(0.08126689008605865), 'aux_bce_loss_4': np.float64(0.21137547492980957), 'aux_dice_loss_4': np.float64(0.1491101086139679), 'rg_l1_loss_4': np.float64(0.07804708551255915), 'focal_loss_0': np.float64(nan), 'focal_loss_1': np.float64(0.08443963527679443), 'focal_loss_2': np.float64(0.08257567882537842), 'bbox_loss': np.float64(0.7516565071981935)}

Hello, thanks for attending to our work. Yes, I also met such a problem. I think this is due to the version of detectron2. You can follow the official devit's install instructions to fix the environment.