grad_norm: nan

[coebangalore93]

Hi Authors,

I am training the PETR model with the config PETR-vov-p4-1600x640, on a single NVIDIA RTX 6000 Ada Generation with 48GB VRAM with a peak usage of 7.5 GB peak usage out of the 48GB. during epoch 1 training i saw the following behavior

2026-04-01 16:18:01,524 - mmdet - INFO - Epoch [1][9100/28130] lr: 2.000e-04, eta: 4 days, 13:15:14, time: 0.586, data_time: 0.011, memory: 7496, loss_cls: 1.5314, loss_bbox: 1.4246, d0.loss_cls: 1.5302, d0.loss_bbox: 1.4339, d1.loss_cls: 1.5307, d1.loss_bbox: 1.4190, d2.loss_cls: 1.5305, d2.loss_bbox: 1.4277, d3.loss_cls: 1.5316, d3.loss_bbox: 1.4359, d4.loss_cls: 1.5296, d4.loss_bbox: 1.4229, loss: 17.7481, grad_norm: 74.8552
/home/shrey/Documents/naveeen_amrita/my_env/lib/python3.8/site-packages/mmcv/runner/hooks/optimizer.py:31: FutureWarning: Non-finite norm encountered in torch.nn.utils.clip_grad_norm_; continuing anyway. Note that the default behavior will change in a future release to error out if a non-finite total norm is encountered. At that point, setting error_if_nonfinite=false will be required to retain the old behavior.
return clip_grad.clip_grad_norm_(params, **self.grad_clip)
2026-04-01 16:18:30,110 - mmdet - INFO - Epoch [1][9150/28130] lr: 2.000e-04, eta: 4 days, 13:13:36, time: 0.572, data_time: 0.011, memory: 7496, loss_cls: 1.4051, loss_bbox: 1.5441, d0.loss_cls: 1.4125, d0.loss_bbox: 1.5319, d1.loss_cls: 1.4115, d1.loss_bbox: 1.5443, d2.loss_cls: 1.4020, d2.loss_bbox: 1.5495, d3.loss_cls: 1.4087, d3.loss_bbox: 1.5399, d4.loss_cls: 1.3985, d4.loss_bbox: 1.5450, loss: 17.6930, grad_norm: nan
2026-04-01 16:18:58,264 - mmdet - INFO - Epoch [1][9200/28130] lr: 2.000e-04, eta: 4 days, 13:11:28, time: 0.563, data_time: 0.011, memory: 7496, loss_cls: 1.3440, loss_bbox: 1.6055, d0.loss_cls: 1.3440, d0.loss_bbox: 1.6055, d1.loss_cls: 1.3440, d1.loss_bbox: 1.6055, d2.loss_cls: 1.3440, d2.loss_bbox: 1.6055, d3.loss_cls: 1.3440, d3.loss_bbox: 1.6055, d4.loss_cls: 1.3440, d4.loss_bbox: 1.6055, loss: 17.6972, grad_norm: nan
2026-04-01 16:19:26,448 - mmdet - INFO - Epoch [1][9250/28130] lr: 2.000e-04, eta: 4 days, 13:09:22, time: 0.564, data_time: 0.011, memory: 7496, loss_cls: 1.4146, loss_bbox: 1.6482, d0.loss_cls: 1.4146, d0.loss_bbox: 1.6482, d1.loss_cls: 1.4146, d1.loss_bbox: 1.6482, d2.loss_cls: 1.4146, d2.loss_bbox: 1.6482, d3.loss_cls: 1.4146, d3.loss_bbox: 1.6482, d4.loss_cls: 1.4146, d4.loss_bbox: 1.6482, loss: 18.3770, grad_norm: nan

at the time of writing this, we are at step 14900, we are still seeing NAN values for grad norm but the other fields like the cls and bbox are producing numerical values. i haven't changes any of the code
the only changes is that i am using the below scripts as i am using a single GPU
python tools/train.py
projects/configs/petr/petr_vovnet_gridmask_p4_1600x640.py

should i need to stop the training or is it expected and also please advise on accelerating the training as i am using a 48 GB without effecting the Evaluation Metrics that i am trying to reproduce

[coebangalore93]

and also please find the other training parameters
batch size = 1
workers per GPU = 4
Loss scale = 512 (fixed FP16)
Iterations per epoch = 28,130
estimated training time is 5 days

[delei-kong]

Hi, I ran into a similar issue. Before my grad_norm turned into NaN, I actually saw a few inf values first. The gradients didn't disappear right away, but then NaN showed up just like in your case. You might want to check your earlier logs to see if inf appeared before the NaN.

The NaN is very likely caused by gradient explosion: gradients blow up into inf, and then some operation turns that into NaN. My guesses for why the explosion happened:

1. Initial learning rate too high. The original paper used batch size 8 with lr 2e-4. With a single GPU and effective batch size of 1, sticking to 2e-4 is probably too large.

2. FP16 range limit. The max representable value in FP16 is about 65504. If some "dirty" sample produces a huge gradient, it can easily overflow.

3. Small batch size makes you more sensitive to individual samples. With 8 GPUs and total batch size 8, a weird sample's loss gets averaged out by others. With batch size 1, there's no dilution, so an anomalous sample can directly trigger an explosion.

Those are just my guesses. I had the same problem myself, and here's what fixed it for me:

1. Increased per-GPU batch size to 2 (samples_per_gpu=2).

2. Linearly scaled the learning rate down to 1/4 of the original (since 2/8), i.e., 5e-5.

3. Turned off mixed precision (FP16) and used FP32.

After these changes the problem went away. Switching to FP32 does slow down training though. Since your GPU has 48 GB and you're only using 7.5 GB, you can easily increase the batch size further (say 4, 6, or even more) – that'll speed up training and also help stabilize gradients. You could also manually tweak the gradient clipping threshold, but that might affect your final metrics.

One more thing worth keeping an eye on: your loss doesn't seem to be dropping much. It likely started around 20-something, and at step 9250 it's still around 18 – no clear downward trend. That could mean the gradients haven't been doing much lately, maybe the network has already degenerated from the NaN. Might be a useful signal to watch while you train.