What is "nn.BatchNorm2d" for?

[ghbcpeng]

Dear TA,

In the example code from GitHub https://github.com/kuangliu/pytorch-cifar, nn.BatchNorm2d is invoked in file resnet.py to build BasicBlock. This function seems not shown in Figure 2 in the lab handout, and the images are already normalized by trainloader and testloader. Then why do we need nn.BatchNorm2d? Thanks for your reply.

Regards,
BC

[JiaRenChang]

Hi,
Yes, you need batch normalization.
The only one reason why the Figure 2 do not draw Batch Normalization (BN) layer is for simplification.
Because BN is very important to train deep networks.
If you want to know why BN is essential to deep learning, you can refer to this paper "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift ".

Jia-Ren

[jessejchuang]

Hi TA,

In Pytorch BN module, we can see four parameters as below.
>>> self.bn1.state_dict().keys()
... odict_keys(['weight', 'bias', 'running_mean', 'running_var'])

I know ‘running_mean’ and ‘running_var’ are beta and gamma respectively and they are learnable parameters in training mode (then fixed in evaluation mode).

But for 'weight' and 'bias', they are inherited from parent module but seem useless in batchNorm calculation. Do I misunderstand weight/bias? Any comments?

[JiaRenChang]

Hi,
You misunderstand.
The weight is gamma and the bias is beta according to the paper.
The running mean and variance is for inference.
You can refer to the stage 10 in algorithm 2 in the paper.
Jia-Ren