Additional Log in the softmax function

[ATAboukhadra]

Hi,

I just have a question, why did you use an additional log before the softmax function in st.py loss on one input but not the other?

def forward(self, out_s, out_t):
		loss = F.kl_div(F.log_softmax(out_s/self.T, dim=1),
						F.softmax(out_t/self.T, dim=1),
						reduction='batchmean') * self.T * self.T

		return loss

[nguyenvulong]

Yeah it is strange that is has the log there. I found another implementation that did not use log but softmax only for both Teacher and Student.

As you might have already known it, when we modify the F.log_softmax(out_s/self.T, dim=1) in to F.softmax(out_s/self.T, dim=1), the KD output is negative.

Do you have any ideas why?

[nguyenvulong]

I asked another researcher and here is the answer yoshitomo-matsubara/torchdistill#233