The detail of classification.

[hzwfl2]

Hi, the model CADA-VAE is used to extract latent features. As for classification, I did not find the detail in the paper. Can you tell me more details about your setting to get the accuracy in the test of zero-shot?

[edgarschnfld]

Hi,
Let's assume we have 2 trained encoders to extract latent features. The question now is "how is the classifier trained and tested?". Let's take the example of the CUB dataset, which has 150 training and 50 test classes. To train the classifier (one linear layer with softmax), you first need to extract latent features for the classes you want to test. In the case of GZSL, that is all 200 classes. In the case of ZSL, that is 50 classes. You should extract multiple features per class, which is possible due to the stochastic variation in the encoders (For details on how many features are adequate, see the last paragraph of section 4.1 in the paper). Then you train your classifier on those extracted features.
Now it's time to test the classifier. For that, we use the provided test set, which consists of image features. Convert them to latent features first (only take the predicted mean of the variational encoders, and discard the sigma). Compute the accuracy (in case of ZSL) or the harmonic mean accuracy (for GZSL). The test set provides fresh unused image features for any class you want.
You can switch between --generalized True/False to do GZSL/ZSL in this repository.

If something is still unclear, just ask :)

[MuqaddasGull]

Thank you for explaining everything in detail. But I have a query like in your code where exactly you done this like where you convert those test features into latent features vector. One more thing is that you discard sigma and only consider mean?

[hzwfl2]

Hi,
Let's assume we have 2 trained encoders to extract latent features. The question now is "how is the classifier trained and tested?". Let's take the example of the CUB dataset, which has 150 training and 50 test classes. To train the classifier (one linear layer with softmax), you first need to extract latent features for the classes you want to test. In the case of GZSL, that is all 200 classes. In the case of ZSL, that is 50 classes. You should extract multiple features per class, which is possible due to the stochastic variation in the encoders (For details on how many features are adequate, see the last paragraph of section 4.1 in the paper). Then you train your classifier on those extracted features.
Now it's time to test the classifier. For that, we use the provided test set, which consists of image features. Convert them to latent features first (only take the predicted mean of the variational encoders, and discard the sigma). Compute the accuracy (in case of ZSL) or the harmonic mean accuracy (for GZSL). The test set provides fresh unused image features for any class you want.
You can switch between --generalized True/False to do GZSL/ZSL in this repository.

If something is still unclear, just ask :)

Thanks for your reply. I have a question. When you get the latent features to train the classifier, do you use the attribution information? Because in ZSL, you have 150 training (seen classes) and 50 test classes (unseen classes), you only know seen classes during training the classifier. If you do not use the attribution, how to associate seen classes and unseen classes?

Thanks for your time to reply to this issue!

[seyeeet]

how/where should we get the attributes for each dataset?

[edgarschnfld]

The attributes for each dataset can be found in the dropbox folder that is linked in the readme. That folder just has to be copied in the repository, and then you can run the training script and it should work out of the box for all datasets.

[edgarschnfld]

Not quite sure what the question about attribution information aims at, but I'll try to answer it: I guess with "attribution information" you just mean that we know beforehand which classes are seen and unseen? Then yes, we use attribution information. Not sure what is meant by "associate": Seen and unseen classes both live in the attribute space, and in this work also in the learned shared latent space. Thus, for a classifier in this space, they are all equally "seen". That way, they become associated. If with "associated" you mean associated with either the seen or unseen set, then this association is made by the humans that collect the data.

[edgarschnfld]

Thank you for explaining everything in detail. But I have a query like in your code where exactly you done this like where you convert those test features into latent features vector.

Hi, that would be here:

CADA-VAE-PyTorch/model/vaemodel.py

Lines 310 to 323 in 26f0085

	####################################
	# preparing the test set
	# convert raw test data into z vectors
	####################################
	self.reparameterize_with_noise = False
	mu1, var1 = self.encoder['resnet_features'](novel_test_feat)
	test_novel_X = self.reparameterize(mu1, var1).to(self.device).data
	test_novel_Y = test_novel_label.to(self.device)
	mu2, var2 = self.encoder['resnet_features'](seen_test_feat)
	test_seen_X = self.reparameterize(mu2, var2).to(self.device).data
	test_seen_Y = test_seen_label.to(self.device)

One more thing is that you discard sigma and only consider mean?

Yes, but only for the test samples. The representation that CADA-VAE learns for a given class is a multivariate Gaussian distribution with a mean and variance. By convention, when learning representations with VAEs, the mean is usually assumed to be the representation itself, if you need a single vector. For the test set, we chose to only use the mean and discard the sigma. Otherwise, we have stochastic variation in the test set, and that would negatively affect the comparability to other runs and methods.

[Moreqg]

hi, how are the parameters of attributes trained? Did you just use the image labels of CUB, or use the pretrained bert weights?