Question about the num_workers setting in the CLIP Zero-Shot Evaluation experiment

[DpZhang-GDUT]

Hi, I encountered the following issue when running the CLIP Zero-Shot experiment during Evaluation.

This error occurs in the cli.py file when num_workers=4, which is the default setting. Did you encounter the same issue when running this experiment?

ValueError: Caught ValueError in DataLoader worker process 3.
ValueError: No samples found in dataset; perhaps you have fewer shards than workers.

[chs20]

Hi, I have not encountered this error before, sorry.

[DpZhang-GDUT]

Thank you very much for your prompt reply! Could you please tell me if the value of num_worker was always set to 4 during your experiments?

[chs20]

Yes, it was not changed.

Does it work with less (even 0) workers?

[DpZhang-GDUT]

When I set numwoker = 1, the code can successfully test all the datasets.
However, I found that for the same dataset, different numworkers would yield different results.
The reason for this situation might be that the change in numwoker has caused the 1000 samples used in the test to be altered.
The current code execution process is roughly as follows:
WebDataset shard list
-> dataset.shuffle(10000, initial=10000, rng=random.Random(seed))
-> dataset.batched(50)
-> PyTorch DataLoader(num_workers=N)
-> zeroshot_classification.py Select the first 1000 samples
-> AutoAttack print “initial accuracy”
When the value of num_workers changes, WebDataset will assign shards to different workers. This may cause the loading order of the data to change, thereby resulting in a variation in the final selection of the first 1000 samples.