Features: Data parallel for more flexibility and more efficiency

[seoulsky-field]

What

When I do experiments using multi-gpu, I was faced of training slow down, imbalanced data assignment, some problems of multi processing.

Why

When we use multiple gpu, we should correctly use PyTorch programming in multi processing. (For server and us)
However, because of time consuming, this is a future work.

How

• Use DistributedDataParallel
• Use DistributedSampler
• reference: https://kongsberg.tistory.com/7

[kdg1993]

Thanks for considering the multi-GPU issue!

As far as I know, the DistributedDataParallel is a bit complex compared to the DataParallel (multi-processing vs multi-threading), So I think the official PyTorch document is worth reading once.

I think you already know this well but I leave the similar issue #54 just in case 😄 👍