Hi again
I have 2GPUs, each 24GB. I am using lit-llama-7B model. I was trying to set using two GPUs, by setting the number of devices here, fabric = L.Fabric(accelerator=accelerator, devices=2), but the model uses only one GPU (cuda:0), and the other one is not used at all. However, I get this error:
CUDA out of memory. Tried to allocate 192.00 MiB (GPU 0; 23.68 GiB total capacity; 23.03 GiB already allocated; 142.56 MiB free; 23.03 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Am I doing something wrong? How can I make the model use both GPUs?
Thank you so much for your support.
Hi again
I have 2GPUs, each 24GB. I am using
lit-llama-7Bmodel. I was trying to set using two GPUs, by setting the number of devices here,fabric = L.Fabric(accelerator=accelerator, devices=2), but the model uses only one GPU (cuda:0), and the other one is not used at all. However, I get this error:CUDA out of memory. Tried to allocate 192.00 MiB (GPU 0; 23.68 GiB total capacity; 23.03 GiB already allocated; 142.56 MiB free; 23.03 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONFAm I doing something wrong? How can I make the model use both GPUs?
Thank you so much for your support.