ERROR: File "/home/mansisak/.conda/envs/attnlens/lib/python3.10/site-packages/torch/optim/adam.py", line 505, in _multi_tensor_adam exp_avg_sq_sqrt = torch._foreach_sqrt(device_exp_avg_sqs) torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 246.00 MiB (GPU 3; 39.59 GiB total capacity; 37.60 GiB already allocated; 99.69 MiB free; 38.17 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
ERROR: File "/home/mansisak/.conda/envs/attnlens/lib/python3.10/site-packages/torch/optim/adam.py", line 505, in _multi_tensor_adam exp_avg_sq_sqrt = torch._foreach_sqrt(device_exp_avg_sqs) torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 246.00 MiB (GPU 3; 39.59 GiB total capacity; 37.60 GiB already allocated; 99.69 MiB free; 38.17 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF