OOM error on pytorch-add.py (RTX 2080 Ti)

[robasp-fhnw]

Hi

We (a university of applied sciences in switzerland) found this great project and are now planning to use it in our Slurm-based HPC-Cluster due to its large potential to improve GPU utilization/efficiency.
At the moment we are only doing some local testing on a single Ubuntu server (no Kubernetes, no Slurm).

We noticed a unexpected CUDA OOM error when running 2 Pytorch Scripts, each allocating ~7 GB on a GPU with 11GB Memory.
We are using RTX 2080 Ti with CUDA 12.8

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.133.20             Driver Version: 570.133.20     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 2080 Ti     Off |   00000000:21:00.0 Off |                  N/A |
| 29%   36C    P8              8W /  250W |      28MiB /  11264MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

For testing we are using a slightly modified version of your pytorch-add.py with n set to different values.

1. case
n = 21000 -> ~6.9GB is allocated on the GPU

We run the script twice simultaneously in two shells
NVSHARE_DEBUG=1 LD_PRELOAD=libnvshare.so python /mnt/data/examples/pytorch-add-6.9G.py

Everything works as expected. Both show correct output: nvshare client is initialized, the first script runs for 30 seconds, switches to the second script and so on...

2. case
n = 21500 -> ~7.2GB is allocated on the GPU

Again, we run the script twice simultaneously in two shells.
NVSHARE_DEBUG=1 LD_PRELOAD=libnvshare.so python /mnt/data/examples/pytorch-add-7.2G.py

Both nvshare-clients are initializing, the first script is running on the GPU for 30 seconds.
Then the scheduler releases the lock and the second script tries to allocate memory, this time however, we get a CUDA OOM error.

# Script 1
[NVSHARE][DEBUG]: Found NVML
[NVSHARE][DEBUG]: NVSHARE_POD_NAME = none
[NVSHARE][DEBUG]: NVSHARE_POD_NAMESPACE = none
[NVSHARE][DEBUG]: Sent REGISTER
[NVSHARE][DEBUG]: Received SCHED_ON
[NVSHARE][INFO]: Successfully initialized nvshare GPU
[NVSHARE][INFO]: Client ID = 3461be0e3e2750d9
[NVSHARE][DEBUG]: real_cuMemGetInfo returned free=10575.12 MiB, total=10822.94 MiB
[NVSHARE][DEBUG]: nvshare's cuMemGetInfo returning free=9286.94 MiB, total=10822.94 MiB
[NVSHARE][DEBUG]: cuMemAlloc requested 1849688064 bytes
[NVSHARE][DEBUG]: cuMemAllocManaged allocated 1849688064 bytes at 0x7d56c4000000
[NVSHARE][DEBUG]: Total allocated memory on GPU is 1764.00 MiB
[NVSHARE][DEBUG]: Received LOCK_OK
[NVSHARE][DEBUG]: cuMemAlloc requested 1849688064 bytes
[NVSHARE][DEBUG]: cuMemAllocManaged allocated 1849688064 bytes at 0x7d5654000000
[NVSHARE][DEBUG]: Total allocated memory on GPU is 3528.00 MiB
  0%|                                                                                                                                                                                                               | 0/4000 [00:00<?, ?it/s][NVSHARE][DEBUG]: cuMemAlloc requested 1849688064 bytes
[NVSHARE][DEBUG]: cuMemAllocManaged allocated 1849688064 bytes at 0x7d55e4000000
[NVSHARE][DEBUG]: Total allocated memory on GPU is 5292.00 MiB
[NVSHARE][DEBUG]: Pending Kernel Window is 2.
  0%|                                                                                                                                                                                                       | 1/4000 [00:00<26:28,  2.52it/s][NVSHARE][DEBUG]: cuMemAlloc requested 1849688064 bytes
[NVSHARE][DEBUG]: cuMemAllocManaged allocated 1849688064 bytes at 0x7d5574000000
[NVSHARE][DEBUG]: Total allocated memory on GPU is 7056.00 MiB
[NVSHARE][DEBUG]: Pending Kernel Window is 4.
  0%|▏                                                                                                                                                                                                      | 3/4000 [00:00<15:27,  4.31it/s][NVSHARE][DEBUG]: Pending Kernel Window is 8.
[NVSHARE][DEBUG]: Pending Kernel Window is 16.
  0%|▋                                                                                                                                                                                                     | 15/4000 [00:00<02:41, 24.75it/s][NVSHARE][DEBUG]: Pending Kernel Window is 32.
  1%|█▌                                                                                                                                                                                                    | 31/4000 [00:01<01:25, 46.24it/s][NVSHARE][DEBUG]: Pending Kernel Window is 64.
  2%|███                                                                                                                                                                                                   | 63/4000 [00:01<00:56, 69.83it/s][NVSHARE][DEBUG]: Pending Kernel Window is 128.
  3%|██████▎                                                                                                                                                                                              | 127/4000 [00:02<00:45, 85.75it/s][NVSHARE][DEBUG]: Pending Kernel Window is 64.
  6%|████████████▌                                                                                                                                                                                        | 255/4000 [00:03<00:40, 93.34it/s]

.......

 61%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████                                                                             | 2431/4000 [00:25<00:15, 98.07it/s][NVSHARE][DEBUG]: Pending Kernel Window is 64.
 64%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍                                                                      | 2559/4000 [00:26<00:14, 98.08it/s][NVSHARE][DEBUG]: Pending Kernel Window is 128.
 66%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌                                                                   | 2623/4000 [00:27<00:14, 98.07it/s][NVSHARE][DEBUG]: Received DROP_LOCK
[NVSHARE][DEBUG]: Pending Kernel Window is 1.
[NVSHARE][DEBUG]: Sent LOCK_RELEASED

# Script 2
[NVSHARE][DEBUG]: Found NVML
[NVSHARE][DEBUG]: NVSHARE_POD_NAME = none
[NVSHARE][DEBUG]: NVSHARE_POD_NAMESPACE = none
[NVSHARE][DEBUG]: Sent REGISTER
[NVSHARE][DEBUG]: Received SCHED_ON
[NVSHARE][INFO]: Successfully initialized nvshare GPU
[NVSHARE][INFO]: Client ID = 0ff51254656fe01d
[NVSHARE][DEBUG]: real_cuMemGetInfo returned free=6557.12 MiB, total=10822.94 MiB
[NVSHARE][DEBUG]: nvshare's cuMemGetInfo returning free=9286.94 MiB, total=10822.94 MiB
[NVSHARE][DEBUG]: cuMemAlloc requested 1849688064 bytes
[NVSHARE][DEBUG]: cuMemAllocManaged allocated 1849688064 bytes at 0x73e058000000
[NVSHARE][DEBUG]: Total allocated memory on GPU is 1764.00 MiB
[NVSHARE][DEBUG]: Received LOCK_OK
[NVSHARE][DEBUG]: cuMemAlloc requested 1849688064 bytes
[NVSHARE][DEBUG]: cuMemAllocManaged allocated 1849688064 bytes at 0x73dfe8000000
[NVSHARE][DEBUG]: Total allocated memory on GPU is 3528.00 MiB
  0%|                                                                                                                                                                                                               | 0/4000 [00:00<?, ?it/s][NVSHARE][DEBUG]: cuMemAlloc requested 1849688064 bytes
[NVSHARE][DEBUG]: cuMemAllocManaged allocated 1849688064 bytes at 0x73df78000000
[NVSHARE][DEBUG]: Total allocated memory on GPU is 5292.00 MiB
  0%|                                                                                                                                                                                                               | 0/4000 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/mnt/data/examples/pytorch-add-7.2G.py", line 33, in <module>
    z = torch.add(x, y)
        ^^^^^^^^^^^^^^^
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

I assumed that each of my scripts would be able to use basically all GPU memory (~11G) and nvshare would use page faults using unified memory to switch between the two.
Is this behaviour expected? Why does it happen exactly at 2 * 7 = 14GB?

Thanks!

[grgalex]

Hi,

1. What version of torch do you have installed?

2. Can you repeat the experiment inside the container that nvshare uses for its Kubernetes test workloads [1]? Run it locally, no need for Kubernetes.

3. Can you run both the 6.9G pair and the 7.2 pair under ltrace:

   ltrace -e 'cu*' python ...
   

   and then attach the unique sets of CUDA functions invoked in each case? This step wants to check if some different CUDA function (that nvshare currently does not hook) is invoked in the second case.

[1] https://github.com/grgalex/nvshare/blob/main/tests/dockerfiles/Dockerfile.pytorch.small

[robasp-fhnw]

Thanks for your feedback!

1. I am using torch 2.7.1.

2. I tried to run it in docker but i am facing an issue where the GLIBC of libnvshare is too new for the GLIBC of the container (pytorch/pytorch:1.9.1-cuda11.1-cudnn8-runtime)I get the following error:
   python: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.32' not found (required by /libnvshare.so)
   I will try to re-build nvshare with the correct glibc.
   Did you suggest trying the docker image because it uses an older pytorch version or because of the CUDA version?

3. I ran the following commands:

LD_PRELOAD=libnvshare.so ltrace -e 'cu*' python /mnt/data/examples/pytorch-add-7.2G.py > pytorch-7.2G.log 2>&1
cat pytorch-7.2G1.log | grep -- '->' | sed -E 's/^([^ ]+->[^ (]+).*/\1/' | sort -u

This gives me the list of invoked functions.
i tested this with both script variants in single mode and as a pair running simultaneously.

6.9G single
libc10_cuda.so->cudaDeviceGetStreamPriorityRange
libc10_cuda.so->cudaDeviceSynchronize
libc10_cuda.so->cudaGetDevice
libc10_cuda.so->cudaGetDeviceCount
libc10_cuda.so->cudaGetLastError
libc10_cuda.so->cudaMalloc
libc10_cuda.so->cudaStreamIsCapturing
libnvshare.so->cuda_driver_check_error
libtorch_cuda.so->cudaGetDeviceProperties_v2
libtorch_cuda.so->cudaGetLastError
libtorch_cuda.so->cudaLaunchKernel
libtorch_cuda.so->cudaMemcpyAsync
libtorch_cuda.so->cudaStreamSynchronize

7.2G single
libc10_cuda.so->cudaDeviceGetStreamPriorityRange
libc10_cuda.so->cudaDeviceSynchronize
libc10_cuda.so->cudaGetDevice
libc10_cuda.so->cudaGetDeviceCount
libc10_cuda.so->cudaGetLastError
libc10_cuda.so->cudaMalloc
libc10_cuda.so->cudaStreamIsCapturing
libnvshare.so->cuda_driver_check_error
libtorch_cuda.so->cudaGetDeviceProperties_v2
libtorch_cuda.so->cudaGetLastError
libtorch_cuda.so->cudaLaunchKernel
libtorch_cuda.so->cudaMemcpyAsync
libtorch_cuda.so->cudaStreamSynchronize

6.9G pair
libc10_cuda.so->cudaDeviceGetStreamPriorityRange
libc10_cuda.so->cudaDeviceSynchronize
libc10_cuda.so->cudaGetDevice
libc10_cuda.so->cudaGetDeviceCount
libc10_cuda.so->cudaGetLastError
libc10_cuda.so->cudaMalloc
libc10_cuda.so->cudaStreamIsCapturing
libnvshare.so->cuda_driver_check_error
libtorch_cuda.so->cudaGetDeviceProperties_v2
libtorch_cuda.so->cudaGetLastError
libtorch_cuda.so->cudaLaunchKernel
libtorch_cuda.so->cudaMemcpyAsync
libtorch_cuda.so->cudaStreamSynchronize


7.2G pair
libc10_cuda.so->cudaDeviceGetStreamPriorityRange
### libc10_cuda.so->cudaDeviceSynchronize is missing for the script that crashes ####
libc10_cuda.so->cudaGetDevice
libc10_cuda.so->cudaGetDeviceCount
libc10_cuda.so->cudaGetErrorString          ### only for the script that crashes
libc10_cuda.so->cudaGetLastError
libc10_cuda.so->cudaMalloc
libc10_cuda.so->cudaStreamIsCapturing
libnvshare.so->cuda_driver_check_error
libtorch_cuda.so->cudaGetDeviceProperties_v2
libtorch_cuda.so->cudaGetLastError
libtorch_cuda.so->cudaLaunchKernel
libtorch_cuda.so->cudaMemcpyAsync
libtorch_cuda.so->cudaStreamSynchronize

The list of functions is the same for both cases except for the crashing script

[grgalex]

If you want to give me SSH access to a GPU machine so I can help you debug this, you can send me an email to grgalex@ba.uoa.gr and I can send you my public SSH key.

[robasp-fhnw]

Sorry for my late answer.
Thanks for your help.
Unfortunately, I am not allowed to grant you SSH access...

I now was able to run my 7.2G pair inside a container using pytorch/pytorch:1.9.1-cuda11.1-cudnn8-runtime as base image
Interestingly, this worked perfectly this time. (I first had to build nvshare against an older glibc version as the one on my host was too new)

However, if i run the same script with same base image with 8 GB allocation, i get a CUDA OOM error. This happens if i run a single instance of the script (not as pair). I checked with nvidia-smi that nothing else is running on the GPU.

docker run -it --gpus all --entrypoint=/usr/bin/env -v /usr/local/lib/libnvshare.so:/libnvshare.so -v /var/run/nvshare:/var/run/nvshare pytorch-add:8G bash -c "NVSHARE_DEBUG=1 LD_PRELOAD=/libnvshare.so python /pytorch-add-8G.py"
[NVSHARE][DEBUG]: Found NVML
[NVSHARE][DEBUG]: NVSHARE_POD_NAME = none
[NVSHARE][DEBUG]: NVSHARE_POD_NAMESPACE = none
[NVSHARE][DEBUG]: Sent REGISTER
[NVSHARE][DEBUG]: Received SCHED_ON
[NVSHARE][INFO]: Successfully initialized nvshare GPU
[NVSHARE][INFO]: Client ID = 8103957d0539f5c9
[NVSHARE][DEBUG]: real_cuMemGetInfo returned free=9743.12 MiB, total=10822.94 MiB
[NVSHARE][DEBUG]: nvshare's cuMemGetInfo returning free=9286.94 MiB, total=10822.94 MiB
[NVSHARE][DEBUG]: cuMemAlloc requested 2917138432 bytes
[NVSHARE][DEBUG]: cuMemAllocManaged allocated 2917138432 bytes at 0x747112000000
[NVSHARE][DEBUG]: Total allocated memory on GPU is 2782.00 MiB
[NVSHARE][DEBUG]: Received LOCK_OK
[NVSHARE][DEBUG]: cuMemAlloc requested 2917138432 bytes
[NVSHARE][DEBUG]: cuMemAllocManaged allocated 2917138432 bytes at 0x747064000000
[NVSHARE][DEBUG]: Total allocated memory on GPU is 5564.00 MiB
  0%|                                                                                                                                                                                                               | 0/4000 [00:00<?, ?it/s][NVSHARE][DEBUG]: cuMemAlloc requested 2917138432 bytes
[NVSHARE][DEBUG]: cuMemAllocManaged allocated 2917138432 bytes at 0x746fb6000000
[NVSHARE][DEBUG]: Total allocated memory on GPU is 8346.00 MiB
[NVSHARE][DEBUG]: Pending Kernel Window is 2.
  0%|                                                                                                                                                                                                       | 1/4000 [00:00<25:38,  2.60it/s][NVSHARE][DEBUG]: real_cuMemGetInfo returned free=2703.12 MiB, total=10822.94 MiB
[NVSHARE][DEBUG]: nvshare's cuMemGetInfo returning free=9286.94 MiB, total=10822.94 MiB
  0%|                                                                                                                                                                                                       | 1/4000 [00:00<26:15,  2.54it/s]
Traceback (most recent call last):
  File "/pytorch-add-8G.py", line 33, in <module>
    z = torch.add(x, y)
RuntimeError: CUDA out of memory. Tried to allocate 2.72 GiB (GPU 0; 10.57 GiB total capacity; 8.15 GiB already allocated; 9.07 GiB free; 8.15 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

[grgalex]

I suspect this has to do with the static amount of memory (currently 1.5 GiB) that nvshare hides from the user application to account for the non-pageable memory of the CUDA contexts.

Take a look at this comment in the code to better understand how this works: https://github.com/grgalex/nvshare/blob/1a8f211ae45a8aded4f54d0e1832c6073a4f19ef/src/hook.c#L740\

Can you change the value of MEMINFO_RESERVE_MIB to a smaller value, recompile nvshare and re-run?

[robasp-fhnw]

Hi

You were right, after changing MEMINFO_RESERVE_MIB to 512 i was able to run a single 10G pytorch script.

For the pair case i made an interesting discovery:

I can run the 7.2G Pytorch Script as a pair without problems but this is the max. If i execute one 7.2G and one 7.4G script, i get CUDA out of memory error.

Whats interesting is that i can run a 9.0G and 5.8G script with nvshare simultaneously without out of memory error. However, this only works if i start the smaller script (5.8G) first. If I run the 9.0G script first, i get OOM.

So i guess even with MEMINFO_RESERVE_MIB set to a smaller value i am hitting some boundary if i try to allocate more than ~14.5G in total.
Why the order matters i don't know.
Do you have an idea?

Thanks

[grgalex]

Can you rerun with NVSHARE_ENABLE_SINGLE_OVERSUB=1?

(https://github.com/grgalex/nvshare?tab=readme-ov-file#single_oversub)

[robasp-fhnw]

Hi

I reran with NVSHARE_ENABLE_SINGLE_OVERSUB=1
Unfortunately, i see no change. If i start the 5.8G-Script before the 9.0G-Script it works.
If i start the 9.0G first, i get OOM on after the first TQ.

[grgalex]

How much physical GPU memory does the device have?

[robasp-fhnw]

The NVIDIA GeForce RTX 2080 Ti has 11 GB GPU Memory.

[parallelArchitect]

The OOM at exactly 2×7GB is a committed memory ceiling issue, not a paging failure. cuMemAllocManaged reserves virtual address space and commits physical backing immediately on Turing. When Script 2 tries its fourth allocation, total committed managed memory (~14GB) exceeds what the system can back — VRAM (11GB) plus whatever host RAM the driver has reserved as overflow. The scheduler hasn't evicted Script 1's pages yet at that point, so there's no room.
The fix is either: stagger allocations so Script 2 doesn't pre-allocate until Script 1 has released, or reduce per-script allocation to leave headroom for the overlap window.
I've been measuring this exact boundary on Pascal/Turing — the allocatable ceiling is typically ~70% of (VRAM + available host RAM), not the full sum. Relevant tool if useful for diagnosing the headroom before launch:
https://github.com/parallelArchitect/cuda-unified-memory-analyzer