Fix incorrect memory reporting on coherent UMA platforms (GB10 / DGX …

[parallelArchitect]

Fix incorrect memory reporting on coherent UMA platforms (GB10 / DGX Spark)

On GB10 / DGX Spark, nvmlDeviceGetMemoryInfo returns NVML_SUCCESS with total equal to system MemTotal (~121GB). This causes nvitop to display full system RAM as GPU memory instead of actually allocatable memory.

The existing NVMLError_NotSupported path correctly handles some UMA platforms, but GB10 returns NVML_SUCCESS — not NOT_SUPPORTED — so it falls through to the discrete GPU path and displays wrong values.

Issue Type

• Bug fix

Description

Detect coherent UMA by comparing NVML-reported total against system virtual memory total. If total >= 90% of system RAM, classify as unified memory and use system virtual memory (MemAvailable) for display instead.

Preserves existing behavior for discrete GPUs.

Motivation and Context

Same root cause documented and fixed in:

• nvtop PR: Fix NVML memory reporting regression on coherent UMA platforms (Fixes… Syllo/nvtop#463
• btop PR: Fix NVML memory reporting regression on coherent UMA platforms aristocratos/btop#1611
• NVML shim workaround: https://github.com/parallelArchitect/nvml-unified-shim

Note

Requires validation on GB10 / DGX Spark hardware. The fix has not been independently validated on a coherent UMA system.

[XuehaiPan]

Hi @parallelArchitect, thanks for raising this!

Before we investigate this, can you elaborate on the context for the current behavior of the GB10 machines? You mean memory_info.total is constantly returning the wrong metrics with system MemTotal (~121GB)? I'm wondering what the difference is between system MemTotal and virtual_memory.total? Could you add a screenshot or the hardware specs?

from nvitop import *

print(host.virtual_memory())
print(Device(0).as_snapshot())

[parallelArchitect]

Thanks for the quick response.
On GB10 / DGX Spark, nvmlDeviceGetMemoryInfo returns NVML_SUCCESS with memory_info.total approximately equal to the full system MemTotal (~121GB on a 128GB system). This is documented in the community NVML shim project: https://github.com/CINOAdam/nvml-unified-shim
system MemTotal and virtual_memory().total should be equivalent — both reflect the total installed RAM. The issue is that on GB10, NVML reports this full value as GPU memory rather than returning NVML_ERROR_NOT_SUPPORTED or zero, which is what the existing code path expects for unified memory detection.
I don't have GB10 hardware to run the requested script. The fix is based on analysis of GB10 NVML behavior documented by community members with hardware. If someone with a GB10 can run the script you suggested, that would confirm the behavior.
The same fix has been applied to nvtop (Syllo/nvtop#463) and btop (aristocratos/btop#1611) where the root cause is identical.

One additional note: the existing unified memory path returns vm.total (MemTotal) as the display total. On GB10, MemAvailable would be more accurate since it reflects actually allocatable memory after kernel reservations and page cache. Happy to update that too if you agree, but wanted to flag it separately rather than expanding the PR scope without discussion.

[bernardogv]

I have a DGX Spark and can add hardware data here. On my machine NVML returns
NOT_SUPPORTED rather than NVML_SUCCESS, which is the opposite of the behaviour this
PR describes — so the two of us may be seeing driver- or config-dependent differences worth
pinning down before settling on a detection heuristic.

Raw NVML, via ctypes directly against libnvidia-ml.so.1

Deliberately bypassing any Python wrapper, so this is what the driver itself returns:

device : NVIDIA GB10
driver : 580.173.02
nvmlDeviceGetMemoryInfo    -> NVML_ERROR_NOT_SUPPORTED   total=0  free=0  used=0
nvmlDeviceGetMemoryInfo_v2 -> NVML_ERROR_NOT_SUPPORTED   total=0  free=0  used=0
  MemTotal:       127600748 kB
  MemAvailable:   118566368 kB

nvidia-smi agrees — every framebuffer field is N/A:

$ nvidia-smi --query-gpu=memory.total,memory.used,memory.free,utilization.gpu,power.draw,temperature.gpu --format=csv
memory.total [MiB], memory.used [MiB], memory.free [MiB], utilization.gpu [%], power.draw [W], temperature.gpu
[N/A], [N/A], [N/A], 0 %, 11.17 W, 35

nvitop 1.7.1 does not misbehave here — the existing UMA path already catches it

Because the call returns NOT_SUPPORTED, the existing path is taken and nothing is broken
on this machine:

│   0  GB10                  On │ 0000000F:01:00.0 Off │                  N/A │
│ N/A   37C   P0      11W / N/A │   8860MiB / 121.7GiB │      0%      Default │

│   0 3695970 C bernar+ 365.7MiB   0     0   0.0   3.6  44:17:20  /home/ber.. │

Total (121.7 GiB) comes from psutil, and per-process reporting works —
nvmlDeviceGetComputeRunningProcesses_v3 returns NVML_SUCCESS on GB10 and the 365.7 MiB
figure matches what the process actually holds.

Why this matters for the proposed detection

The heuristic in this PR — classify as UMA when NVML total ≥ 90% of system RAM — cannot
trigger on my system, because nvmlDeviceGetMemoryInfo never succeeds and total is 0.
So if both behaviours exist in the wild, detection likely needs to handle two cases:

1. NOT_SUPPORTED (this machine, driver 580.173.02) — already handled by the existing path.
2. NVML_SUCCESS with total ≈ MemTotal (as reported here) — what this PR adds.

Guarding both, rather than replacing the first with the second, would avoid regressing
machines like mine. It'd also be worth confirming the reporter's driver version, since that
seems the most likely variable.

For cross-project context: nvtop hit the same problem and merged a UMA fix
(Syllo/nvtop#466, 2026-04-29, not yet released).
I tested it on this hardware today and it lands on the same semantics nvitop already uses —
total from MemTotal, used from system memory. Details in
Syllo/nvtop#449.

One suggestion on the displayed "used" figure

Currently the device row shows psutil.virtual_memory().used (8.60 GiB here), while the
GPU is actually holding 0.36 GiB — the difference being CPU-side allocations. Both numbers
are legitimate but they answer different questions, and on unified memory "what is the GPU
holding" is often the one you want. Summing
nvmlDeviceGetComputeRunningProcesses_v3 gives it directly and works reliably on GB10 —
during a load test the sum matched an on-disk model size to within the KV-cache delta.

Not arguing it should replace the system figure; just noting it's available if a
GPU-held-vs-host-used split is ever of interest.

Happy to test

I can run patches against real GB10 hardware and report back — driver 580.173.02,
Ubuntu 24.04.4, kernel 6.17.0-1029-nvidia, aarch64. Just point me at a branch.

[parallelArchitect]

Thanks for providing the direct hardware data.

One relevant difference is the software baseline. My earlier GB10 characterization work was performed on the 580.142 driver series, roughly 3 months ago:

https://forums.developer.nvidia.com/t/gb10-hardware-baseline-first-direct-measurements-and-findings/367851

Your results are from 580.173.02 — the NVML behavior difference is likely driver-version dependent, given the driver has moved on since then.