Feedback on the experimental Performance Monitor

[mostlygeek]

Usage and limitations

• currently GPU metrics are only supported on linux via LACT. LACT offers good cross platform GPU information and llama-swap will communicate directly with its /run/lactd.sock.
• If LACT is not available llama-swap will attempt to use nvidia-smi or rocm-smi to get GPU data.
• System stats (cpu, ram, network) is available on all platforms.

New performance config settings:

The performance monitor can be tweaked with these new configuration settings. The performance monitor is enabled by default with the settings below:

# performance: configuration for system monitoring statistics
# - timing values are duration strings like 1s, 1h30m, 90m, 2h10s, etc.
performance:
  # disabled: boolean
  # - default: false
  disabled: false

  # every: delay between polling for new performance statistics
  # - default: 5s
  # - minimum duration 5s
  every: 15s

New Performance tab in the UI

Prometheus and Grafana support

Prometheus Metrics are available at the new /metrics endpoint. These can be wired up to Grafana to produce some nice graphs. An example dashboard is provided at docs/grafana/example-dashboard.json:

[andrewfromtver]

Hello there! Is it available when running llama-swap in container?

my instance running in k8s cluster with mobile gpus nvidia-smi works fine and Nvidia device plugin and dcgm exportr works fine but there is no performance metrics in llama-swap UI

[eslowney]

Same issue here, I can't really install LACT on our servers, but there's no data when (I assume) it's using nvidia-smi. I wrote my own little systemd service/timer that calls nvtop -s every 10 seconds and dumps the results into node-exporter, which grafana has good support for. DCGM-Exporter was a little too heavy for what I needed.

[gbetous]

Hi,
Nice feature !

On my setup (Arch/KDE/AMD 7900 XTX) CPU and memory usage is not working, while I can read GPU temperatures (rocm-smi is installed and functional)

[candrews]

On my setup using rocm-smi with an AMD RX 580 8gb, the stats aren't correct:

What's not correct:

• The "GPU Memory Utilization" is always 0
• "GPU VRAM Temperature" is wrong. It shows 0, then sometimes 100, and the values in between don't seem high enough. I believe the lowest temperature possible is ambient, which is currently ~22.

What is correct:

• "GPU Utilization" looks good
• "GPU Temperature" looks good

/CC @knguyen298 as they added rocm-smi monitoring in #767

[mostlygeek]

is it rocm-smi that is reporting bad or no data?

[candrews]

I think "GPU VRAM Temperature" is reporting "memory use %" - not temperature.

# rocm-smi -i -P -t -f -u --showmemuse --showmeminfo vram --showproductname


============================ ROCm System Management Interface ============================
WARNING: AMD GPU device(s) is/are in a low-power state. Check power control/runtime_status

=========================================== ID ===========================================
GPU[0]          : Device Name:          AMD Radeon RX 580 Series
GPU[0]          : Device ID:            0x67df
GPU[0]          : Device Rev:           0xe7
GPU[0]          : Subsystem ID:         0x2378
GPU[0]          : GUID:                 2621
==========================================================================================
====================================== Temperature =======================================
GPU[0]          : Temperature (Sensor edge) (C): 57.0
==========================================================================================
=================================== Current Fan Metric ===================================
GPU[0]          : Fan Level: 122 (48%)
GPU[0]          : Fan RPM: 1681
==========================================================================================
=================================== Power Consumption ====================================
GPU[0]          : Current Socket Graphics Package Power (W): 28.16
==========================================================================================
=================================== % time GPU is busy ===================================
GPU[0]          : GPU use (%): 1
==========================================================================================
=================================== Current Memory Use ===================================
GPU[0]          : GPU Memory Allocated (VRAM%): 98
GPU[0]          : GPU Memory Read/Write Activity (%): 0
GPU[0]          : Memory Activity: N/A
GPU[0]          : Not supported on the given system
==========================================================================================
================================== Memory Usage (Bytes) ==================================
GPU[0]          : VRAM Total Memory (B): 8589934592
GPU[0]          : VRAM Total Used Memory (B): 8421638144
==========================================================================================
====================================== Product Info ======================================
GPU[0]          : Card Series:          AMD Radeon RX 580 Series
GPU[0]          : Card Model:           0x67df
GPU[0]          : Card Vendor:          Advanced Micro Devices, Inc. [AMD/ATI]
GPU[0]          : Card SKU:             N/A
GPU[0]          : Subsystem ID:         0x2378
GPU[0]          : Device Rev:           0xe7
GPU[0]          : Node ID:              1
GPU[0]          : GUID:                 2621
GPU[0]          : GFX Version:          gfx803
==========================================================================================
================================== End of ROCm SMI Log ===================================

# rocm-smi -i -P -t -f -u --showmemuse --showmeminfo vram --showproductname --csv
WARNING: AMD GPU device(s) is/are in a low-power state. Check power control/runtime_status

device,Device Name,Device ID,Device Rev,Subsystem ID,GUID,Temperature (Sensor edge) (C),Current Socket Graphics Package Power (W),GPU use (%),GPU Memory Allocated (VRAM%),GPU Memory Read/Write Activity (%),Memory Activity,VRAM Total Memory (B),VRAM Total Used Memory (B),Card Series,Card Model,Card Vendor,Card SKU,Node ID,GFX Version
card0,AMD Radeon RX 580 Series,0x67df,0xe7,0x2378,2621,50.0,28.051,7,98,0,N/A,8589934592,8421638144,AMD Radeon RX 580 Series,0x67df,Advanced Micro Devices Inc. [AMD/ATI],N/A,1,gfx803

[gbetous]

I checked the code, and the field numbers of CSV are hardcoded and don't correspond to my case (rocm-smi also).

For example the code reads fields[14] for memUtil, while in my case it corresponds to "Average Memory Bandwith" of the generated CSV.

[knguyen298]

What version ROCm is this?

As a first proof of concept implementation, this just matched the columns by number, same as the nvidia-smi implementation, but ideally it should match by column name to avoid cases like this.

I’ll see if that’s something easily doable if I have some time today.

[candrews]

What version ROCm is this?

# rocm-smi --version
ROCM-SMI version: 4.0.0+unknown
ROCM-SMI-LIB version: 7.8.0

[axxelh]

The logic in monitor_unix.go allows only nvidia-smi or rocm-smi monitoring, but not both. Ideally for heterogeneous setups both would be supported at the same time. The LACT and (future) sysfs versions should probably not combine to avoid duplicate entries, but the two "smi" tools definitely can combine without overlap.

[mostlygeek]

This would be a good improvement. Does anyone have a setup with nvidia/amd GPUs to help test patches?

[axxelh]

Sorry, should have been more specific. Yes, I have mixed GPUs, and yes, I am willing to test patches.

[eslowney]

Upgrading from v212 to v216 fixed everything for me. Thanks!

[Gerporgl]

Shouldn't amd-smi be used instead of rocm-smi?

[Gerporgl]

UPDATE:
This worked, I had to install rocm-smi-lib package instead of rocm-smi, this is probably what I was doing a while ago when I got it working.

Now I also realized that this wakes up the GPU / driver module every 15 seconds, which isn't going to work in my case. I can see it in dmesg and other power usage tools on my server. I don't think there is anything you'll be able to do for that... perhaps except finding a way to query if the cards / GPU is in sleep mode and not call rocm-smi in that case.

Low idle power usage is kind of critical on my small of home lab server, and I need the GPU to be in sleep when not in use.
My current llama-swap configuration and usage where models are unloaded after some times works extremely well for me. So I think I'll just remove the rocm-smi package for now and only keep the CPU monitoring.

[mostlygeek]

@Gerporgl do you have the ability to try LACT instead of rocm-smi? I wonder if that also wakes up the GPUs.

[Gerporgl]

I wanted to try that as well yesterday, but could not find a package provided by the Ubuntu base image directly and I thought flatpak was the only option, but just found out LACT provides deb packages for ubuntu 24.04, which worked for me.

With LACT, the behavior is a bit different, the GPU always remain in a wakeup state, as soon as llama-swap is started, which again, for me, is not desirable.

At least it does not cause multiple repeated dmesg that shows the device is constantly resuming every 15 seconds (this is how it was with rocm-smi), but overall the power usage is even worst with LACT.

I understand for users on a desktop, the GPU will never go to sleep, but for me I liked very well the fact that adding a GPU to my small proxmox server literally increased the power by 0W (or close to) when not in use, because the GPU and pci express bus go to sleep as soon as it is unused.

Perhaps having a configuration to only enable CPU monitoring may be a good thing, because if for some reason I wanted to install rocm-smi (or if in the future amd-smi starts to be used), I would have to disable entirely the performance monitoring, including CPU.

[matthew-bailey]

I also observed that this feature doesn't allow the GPUs to enter a low power state. One alternative would be to only have it fetch GPU performance metrics when there is a model loaded (per GPU, or globally if llama-swap can't know which GPUs are actually in use).

[Gerporgl]

I agree, I think llama-swap itself monitoring its own active models is probably the best way.

Perhaps something as simple as having another configuration for performance:

performance:
    disabled: false
    cpu: true/false (default true)
    gpu: true/false/per_model (default true)

You could then have another property under each model to allow or disable completely gpu monitoring for this model.

I also thought you could query something like

cat /sys/class/drm/card1/device/power/runtime_status

In my case it shows as "suspended" when in low power state and "active" when in use...

However there would be several challenges with this I think. Like monitoring the correct card (some cards like iGPU are always active), but also, as soon as you would query rocm-smi this would go in active state, and you may never get out of it.

[Pauliehedron]

Following up on the macOS GPU gap you noted here — I posted a design sketch in Ideas (#814) with the data source mapping for Apple Silicon via ioreg. I missed the link with Perfomrance Monitor, I searched with a Mac specific view, sorry about that!

[mostlygeek]

I landed #816 and released in v223. Please give it a try. It uses mactop (brew install mactop) and fallsback to ioreg when its not available.

[Pauliehedron]

Tested on Mac Studio — M1 Max (32-core GPU), 64GB unified memory, macOS 26.5. Brew v223 with mactop + ioreg.

Test Results — GPU Monitoring PR #816

Source build (from main, commit 29d3d9b)

Backend	GPU Util	GPU Mem	Temp	Fan	Power	Status
mactop	66%	24% / 15.4 GB	47°C	9.7%	4.4 W	✅ Full metrics
ioreg fallback	43%	24% / 15.3 GB	0	0	0	✅ GPU util + memory populated; temp/fan/power correctly zero

Brew v223 (same commit)

Identical results to source build — same commit 29d3d9b, same binary behavior.

First-run caveat

mactop triggers a macOS Screen Recording + Microphone permission prompt on first launch (Apple Silicon coregraphics hooks). Grant once, works silently afterward. Without granting, ioreg fallback kicks in automatically.

Verdict

Both paths work. Both brew v223 and a source build from main (same commit 29d3d9b) produce identical results. No failures, no regressions in existing CPU/memory/swap/load metrics.

Comment drafted with assistance from Ensemble (ENSEMBLE Framework v3.3.3 protocol on Hermes Agent). Accountable human: @Pauliehedron, Conductor

[skobkin]

On my RX 7800 XT when running in Docker (ghcr.io/mostlygeek/llama-swap:vulkan):

No GPU data available

/dev/dri and /dev/kfd are mounted and GPU acceleration works fine for inference.

I guess, it's related to this:

currently GPU metrics are only supported on linux via LACT.
If LACT is not available llama-swap will attempt to use nvidia-smi or rocm-smi to get GPU data

Am I correct to assume that LACT is missing in the Docker image for Vulkan?

[Petrox]

Feedback: this is a cool feature (strixhalo native install works fine)

[murflaw7424]

Would be nice to prefill/preprocessing and how long it takes to load context.

Also, would be nice to a graph of context used v. How much context you set.

Overall great system.

[diadras]

A bit "off-topic" but; Thank you for creating and maintaining llama-swap!! I have been really enjoying using it and it really is the backbone of my local LLM inference.
I had never heard about LACT but you choosing to add it to llama-swap really show me you care about the quality of the software. It works very well!

Thank you!

[ejstreet]

I made a PR (#938) that uses intel_gpu_top to get metrics from Intel GPUs, since LACT doesn't really give any information on those (at least not from my iGPU).

[phoehnel]

Great feature! I've been using it for quite a while. What would make it even better:

• add a system-info block at the top, showing the machines total RAM/VRAM CPU-Cores etc. If you have many llama machines, its often unclear what "50% vRAM-Usage" in this view actually means. RAM already has this info displayed below the graph
• Add simple stats below the graphs (e.g. min/max/avg over the displayed timeframe)
• Also show buffer/cache RAM usage
• Optional "Cherry on top" (probably not easy to achieve): Add graph for avg token per second over time.

[arbv]

Please consider hiding this tab from the UI when this functionality is disabled.

[HenkieTenkie62]

I really like this functionality. Is there a way to log the averages (1, 5, or 10) to a file so it can be reviewed later on?

[krampenschiesser]

I love this and am using it alot! A few feature/change suggestions from my side:

1. If possible add the fan speed to the GPU temp graph as part of the labels. so instead of 'NVIDIA GeForce RTX 5060 TI' display 'NVIDIA GeForce RTX 5060 TI (Fan 55%)'
2. Shorten the GPU names if they are all of the same vendor and/or type 'NVIDIA GeForce RTX 5060 TI' -> 'GeForce RTX 5060 TI' -> 'RTX 5060 TI'
3. Add a mean/average option to each chart to switch between one graph per GPU to a single graph averaging all GPU's
4. for the power consumption a stacked chart would be great. if not possible then at least an option to display a single graph with sum of power consumption