LemonTop

LemonTop is a read-only, btop-style terminal monitor for Lemonade Server. It combines Lemonade request statistics and live inference logs with Linux host telemetry, AMD GPU/APU sysfs counters, GTT/VRAM usage, and recent journalctl events.

The app makes no system configuration changes. It does not load/unload models, restart services, or modify Lemonade.

What it monitors

• Lemonade health, API latency, loaded model/backend when reported
• Last-request output throughput, prompt throughput, TTFT, and token counts
• Phase-aware live inference activity: generic processing, prompt ingestion, token generation, and context-limit failures when recognizable log records are available
• Live prompt progress, token counts, speed, elapsed time, and ETA, followed by generated-token counts and rolling output speed
• Context and KV-cache indicators when exposed by the active backend
• Active/queued requests when exposed by the server
• CPU, load average, RAM, swap, disk I/O, and network I/O
• AMD GPU utilization, GTT, fixed VRAM, temperature, power, and clock via sysfs
• NPU utilization when Lemonade reports it
• Model-storage usage from Lemonade system information
• Recent lemond.service journal entries

Every optional metric displays — when unavailable. Older Lemonade releases without /v1/system-stats continue to work using local system sources.

Install on the Lemonade host

Python 3.10 or newer is required. On Ubuntu, one command does everything — prerequisites, clone, venv, and lemontop on your PATH:

curl -fsSL https://raw.githubusercontent.com/cbigeagle/LemonTop/main/install.sh | bash

See INSTALL-linux.md for the manual steps, permissions, and troubleshooting.

Manual install:

cd /path/to/LemonTop
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e .

Run it:

lemontop

Or without installing the console command:

PYTHONPATH=src python -m lemontop

Demo mode

Review the complete interface without a live Lemonade server:

lemontop --demo

Configuration

The default server is http://localhost:13305.

lemontop --server http://localhost:13305
lemontop --refresh 0.5
lemontop --service lemond.service
lemontop --no-logs

Equivalent environment variables:

Variable	Default	Purpose
LEMONTOP_SERVER	http://localhost:13305	Lemonade root URL
LEMONTOP_REFRESH	1.0	Poll interval in seconds
LEMONTOP_TIMEOUT	0.8	Per-request timeout
LEMONTOP_SERVICE	lemond.service	systemd unit for logs

LemonTop tries both the compatibility prefix /api/v1 and the current /v1 prefix for health, stats, system-stats, and system-info. The prefix that answers first is reused for later polls, so only one request per endpoint is sent once the server is identified.

When Lemonade advertises a log WebSocket in its health response, LemonTop subscribes to it and extracts phase-aware inference telemetry in real time. It tracks generic request processing, llama.cpp prompt processing and generation, and context checkpoint/limit records. On the Lemonade host, recent journal lines provide a fallback if the stream is absent or reconnecting. The inference panel shows only values present in the backend logs; unavailable details remain —. --no-logs disables both log sources and therefore the enhanced activity view.

Keyboard controls

Key	Action
q	Quit
p	Pause/resume updates
r	Refresh now
l	Focus event log
t	Focus request table
?	Show key help

Permissions

Normal user access is enough for Lemonade and host metrics. Journal entries may require membership in systemd-journal (or equivalent local policy):

sudo usermod -aG systemd-journal "$USER"

Log out and back in after changing group membership. This is optional; LemonTop continues without journal access.

AMD metrics are read from:

/sys/class/drm/card*/device/

Availability and permissions vary by kernel/driver. LemonTop does not require amd-smi; server-provided telemetry and Linux sysfs are preferred.

Verification

After installation:

python tests/test_smoke.py
lemontop --demo

The tests launch the Textual app in headless test mode and verify the status line, request table, event-log tailing, and metric formatting. They also run under pytest tests/ if you have it installed.

Current limitations

• Lemonade's stats endpoint documents performance for the last request, not a durable request ledger. The live collector keeps a small in-memory history of changed stats observed while LemonTop is running. Because the endpoint carries no request id, two identical back-to-back requests are indistinguishable and collapse into a single row.
• Journal events require Linux with journalctl on PATH; elsewhere the event panel stays empty rather than reporting an error every second.
• Only the first AMD card under /sys/class/drm/card* is read. On a system with an APU plus a discrete GPU, the second device is not shown.
• Context size, KV-cache size, prompt throughput, queue depth, and loaded-model details depend on the Lemonade version/backend and may not be reported.
• Detailed live activity and context metrics depend on the active backend's log format. LemonTop recognizes current llama.cpp prompt-processing, generation, context-checkpoint, and context-size records plus Lemonade's normalized progress records. Other modalities may expose only generic PROCESSING.
• GTT use is read from amdgpu sysfs where supported. Some kernels expose only the configured total or provide counters with different semantics.
• No history is persisted between LemonTop runs.
• Remote monitoring shows remote Lemonade data but local CPU/RAM/sysfs and journal data from the machine where LemonTop itself is running. For a coherent view, run LemonTop on the Lemonade host over SSH.