Skip to content

Commit 0177fa0

Browse files
committed
feat: sparkDash-derived host + serving metrics for every gateway
Phase 1/2 of the sparkDash integration, gateway-agnostic end to end (all data flows through the existing agent payloads; no daemon, no web server): Agent 1.2.0 (agent_core collectors, pure parsers, graceful None): - uptime (proc/uptime or kern.boottime), storage (statvfs), network rx/tx rates (/proc/net/dev deltas, loopback/veth excluded) - tailnet self-health (Self.Online, backend state, health warnings - the healthy-on-LAN-but-off-tailnet blind spot), TTL-cached - GPU process names via nvidia-smi --query-compute-apps - live serving rates for running models: llama.cpp /slots per-slot counter diffs, vLLM Prometheus generation/prompt token counters (+ KV cache %, running/waiting), sglang /server_info counters with sticky-throughput expiry. TTL-coalesced, loopback-only unless ALLOW_REMOTE_HEALTHCHECK, idle reads 0, all failures degrade. Swift (additive decode, old agents tolerated): - HostMetricsPayload: uptime_seconds/storage/network/tailscale, HostGPUProcess.name; ModelProfileStatus.serving (ServingMetrics) - Remote Hosts card: uptime in header, STORAGE/NET lines, tailnet status dot with warning reasons, GPU process name under models, tok/s on running rows; dashboard row subtitles prefer live tok/s over stored benchmark rows Tests: 21 new collector/parser tests (failure-first), 5 decoder/ presentation tests, conformance allowlist updated for . Verified collectors live on Spark (uptime 1.0h, storage 71.7%, net 66.7 kbps, tailnet Running/100.122.96.76). 187 swift, 57 python.
1 parent df12b5a commit 0177fa0

12 files changed

Lines changed: 1124 additions & 26 deletions

File tree

CHANGELOG.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,11 @@
33
All notable changes to this project are documented in this file.
44

55
## [Unreleased]
6+
### Added
7+
8+
- sparkDash-derived host visibility for every remote gateway (agent 1.2.0): uptime on the gateway header, storage and network rate readouts, tailnet self-health (online state, backend state, warning reasons - catches the "healthy on LAN but off-tailnet" blind spot), and GPU process names next to running models.
9+
- Live LLM serving rates on model rows and the Remote Hosts panel: decode/prefill tok/s from llama.cpp `/slots`, vLLM Prometheus `/metrics`, and sglang `/server_info` counters, diffed over time so idle reads 0; plus KV-cache usage and request queue depth for vLLM. Loopback-only unless `ALLOW_REMOTE_HEALTHCHECK`, same SSRF posture as health probes; every probe failure degrades to a hidden label, never an error.
10+
611
### Fixed
712

813
- Gateway agent updates can no longer hang forever on "Pushing agent…": each deploy step is killed at a 120s deadline with a pointed message, and Tailscale SSH re-auth prompts ("To authenticate, visit https://login.tailscale.com/…") are classified into remediation instead of stalling BatchMode ssh silently.

0 commit comments

Comments
 (0)