[Testing] Periodic metric-name drift detection for vLLM, SGLang, and TGI

[Bslabe123]

Part of the #621 testing work for the v0.7.0 tracker (#606). Builds on the replayed-/metrics e2e fixture landing with #382.

Problem

inference-perf hardcodes the Prometheus metric names it queries for every supported model server, and nothing tells us when an upstream server renames, removes, or re-namespaces one:

• vllm_client.py: ~25 names under vllm: (vllm:num_requests_waiting, vllm:time_to_first_token_seconds, vllm:request_time_per_output_token_seconds, vllm:inter_token_latency_seconds, vllm:prompt_tokens, vllm:generation_tokens, vllm:kv_cache_usage_perc, ...)
• sglang_client.py: 9 names under sglang:
• tgi_client.py: 6 names under the tgi_ prefix

The failure is silent by construction. A stale name produces no error: the PromQL matches nothing, the query returns an empty result, and the affected field in summary_prometheus_metrics.json is absent or zero. A user reading the report cannot distinguish "the server did not report this" from "we asked for a name that no longer exists."

We already have one instance in flight: #382 updates sglang:cache_hit_rate to sglang:token_usage, a rename that was caught by hand rather than by CI.

Why it matters

This is the #564 family. Nothing fails, a number is just missing or wrong, and it stays that way until someone notices a blank column. The blast radius is every server-metrics section we publish, and the probability rises over time because all three servers version independently of us.

TGI is the sharpest case: it is the least active of the three upstreams and the one we exercise least, so a rename there could sit undetected for a full release cycle.

Proposed shape

A scheduled job whose output is a fixture-refresh PR, not a test failure:

1. On a schedule (weekly is enough), start the latest release of each supported server: vLLM, SGLang, TGI.
2. Scrape /metrics once and write the response to the checked-in fixture for that server, stamped with the server version it came from.
3. If the fixture is unchanged, exit quietly. If it changed, open a PR with the new dump.

That PR then runs the normal merge-blocking e2e test, which replays the fixture through a real Prometheus and asserts the assembled report. A metric we no longer recognize shows up there as a report field going empty against the golden. So drift detection needs no new assertion machinery: it reuses the blocking test, and the failure lands on one attributable PR that someone owns instead of on every open PR at once.

Two things that keep this cheap:

• No benchmark run is required. All three servers register their metric families at engine init, so a bare startup plus a single scrape is enough to see the names.
• No new test to maintain. The job's only logic is start, scrape, diff, open PR.

Explicitly not merge-blocking

Deliberate. If latest upstream gated merges, a rename would turn every open PR red simultaneously, including the PR that fixes it. A moving external dependency cannot sit in the merge gate. Advisory and periodic is the correct contract here, and it is why this row does not gate a release even though it lives in the live tier.

Update (Aug 17): for vLLM this changed with #697: every release in e2e/vllm_releases.txt, including latest-x86_64, gates at merge. The "fixing PR is red too" premise does not hold once the table and declarations live in the tree. The scheduled, non-gating shape stays the plan for SGLang and TGI, which have no CPU tier.

Relationship to existing work

• Cleanup Prometheus Metric Querying #382 lands the hermetic half: a captured /metrics dump replayed through a real Prometheus, with the assembled report asserted against a golden. That is what makes this job's PR meaningful. This issue should not block Cleanup Prometheus Metric Querying #382.
• [Testing] Identify accelerator-backed CI capacity for the release-blocking live e2e jobs #641 provides the capacity. vLLM, SGLang, and TGI all want an accelerator for a normal startup. If we would rather not wait, a CPU build of vLLM would decouple at least the vLLM half at the cost of image build time in the job; worth deciding when we pick this up.
• [Testing] Unit tests for the Prometheus query path (query builder, execute_query, collection flows) #660 is the unit tier for the same surface and is unaffected. Note that neither this nor the e2e work moves the coverage number, since the gate in scripts/check_coverage_regression.py runs pytest --cov=inference_perf tests/ and does not include e2e/.
• The repo has no scheduled workflow today and no dependabot/renovate precedent. GitHub Actions schedule: is the lower-friction path since e2e already lives there; a prow periodic in kubernetes/test-infra is more idiomatic for k8s-sigs but adds a cross-repo review.

Known limit

This catches renames, removals, and re-namespacing, because those show up as a name appearing or disappearing from the dump. It does not catch a metric that keeps its name while changing units or semantics. Nothing short of value-level review catches that, and it is the same silent-wrongness family this issue is trying to shrink.

[Bslabe123]

Follow-up from the #606 lane discussion: jjk-g raised investigating vLLM in CPU mode in GitHub Actions.

If that works, it removes this issue's dependency on #641. The job only needs a server to start and expose /metrics; no benchmark run and no accelerator throughput is required, so CPU mode is sufficient for the vLLM half even if it is far too slow for anything per-change. Worth checking before assuming this waits on accelerator capacity.

[Bslabe123]

Follow-up for the drift oracle, surfaced by #568.

Both halves decide whether a declared name resolves by restating the exposition's
naming convention in the test util: metric_families.is_exposed/in_golden
(#697) and server_metric_names.resolves (#737) each hardcode "a counter matches
X or X_total". That is the convention Metric.get_queries already
implements, written a second and third time, and the copies drifted.
CounterMetric("vllm:prompt_tokens") built increase(vllm:prompt_tokens{...}),
the bare name only, while the oracle accepted the declaration because
vllm:prompt_tokens_total is in the v0.26.0 golden. The drift check passed on a
metric whose query selects nothing, and prompt_len/output_len have read 0
against a stock v0.26.0 server on any Prometheus. #455 fixed several of these by
hand in April and left vllm:prompt_tokens and vllm:generation_tokens bare.

#568 fixes the query side, which makes the tolerance honest again, but leaves the
duplication in place.

Proposal: derive the candidates from the metric object rather than restating
them. Metric.candidate_names() -> Sequence[FrozenSet[str]], groups OR'd, names
within a group AND'd:

• CounterMetric("X") -> [{"X_total"}, {"X"}]
• GaugeMetric("X") -> [{"X"}]
• HistogramMetric("X") -> [{"X_bucket", "X_count", "X_sum"}]

is_exposed becomes any(g <= names for g in metric.candidate_names()), and
declared_metrics yields metric objects instead of (name, type) pairs. Run
against the pre-#568 declarations this goes red on vllm:prompt_tokens, which is
the property the check was supposed to have.

Natural home is the module collapse already noted in #737: fold
server_metric_names.py and metric_families.py into one prefix-parameterized
module and state the convention once.