[core][telemetry] Reduce cardinality of Serve metrics to node level

[eicherseiji]

What

Make Serve and vLLM metrics honor Ray's metric cardinality reduction, with RAY_metric_cardinality_level as the single control. At low:

• The OpenTelemetry export drops WorkerId and ReplicaId from any metric carrying a ReplicaId tag, and aggregates the collapsed per-replica series to the node level.
• The Serve controller drops its high-cardinality handle/replica tags (the behavior from Added RAY_SERVE_CONTROLLER_METRICS_INCLUDE_HIGH_CARDINALITY_TAGS to c… #63642), now driven by the same level.

Why

1. The reducer touched only tasks and actors. Every Serve and vLLM metric kept WorkerId/ReplicaId at every level. Those labels are 1:1 per replica, so the series count grew with replicas and never reduced.
2. Added RAY_SERVE_CONTROLLER_METRICS_INCLUDE_HIGH_CARDINALITY_TAGS to c… #63642 added a separate Serve-only env var for the same intent, with a different name and opposite default. Two knobs for one goal.

Posture

Keep tags by default; reduce only at low. Matches #63642's opt-out posture.

• legacy, recommended (default): tags kept.
• low: WorkerId+ReplicaId dropped from Serve/vLLM metrics; controller handle/replica dropped.

tasks/actors keep their existing per-level behavior.

Unifying #63642

RAY_metric_cardinality_level is now the canonical, documented control. RAY_SERVE_CONTROLLER_METRICS_INCLUDE_HIGH_CARDINALITY_TAGS becomes a deprecated alias: its default is derived from the level, and when set explicitly it still wins (backwards compatible) but emits a deprecation warning pointing to the level. Full removal is left to a coordinated follow-up with a deprecation cycle.

Scope

ReplicaId presence scopes the reducer to Serve/vLLM metrics; other user metrics are untouched. The controller change is limited to the four metrics #63642 already gated.

Aggregation

Gauges sum by default (correct for additive gauges like num_requests_running); counters sum; histograms merge buckets. Ray core does not name any metric from another project.

Changes

• metric_cardinality.py: get_high_cardinality_labels_to_drop takes the series tag keys; at low, drops WorkerId+ReplicaId for ReplicaId-bearing metrics. Gauges default to sum.
• open_telemetry_metric_recorder.py: pass the series tag keys at the gauge/counter callback and both histogram paths.
• serve/_private/constants.py: derive the controller knob from the level; deprecate the env var (still honored, warns).
• doc/source/serve/monitoring.md: document the level as canonical; mark the env var deprecated.

Known limitations

• Narrow by design. Reduces ReplicaId-bearing (vLLM) series and the controller tags. Does not reduce the larger Ray-internal histogram bulk (ray_grpc_server_*, etc.) or ray_serve_* latency buckets, which the tag scoping does not reach. Measured: low leaves ray_grpc_server_req_process_time_ms_bucket at 203 series unchanged. Broadening is a follow-up.
• Ratio gauges overcount under sum (e.g. vllm:kv_cache_usage_perc); aggregate at the query layer until the metric owner can declare its aggregation.
• OpenCensus export path is not covered; it is not the default backend.

Relationship to #63642

#63642 (merged) drops handle/replica from controller metrics at emit time, on the head node. This PR drops WorkerId/ReplicaId from replica/vLLM metrics at the reporter agent, and folds #63642's knob under the shared level. Drop location stays different by necessity: the controller aggregates centrally, replica metrics can only be summed at the agent.

Tests

Run locally against a per-commit wheel.

Reducer path (test_metric_cardinality.py):

• Unit: 8 passed. Integration test_serve_metric_cardinality: 3 passed. Original test_cardinality_recommended_and_legacy_levels: 9 passed (no regression).

Controller unification:

• New unit test serve/tests/unit/test_metric_cardinality_constants.py: 5 passed (level drives the default; alias overrides; deprecation warning emitted).
• E2E: ran Serve with two replicas and scraped ray_serve_deployment_replica_healthy. recommended shows two per-replica series with a replica tag; low collapses to one series with no replica tag and value 2.0 (healthy count); explicit env var restores the tag.

AI assistance (Claude Code) was used to write this change.