[serve] Carry replica self-health on handle metric reports

[johntaylor-cell]

[serve] Carry replica self-health on handle metric reports

Second PR of the push-health stack (stacked on #64913).

Why

With P1, replicas that push autoscaling metric reports get health delivery for
free, and everyone else heartbeats. But the most common serving topology —
replicas called through deployment handles — already has another message flowing
to the controller on a fixed cadence: the handle metric report each router
sends for autoscaling. This PR folds replica health into those reports, removing
the P1 heartbeat for two classes of replica:

• replicas behind active handles: their health rides the responses they
  already return;
• handle-owning replicas (e.g. ingress/pipeline deployments): their own
  health rides the handle reports their in-process router already sends.

No new messages, no new RPCs — strictly more payload on existing ones.

What

• Response admission piggyback: ReplicaQueueLengthInfo (the
  rejection-protocol admission message) carries the replica's latest self-check
  (healthy, checked_at, consecutive_failures) on both accept and reject.
• Router carried-health map: the router records the newest entry per replica
  (out-of-order responses resolved by checked_at; the recording wall-clock is
  kept alongside so age decisions never compare across machine clocks), prunes
  entries for departed replicas on membership updates, and drops entries not
  refreshed within 30 s — the controller already has those, and re-sending them
  bloats every report at high fan-in.
• Self-health provider: the replica publishes a process-local provider;
  routers living inside that replica merge the host's own health into their
  handle reports. The merge is exception-isolated — health can never break the
  metrics path.
• Controller: entries arriving on handle reports feed the same push registry
  P1 introduced; from there the existing freshness/fallback logic applies
  unchanged.

Suppression (no double delivery, no lost heartbeats)

• A replica whose metric reports already carry health returns no self-entry
  (reports_carry_health gate) — one channel per fact.
• Consuming the self-entry stamps handle-carried recency, which suppresses the
  P1 heartbeat; if handle reports stop (traffic ceases, handle closed), the
  stamp ages out and the heartbeat resumes on its own. Fail-open in both
  directions.

Testing

• New unit suites: self-entry gating and consumption stamping; router merge
  (newest-wins, age-prune, membership prune, provider exception isolation);
  controller forwarding into the registry.
• test_deployment_state.py + test_metrics_utils.py + test_autoscaling_policy.py
  green on this tier (375 tests).

Benchmarks

Full-stack numbers are in #64909's table (8K/16K, HAProxy off: 52.1/42.9 ms
loops, ≤0.55% health-deadline misses). This PR's isolated contribution
(P1-only vs P1+P2 arms on the same harness) is being measured and lands here
before review.

[gemini-code-assist]

Code Review

This pull request introduces a replica-pushed self-health mechanism in Ray Serve, allowing replicas to periodically evaluate and push their health status to the controller to bypass pull probes. It also optimizes rank-consistency checks by gating them on replica membership changes. The review feedback focuses on critical performance and reliability improvements: optimizing the pruning logic in ReplicaHealthPushRegistry to run periodically rather than on every record() call under high scale, falling back to a default health check period when target state info is unavailable to prevent false systemic stall verdicts, and bypassing heartbeat suppression when a replica is unhealthy to ensure immediate failure detection.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit c39f9d0. Configure here.}

[johntaylor-cell]

Abandoning