[serve] Cache per-tick replica id walks behind a container mutation version

[johntaylor-cell]

Stacked on #64911 (the rank-consistency
membership gate; this PR replaces its per-tick id-set rebuild with the container key
below).

Why

A 200 Hz CPU profile of a controller managing 16K replicas showed the control loop
dominated not by any single pass but by five separate O(N) id-collection walks
recomputed every tick, while metric-report ingest (<1% of samples) starved behind
them:

• DeploymentState.get_alive_replica_actor_ids — called every tick for
  stale-handle-metric pruning
• DeploymentState.get_active_node_ids — every tick via _update_proxy_nodes
• DeploymentState.get_running_replica_ids — every tick for autoscaler registration
• the autoscaler's id-set rebuild (to_full_id_str() × N, every tick)
• the rank-consistency membership key from
  #64911 — an id-set rebuild over N
  replicas, after copying the full replica list

Each walk drags N replica objects through Python property chains to recompute a
result that only changes when membership changes — and in steady state membership
doesn't change for hours.

What

• ReplicaStateContainer gains a membership key: the summed per-replica hash of its
  {replica id -> state} content, maintained incrementally on add/pop/remove, and
  returned paired with the exact replica count. It is a hash of the content rather than a
  counter of mutations because the health and migration passes pop a whole bucket and
  re-add every replica to the state it came from: gang deployments take that path on
  every tick, and every other deployment takes it whenever any node in the cluster is
  draining, which is 5 mutations per otherwise-idle tick at 4 replicas. A counter reads
  that churn as a membership change and gives back none of the win. Summing per-replica
  terms makes pop-and-re-add cancel exactly, while a real arrival, departure or state
  transition still changes it.
• Unlike [serve] Gate rank-consistency check on replica membership changes #64911's id-set comparison, which was exact, this is a hash: two memberships
  of the same size collide with probability ~2^-64. The pairing with the exact count
  removes every collision where cardinality differs. The asymmetry worth knowing is that
  a stale memo self-corrects on the next real membership change, whereas a skipped rank
  pass would not, which is why the count is carried and why the fingerprint is pinned to
  a full recompute in the tests.
• The derived collections memoize on that key at both DeploymentState and
  DeploymentStateManager level, through one shared _memoized_walk helper.
• Caching disarms while any STARTING, UPDATING or RECOVERING replicas exist: the
  startup check calls check_ready() for all three, and that rewrites
  actor_id/node_id in place without a container mutation. During ramp, rollout and
  recovery every getter recomputes exactly as before; the cache engages only at steady
  state, which is where the waste was.
• The autoscaler skips its id-set rebuild when handed the same (cached) list object it
  already processed. That is safe because _running_replicas and
  _cached_running_replica_strs are only ever written together and both start empty, so
  the pair cannot desync — note that re-registering a deployment reuses the existing
  DeploymentAutoscalingState rather than presenting a new list object.
• #64911's membership key and its
  error backoff both become this key — no walk at all, and the gate's check now sits
  before the get() list copy instead of after it.

Two deliberate deltas from the gate as it stands in #64911: the empty-membership early
return now caches before returning, and the errored-pass path clears the key.

Cached collections are shared objects, so the getters return them frozen
(FrozenSet/Tuple) and the one caller that needs a mutable set copies at its own
boundary; every other caller only reads them.

Results

Isolated microbench of the three DeploymentState walks at 16K replicas:
3.23 ms → 0.008 ms per tick (real DeploymentReplica property chains cost
more than the benchmark stand-ins, so live savings are larger).

On the full harness (16K replicas, single deployment, pinned, steady state,
36 samples): control loop 352.6 ms → 42.9 ms, deployment-state update
253.8 ms → 17.1 ms, 1.36 → 4.99 loops/s — and the first run on this
harness to place all 16,384 replicas and complete cleanly; without this change
the controller starves its own report ingest at that scale. (Composite numbers
measured with the push-health stack on top; isolated this-PR-vs-base cells on the
same harness are running and will replace this table before review.)

Testing

• The key changes only on a real membership change: empty pop/remove do not change it,
  nor does popping a bucket and re-adding every replica to the state it came from, while
  a transition, an arrival and a departure each do. A separate test recomputes the key
  from every bucket and asserts it matches after a mixed add/pop/remove/re-bucket
  sequence, so a future bucket write that forgets to maintain it fails loudly.
• Getters memoize and invalidate on mutation; STARTING/UPDATING/RECOVERING disarms
  caching; autoscaler identity-skip (same object skipped, new object rebuilt).
• Churn coverage, since that is what a counter got wrong: the memos and [serve] Gate rank-consistency check on replica membership changes #64911's rank
  gate both hold across ticks with a node draining elsewhere in the cluster, and across
  steady-state ticks of a gang deployment. Each of these witnesses the churn through
  bucket-list identity so none can pass vacuously, and each is verified to fail if the
  key's per-replica terms are made non-cancelling.
• An errored pass drops the previously validated key, so a membership it validated
  before a failure is rechecked rather than skipped forever.
• test_deployment_state.py + test_autoscaling_policy.py: 287 tests, green under the
  default env and all four CI env variants (_with_pack_scheduling, _metr_disab,
  _metr_agg_at_controller, _metr_agg_at_controller_and_replicas).

[gemini-code-assist]

Code Review

This pull request introduces performance optimizations to Ray Serve's deployment state management by caching and memoizing expensive per-tick id-collection walks (such as alive replica actor IDs, running replica IDs, and active node IDs) using a mutation version tracker on the replica container. It also optimizes the rank-consistency check to run only when replica membership changes. The reviewer feedback focuses on ensuring backward compatibility during rolling upgrades or controller recoveries by defensively using getattr to access newly introduced attributes on deserialized objects, and preventing state corruption by returning immutable collections (frozenset and tuple) from cached methods.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit c7f6a48. Configure here.}