[serve] Gate rank-consistency check on replica membership changes

[johntaylor-cell]

[serve] Gate rank-consistency check on replica membership changes

Why

The Ray Serve controller runs on the head node, continuously reconciling deployments and
making autoscaling decisions. check_rank_consistency_and_reassign_minimally runs at the
tail of every DeploymentState.update() — once per deployment per control-loop tick,
which at measured loop rates is several times a second.

The pass is O(N) in replicas with large constant factors: it materializes the active-key
set and the rank-key set, takes set differences both ways, copies the entire _ranks
dict, tallies per-rank counts across every active key, and sorts all rank values — then
the manager above it regroups replicas per node. In steady state every one of those
passes re-derives an identical answer. At 16K, py-spy showed it monopolizing the event
loop and starving handle-report ingest for 20 s+.

Ranks can only become inconsistent when replica membership changes, so gate the pass on
the active replica-id set: run it when the set differs from the last checked one, skip
when it does not. Transitions are unaffected — membership churn during rollouts and
autoscaling changes the set, so the pass still runs exactly when it can matter. The
guards that were already there (deployment not HEALTHY, or any STARTING replica) run
first, before the replica list is materialized.

RAY_SERVE_FAIL_ON_RANK_ERROR defaults off, so in production the pass can log an error
and return a safe default. When that happens the membership is deliberately not
cached and the check is retried next tick, rather than latching a membership the pass
never validated. The error flag is read before the reassignment reconfigure, which calls
back into the rank manager and resets it.

The gate's own cost is the remaining O(N) here: building the id set measures 1.9 ms at
16,384 replicas, ~4.5% of the optimized loop. #64910, stacked on this PR, replaces the
set comparison with a ReplicaStateContainer mutation-version integer, which removes it.

What

• Extract the pass into _maybe_check_rank_consistency().
• Gate it on the set of active replica ids: if the set is unchanged since the pass last
  ran, skip it.
• Only record a membership as checked when the pass did not swallow an error.
  RAY_SERVE_FAIL_ON_RANK_ERROR is off by default, so the pass can catch an exception and
  return []; caching that would mean a stable-but-inconsistent deployment never retries.
  DeploymentRankManager now records whether the last rank op errored.
• The existing guards are preserved verbatim: only when the deployment is HEALTHY, and
  never while STARTING replicas exist (the node-migration case documented in the original
  comment).

Why this is safe

Rank consistency can only be violated by a membership change — replicas added, removed, or
replaced — and every such change alters the active id set, so the pass still runs on
exactly the ticks where it can find work. Rank release and removal from _replicas happen
in the same pop(states=[STOPPING]) branch, so the id set and the rank table cannot drift
apart across ticks. The per-node and node-rank passes group off _replica_to_node, which
is only mutated by rank assign/recover/release, so a fixed replica-id set implies a fixed
node grouping.

Set comparison is exact — there is no fingerprint collision to reason about. The residual
cost is the O(N) set construction itself (~1.9 ms/tick at 16K replicas); the follow-up
#64910 replaces it with an exact ReplicaStateContainer mutation counter, making the gate
O(1).

Results

• Follow-up profile of the same 16K workload: the rank pass no longer appears in
  controller CPU samples; report ingest resumes (stale-report drops stop).
• Composite (with the rest of the optimization stack): 16K steady-state control loop
  352.6 ms → 42.9 ms.

Testing

• 6 new unit tests. Four over the gate itself: runs once then skips on stable
  membership; re-runs on membership change; a swallowed rank error is not cached and
  retries next tick; skips entirely (without recording a membership) while STARTING
  replicas exist. Two over a real DeploymentRankManager(fail_on_rank_error=False) —
  the production default — covering that a swallowed error is not cached, and that the
  error flag is read before _reconfigure_replicas_with_new_ranks can clear it.
• Full test_deployment_state.py suite green (237 tests).

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit 7fa7c85. Configure here.}

[gemini-code-assist]

Code Review

This pull request optimizes the deployment state machine by gating the expensive rank-consistency check so that it only runs when replica membership changes. The current implementation uses a fingerprint calculated via reduce and xor over replica IDs. The review feedback suggests a more robust and idiomatic Python approach: storing and comparing a set of active replica IDs instead of an integer fingerprint. This eliminates the risk of hash collisions, removes unnecessary imports, and improves performance by leveraging optimized set comparisons. Corresponding updates to the unit tests are also provided.

[abrarsheikh]

Attach the before/after CPU profile in the PR description.

QQ: does the issue go away if we flip the default for _fail_on_rank_error to true?

[johntaylor-cell]

Attach the before/after CPU profile in the PR description.

QQ: does the issue go away if we flip the default for _fail_on_rank_error to true?

No, the flag only decides what happens after the pass runs.
check_rank_consistency_and_reassign_minimally does its full O(N) scan inside
_check_rank_consistency_impl and then hands the result to
_execute_with_error_handling, which propagates or swallows depending on the flag. The
per-tick cost is identical either way; fail-fast just turns a logged violation into a
raised one.

Skipping the work is what this PR does: gate the pass on the active replica-id set, since
rank consistency can only be violated by membership changes.