[Serve] Fix proxy state crash on ActorUnschedulableError during RayService incremental upgrade

[ryanaoleary]

Description

Fixes an issue where the Ray Serve controller loop crashes with an uncaught ActorUnschedulableError when attempting to shut down a proxy pinned to a dead or non-existant node. This PR wraps the shutdown ray.get() call in a try...except block, allowing the controller to gracefully clean up the dead proxy handle.

When does this bug occur?

This race condition occurs when a proxy actor is created on a node that dies while the creation is still pending. It can be seen during RayService incremental upgrades when GCS Fault Tolerance with Redis is enabled:

1. During an incremental upgrade a new RayCluster spins up to serve traffic in parallel with the old cluster and connects to Redis, restoring old proxy actors pinned to pre-upgrade node IDs.
2. Since the old node is dead/doesn't exist on the new cluster, proxy creation gets stuck in PENDING_CREATION.
3. The Serve controller detects the node ID is not in alive_node_ids and calls shutdown() on the pending proxy handle. ray.get() throws an ActorUnschedulableError, crashing the controller.

Related issues

This issue was brought up in discussion in the KubeRay slack channel.

Additional information

[gemini-code-assist]

Code Review

This pull request improves the robustness of the proxy actor shutdown and health check logic in proxy_state.py by handling ActorUnschedulableError when a node dies during creation, and wrapping shutdown and kill operations in try-except blocks. The review feedback suggests avoiding redundant ray.kill calls when the actor is already shut down, and using the imported ActorUnschedulableError directly instead of its fully qualified name.

[jeffreywang88]

great find! can we add some tests?

[abrarsheikh]

@ryanaoleary please rebase your PR after #64403 merged. your covers the kill case as well which is worth fixing.

[abrarsheikh]

connects to Redis, restoring old proxy actors pinned to pre-upgrade node IDs.

this is true? why should the new cluster restore its state from old clusters checkpoint? that sounds like a design flow in RayService incremental upgrade no?