[Serve] Fix proxy state crash on ActorUnschedulableError during RayService incremental upgrade

[ryanaoleary]

Description

Fixes an issue where the Ray Serve controller loop crashes with an uncaught ActorUnschedulableError when attempting to shut down a proxy pinned to a dead or non-existant node. This PR wraps the shutdown ray.get() call in a try...except block, allowing the controller to gracefully clean up the dead proxy handle.

When does this bug occur?

This race condition occurs when a proxy actor is created on a node that dies while the creation is still pending. It can be seen during RayService incremental upgrades when GCS Fault Tolerance with Redis is enabled:

1. During an incremental upgrade a new RayCluster spins up to serve traffic in parallel with the old cluster and connects to Redis, restoring old proxy actors pinned to pre-upgrade node IDs.
2. Since the old node is dead/doesn't exist on the new cluster, proxy creation gets stuck in PENDING_CREATION.
3. The Serve controller detects the node ID is not in alive_node_ids and calls shutdown() on the pending proxy handle. ray.get() throws an ActorUnschedulableError, crashing the controller.

Related issues

This issue was brought up in discussion in the KubeRay slack channel.

Additional information

[gemini-code-assist]

Code Review

This pull request updates the proxy actor shutdown logic in proxy_state.py by handling ActorUnschedulableError and wrapping the shutdown and kill operations in try...except blocks to log warnings on failure. Feedback was provided to refactor the kill method because placing the is_shutdown() check inside the graceful shutdown try block could lead to misleading error logs and redundant kill attempts if the initial kill fails.

[ryanaoleary]

reopened here: #65076, meant to push to my fork