fix(supervisor): evict dead replica from round-robin when auto-recover exhausted

[m199369309]

Summary

• When worker exhausts AUTO_RECOVER_LIMIT and stops recreating a model actor, the dead replica remains in supervisor's round-robin scheduler and model_unexpected_termination gauge is never lit. Requests routed to the dead replica fail silently, and ops has no observability signal that N-1 out of N replicas are healthy.
• Add supervisor.mark_replica_dead() that evicts the dead replica from the round-robin scheduler, lights up model_unexpected_termination, and advances base_uid to TERMINATED when the single/last replica dies. The method is idempotent and deliberately does not call back worker.terminate_model (the worker already terminated the model locally before giving up recreating).
• Worker calls back supervisor.mark_replica_dead() after "Stop recreating model actor." with a 5s xo.wait_for timeout guard so a stalled supervisor does not hold up the worker's local shutdown. Failure/timeout is non-fatal — the next death detection or redeploy will reconcile.
• Add regression tests in both sync and async client test suites (test_model_oom_recover_exhausted_evicts_replica).

Depends on #5005 (fix/model: use os._exit for OOM to trigger pool recovery) which provides the os._exit(1) fix that makes OOM-triggered recovery cycles actually work. This PR includes #5005's commits as its base; the PR3-specific changes are in the supervisor.py mark_replica_dead method, the worker callback, and the eviction test.

Test plan

• Run test_model_oom_recover_exhausted_evicts_replica in both test_client.py and test_async_client.py — verifies dead replica is evicted after recover limit exhausted
• Run existing test_model_oom_triggers_restart (from fix(model): use os._exit for OOM to trigger pool recovery #5005) — verifies OOM still triggers subprocess exit
• Run existing test_auto_recover — verifies normal auto-recovery still works
• Manual test: deploy a model with XINFERENCE_MODEL_ACTOR_AUTO_RECOVER_LIMIT=1, trigger repeated OOM, verify model_unexpected_termination gauge lights up and list_models() drains to empty

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request improves Out-Of-Memory (OOM) recovery by switching from sys.exit(1) to os._exit(1) in the model actor, ensuring xoscar can detect subprocess failures and trigger pool recovery. It also implements replica eviction in the supervisor when recovery limits are exhausted and adds tests for these scenarios. The review feedback correctly points out that the new asynchronous tests use blocking time.sleep(1) calls, which should be replaced with await asyncio.sleep(1) to prevent blocking the event loop.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[qinxuye]

Two issues to address before LGTM.

[qinxuye]

LGTM