fix(worker): pop launch_ts in recover_model to avoid strict-constructor crashes

[m199369309]

Summary

• Bug: OOM auto-recovery (recover_sub_pool → recover_model) crashes strict-constructor models (e.g., jina-reranker-v3) with TypeError: unexpected keyword argument 'launch_ts'. The bug is that recover_model splats the cached launch snapshot (which includes launch_ts, an internal timestamp) back into launch_builtin_model(**launch_args), injecting launch_ts into the model's self._kwargs. Models that forward the full self._kwargs into a strict constructor (like AutoModelForCausalLM.from_pretrained(**model_kwargs)) then crash.
• Fix: Pop launch_ts from launch_args in recover_model before the splat, mirroring the existing cleanup in recover_models_on_startup (line 788). Copy the dict first to avoid mutating the cached snapshot in self._model_uid_to_launch_args (which legitimately uses launch_ts as created_ts).
• Impact: Fixes OOM auto-recovery for all strict-constructor models (jina-reranker-v3, qwen3 rerank, Qwen3-VL-Embedding, etc.). Without this fix, the replica stays permanently down after OOM even though recover_sub_pool detects the crash and triggers recovery.

Root cause analysis (from production logs)

16:47:40.000 Process xinference-worker-4090-1:42155 is down.                    # 10: monitor detects crash
16:47:40.012 Recreating model actor 24a22820-9128-...-0 ...                    # recover_sub_pool triggers
16:47:52.717 Starting ModelActor at ...:34755 (new pid 966847)                 # new subpool created
16:47:53.234 Loading qwen3 rerank with kwargs {'device_map': 'auto', 'launch_ts': 1780900592}  # ⚠️ launch_ts leaks
16:47:53.246 Failed to load model ... TypeError: ... unexpected keyword argument 'launch_ts'    # ❌ crash
16:47:53.330 Launch finished: model_name=jina-reranker-v3 ... (active: 0/5)    # replica stays down

Why embedding v3/v4 survived: They use .get() to selectively extract kwargs (not a full splat), so launch_ts was never passed to the constructor. Only models that forward the entire self._kwargs via ** are affected.

Test plan

• Unit test added: test_recover_model_pops_launch_ts_from_kwargs in xinference/core/tests/test_worker.py — verifies launch_ts is removed from kwargs passed to launch_builtin_model, and that the original cached snapshot still contains launch_ts (copy isolation).
• Real-world test: Deploy jina-reranker-v3 on 4090×2 worker with XINFERENCE_MODEL_ACTOR_AUTO_RECOVER_LIMIT=2, trigger OOM, verify auto-recovery completes within 30s (should see ModelActor loaded instead of TypeError).
• Regression: Ensure embedding v3/v4 OOM recovery still works (should be unaffected since they don't splat self._kwargs).

Related PRs

• fix(worker): ensure subpool monitor starts after append_sub_pool #5004 (merged): Ensured monitor_sub_pools runs so recover_sub_pool detects crashes.
• fix(model): use os._exit for OOM to trigger pool recovery #5005 (merged): Used os._exit(1) so OOM crashes are detectable.
• fix(supervisor): evict dead replica from round-robin when auto-recover exhausted #5006 (merged): Evicts dead replicas from round-robin scheduler when auto-recovery is exhausted.
• This PR: Fixes the latent bug exposed by fix(worker): ensure subpool monitor starts after append_sub_pool #5004/fix(model): use os._exit for OOM to trigger pool recovery #5005/fix(supervisor): evict dead replica from round-robin when auto-recover exhausted #5006 — the recovery chain now works end-to-end, but the recovered model failed to load due to launch_ts leakage.

Blast radius

• Scope: recover_model only. Does not affect recover_models_on_startup (already had the cleanup), launch_builtin_model (no changes), or model-side code.
• Backwards compatibility: The cached snapshot in self._model_uid_to_launch_args is unchanged (copy isolation), so any code that reads launch_ts from the cache (e.g., created_ts computation) is unaffected.
• Risk: Minimal — the fix is a direct port of existing logic from recover_models_on_startup to recover_model.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request addresses an issue where the internal timestamp launch_ts was passed to model constructors during recovery, causing a TypeError in models with strict constructors. The fix ensures launch_args is copied and launch_ts is popped before launching the model. A unit test has been added to verify this fix. I have no feedback to provide as there are no review comments.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[qinxuye]

LGTM