fix(worker): strip test envs before recover and clean up GPU orphans

[m199369309]

Summary

• Strip XINFERENCE_TEST_* envs from cached launch_args before recover and startup recovery. Test-injected env vars (e.g. OOM simulation flags) were being cached alongside model parameters and replayed on every recover, causing infinite recover loops where the model repeatedly crashes with the same test env.
• NVML-based VRAM polling after terminate in recover_sub_pool. Replaces implicit reliance on process exit timing with explicit VRAM free-ratio check (90% threshold, 30s timeout). Gracefully degrades to psutil if pynvml is unavailable.
• SIGKILL orphan GPU processes after terminate_model in both recover and normal terminate paths. Spawn-created EngineCore subprocesses can survive stop() + remove_sub_pool as orphans, holding ~21.5G VRAM and blocking subsequent launches. Uses cmdline verification (vllm/enginecore/model_uid) to safely identify orphans.
• Per-uid persist dirty tracking with retry limit (_PERSIST_RETRY_MAX=3) prevents infinite retry storms when disk writes fail.
• Startup recovery path (_try_recover_models) also strips test envs from persisted launch_args.

Test plan

• Unit tests: _strip_test_envs (13 test cases covering prefix matching, reference isolation, non-dict envs, exception safety)
• Unit tests: GPU utility functions (_parse_gpu_indices, _snapshot_gpu_occupying_pids, _snapshot_gpu_free_ratio, _kill_orphan_gpu_pids)
• pytest xinference/core/tests/test_strip_test_envs.py -v passes
• pytest xinference/core/tests/test_launch_orphan_cleanup.py -v passes
• TP=1 OOM self-heal: terminate → VRAM returns to 0 → clean relaunch succeeds
• TP=2 OOM self-heal: VRAM polling confirms release before recover
• Multi-model same worker: orphan cleanup doesn't affect other models' processes

Root cause

When models are recovered (OOM self-heal or worker restart), cached launch_args containing test-injected env vars are replayed verbatim, causing the model to crash again with the same test flag. Additionally, spawn-created EngineCore subprocesses (introduced for fork deadlock avoidance) survive stop() as orphans, holding GPU VRAM and causing ValueError: Free memory on device ... less than desired on subsequent launches.

Design notes

• pynvml is an optional dependency — all GPU utility functions gracefully degrade when unavailable
• Test env convention: all test-only env vars must use XINFERENCE_TEST_ prefix
• Orphan identification uses cmdline matching (not PID diff) to correctly handle the terminate path where the orphan was already running before stop
• Worker PID and other models' known PIDs are excluded from orphan candidates

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces GPU orphan process cleanup and VRAM reclaim waiting during model recovery and termination, along with associated unit tests. The review feedback highlights several critical issues: the process-killing logic is too broad and risks terminating unrelated processes, the unit tests reference non-existent helper functions, the VRAM polling loops do not handle missing pynvml gracefully (leading to unnecessary 30-second delays), and pynvml resources are not properly released via nvmlShutdown.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[qinxuye]

One new issue found.

[m199369309]

Addressed review finding:

model_uid provenance for vLLM child processes (qinxuye): Fixed with a two-pronged approach:

1. Process title marker: VLLMModel.load() now sets XINFERENCE_MODEL_UID env var before engine creation. Both distributed_executor.py and distributed_executor_v1.py WorkerActors now include [model_uid] in their process title: Xinf vLLM worker: <rank> [<uid>].

2. Ancestry walk for EngineCore: EngineCore is forked by vLLM itself — we can't control its process title, but it inherits the environment from its parent. Added _process_or_ancestor_has_uid() helper that walks up the parent chain checking cmdline. This catches EngineCore whose parent (the vLLM worker) now has model_uid in its title.

3. Test coverage: Added test_kill_orphan_gpu_pids_model_uid_filter (covers: worker with uid in cmdline → killed, EngineCore with uid only in parent → killed, unrelated process → skipped) and test_kill_orphan_gpu_pids_model_uid_skips_other_model (covers: other model's worker → skipped).

Other findings (already fixed in prior commits, no changes needed):

• Gemini Critical ×2 (recover/terminate path missing model_uid check): both call sites already pass model_uid
• Gemini High ×2 (missing helper functions): _kill_orphan_gpu_pids and _wait_pids_dead are defined in worker.py
• Gemini High ×2 (VRAM reclaim loop not breaking on -1): both loops already have if _free_ratio < 0: break
• Gemini Medium ×3 (pynvml not shut down): all three call sites already have try...finally with nvmlShutdown()

[m199369309]

Terminate path missing GPU indices when gpu_idx=None (qinxuye): Fixed.

When gpu_idx=None is passed to launch (the default), _create_subpool() auto-selects GPUs via allocate_devices() but doesn't write them back to launch_args. The terminate path only checked launch_args["gpu_idx"], so _gpu_indices_for_terminate stayed empty and orphan cleanup was skipped entirely for the default launch path.

Fix: fall back to model_spec["accelerators"] (the allocated devices saved at model.model_family.accelerators = devices after _create_subpool() returns) when launch_args["gpu_idx"] is empty. This covers both user-specified and auto-allocated GPU paths.

Other findings (already fixed in prior commits, no changes needed):

• Gemini Critical ×2 (recover/terminate missing model_uid check): both call sites pass model_uid
• Gemini High ×2 (missing helper functions): both functions exist in worker.py
• Gemini High ×2 (VRAM reclaim loop not breaking on -1): both loops have if _free_ratio < 0: break
• Gemini Medium ×3 (pynvml not shut down): all three call sites have try...finally with nvmlShutdown()
• qinxuye (model_uid provenance): fixed with XINFERENCE_MODEL_UID env var + ancestry walk

[qinxuye]

LGTM