Skip to content

[XPU][CI] Fix hang in compare_all_settings by isolating server runs in subprocesses#4

Open
chaojun-zhang wants to merge 10000 commits into
mainfrom
tests/fix-xpu-ci-hang
Open

[XPU][CI] Fix hang in compare_all_settings by isolating server runs in subprocesses#4
chaojun-zhang wants to merge 10000 commits into
mainfrom
tests/fix-xpu-ci-hang

Conversation

@chaojun-zhang

Copy link
Copy Markdown
Owner

Problem

On XPU (Intel GPU / Level Zero), compare_all_settings would hang when testing multiple server configurations sequentially. After the first server shut down, residual EngineCore worker processes still held GPU resources, causing the next server startup to deadlock.

Fix

Run each server configuration in a dedicated subprocess so all GPU resources are fully released before the next configuration starts.

Key changes in tests/utils.py:

  • _server_worker / _run_server_collect_results: launch each server in a subprocess and return results via pickle IPC.
  • Refactor compare_all_settings: use _run_server_collect_results instead of sequentially running servers in the same process.
  • _limit_gpu_env: restrict visible XPU devices to tensor_parallel_size to prevent unnecessary multi-device (Level Zero) initialisation hangs.
  • Wire _limit_gpu_env into RemoteOpenAIServer and spawn_new_process_for_each_test.
  • Add XPU mem_get_info path to RemoteVLLMServer.get_total_gpu_memory().
  • Store _vllm_num_gpus on test functions decorated with multi_gpu_test for correct device-list restriction.

Notes

  • This is not duplicating any existing open PR (searched for XPU hang / compare_all_settings).
  • AI assistance was used; changes reviewed and tested end-to-end by the submitter.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.