[XPU][CI] Fix hang in compare_all_settings by isolating server runs in subprocesses#4
Open
chaojun-zhang wants to merge 10000 commits into
Open
[XPU][CI] Fix hang in compare_all_settings by isolating server runs in subprocesses#4chaojun-zhang wants to merge 10000 commits into
chaojun-zhang wants to merge 10000 commits into
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On XPU (Intel GPU / Level Zero),
compare_all_settingswould hang when testing multiple server configurations sequentially. After the first server shut down, residual EngineCore worker processes still held GPU resources, causing the next server startup to deadlock.Fix
Run each server configuration in a dedicated subprocess so all GPU resources are fully released before the next configuration starts.
Key changes in
tests/utils.py:_server_worker/_run_server_collect_results: launch each server in a subprocess and return results via pickle IPC.compare_all_settings: use_run_server_collect_resultsinstead of sequentially running servers in the same process._limit_gpu_env: restrict visible XPU devices totensor_parallel_sizeto prevent unnecessary multi-device (Level Zero) initialisation hangs._limit_gpu_envintoRemoteOpenAIServerandspawn_new_process_for_each_test.mem_get_infopath toRemoteVLLMServer.get_total_gpu_memory()._vllm_num_gpuson test functions decorated withmulti_gpu_testfor correct device-list restriction.Notes