Repository navigation
REF: extract shared Xavier core and Torch backend - #5629
Merged
Merged
Conversation
rogercloud
approved these changes
Oct 4, 2026
rogercloud
left a comment
Contributor
There was a problem hiding this comment.
Minor
xinference/model/llm/xavier/collective_manager.py:29: the defaulttransfer_actor_uidis a hard-coded literal that must match the vLLM actor'sdefault_uid()(see inline).
Blocking: no — recommended event: APPROVE
This was referenced Oct 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Xavier's reusable coordination and snapshot/transfer implementations currently live inside the vLLM package. Extract them into
xinference/model/llm/xavier/, with Torch storage and CUDA/NIXL transfer under the optionalbackends/torch/package, so additional engine adapters can reuse the implementation.Keep the vLLM connector, scheduler integration, transfer actor and rank-zero bootstrap in the vLLM adapter. Legacy modules alias the shared implementation so imports, event-loop/profiling state and serialized actor references remain compatible; existing actor UIDs and connector configuration paths are preserved. Coordination and transfer lookup now use the adapter's actor UID. The default UID is defined once in the shared core and used by the coordinator and both vLLM transfer actor classes. Snapshot, tiered storage, payload packing and history algorithms are unchanged.
Validation:
7f6b39d19): 229 targeted tests passed, 1 skipped, covering the default UID contract and rank-zero/ordinary-rank lookups, dependency isolation, actor recovery and transfer regressions. All configured pre-commit hooks andgit diff --checkpassed. The existing UID value is unchanged.d21461203): 459 targeted tests passed, 16 skipped, including PD routing/lifecycle, snapshot and transfer regressions, dependency isolation, legacy imports and real subprocess actor registration/recovery.git diff --checkpassed.d21461203: 3 passed, covering default Xavier history, history disabled, and native NIXL, including streaming, concurrent remote KV loads and actual P-side history restores.f90f67304214bc47ab00218f06b57150b6989056withd2146120303d1b5cd4196072b9231611499d0f7don two RTX 3090 Ti GPUs using Qwen2.5-Instruct 0.5B BF16, vLLM 0.21.0, Torch 2.11.0, NIXL 1.1.0 and xoscar 0.11.1. Identical default Xavier settings, profiling off; initial main/PR/PR/main order plus repeated controls. All 14,668 measured requests succeeded and completed direct GPU KV handoff. Both GPU and CPU history tiers restored successfully, with zero history failures or ticket expiry.Tail latency attribution remains inconclusive. Original-model background responses can stop below the requested 1,024-token ceiling, so decode work varies. The separate fixed-output control uses the same weights with synthetic EOS metadata, verifies every background response has 1,024 tokens and every incoming response 64, and includes 96 cold arrivals per variant. It is additional evidence, not a replacement for the original-model result. These measurements confirm throughput parity and functional transfer/cache behavior for this reference configuration; they do not establish universal latency equivalence or performance for larger models and other engines.