Skip to content

REF: extract shared Xavier core and Torch backend - #5629

Merged
qinxuye merged 2 commits into
xorbitsai:mainfrom
qinxuye:refactor/xavier-shared-core
Oct 4, 2026
Merged

qinxuye merged 2 commits into
xorbitsai:mainfrom
qinxuye:refactor/xavier-shared-core

Conversation

@qinxuye

@qinxuye qinxuye commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Xavier's reusable coordination and snapshot/transfer implementations currently live inside the vLLM package. Extract them into xinference/model/llm/xavier/, with Torch storage and CUDA/NIXL transfer under the optional backends/torch/ package, so additional engine adapters can reuse the implementation.

Keep the vLLM connector, scheduler integration, transfer actor and rank-zero bootstrap in the vLLM adapter. Legacy modules alias the shared implementation so imports, event-loop/profiling state and serialized actor references remain compatible; existing actor UIDs and connector configuration paths are preserved. Coordination and transfer lookup now use the adapter's actor UID. The default UID is defined once in the shared core and used by the coordinator and both vLLM transfer actor classes. Snapshot, tiered storage, payload packing and history algorithms are unchanged.

Validation:

  • Review follow-up (7f6b39d19): 229 targeted tests passed, 1 skipped, covering the default UID contract and rank-zero/ordinary-rank lookups, dependency isolation, actor recovery and transfer regressions. All configured pre-commit hooks and git diff --check passed. The existing UID value is unchanged.
  • Initial extraction (d21461203): 459 targeted tests passed, 16 skipped, including PD routing/lifecycle, snapshot and transfer regressions, dependency isolation, legacy imports and real subprocess actor registration/recovery.
  • The same 412-test CPU regression set passed before and after extraction.
  • All configured pre-commit hooks and git diff --check passed.
  • Real two-GPU integration at d21461203: 3 passed, covering default Xavier history, history disabled, and native NIXL, including streaming, concurrent remote KV loads and actual P-side history restores.
  • Compared f90f67304214bc47ab00218f06b57150b6989056 with d2146120303d1b5cd4196072b9231611499d0f7d on two RTX 3090 Ti GPUs using Qwen2.5-Instruct 0.5B BF16, vLLM 0.21.0, Torch 2.11.0, NIXL 1.1.0 and xoscar 0.11.1. Identical default Xavier settings, profiling off; initial main/PR/PR/main order plus repeated controls. All 14,668 measured requests succeeded and completed direct GPU KV handoff. Both GPU and CPU history tiers restored successfully, with zero history failures or ticket expiry.
Metric main this PR Change
C16 output tokens/s, mean across 4 fresh servers 1,678 1,680 +0.1%
C32 output tokens/s, mean across 7 batches of 600 requests 2,426 2,421 -0.2%
Cold long-prompt TTFT P95, mean across fresh servers 169 ms 168 ms -0.6%
Original-model cold arrivals during decode, pooled TTFT P95 506 ms 577 ms +14.0%
Fixed-output diagnostic cold arrivals, TTFT P95 845 ms 820 ms -2.9%

Tail latency attribution remains inconclusive. Original-model background responses can stop below the requested 1,024-token ceiling, so decode work varies. The separate fixed-output control uses the same weights with synthetic EOS metadata, verifies every background response has 1,024 tokens and every incoming response 64, and includes 96 cold arrivals per variant. It is additional evidence, not a replacement for the original-model result. These measurements confirm throughput parity and functional transfer/cache behavior for this reference configuration; they do not establish universal latency equivalence or performance for larger models and other engines.

@XprobeBot XprobeBot added this to the v3.x milestone Oct 4, 2026
@qinxuye
qinxuye requested a review from rogercloud October 4, 2026 15:43

@rogercloud rogercloud left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor

  • xinference/model/llm/xavier/collective_manager.py:29: the default transfer_actor_uid is a hard-coded literal that must match the vLLM actor's default_uid() (see inline).

Blocking: no — recommended event: APPROVE

Comment thread xinference/model/llm/xavier/collective_manager.py Outdated
@qinxuye
qinxuye merged commit 2e574b7 into xorbitsai:main Oct 4, 2026
14 of 15 checks passed
@qinxuye
qinxuye deleted the refactor/xavier-shared-core branch October 4, 2026 17:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants