FEAT: support Xavier P/D across vLLM and SGLang

[qinxuye]

Prefill and decode replicas can now use different engines: vLLM → SGLang and SGLang → vLLM. Per-replica model_engine / engine_config select each role, and Xavier converts canonical 64-token KV pages on GPU and transfers them directly through NIXL. Weight/tokenizer/context contracts and prompt token IDs must match; incompatible configurations and failed transfers raise request errors.

The initial scope is unquantized FP16 full-attention Qwen2/Qwen3/Llama text models with TP=PP=DP=1 and n=1, including streaming and non-streaming chat/completions. Cancellation drains GPU work and releases handoff state without terminating either engine. Documentation and all nine translations are updated.

Dependencies

Based on main after #5629, #5631 and #5638 merged; no unmerged PR dependencies. Homogeneous MLX support is preserved. Mixed-engine MLX P/D remains outside this PR.

Final performance

Deployment	C1 TTFT p50 (ms)	C16 TTFT p95 (ms)	C16 output tokens/s
vLLM ordinary replica	31.56	81.01	1792.01
SGLang ordinary replica	21.91	86.35	1496.82
vLLM native NIXL P/D	60.74	392.57	905.47
SGLang native NIXL P/D	21.44	152.71	1506.36
vLLM P → SGLang D	28.38	160.73	1441.22
SGLang P → vLLM D	59.57	227.52	1391.80

Same Xinference API/router, Qwen2.5-0.5B-Instruct FP16, RTX 3090 Ti, vLLM 0.28.0 / SGLang 0.5.21, eager execution, 64-token pages, 32,768 KV tokens per replica and 8,192 context limit. Warmed mixed prompts of 40/173/733/2,833 tokens, fixed 32-token output, 128 requests at each concurrency. Ordinary replicas use one GPU; P/D uses two. GPU interference checks passed for every run. TTFT includes prefill, queueing, transfer/conversion and decode. Ordinary replicas remain faster on this small model. Runtime revision: 5c55e954a.

Validation

• Latest-head regression sweep: 1,085 passed, 49 skipped. Whole-repository pre-commit passed.
• Two-GPU KV transfer/layout tests: 9 passed with bitwise equality in both directions at 65/129/193 tokens, covering both minimal and full prompt-page allocation on vLLM decode and verifying that the extra target page remains untouched.
• Both heterogeneous directions passed latest-head streaming/non-streaming chat and completion checks at 64/65/66/128/129/130/193 prompt tokens. Greedy outputs match each prefill-engine baseline; GPU bytes/imported-token counters increase, CPU batches remain zero, and disconnect plus three early cancellations leave no active handoff/transfer/receive state. Fresh requests succeed after cancellations and after rejecting the one-token vLLM decode request.
• CPU regressions cover preparation/load/registration/poll failures, source-page trimming, configured deadlines/progress renewal, lease expiry, release-RPC failure, adapter contract equality, cached weight fingerprints, early engine/format/quantization validation, serialized scheduler-to-worker cancellation retries, late producer calls after abort, connector selection/fail policy, Xavier config preservation, cache-budget rejection and JSON response-format rejection.
• The performance table is measured at the revision named above; its 1,536 requests passed, including greedy prefill-baseline agreement, streaming and cancellation cleanup. This review follow-up reran functional and bitwise validation on the latest head; the full performance comparison was not rerun.
• One existing test_block_tracker assertion also fails on main: it expects an empty bucket that the shared tracker removes. Three optional vLLM chat tests also fail on main in a local environment without installed vLLM. These are recorded separately from the passing regression sweep.
• All nine modified PO catalogs passed msgfmt validation and compilation. P/D and Xavier pages rendered with verified translations in English and all nine maintained languages (20 pages).

[rogercloud]

Major

• xinference/model/llm/xavier/backends/torch/pd.py:286 — SGLang prefill to vLLM decode fails for prompts with len % 64 == 1 (page-count mismatch).
• xinference/model/llm/vllm/xavier/cross_engine.py:176 — per-request failures raised in vLLM scheduler hooks kill EngineCore.
• xinference/model/llm/xavier/backends/torch/pd.py:262 — hard-coded 120s receive deadline, busy-polling, missing lease_timeout and wait_done deadline.

Minor

• xinference/model/llm/xavier/pd_contract.py:57 — fingerprint_files runs without cache_dir, so every cross-engine launch hashes all weights (also xinference/model/llm/sglang/xavier/config.py:156, which discards the contract built at :120-123).
• xinference/model/llm/xavier/backends/torch/pd.py:176 — release RPC failure in poll_direct_gpu_v1 loses the finished set; rooms also leak when a direct lease expires.
• xinference/core/pd_model.py:351 — any truthy response_format is rejected, including {"type":"text"}.
• xinference/core/supervisor.py:3332 — cfg.engine_config merged without reserved-key checks.
• xinference/core/supervisor.py (add_model_replica(s)) — silently ignores model_engine/engine_config for non-PD models while launch rejects them.
• xinference/core/supervisor.py:3219 — "does not yet support retained GPU history" message is misleading for vLLM/SGLang cross-engine; a context-length/tokenizer/weights mismatch surfaces as "KV namespaces differ" without naming the field.
• xinference/model/llm/xavier/tests/test_pd_cross_engine_gpu.py:19 — no test that SGLang configure_xavier and vLLM configure_cross_engine yield the same fingerprint (different context keys); no CPU tests for CrossEngineGPUActor deadlines or poll_direct_gpu_v1 error handling; the GPU test is opt-in and not run in CI.

Blocking: yes — recommended event: COMMENT

• xinference/model/llm/xavier/backends/torch/pd.py:286 major — SGLang prefill to vLLM decode fails for prompts of 65/129/193... tokens [new]
• xinference/model/llm/vllm/xavier/cross_engine.py:176 major — a single bad request kills the whole vLLM decode EngineCore [new]

[qinxuye]

Addressed the additional review-summary items in d0bc1f4:

• Both scale-up entry points now reject per-replica engine settings before any launch, consistently with initial launch. The top-level scale-up engine option is unchanged.
• Retained-history errors identify cross-engine PD, and contract mismatches name the differing weights/tokenizer/position field.
• Added adapter fingerprint parity/hash-cache tests and CPU actor deadline, progress, lease-expiry and release-RPC regressions. These run in the normal non-GPU CI selection; the vLLM API/connector cases run in the supplementary CI selection.

Local validation: 1,058 passed / 46 skipped, 29 benchmark tests passed, whole-repository pre-commit passed, and all six two-GPU bitwise cases passed (both directions at 65/129/193 tokens with actual minimum target-page allocation). The two-GPU test remains opt-in because the existing GPU CI runner has one T4; it was executed on the reserved two-GPU host. Both directions also passed actual streaming/non-streaming boundary requests at 64/65/66/128/129/130/193 tokens, prefill-baseline output checks, transfer counters and cancellation cleanup. A one-token decode failure leaves the service able to answer fresh requests. Latest-head lint and hosted documentation checks passed; the test matrix is running. Performance samples were discarded because of concurrent foreign GPU workloads, so the PR retains the existing table with its measured revision explicitly identified; latest-head performance remains pending an isolated window.

[rogercloud]

Minor

• xinference/model/llm/vllm/xavier/cross_engine.py:302: _pending_aborts is written in the scheduler-role connector but only drained in the worker-role instance, so the abort retry never runs. When that happens, the request's prefill KV blocks stay pinned (inline).
• xinference/model/llm/xavier/backends/torch/pd.py:204: CrossEngineGPUActor.abort pops the room, so late producer calls raise bare KeyError and the aborted checks are dead code. The related test seeds a state production never creates (inline).
• xinference/core/replica_config.py:185: the reserved engine_config keys miss n_worker, shard and driver_info, so n_worker>1 can get past the supervisor rejection (inline).
• xinference/core/supervisor.py:3166: heterogeneous PD skips the model_format/quantization/n_worker pre-launch checks. Bad weights then fail only after side effects have started (inline).
• doc/source/user_guide/pd_separation.rst:141: the "supports no CPU fallback, ..." wording is ambiguous (inline).
• Test coverage gaps: (a) VLLMModel.set_xavier_config(None) keeping the heterogeneous config (vllm/core.py ~550-557); (b) XavierEngine selecting CrossEngineConnector with kv_load_failure_policy="fail" for heterogeneous configs (vllm/xavier/engine.py ~129-148); (c) heterogeneous PD rejecting xavier_gpu_cache_bytes>0 / xavier_cache_bytes; (d) response_format json_object/json_schema still being rejected on cross-engine PD (only the text-allowed case is tested).
  (e) The vLLM decode receive tests pass exactly ceil((len-1)/64) destinations, but vLLM allocates ceil(len/64) when len%64==1. As a result the destinations[:receive_pages] trim path is never exercised.

Simplification

• xinference/model/llm/vllm/xavier/cross_engine.py L87: delete: self._prepared is never added to and is only discarded (L305/L310). Remove the set, both discards and the test seeding (test_cross_engine.py L39/L76).
• xinference/model/llm/vllm/xavier/cross_engine.py L242: shrink: the _receive_pages try/except duplicates the outer catch in receive() (both log and return set(destinations)). Inline actor.receive(room, destinations, 0) and keep the single catch, moving its comment there. Tests that call _receive_pages directly can go through start_load_kv instead.

net: -15 lines possible

Blocking: no — recommended event: APPROVE

[qinxuye]

Addressed the second review in aef81e7. All five inline threads are fixed, replied to and resolved.

The review-body suggestions are also covered: removed the unused _prepared set and redundant receive exception wrapper; added regressions for Xavier config preservation, connector selection with kv_load_failure_policy="fail", CPU/GPU cache-budget rejection, JSON response-format rejection, and both minimal/full vLLM destination allocations. Cancellation metadata is serialized between separate scheduler and worker connectors, with retry persistence and no duplicate completion. Producer tests now exercise an actual aborted room.

Validation on this head: 1,085 passed / 49 skipped, whole-repository pre-commit passed, 9 two-GPU bitwise cases passed, and both heterogeneous directions passed real streaming/non-streaming API boundary and cancellation checks. All nine modified catalogs passed validation/compilation; 20 English/translated pages rendered successfully. Hosted English/Chinese docs and CI lint have passed; the remaining CI matrix is running.

The PR description contains the final performance table with its measured revision 5c55e954a; this follow-up reran correctness checks and does not claim a new full performance comparison. #5641 is untouched.