FEAT: support SGLang Xavier KV sharing and native GPU PD

[qinxuye]

Adds Xavier KV-cache sharing and prefill/decode separation to SGLang, with native SGLang NIXL available through the same Xinference API and P/D router.

Changes

• Connect SGLang plugin hooks to the engine-neutral Xavier core and KV compatibility contract; preserve existing vLLM import paths. Ordinary SGLang replicas share CPU HiCache pages, while Xavier P/D transfers KV directly between GPUs.
• Support concurrent P/D startup, first-token handoff, streaming, cancellation and request-slot cleanup for both Xavier and native NIXL.
• Apply CUDA IPC, persistent GPU slabs and asynchronous completion, together with deployment-scoped directory/namespace caching and bounded prompt-fingerprint caching.
• Initial Xavier GPU support: unquantized FP16 full-attention Qwen2, Qwen3 and Llama text models, TP=PP=DP=1, SGLang >=0.5.21 and xoscar[nixl]>=0.11.1. Xavier P/D has no CPU fallback or retained GPU history; decode radix caching is rejected at launch.
• Update deployment documentation, nine locale catalogs, benchmarks and CI coverage.
• Use one shared model-stack constraints file throughout CPU/CUDA CI installs, check runtime imports, and synchronize the tokenization event-loop regression without a scheduling-rate threshold.

Final performance

Head f26b5f669; two RTX 3090 Ti GPUs with NVLink, Qwen2.5-0.5B, FP16, SGLang 0.5.21, and identical 6 GiB KV pools per role. Values are output tokens/s on repeated warm prompts.

Mode	Concurrency	Native NIXL	Xavier	Xavier difference
Eager	16	1,725	1,649	-4.4%
Eager	32	2,816	2,612	-7.3%
CUDA Graph	16	3,798	3,567	-6.1%
CUDA Graph	32	3,943	3,839	-2.6%

Warm throughput is close to native; first-token tail and cold-arrival latency remain higher. Prompt-fingerprint caching benefits repeated text; unique prompts still use the existing tokenization validation.

Validation

Head 878372e67: 697 focused tests pass (18 hardware/platform-dependent tests skipped locally), full-repository pre-commit and benchmark Black/Ruff pass. Five installed-SGLang/CUDA regressions cover salted cache isolation, remote HiCache restoration, bitwise GPU transfer/chunks/cancellation, end-to-end P/D cleanup and authenticated native Runtime startup. GPU monitoring found no unrelated workload interference. The merged CI dependency stack passes its runtime import checks. All 18 updated locale catalogs pass validation and compilation; the changed pages render in English and all nine maintained languages.

The full performance controls for the measured head above completed 5,536 requests with matching outputs, verified source/workload/KV capacity, and clean GPU isolation.

Manual SGLang GPU regressions

Run on Linux with two idle CUDA GPUs, SGLang 0.5.21, compatible PyTorch/NIXL, xoscar[nixl]>=0.11.1, and local unquantized FP16 Qwen2.5-0.5B weights. The real GPU tests are opt-in and are not enabled by the automatic GPU workflow.

XINFERENCE_TEST_SGLANG_XAVIER=1 \
XINFERENCE_TEST_PD_MODEL_PATH=/absolute/path/to/qwen2.5-0.5b \
SGLANG_PLUGINS=__xinference_test__ \
python -m pytest -vv \
  xinference/model/llm/sglang/tests/test_xavier_gpu.py \
  xinference/model/llm/sglang/tests/test_xavier_gpu_pd.py

Aborting an in-flight GPU handoff can synchronously wait for its drain fence before native SGLang reclaims the KV slots. The review records this native slot-ownership constraint; clear() cleanup is asynchronous. A nonblocking abort requires a separate deferred-reclamation protocol.

[rogercloud]

Major

• xinference/model/llm/sglang/core.py:994 — room deadline is fixed at 300s from the first prepare() and check() expires first, so completed long P/D requests are discarded; extend or exempt completed rooms (see inline). Trigger: non-stream P/D request taking >300s end-to-end, or a stream whose first chunk arrives >300s after prepare. Impact: finished output is dropped with "SGLang Xavier decode did not complete its GPU KV transfer".
• xinference/core/worker.py:6749 — recover_model relaunches Xavier P/D replicas, but register_peer rejects the new nixl://host:0 pool, so recovery always fails and evicts the replica; allow re-registration after unregister or fail fast (see inline).

Minor

• xinference/model/llm/sglang/xavier/gpu.py:526 — blocking open() in the scheduler thread with unhandled errors and a leaked self.rooms[room]; convert to KVPoll.Failed (see inline).
• xinference/model/llm/sglang/pd.py:44 — worker routable host forwarded to sgl.Runtime, possibly exposing the unauthenticated SGLang HTTP API; keep HTTP on loopback (see inline).
• xinference/core/worker.py:1867 — truthiness check on _xavier_cache_config also drops non-P/D HiCache replicas from restart recovery; gate on role (see inline).
• xinference/core/supervisor.py:3131 — model_format != "pytorch" rejects the default None; use not in (None, "pytorch") (see inline).
• xinference/core/pd_model.py:518 — p_task is never awaited when decode ends first, losing prefill errors; await it before completed = True (see inline).
• xinference/model/llm/sglang/xavier/gpu.py:263 — hard-coded 120s receive deadline covers prefill queue plus compute; make configurable (see inline).
• xinference/model/llm/xavier/backends/torch/storage.py:54 — _keys(limit=128) raises and the adapter swallows it as all-miss; chunk or raise limit (see inline).
• xinference/model/llm/sglang/xavier/plugin.py:31 — register() patches every SGLang process unconditionally; guard on GPU_CONFIG_ENV (see inline).
• xinference/model/llm/sglang/xavier/config.py:130 — silent BF16 to FP16 cast; warn and document (see inline).
• xinference/model/llm/sglang/xavier/config.py:114 — synchronous full SHA-256 of all safetensors on every load; cache by (path, size, mtime) (see inline).
• .github/workflows/python.yaml:384 — test_gc_lifecycle.py never runs in CI and abort_request, request-body construction and recover_model roles lack unit tests (see inline).
• xinference/core/supervisor.py:5259 — failed cache-actor destroy logged at debug only; relaunch may hit actor-already-exists (see inline).
• xinference/model/llm/sglang/core.py:983 — request id registered before the lazy generator starts and leaks if never iterated (see inline).
• xinference/api/restful_api.py:574 — stale comment about respecting an existing freeze (see inline).
• benchmark/benchmark_sglang_pd.py:485 — cwd-relative source hashing breaks outside repo root (see inline).

Simplification

• xinference/model/llm/xavier/backends/torch/canonical.py:L35: delete encode_layer/decode_layer/export_token_slots, only tests call them. Drop until a consumer lands. (~120 lines)
• xinference/model/llm/xavier/contract.py:L167: delete KVBlockKey/KVLayerMetadata/build_block_key/build_prefix_keys, reached only from tests and canonical.py. Defer with canonical.py. (~160 lines)
• xinference/model/llm/xavier/backends/torch/storage.py:L73: delete reserve/prepare/validate/release_handoff, no production callers; the pd_model.py:267 release_handoff branch is unreachable since stored SGLang handoffs always have mode "gpu". Remove both. (~60 lines)
• xinference/model/llm/sglang/pd.py:L34: delete the "_xavier_cache_config" check, it is popped in core.py before configure_nixl so it never matches. Drop it. (1 line)
  net: -340 lines possible

Blocking: yes — recommended event: COMMENT
Blocking issues:

• xinference/model/llm/sglang/core.py:994 — major — completed long PD requests discarded after the 300s room deadline [new]

[qinxuye]

Addressed all 17 inline review threads on eaf0086e9, including completed-room lifetime, registered-peer recovery, nonblocking ordered sender RPCs, large HiCache batches, lazy stream ownership, and authenticated native SGLang HTTP startup/generation/abort. The fixes have public replies and the corresponding threads are resolved.

Validation: 354 focused tests passed (9 hardware tests skipped locally), full-repository pre-commit and benchmark lint passed, and three real GPU tests passed for byte/chunk/cancellation correctness, end-to-end P/D cleanup, and native Runtime authentication. Updated catalogs and rendered pages were verified for all nine maintained locales.

For the simplification suggestions: removed the unreachable _xavier_cache_config guard from configure_nixl. Retained typed prefix identities and the common CPU handoff interface because the MLX adapter in #5638 already consumes them. The canonical codec remains a tested protocol reference; the SGLang GPU serving path does not call it. These shared interfaces remain stable while their dependent adapter is under review.

[rogercloud]

Major

• xinference/model/llm/sglang/xavier/gpu.py:211 — SGLang direct tickets still expire after the hard-coded 120s, not XINFERENCE_SGLANG_XAVIER_TRANSFER_TIMEOUT; see inline. [prior]
• xinference/core/pd_model.py:494 — prefill failure after decode already returned its stream leaks the decode stream and its request_limits slot; see inline.
• xinference/model/llm/sglang/xavier/gpu.py:624 — sender/receiver abort/clear still block the scheduler on kv_mgr.call(...) (up to 180s); see inline. [prior]

Minor

• xinference/model/llm/sglang/core.py:1040 — unguarded pd.release(...) in finally can turn a successful result into an error; see inline.
• xinference/model/llm/sglang/xavier/storage.py:37 — per-RPC actor-loop acquire/release recreates the loop and connection for every HiCache batch; see inline.
• xinference/core/supervisor.py:3281 — native NIXL host skips resolve_nixl_host, so a 0.0.0.0 worker advertises an unroutable bootstrap host; see inline.
• xinference/model/llm/sglang/core.py:868 — PD decode stream lacks raise_for_status and treats a missing/abort finish as success; see inline.
• xinference/model/llm/sglang/core.py:799 — cache_salt is forwarded but Xavier HiCache keys may not include it (unverified); see inline.
• xinference/model/llm/sglang/xavier/config.py:87 — explicit null config values bypass .get() defaults; see inline.
• Test coverage: xinference/model/llm/sglang/tests/test_xavier_gpu.py and test_xavier_gpu_pd.py are gated on XINFERENCE_TEST_SGLANG_XAVIER, which no workflow sets (.github/workflows/pd-gpu.yaml only runs the vLLM suites), so the real CUDA IPC/NIXL SGLang path never runs in CI. Add them to the GPU job or document how to run them manually.

Simplification

• xinference/model/llm/sglang/xavier/pd.py L18: delete: the page_size parameter of SGLangXavierHandoff.__init__ is never read; drop it and the argument passed from core.py.
  net: -2 lines possible

Blocking: yes — recommended event: COMMENT
Blocking issues:

• xinference/model/llm/sglang/xavier/gpu.py:211 — major — SGLang GPU handoff fails when decode pulls >120s after prefill registration, regardless of the configured transfer timeout [prior]

[qinxuye]

Updated on 878372e67, merged with main 92ba75776. The merge preserves main's grouped/recurrent KV ownership and vLLM executor coverage, while applying the configurable SGLang direct-ticket lease. CI now uses main's single model-stack constraints file with the transitive datasets/diffusers bounds; duplicate schema Config definitions from the automatic merge were removed. No downstream PR branches were rewritten.

Eight of the nine new inline threads are addressed: the >120-second ticket lease, decode stream/serve-slot cleanup after prefill failure, cleanup-error preservation, persistent HiCache RPC connections, worker-local wildcard bootstrap resolution, HTTP/terminal response failure handling, verified salted HiCache isolation, and nullable model defaults. Removed the unused handoff constructor page_size argument. The PR description now documents the exact opt-in SGLang GPU test command.

The abort thread remains open. clear() is asynchronous, but abort() still drains active GPU access before returning because native SGLang immediately reclaims the KV slots afterward. An asynchronous RPC alone would allow those slots to be overwritten before transfer completion. This is an explicit remaining scheduler-liveness limitation under the native ownership contract.

Validation: 697 focused tests passed, 18 hardware/platform skips locally; full-repository pre-commit and benchmark lint passed. Five installed-SGLang/CUDA tests passed for salt isolation, remote HiCache reuse, byte/chunk/cancellation correctness, P/D lifecycle and native Runtime authentication, with no foreign GPU interference. The merged CI dependency stack passes runtime imports; 18 catalogs and the two deployment pages were validated in all nine locales. Performance figures remain the final controlled measurements labeled with their measured head.

Previous-head CI failures were inspected individually:

• Windows 3.14 manifest failure: fixed this PR's platform-dependent path separators with as_posix().
• Windows 3.10 image-engine failure: test_fatal_engine_error_closes_model can observe the waiter exception before model.close() completes. Both implementation and test are byte-identical to main 92ba757; a controlled drain/close reproduction confirms the race.
• macOS 3.10 log-rotation failure: a concurrent rotation renames xinf.log between TimedRotatingFileHandler's existence check and stat during initialization. Both handler and test are byte-identical to main 92ba757; a controlled rename reproduces the same FileNotFoundError.

The last two are existing main concurrency issues outside the SGLang adapter changes. Their implementations are unchanged in this PR. Fresh CI is pending on the new head.

[rogercloud]

Minor

• xinference/model/llm/sglang/xavier/gpu.py:254: wait_done has no deadline, and lease expiry never completes or aborts the room (inline).
• xinference/model/llm/sglang/xavier/directory.py:47: completed rooms are never expired, so a lost decode release leaks a slot toward the 4096 cap (inline).
• xinference/model/llm/sglang/xavier/gpu.py:275: fixed 1-5 ms supervisor/actor polling with no backoff for the whole prefill duration (inline).
• xinference/core/pd_model.py:269: the vLLM abandon path now runs for handoffs with ticket="" (inline).
• xinference/model/llm/sglang/core.py:254: get_transport_host re-parses an already-unbracketed IPv6 host (inline).
• xinference/model/llm/xavier/backends/torch/storage.py:155: get() hits don't refresh LRU recency (inline).
• xinference/core/supervisor.py:3098: SGLang enable_xavier lacks the vLLM replica<=1 downgrade (inline).
• doc/source/user_guide/pd_separation.rst:49: still says "In the Web UI, select vLLM", although the frontend and the new SGLang section (L56) now support SGLang through the Web UI. Change it to "select vLLM or SGLang".
• [prior] xinference/model/llm/sglang/tests/test_xavier_gpu.py / test_xavier_gpu_pd.py: these are gated on XINFERENCE_TEST_SGLANG_XAVIER and no workflow sets it. The run instructions (XINFERENCE_TEST_SGLANG_XAVIER=1, XINFERENCE_TEST_PD_MODEL_PATH, SGLANG_PLUGINS) live only in the PR description. Put them in the test module docstring or a doc page so they survive merge.

Simplification

• xinference/core/supervisor.py:L3125: delete: transport_backend != "xavier" or (unreachable, since sglang_xavier already requires the xavier transport). Check n_worker != 1 only.
• xinference/core/supervisor.py:L3357: delete: the hasattr(self, "_xavier_cache_mapping") guard and its initializer (__init__ at L1069 always sets it). Assign into self._xavier_cache_mapping directly.
• xinference/core/supervisor.py:L5246: delete: the getattr(self, "_xavier_cache_mapping", {}) fallback. Use self._xavier_cache_mapping.pop(model_uid, None).
  net: -2 lines possible

Blocking: no — recommended event: APPROVE

[qinxuye]

Addressed this review in 0910aaa and replied to/resolved all seven inline threads. The review-body suggestions are also handled: removed the redundant supervisor checks/fallbacks (cache mapping now initializes in the constructor), documented both GPU test invocations in the test modules, and updated the Web UI instructions and timeout/lease behavior in English and all nine locales.

Validation: 588 focused tests and two rank0 cleanup tests passed; all five opt-in two-GPU SGLang tests passed, covering remote prefix restore, real GPU bytes, concurrent P/D, streaming disconnect/recovery, ordinary-engine output equivalence and authenticated native startup. Pre-commit, catalog validation/compilation and rendered English/nine-locale documentation passed. GPU test processes exited cleanly.

Also corrected the Windows 3.10 shutdown test's floating-point boundary (0.0500000000001819 <= 0.05) from the preceding CI run. Its macOS 3.14 failure is llama_cpp/tests/test_gguf.py::test_gguf_multimodal (failed to process mtmd chunk / image decode ret=-3); both that test and the llama.cpp implementation are unchanged from current main. That failure has not been reproduced locally. CI for the new head is reported separately from the local validation above.