[qinxuye]

Adds bidirectional NVIDIA/Metal P/D: vLLM → MLX, SGLang → MLX, MLX → vLLM and MLX → SGLang, including streaming/non-streaming chat and completions. CUDA/Metal handoff uses canonical FP16 64-token pages through bounded CPU staging and actor RPC. MLX prefill owns pages on its worker under a configurable xavier_cache_bytes in-flight budget (default 512 MiB), reserved before Metal prefill; concurrent requests wait for capacity. MLX uses event-driven handoff waits; SGLang decode commits full prompt KV and P’s first token, while vLLM/MLX decode recomputes the last prompt token. Cancellation drains CUDA operations before releasing engine slots, and termination/failed launch removes worker-local source actors with a bounded worker RPC. MLX-prefill requests reject unsupported first-token sampling parameters before scheduling either role.

Dependencies

Based on main at 4969faffd after #5631, #5638 and #5639 merged; no unmerged PR dependencies. The PR now contains only the NVIDIA/MLX host-handoff extension and its integration tests/docs.

Initial scope: identical original unquantized Qwen2/Qwen3/Llama full-attention checkpoints and tokenizer assets, effective FP16 weights, TP=PP=DP=1 and n=1, without LoRA or multimodal/hybrid caches. Per-replica engine_config accepts worker-local model paths/formats. Weight/tokenizer/context/token-ID mismatches and transfer failures raise errors. Supervisor and worker/model environments must use matching Python and RPC serialization versions. Documentation and all nine translations are updated; the benchmark runs either or both directions on an existing cluster.

Final performance

Median TTFT (ms):

Prompt tokens	MLX local	vLLM local	SGLang local	vLLM P → MLX D	SGLang P → MLX D	MLX P → vLLM D	MLX P → SGLang D
51	50.5	75.3	62.5	188.8	235.7	365.9	296.5
184	53.0	76.3	62.4	211.2	254.6	382.2	321.6
744	74.3	82.2	71.1	283.2	345.1	519.3	447.9
2844	170.4	116.0	97.7	587.0	609.1	1013.7	898.6

Qwen2.5-0.5B-Instruct, FP16, RTX 3090 Ti and M5 Pro/64 GB, vLLM 0.28.0 / SGLang 0.5.21 / mlx-lm 0.31.3. Context 8,192, eager CUDA execution, C1, greedy streaming, 32 output tokens, five samples per prompt length after warmup, distinct early prefixes and no complete-page cache reuse. Standalone vLLM uses its default 16-token blocks; P/D uses canonical 64-token pages. Median end-to-end TTFT includes routing, prefill, CPU/network copies and decode. Measured LAN payload throughput is approximately 85 MB/s in both directions. Each retained GPU sample passed the external ownership monitor; runs with observed measurement interference are retried. These P/D combinations have no TTFT advantage over standalone decode on this small model and current link. The table retains the previous final measurements; this revision reruns functional/lifecycle validation and does not claim a new full performance comparison.

Validation

• Latest-head Python/Metal regression sweep: 1,197 passed, 49 skipped; one unchanged baseline BlockTracker test deselected; whole-repository pre-commit passed.
• Latest-head real LAN chat/completions and streaming/non-streaming requests passed in all four directions, including 64/65/66/128/129/130/193-token boundaries (88 requests total; all 44 streaming/non-streaming text pairs matched). Completed imports are verified by imported_tokens / host_bytes, with zero gpu_bytes and no active handoffs after completion. The merged vLLM/SGLang cancellation retry and page-allocation fixes remain in place; host imports also cover minimum/full target allocation. Invalid per-replica weight formats fail before directory/source/model creation.
• Capacity and sampling: oversized prompts fail before Metal prefill, concurrent requests wait for capacity, reservation/publish failures release the byte budget, and unsupported MLX-prefill sampling settings fail before scheduling. Real LAN checks with a 1.5 MiB budget pass for both NVIDIA decoders, with oversized/unsupported requests rejected and parallel/recovery requests completing without retained handoffs or bytes. Unit/RPC regressions cover lease expiry, repeated cancellation, failed source cleanup, multi-chunk/prefix geometry, host-mode SGLang prepare and worker cleanup timeout.
• Lifecycle: parallel requests, stream disconnect, cancellation during CPU handoff, empty source/transfer state and successful fresh requests after cancellation, in both directions.
• In the previous full performance run, same-prompt greedy text matched standalone FP16 MLX in 18/20 vLLM → MLX, 19/20 SGLang → MLX, 17/20 MLX → vLLM and 19/20 MLX → SGLang final samples. Some FP16 continuations differ between engines; cross-engine greedy text equality is not guaranteed.
• All nine changed PO catalogs validated and compiled; P/D and Xavier pages rendered in English and every maintained language (20 pages), with translated text verified.

Latest-head CI is running.

[rogercloud]

Major

• xinference/model/llm/xavier/backends/bytes/pd.py:59 — fixed 512 MiB reject-only byte budget: an MLX prefill whose pages exceed it fails only after the full Metal prefill, and concurrent requests fail instead of waiting. Make it configurable, check ceil(n/64)*page_bytes before prefill, and wait for capacity.

Minor

• xinference/model/llm/mlx/xavier.py:522 — 5 ms busy-poll hits the supervisor directory ~200-400 RPC/s per request (also _fetch_host source/chunk polls). Use wait_source/wait_complete/wait_chunk with a transfer_timeout() timeout.
• xinference/model/llm/mlx/xavier.py:410 — release is not shielded in _fetch_host, and source.abort failure in _prefill_host skips the directory release. Shield it and use try/finally.
• xinference/core/supervisor.py:5517 — worker-local destroy_actor has no timeout; an unreachable worker stalls termination. Use xo.wait_for(..., timeout=5).
• xinference/model/llm/xavier/backends/torch/pd.py:335 — directory.request_info RPC runs per batch under send_lock. Fetch once per room before the lock.
• xinference/model/llm/mlx/xavier.py:359 — xavier-cross-engine-transfer-{rank} is hard-coded here and in core/pd_model.py, duplicating CrossEngineGPUActor.default_uid(). Share a torch-free constant.
• xinference/model/llm/mlx/xavier.py:165 — host mode calls configure(fingerprint) without the contract (also bytes/pd.py:60), so mismatches get the generic GPU-namespace message; sglang/xavier/pd.py:42 still says "Missing SGLang Xavier GPU bootstrap metadata". Pass contract.to_dict() and derive messages from mode.
• xinference/model/llm/mlx/xavier.py:504 — first token ignores min_p/presence/frequency penalty/seed, which are honored afterwards. Reject them in the router for MLX-prefill requests or document.
• Tests: none for XavierHostPDSource.publish rejection/rollback branches, SGLangXavierHandoff.prepare in host mode, _fetch_host multi-chunk and page-count/incomplete-prefix errors, or budget rejection after MLX prefill.

Simplification

• xinference/core/supervisor.py L3413: delete: redundant hasattr guard; __init__ sets _xavier_source_mapping. Call self._xavier_source_mapping.setdefault(model_uid, []).append(source) directly. (-2)
• xinference/core/supervisor.py L5515: delete: getattr default unnecessary. Use self._xavier_source_mapping.pop(model_uid, []). (0)
• xinference/model/llm/mlx/xavier.py L317: shrink: _fetch_host and _prefill_host (L450) duplicate the 11-line handoff validation. Extract one helper returning handoff["room"]. (~-7)
  net: -9 lines possible

Blocking: no — recommended event: APPROVE