TEST: Cover multi-prefill and multi-decode GPU routing

[qinxuye]

The GPU PD regression currently covers only 1P1D. Add shared-GPU 2P1D, 1P2D and 2P2D cases for Xavier and native NIXL, checking request-specific answers, streaming, concurrency and real KV transfers. The 2P2D test rotates decoder registration to cover all four P→D pairs; this is route registration coverage, not process-failure recovery.

Each deployment now writes an isolated, non-rotating evidence log. This fixes lost phase counters caused by shared log rotation during concurrent test sessions. The manually triggered PD GPU workflow runs the new cases and uploads those logs. Production scheduling remains round robin; the only production-code addition is DEBUG route evidence.

Validation

• All six backend/topology GPU cases passed on the two-GPU host: 112 requests, including Xavier history restores with engine prefix caching disabled.
• Re-ran native NIXL 2P2D on the pre-review head (7fcdf37a0): 24 requests, all four pairs, actual KV receive checks passed. The subsequent review fixes disable thinking in the test request, require DEBUG evidence logging and retain passing-test output with -rP; GPU cases were not rerun for those harness changes.
• Local focused suite: 81 passed, 9 GPU cases skipped. Changed-file pre-commit checks passed; workflow YAML parsed successfully.
• The separate routing prototype's focused tests passed 95 tests, including load cleanup on cancellation and GPU-monitor failure handling. Those prototype changes are outside this PR.

Performance validation completed before opening

Qwen2.5-0.5B-Instruct FP16, vLLM 0.21.0, xoscar 0.11.1, NIXL 1.1; two RTX 3090 Ti 24 GiB GPUs connected by NVLink NV3. Two producers share GPU 0 and two decoders share GPU 1. Each replica uses a 0.35 GPU memory fraction, eager mode, disabled engine prefix caching, and 256 MiB Xavier GPU history. All P/D transfer pairs are warmed before measurement; output length is 16 tokens.

These compare Xavier round-robin routing against a prefix-affinity prototype, not Xavier against native vLLM NIXL.

Workload at concurrency 8	Round robin req/s	Affinity req/s	Round robin TTFT p95 ms	Affinity TTFT p95 ms
12 fixed-length prefixes, ~1,840 input tokens	33.61–34.68	33.85–34.08	154.3–159.1	156.2–157.3
12 mixed-length prefixes, ~500/900/1,800/3,200 tokens	33.71–34.79	33.75–34.01	173.7–179.1	177.8–182.0
20 prefixes, GPU-history capacity pressure	32.19–33.00	32.32–32.45	173.8–176.0	175.7–177.0

Ranges show two repetitions per policy in ABBA order. The first two profiles each use 12 cold, 36 sequential repeated and 2,400 concurrent requests per deployment; the mixed profile follows fixed lengths without clearing history. Capacity pressure uses 20 cold, 60 repeated and 1,200 concurrent requests. Its roughly 430 MiB KV working set exceeds one producer's GPU-history budget but fits the combined budget.

24,704 accepted measurement requests succeeded, with no foreign GPU processes observed in 805 one-second samples, excluding explicitly recorded idle daemons. Interfered pressure cases were discarded and rerun; a retry also stopped before launch when GPUs were busy. The accepted pressure sequence remains ABBA but has extra gaps. Process sampling is not a hardware reservation. DEBUG evidence logging affects absolute rates, and colocated replicas do not establish four-GPU scaling.

Findings:

• Sequential fixed-length reuse improves TTFT p95 from 107.4–109.6 ms to 85.3–85.9 ms. Under capacity pressure it improves from 106.6–109.4 ms to 86.7–88.3 ms.
• Under pressure, CPU-restored blocks fall from 54,340–54,620 to 36,392–42,298 per 1,200 requests. GPU-restored blocks rise from 81,646–81,928 to 90,080–96,790, but total restored blocks also fall slightly. Reduced CPU reads alone are not an end-to-end improvement.
• Sustained concurrent throughput and tail latency show no consistent benefit. The prototype is therefore not enabled or included in this PR's runtime.
• Median P RPC time is ~33–39 ms in the sustained profiles; P completion to the first D response is ~64–67 ms. The latter includes RPC, decode queueing, KV load and first execution, and cannot be called pure transfer time. Further optimization needs finer timing and actual cache-residency/work-size information.

The earlier short runs also failed to establish a consistent concurrent benefit. Their shared-log rotation failure was a measurement issue despite successful responses; isolated logs fix that issue.

The experiment runner and prototype are preserved separately; the Xavier README on that branch describes reproduction settings. This PR's Xavier README records the completed sustained and capacity-pressure results. Rebased onto current main, including the now-merged hybrid-attention PR #5630.

[rogercloud]

Minor

• xinference/model/llm/vllm/xavier/test/test_pd_multi_gpu.py:54 — chat() lacks enable_thinking: False; thinking-model overrides fail the answer assertion.
• xinference/model/llm/vllm/xavier/test/test_pd_gpu.py:43 — undocumented log-level knob; any level above DEBUG breaks the evidence assertions.
• xinference/model/llm/vllm/xavier/test/test_pd_multi_gpu.py:180 — MULTI_PD_RESULT is discarded on passing CI runs (no -rP/-s in .github/workflows/pd-gpu.yaml:50).

Blocking: no — recommended event: APPROVE

[qinxuye]

Addressed all three review comments and resolved their threads. Rebased onto current main (including merged #5630); the conflict was only the duplicate vllm.ir.op logging comment, and the upstream version was retained. Current head: fac9941.

Validation after rebase: 81 passed, 9 GPU cases skipped; changed-file pre-commit passed. Workflow YAML parses, and a passing pytest smoke case confirms -rP retains successful stdout. GPU model cases were not rerun for these harness-only fixes. The PR description now distinguishes the earlier GPU run from this follow-up validation.