[qinxuye]

When SGLang's JIT cache root contains a symlink, SGLang 0.5.21 compares resolved compiler dependencies against an unresolved staging path. Generated .staging-*/cuda.cu files are recorded as persistent dependencies, so publishing a build makes its cache unusable to fresh workers and subsequent starts recompile kernels.

Canonicalize SGLANG_JIT_CACHE_DIR before starting the weight daemon and Runtime, including the default ~/.cache/sglang/jit and relative paths. Children inherit the real path to the same configured directory. This applies to initial startup and retained-weight reloads, including authenticated and distributed Runtime creation, without a new option or a SGLang upgrade.

Validation:

• All 7 new regression cases fail on the original code and pass with the fix. They cover default/empty/tilde paths and real child-process inheritance through a symlink with a not-yet-created suffix, with and without API authentication.
• The model re-entry test verifies canonicalization before weight-daemon preparation; this check fails when normalization is delayed until Runtime construction.
• 66 focused SGLang/runtime/reload tests passed; all 7 new cases also passed on the Linux GPU host with the installed SGLang SDK. Pre-commit passed.
• Review follow-up adds the runtime cases to the explicit CI test list and fixes an environment-variable leak in the default-path test. The in-process leak check fails before the fix and passes afterward; the follow-up test group reports 52 passed, 1 skipped, and pre-commit passes.

GPU verification uses 2 x RTX 3090 Ti 24 GiB with NVLink, unmodified SGLang 0.5.21, Torch 2.13.0, unquantized BF16 Qwen2.5-Instruct models, and TP=2. Runtime/core source hashes from benchmark snapshot 58d7b23accb27a9c8140730dbad949804f352842 match current head acd8868ccc63d4a9263184a00562fff79e83017c; the latter changes only tests and CI. Both startup paths use this fix, with a symlinked JIT cache root, so the comparison measures retained-weight reload versus full backend reconstruction.

CUDA Graph enabled, context 512, max prefill tokens 512, max total KV tokens 2048, memory fraction 0.85, concurrency 4 -> 8:

Model	Full backend reconstruction, two trials (s)	Retained-weight reload, two trials (s)	Medians: reconstruction -> reload (s)	Observed time reduction
7B	27.218 / 25.213	25.217 / 25.215	26.216 -> 25.216	3.81%
14B	41.444 / 33.333	29.285 / 29.281	37.388 -> 29.283	21.68%

Timing runs from stopping the old engine through the first successful external 8-token greedy response. The baseline reconstructs the entire backend in the same actor process, excluding actor creation and REST orchestration. Each model uses ABBA order (reconstruction, reload, reload, reconstruction), with an excluded successful load/generation before each phase. The measurements follow checkpoint-cache warm-up; no cold-cache clearing is performed. Timing probes are disabled, and downloads/hash verification finish before GPU comparisons.

There are only two observations per mode. Ordinary reconstruction varies by about 2 s for 7B and 8 s for 14B; the percentages describe these samples and require more observations to establish stable gains. This small-context configuration also bounds the capacity result.

All eight timed trials pass: identical greedy output within each model, effective concurrency and ready state verified, unchanged weight-daemon PID/socket paths/inodes across reloads. Continuous GPU monitoring records 980 samples for 7B and 1,238 for 14B, with no foreign GPU activity or monitoring errors. Timed trials use SGLANG_CRASH_ON_JIT_COMPILE=1 successfully; phase warm-up may compile new kernels. After the experiments, all 8 SGLang dependency manifests exclude temporary staging paths, and task-owned GPU processes are cleaned up.

The 14B checkpoint was downloaded with the official ModelScope SDK. All 8 safetensors shards (29,540,134,000 bytes) pass full SHA-256 checks against official ModelScope headers. Initial load, generation, and reload succeed; sampled device-wide peak memory is approximately 17.59 GiB per GPU, including pre-existing idle CUDA contexts. Memory is sampled every 0.5 s; per-process IPC allocations are not added together.

Engine startup logs show weight loading falling from 2.53-2.56 s to 0.11-0.12 s for 7B, and from 4.58-8.30 s to 0.13-0.15 s for 14B. CUDA Graph recapture remains approximately 6.8 s for 7B and 12.0 s for 14B. Worker import/setup and readiness costs remain as well.

The separate graph-disabled 7B verification remains valid: two reloads take 17.267 and 17.260 s, with concurrency 4 -> 8 -> 4, identical output, unchanged daemon PID/socket inodes, and no foreign GPU interference. Its cold initial startup is excluded. Separate process diagnostics attribute approximately 10.5 s of the remaining graph-disabled startup to the serial server/scheduler import and setup stages. Diagnostic probes are separate from the uninstrumented comparison above.

[rogercloud]

Minor

• xinference/model/llm/sglang/tests/test_runtime.py is never collected in CI: .github/workflows/python.yaml runs --ignore xinference/model/llm/sglang and only an explicit list of sglang test files (test_xavier, test_xavier_pd, test_pd, test_xavier_gpu_lifecycle, test_gc_lifecycle), so the symlink/relative/auth/default canonicalization cases never run. Fix: add test_runtime.py to that explicit list.
• xinference/model/llm/sglang/tests/test_runtime.py:59: the configured=None case leaks SGLANG_JIT_CACHE_DIR into later tests (see inline comment).

Blocking: no — recommended event: APPROVE