Skip to content

docs: add SGLang direct linker quick start - #129

Merged
mkhazraee merged 1 commit into
ai-dynamo:mainfrom
linhu-nv:linhu/quick-start-sglang-linker
Nov 4, 2026
Merged

mkhazraee merged 1 commit into
ai-dynamo:mainfrom
linhu-nv:linhu/quick-start-sglang-linker

Conversation

@linhu-nv

@linhu-nv linhu-nv commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Add an experimental SGLang Direct Linker quick start and Dockerfile.
  • Use KVCR-owned inventory and Dynamo router hints for peer KV reuse.
  • Scope the example to dense Qwen3-32B BF16 with TP1 workers.

The recipe pins Dynamo #17655 (fe81cfd) and SGLang #45650 (5b65e08); KVCR is installed from the build checkout.

Validation

  • Built the Dockerfile from this PR and ran its import/version canary.
  • Two B200 nodes, one TP1 worker per node: ten 116K-token requests restored 115,968 tokens via router-generated hints and RDMA, with matching output IDs. Mooncake linker local_first and random ran against the same inputs.
  • The cross-node test used a harness and a chroot in retained GPU pods; the guide's same-host deployment was not run as written.

@copy-pr-bot

copy-pr-bot Bot commented Oct 27, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Fold the experimental direct linker into docs/quick-start-sglang.md and
Dockerfile.quick-start-sglang behind QS_MODE=hicache|linker. Each mode pins
its own SGLang and Dynamo revisions; the linker supports TP1 workers for now.

Also update the README for SGLang support, and pin the HiCache mode to the
SGLang revision that implements the renamed KeyAdapter API.

Co-authored-by: Ziqi Fan <ziqif@nvidia.com>
Signed-off-by: linhu-nv <linhu@nvidia.com>
@linhu-nv
linhu-nv force-pushed the linhu/quick-start-sglang-linker branch from 7148789 to d256768 Compare November 4, 2026 05:14
@mkhazraee
mkhazraee merged commit f737e27 into ai-dynamo:main Nov 4, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants