Skip to content

Add Kimi K2.5 PD disaggregation on GB300 Slurm#36

Draft
khluu wants to merge 11 commits into
mainfrom
codex/kimi-k2-5-gb300-slurm
Draft

Add Kimi K2.5 PD disaggregation on GB300 Slurm#36
khluu wants to merge 11 commits into
mainfrom
codex/kimi-k2-5-gb300-slurm

Conversation

@khluu

@khluu khluu commented Jul 20, 2026

Copy link
Copy Markdown
Member

This PR was authored with assistance from Codex.

Summary

  • Add a versioned prefill/decode-disaggregated serving schema to perf-eval workloads.
  • Add a Slurm/Pyxis launcher that allocates and supervises vLLM prefill and decode roles plus the vLLM router.
  • Add an opt-in Kimi K2.5 NVFP4 workload for three GB300 nodes: one DEP4 prefill node and two independent TEP4 decode nodes.
  • Route the workload to the gb300-slurm Buildkite queue and ingest vLLM serving benchmark results with the full serving GPU count.
  • Document the workflow, add focused regression coverage, and include a reusable PD-disaggregation model-onboarding agent skill.

Why

perf-eval previously assumed a single local vLLM server. Multi-node PD disaggregation needs one Slurm allocation, role-aware process placement, KV-transfer configuration, router lifecycle management, and benchmark execution inside that allocation. This adds those primitives while keeping the Kimi workload opt-in (nightly: false).

The benchmark loop also now reads workload TSV rows from a dedicated file descriptor. Slurm/Pyxis child processes could otherwise consume stdin and silently skip every benchmark configuration after the first.

Validation

  • python3 -m unittest discover -s tests -v — 33 tests passed
  • bash -n lib/run.sh lib/server.sh lib/run_lm_eval.sh lib/run_vllm_bench.sh
  • python3 -m py_compile .buildkite/generate_pipeline.py lib/ingest_perf.py lib/parse_workload.py lib/slurm_pd_launcher.py
  • Buildkite perf-eval #295 passed on the GB300 Slurm cluster, including allocation, all three serving nodes, routing, and the 8K-input smoke benchmark.

Follow-up

Build #295 ran before the dedicated-stdin fix and therefore exercised only the smoke row. A full 8K/1K, concurrency-3072 rerun of the current head remains pending.

khluu and others added 11 commits July 14, 2026 03:49
Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Codex <noreply@openai.com>
Migrate the Kimi K2.5 GB300 workload to the current attention and MoE configuration flags.

Co-Authored-By: Codex <noreply@openai.com>
Raise the role and engine startup deadlines so vLLM v0.25.1 can finish FlashInfer NVFP4 autotuning on all twelve GB300 workers.\n\nCo-Authored-By: Codex <noreply@openai.com>
Use BF16 KV cache while calibrated Kimi MLA FP8 scales are not loaded, size decoder concurrency to the resulting cache, and shorten the bring-up smoke. Also extend distributed readiness and tolerate delayed NFS result visibility without accepting stale artifacts.

Co-Authored-By: Codex <noreply@openai.com>
Read orchestrator TSV streams from a dedicated file descriptor so Slurm and other child processes cannot consume the remaining workload rows through stdin. Cover vLLM bench, lm-eval, and BFCL loops with a regression test.

Co-Authored-By: Codex <noreply@openai.com>
Capture the reusable model, topology, Slurm, router, validation, and Buildkite workflow from the Kimi K2.5 bring-up.

Co-Authored-By: Codex <noreply@openai.com>
Resolve shared pipeline and orchestration conflicts while preserving current-main benchmark arguments, BFCL handling, Kubernetes cache behavior, and the Slurm PD path.

Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Codex <noreply@anthropic.com>
Co-Authored-By: Codex <noreply@openai.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant