feat(router): prefix-affinity P/D node selection for multi-P multi-D topologies#405
Open
xiaguan wants to merge 1 commit into
Open
feat(router): prefix-affinity P/D node selection for multi-P multi-D topologies#405xiaguan wants to merge 1 commit into
xiaguan wants to merge 1 commit into
Conversation
…topologies Round-robin node selection breaks prefix locality in xP+yD topologies: each turn of a multi-turn conversation lands on a different P (and D), so most turns miss the local prefix KV and fall back to cross-node full-context RDMA transfers - P nodes end up pulling multi-GiB prefixes from other P nodes (and even from D nodes), and D nodes re-pull the whole context instead of just the fresh suffix. The transfers collide with decode and blow up the ITL tail. Route by a turn-stable key instead: the first chat message, or a fixed-length prompt prefix on /v1/completions (multi-turn contexts grow by appending, so the prefix is stable across turns). Requests with no key fall back to round-robin. The D mapping hashes an extra discriminator so conversations spread independently of the P mapping. Measured on 2P+2D (4x H200, per-GPU 400G IB, Qwen3-14B bf16, 60 multi-turn conversations, first turn 8192 tokens + 2048/turn, concurrency 12): selection out tok/s TPOT p99 ITL p99 round-robin 437 38.9 ms 85.2 ms P affinity 465 30.8 ms 49.2 ms P + D affinity 495 22.8 ms 23.0 ms P-side cross-node prefix fetches drop 30 -> 1; decode-side full-context re-pulls halve. ITL p99 matches the 1P+1D isolation baseline (23 ms) at 2.5x its throughput. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
pegaflow-routerpicks P and D nodes round-robin. In an xP+yD topology this breaks prefix locality for multi-turn traffic: each turn of a conversation lands on a different P (and D), so most turns miss the node-local prefix KV and fall back to cross-node full-context RDMA transfers — P nodes pull multi-GiB prefixes from other P nodes (and even from D nodes, since the mesh is content-addressed and fully bidirectional), and D nodes re-pull the whole context instead of just the fresh suffix. Those transfers collide with decode and blow up the ITL tail.Fix
Route by a turn-stable affinity key, falling back to round-robin when no key exists:
/v1/completions: a fixed-length prompt prefix (512 chars / 64 token ids) — multi-turn contexts grow by appending, so the prefix is stable across turnsThis is the same policy family as the consistent-hash mode in vllm-project/router, kept dependency-free (std
DefaultHasher).Measured (2P+2D, 4x H200, per-GPU 400G IB, Qwen3-14B bf16, OpenInfer engines)
Multi-turn chat load: 60 conversations x 5 turns, first turn 8192 tokens + 2048/turn, 128 out/turn, concurrency 12, temperature 0:
P-side cross-node prefix fetches drop 30 → 1; D-side full-context re-pulls halve (93 → 60, the rest are post-eviction restores that no longer collide with decode). ITL p99 matches the 1P+1D isolation baseline (23.6 ms) at 2.5x its output throughput.
Single-P/single-D deployments are unaffected (affinity only engages when
len > 1).🤖 Generated with Claude Code