Repository navigation

[Python] Add prefix-aware batching and capacity-based GPU autoscaling signal for vLLM RunInference - #40760

Draft

AMOOOMA wants to merge 6 commits into

apache:masterfrom

AMOOOMA:vllm-prod

Draft

[Python] Add prefix-aware batching and capacity-based GPU autoscaling signal for vLLM RunInference#40760
AMOOOMA wants to merge 6 commits into
apache:masterfrom
AMOOOMA:vllm-prod

Commits on Oct 23, 2026
- Isolate vLLM RunInference and BatchElements changes
  AMOOOMA
  committed
  9bf40b1
- Align publisher defaults with module constants, fix import order, and fix uuid identifier test after #40671 annotation scoping
  AMOOOMA
  committed
  46ecede
Commits on Oct 27, 2026
- Fail loudly in ChatPrefixKeyFn on unknown chat message schemas
  AMOOOMA
  committed
  694b666
- Default keyed BatchElements to a 1s flush so incomplete batches never stall in streaming
  AMOOOMA
  committed
  c952984
Commits on Oct 28, 2026
- Document that skewed key distributions need num_shards for prefix batching
  AMOOOMA
  committed
  aa80371
- Default PrefixBatchingConfig.num_shards to 8 so hot prefixes scale with the pool
  AMOOOMA
  committed
  5eab031