Skip to content

[Python] Add prefix-aware batching and capacity-based GPU autoscaling signal for vLLM RunInference - #40760

Closed
AMOOOMA wants to merge 6 commits into
apache:masterfrom
AMOOOMA:vllm-prod
Closed

AMOOOMA wants to merge 6 commits into
apache:masterfrom
AMOOOMA:vllm-prod

Default PrefixBatchingConfig.num_shards to 8 so hot prefixes scale wi…

5eab031
Select commit
Loading
Failed to load commit list.
Sign in for the full log view

Annotations

2 warnings and 1 notice
label
succeeded Oct 28, 2026 in 10s