Surfaced during the vLLM v0.26 audit (#515).
Context
vLLM's `--gpu-memory-utilization` (default 0.9) is the fraction of GPU memory vLLM claims for weights + KV cache. Lowering it is useful when co-locating with other GPU workloads on the same allocation, or when leaving headroom for CUDA graph capture and activation memory.
Users can pass it today via `launch_kwargs`, but it deserves a first-class control alongside the incoming `--max-model-len` field.
Scope
Notes
- Likely lands after or alongside the `--max-model-len` PR; the `buildContainerConfig` merging pattern will be the same.
Surfaced during the vLLM v0.26 audit (#515).
Context
vLLM's `--gpu-memory-utilization` (default 0.9) is the fraction of GPU memory vLLM claims for weights + KV cache. Lowering it is useful when co-locating with other GPU workloads on the same allocation, or when leaving headroom for CUDA graph capture and activation memory.
Users can pass it today via `launch_kwargs`, but it deserves a first-class control alongside the incoming `--max-model-len` field.
Scope
Notes