Surfaced during the vLLM v0.26 audit (#515).
Context
vLLM's `--max-model-len` overrides the model's advertised context length. It is the most common OOM knob on Della — dropping context length from 131072 (Gemma 3's default) to, say, 32768 typically halves KV-cache memory and lets larger models fit or leaves headroom for concurrent requests.
Users can pass it today via the free-form `launch_kwargs` field (or CLI `launch_kwargs`), but it deserves a first-class control given how often it's needed.
Scope
Non-goals
- Enforcing a minimum/maximum. vLLM validates the value at startup and reports a clear error if the request exceeds the model's true context length.
- Auto-suggesting based on GPU memory. Nice-to-have, but the calculation involves dtype and TP shape; leave it to the user for now.
Surfaced during the vLLM v0.26 audit (#515).
Context
vLLM's `--max-model-len` overrides the model's advertised context length. It is the most common OOM knob on Della — dropping context length from 131072 (Gemma 3's default) to, say, 32768 typically halves KV-cache memory and lets larger models fit or leaves headroom for concurrent requests.
Users can pass it today via the free-form `launch_kwargs` field (or CLI `launch_kwargs`), but it deserves a first-class control given how often it's needed.
Scope
Non-goals