Skip to content

web: expose --max-model-len in text-generation Deployment Options #525

Description

@cswaney

Surfaced during the vLLM v0.26 audit (#515).

Context

vLLM's `--max-model-len` overrides the model's advertised context length. It is the most common OOM knob on Della — dropping context length from 131072 (Gemma 3's default) to, say, 32768 typically halves KV-cache memory and lets larger models fit or leaves headroom for concurrent requests.

Users can pass it today via the free-form `launch_kwargs` field (or CLI `launch_kwargs`), but it deserves a first-class control given how often it's needed.

Scope

  • Add a numeric input to the Deployment Options section in `TextGenerationContainerOptionsForm.jsx`. Empty = use model default (unchanged behavior).
  • Wire through `containerOptions.max_model_len` in `text-generation.jsx` initial state.
  • In `buildContainerConfig` (which currently only translates `disable_thinking`), append `--max-model-len ` to `launch_kwargs` when set. Order matters if `disable_thinking` is also set — concatenate rather than overwrite.
  • Reflect the value in `ServiceSummary` so users can see what a running service is configured with.
  • Tests: `TextGenerationContainerOptionsForm.test.jsx`, `web/src/lib/requests.test.js` for the `buildContainerConfig` merging behavior.

Non-goals

  • Enforcing a minimum/maximum. vLLM validates the value at startup and reports a clear error if the request exceeds the model's true context length.
  • Auto-suggesting based on GPU memory. Nice-to-have, but the calculation involves dtype and TP shape; leave it to the user for now.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions