Skip to content

web: expose --gpu-memory-utilization in text-generation Deployment Options #526

Description

@cswaney

Surfaced during the vLLM v0.26 audit (#515).

Context

vLLM's `--gpu-memory-utilization` (default 0.9) is the fraction of GPU memory vLLM claims for weights + KV cache. Lowering it is useful when co-locating with other GPU workloads on the same allocation, or when leaving headroom for CUDA graph capture and activation memory.

Users can pass it today via `launch_kwargs`, but it deserves a first-class control alongside the incoming `--max-model-len` field.

Scope

  • Add a numeric input (0.0–1.0, float) to Deployment Options in `TextGenerationContainerOptionsForm.jsx`. Empty = vLLM default 0.9.
  • Wire through `containerOptions.gpu_memory_utilization` in `text-generation.jsx`.
  • In `buildContainerConfig`, append `--gpu-memory-utilization ` to `launch_kwargs` when set.
  • Reflect the value in `ServiceSummary`.
  • Tests: `TextGenerationContainerOptionsForm.test.jsx`, `web/src/lib/requests.test.js`.

Notes

  • Likely lands after or alongside the `--max-model-len` PR; the `buildContainerConfig` merging pattern will be the same.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions