Skip to content

web: expose thinking_token_budget in text-generation launcher #527

Description

@cswaney

Surfaced during the vLLM v0.26 audit (#515).

Context

vLLM v0.22 added `thinking_token_budget` — a per-request cap on how many tokens the model spends "thinking" before producing the visible answer. It complements the existing Disable Thinking toggle: instead of the binary on/off choice, users could allow reasoning but bound its cost.

The Disable Thinking toggle maps to a launch-time flag (`--default-chat-template-kwargs`). `thinking_token_budget` is per-request, so it belongs on the completions/chat body, not the launcher — likely in the parameters sidebar (`SidebarContainer`'s Parameters section for text-generation), alongside `temperature` and `max_tokens`.

Scope

  • Add a numeric input to the text-generation parameters form. Default: empty (no budget, vanilla thinking behavior).
  • Include `thinking_token_budget: ` in the completion request body when set. Applies to both streaming and non-streaming paths.
  • Add a small explainer in help text: it only takes effect for models that support thinking mode; on non-thinking models it is ignored.
  • Tests for the request body shape.

Notes

  • Complements Disable Thinking rather than replacing it. Disable Thinking is a launch-time choice (affects the chat template); `thinking_token_budget` is a per-request choice.
  • Not a bug — pure feature request.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions