Skip to content

chore(lib): bump vllm/vllm-openai to v0.26 or v0.27 #512

Description

@cswaney

Sub-task of #451.

Bump the pinned vLLM image in `images.py` from `v0.20.0` to `v0.26.0`. `v0.27`+ is unnecessary unless we start using Qwen 3.5.

Draft PR is open at #523. Check items off here as smoke tests pass; mark #523 "Ready for review" once the matrix is green.

Expected Blackfish-side changes

Blackfish's launch template only uses mainline vLLM engine flags — `--model`, `--port`, `--revision`, `--trust-remote-code`, `--tensor-parallel-size`. None appear in vLLM's deprecation/removal lists between v0.21 and v0.26. The only other Blackfish-owned wiring is the Disable Thinking toggle (`--default-chat-template-kwargs '{"enable_thinking": false, "thinking": false}'`), whose flag name and JSON key shape were not touched in the release notes we scanned. Expected diff: pin only, plus incidental test/fixture bumps.

Landmines to watch for during the smoke test:

  • v0.21 deprecates transformers v4 (image dependency change).
  • v0.24 stops setting `CUDA_VISIBLE_DEVICES` internally; we drive device count via `--tensor-parallel-size` so this should be transparent.
  • v0.25 removes legacy PagedAttention; V1/MRv2 backends are the standard path.
  • HPC-specific: watch for CUDA driver mismatches on Della compute nodes and `apptainer pull` issues on the multi-arch manifest.

Runtime override for testing

No code change needed to pilot the bump:

```shell
BLACKFISH_TEXT_GENERATION_IMAGE=vllm/vllm-openai:v0.26.0 blackfish start
```

Stage the SIF on Della once:

```shell
apptainer pull vllm-openai_v0.26.0.sif docker://vllm/vllm-openai:v0.26.0
mv vllm-openai_v0.26.0.sif <cache_dir>/images/
```

Model matrix

For each: launch the service, confirm HEALTHY, hit one completion request through the tunnel.

Regression targets — popular models we already run

  • google/gemma-3-4b-it — Gemma 3 confirmed working
  • meta-llama/Llama-3.3-70B-Instruct — flagship dense; needs TP≥4

New-arch wins — models that were hit-and-miss or failed on v0.20.0

  • google/gemma-4-12b-it — Gemma 4 support landed v0.21+; already confirmed working
  • deepseek-ai/DeepSeek-OCR-2 — already confirmed working
  • Qwen/Qwen3.6-27B — Qwen 3.6 confirmed working
  • Qwen/Qwen3.6-35B-A3B
  • openai/gpt-oss-20b — arch landed v0.24 (model loads—known vocab issue persists)
  • meta-models/Muse-Glimmer-30B (failed)

Parameter checklist

Launch-time

The command is `apptainer run --model ... --port ... --revision ... --trust-remote-code --tensor-parallel-size {launch_kwargs}`. Testing the fixed flags happens implicitly with every launch above; the interesting bits are:

  • Multi-GPU--tensor-parallel-size still works (implicitly exercised by Llama-3.3-70B and Qwen/Qwen3.6-27B)
  • Disable Thinking — launch Qwen/Qwen2.5-72B-Instruct with the Disable Thinking checkbox on. Container log has no "unknown option" warning for --default-chat-template-kwargs; a subsequent completion request comes back without <think>…</think> in the response.

Request-time (OpenAI-compat API served by vLLM)

Against any healthy service (TinyLlama is the cheapest), one chat completion request exercising several sampling params at once:

  • Body: {temperature: 0.7, max_tokens: 64, stop: [\"\\n\\n\"], stream: true, seed: 42, frequency_penalty: 0.3, presence_penalty: 0.3}. Should return normally (no 400). Repeat with seed: 42 and verify deterministic output.

Land the bump

The draft PR (#523) already handles the pin update and the load-bearing test change. Once the checklist above is green, mark #523 "Ready for review" and merge.

Post-merge

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions