Feature: Inference-engine ↔ CUDA/driver/PyTorch compatibility resolution (vLLM + SGLang)

[mitulgarg]

Summary

env-doctor should understand the compatibility relationship between inference
engines (vLLM, SGLang) and the rest of the stack — system CUDA, NVIDIA driver,
PyTorch, and the engine's bundled kernel libraries. Today we validate CUDA,
driver, and library presence/versions, but we don't answer the question that
actually blocks deployments:

"Given the CUDA/driver already installed on this box, which version of
vLLM (or SGLang) will actually install and run — and what's the exact
install command?"

Motivation (real case)

Deploying a newer model onto a customer's L40S box that already had vLLM +
models running on CUDA 12.6:

• Latest vLLM pulls a default wheel built for CUDA 13.x, which won't run
  against the installed 12.6 toolchain/driver.
• The customer can't trivially bump CUDA — it's a shared box with existing
  deployments pinned to 12.6.
• Resolution was to drop down to an older vLLM line (0.19.x) that still ships a
  CUDA 12.6-compatible wheel and supports the target model.

That diagnosis took manual cross-referencing of vLLM release notes, the PyTorch
build matrix, and trial-and-error pip resolves. This is exactly the "flutter
doctor for ML" gap env-doctor exists to close — it should have told the user up
front, in one command.

Why current checks aren't enough

Our existing checks treat CUDA, driver, and libraries as independent facts.
The failure mode here is relational: it's not that any single component is
"wrong," it's that the combination the user is about to install is
incompatible with what's already on the box. We need an engine-aware resolver,
not another presence/version probe.

The compatibility chain (the actual problem)

The thing that has to resolve is a chain, not a single mapping:

inference engine  →  pinned PyTorch  →  CUDA build of that wheel  →  bundled kernel lib
   (vllm / sglang)     (e.g. 2.11)        (cu128 / cu129 / cu130)     (flashinfer / sgl-kernel)
                                                  ↓
                              must be forward-compatible with
                              the INSTALLED system CUDA + driver

Key realities the resolver must encode (verified against current releases —
these move weekly, so they're seed data, not gospel):

• vLLM ships multiple CUDA-build wheels per version, with a default.
  v0.20.0 switched the default PyPI/Docker wheel to CUDA 13.0, but cu128/cu129
  wheels still exist. So the answer is rarely "this version is impossible" — it's
  "the default install breaks; here's the alternate wheel / index that works,"
  or "the newest engine version with a wheel for your CUDA is N."
• The naive install path is the trap. pip install vllm grabs the default
  (newest CUDA) wheel. The fix is usually a --torch-backend=cuXXX flag or an
  --extra-index-url, not a different engine version. env-doctor should prefer
  recommending the install incantation over recommending a downgrade.
• SGLang is a tighter, more brittle chain. sglang ↔ sgl-kernel ↔ flashinfer ↔ torch ↔ cuda-python are co-pinned. Forcing an off-default CUDA
  (e.g. sglang 0.5.x onto cu128) produces hard dependency-resolution conflicts,
  not just a runtime mismatch. The resolver needs SGLang's own per-CUDA wheel
  index URLs as first-class data.
• Driver forward-compat matters too. A wheel built for CUDA 13 can still
  fail if the installed driver predates the minimum for that CUDA major. The
  resolver should factor driver version, not just toolkit version.

Proposed solution

Extend the existing compatibility engine with an inference-engine dimension.

Behaviour

Add detection + advisory for installed (or requested) inference engines:

1. Detect installed vLLM / SGLang and their effective torch/CUDA build.
2. Diagnose mismatches against the installed system CUDA + driver.
3. Recommend the concrete fix, in priority order:
   • the correct install command for the current engine version on this box
     (alternate wheel / index / --torch-backend), if one exists;
   • otherwise, the newest engine version that has a compatible wheel;
   • only as a last resort, a CUDA/driver change (flagged as high-risk on shared
     boxes).

Example output (illustrative)

$ env-doctor check --engine vllm

  ✗ vllm 0.20.x  →  incompatible with installed CUDA 12.6
      default wheel targets CUDA 13.0 (torch 2.11, +cu130)
      installed: CUDA 12.6, driver 550.x

  → Fix options (best first):
    1. Pin a CUDA 12.x build of a recent vLLM:
         uv pip install "vllm==0.19.x" --torch-backend=cu126
    2. Or upgrade toolchain to CUDA 12.9 and use:
         uv pip install vllm --torch-backend=cu129
       (⚠ shared box — existing deployments pinned to 12.6 may break)

  Copy-to-fix:  uv pip install "vllm==0.19.x" --torch-backend=cu126

Same flow for --engine sglang, surfacing the sgl-kernel/flashinfer/torch
chain and SGLang's per-CUDA index URLs.

Data model

This is a new node type / edge set in the compatibility intelligence layer —
it should feed the same database + snapshot distribution we're already building,
not a bolted-on lookup table. Seed it with a curated static table (engine
version → {supported CUDA builds, default CUDA build, pinned torch, kernel-lib
versions, min driver, install command template}) and let opt-in telemetry
enrich it over time (real "this combo worked / this combo failed on this GPU"
verdicts are exactly the moat).

Scope

v1 (ship first — keep it simple):

• Curated static compatibility table for vLLM and SGLang, hand-maintained,
  shipped in the weekly snapshot.
• Detection of installed engine + torch/CUDA build.
• Advisory output with copy-to-fix install command.
• L40S / CUDA 12.6 case passes as the canonical test fixture.

v2 (later):

• Telemetry-fed verdicts (success/failure combos by GPU + CUDA + driver).
• "Will this install resolve?" dry-run prediction before the user runs pip/uv.
• Model-architecture axis (newer models needing newer transformers/engine) —
  likely a separate issue; this one stays focused on engine ↔ CUDA.

Acceptance criteria

• env-doctor check --engine vllm detects installed vLLM + its CUDA build.
• On a CUDA 12.6 box with a CUDA-13 default wheel, output flags the mismatch
  and emits a working copy-to-fix command without a CUDA downgrade where one
  exists.
• Same coverage for --engine sglang, including the kernel-lib chain.
• Compatibility data lives in the snapshot/graph layer, not hardcoded in CLI.
• Recommendations prefer install-incantation fixes over CUDA/driver changes,
  and flag driver/toolkit changes as risky on shared boxes.

Out of scope

• Runtime performance / monitoring (stays out — that's DCGM/Prometheus/Nsight).
• Auto-installing CUDA toolchains as part of this feature (separate workstream).
• Non-NVIDIA backends (ROCm/XPU/Metal) — note as future, don't block on it.

Open questions

• Where does the engine version come from when not yet installed — a --engine vllm@0.20 flag, or infer the target from a requirements file?
• How aggressively do we maintain the static table vs. lean on telemetry? (These
  release matrices change roughly weekly.)
• Do we model the kernel libs (flashinfer, sgl-kernel) as their own nodes, or
  fold them into the engine record for v1?

---

Note: I don't have the current env-doctor codebase in front of me while drafting
this — before implementing, map "existing compatibility engine" and "snapshot
layer" to the actual modules so this extends them rather than duplicating.