Enterprise readiness: NVIDIA Blackwell B200 (sm_100) validation matrix and deltas

[Defilan]

Motivation

NVIDIA's Blackwell datacenter line (B200, GB200) is shipping into enterprise environments and DGX B200 systems are landing in customer datacenters. LLMKube currently validates on H100, L4, and L40S. We should publish a concrete plan for what needs to change to make Blackwell B200 (compute capability 10.0, sm_100) a first-class day-one target so enterprise users running LLMKube on next-generation NVIDIA hardware know what's tested and what's a known gap.

This issue captures the validation matrix and the concrete deltas across runtime images, dependencies, CRDs, and Helm chart defaults that shift between H100-class and B200-class deployments.

Fact-check provenance

The version floors and runtime support claims in this issue were verified on 2026-05-07 against NVIDIA's CUDA / driver / MIG docs, gpu-operator + k8s-device-plugin + dcgm-exporter + NCCL release notes, vllm-project/vllm RFC #18153, llama.cpp build docs, and DGX OS 7 release notes. The companion validation matrix doc at docs/operations/b200-validation-matrix.md (PR #414) records the same numbers and a re-verification cadence. This is a fast-moving stack; re-verify floors quarterly.

Scope

In scope: datacenter Blackwell (B200, GB200) on sm_100.

Out of scope here: consumer Blackwell (RTX 50-series, GB20x, sm_120) is similar but tracked separately. Multi-node B200 sharding (NVLink fabric across chassis) is a larger effort tracked elsewhere.

Validation matrix

When hardware is reachable, the following matrix is published at docs/operations/b200-validation-matrix.md with each cell marked pass / fail / blocked-by-hardware.

#	Test	Notes
1	Single-GPU TinyLlama-1.1B serve on B200 (FP16)	Baseline sanity
2	Single-GPU 8B model FP8 (E4M3) serve	Existing FP8 checkpoints (Llama-3.x-FP8, Qwen2.5-FP8) load unchanged on B200
3	Single-GPU 70B FP8 serve, single chassis	Memory-bound, exercises HBM3e bandwidth
4	8x B200 single-chassis multi-GPU sharding via NVLink5	Layer-based offload, validate NVSwitch5 topology
5	DCGM exporter scrape with all Blackwell-native counters	NVLink5 per-link, FP4 utilization, per-die power, HBM3e bandwidth, PCIe Gen6 link health
6	Recording rules from #409 produce series under load	TTFT, queue wait, restart rate at FP8 throughput
7	MIG profile deploys: 1g.23gb, 2g.45gb, 7g.180gb	Verify device plugin advertises nvidia.com/mig-* resources
8	NVFP4 inference (vLLM 0.10+, ModelOpt-converted)	Real but ModelOpt is proprietary; document conversion path
9	MXFP4 inference	OCP standard, broader vLLM support
10	Crashloop / OOM / NVLink-degrade operational runbooks fire correctly	Triage paths exercised end-to-end

Hardware-blocked rows stay open until B200 access lands (NVIDIA dev partnership, customer install, or rented capacity).

Concrete deltas to land before B200 production use

Cross-referenced against what we ship today.

1. Driver / CUDA floors documented

Add a "Tested platforms" section to the Helm chart README capturing:

• NVIDIA driver: R570 minimum (570.124.06 GA; 570.133.20 required for HGX B200 per gpu-operator platform-support docs); R570.86+ recommended for DGX OS 7
• CUDA toolkit: 12.8 minimum (first public sm_100 codegen, Jan 2025); 12.9+ recommended; forward target CUDA 13.x
• R550 and earlier do not list B200 in supported hardware. (R555 was the open-kernel-module preview branch and is unrelated to Blackwell GA timing.)

2. GPU operator / device plugin floors

Document in charts/llmkube/values.yaml comments and the air-gapped install guide:

• gpu-operator >= v24.6 (v24.9 / v25.x recommended)
• k8s-device-plugin >= v0.17.2 (Blackwell-aware product labels); v0.18.0+ preferred for full architecture detection. (v0.15 / v0.16 do not advertise Blackwell architecture labels.)
• Fabric Manager package version MUST match the NVIDIA driver exactly. A mismatch silently degrades NVLink5 to PCIe with no clear error surface. Highest-priority silent-failure mode; deserves a runbook entry on its own.

3. DCGM exporter floor

• Floor: dcgm-exporter 3.3.x line for basic Blackwell coverage
• Recommended: dcgm-exporter 4.5+ (current line as of Feb 2026). Verify exact DCGM field IDs (NVLink5 throughput, HBM3e bandwidth, PCIe link health) against the DCGM field reference at validation time rather than enumerating in this issue; the named-counter list shifts release-to-release
• Extend the Grafana dashboard from feat(metrics): expose TTFT and per-request error rate; enable vLLM /metrics #409 to surface the new counters once 4.0 is in the bundled chart

4. Runtime image bumps + sm_100 codegen

Audit the default images set by each runtime backend:

• internal/controller/runtime_vllm.go: vllm/vllm-openai:v0.20.0 is on the current line (latest released v0.20.1 as of May 2026). Blackwell support landed incrementally starting around v0.15.x per vllm-project/vllm RFC #18153 (filed May 2025), maturing through v0.16-v0.18. Practical floor: v0.18+. Confirm exact tag at validation time
• internal/controller/runtime_llamacpp.go: ghcr.io/ggml-org/llama.cpp:server must be a build that includes CMAKE_CUDA_ARCHITECTURES="90;100". Pin to a tag verified to ship sm_100 codegen.
• internal/controller/runtime_tgi.go: ghcr.io/huggingface/text-generation-inference:latest should pin to a 3.x tag with rebuilt flash-attn for Blackwell

5. FP8 / FP4 quantization in the CRD

InferenceServiceSpec.VLLMConfig.Quantization already exists. Verify and document:

• FP8 (E4M3, E5M2) values pass through cleanly
• NVFP4 and MXFP4 are in the allowed value set; document the per-runtime support matrix in the field's GoDoc
• Conversion paths for both NVFP4 (ModelOpt, proprietary) and MXFP4 (OCP standard) are mentioned in the model catalog docs

6. NCCL version floor for multi-GPU

Document in the multi-GPU deployment guide:

• NCCL >= 2.25.1 introduced Blackwell support; 2.25.2+ for GB200 MNNVL
• NCCL 2.27+ / 2.28+ recommended for NVLSTree tuning and Blackwell perf fixes. (NCCL 2.24 and earlier predate Blackwell support entirely; the previously-circulated "2.23 floor" is incorrect.)

7. OFED / OS floors for GPUDirect RDMA on Gen6

For the air-gapped install guide and any future multi-node sharding work:

• DGX OS 7 (Ubuntu 24.04, kernel 6.8) for HGX B200; non-DGX kernel floors should be pinned against NVIDIA's GPUDirect RDMA guide at validation time
• MLNX_OFED 24.10 LTS (final standalone release, Oct 2024) or DOCA-OFED (the supported forward path; MLNX_OFED has been EOL for new features since Jan 2025). Verify exact DOCA-OFED version against its release notes at validation time for ConnectX-7 / BlueField-3 GPUDirect RDMA. PCIe Gen6 + ConnectX-8 GPUDirect RDMA may require additional Linux kernel patches per NVIDIA's guide

Common silent-failure modes (each becomes a runbook)

Each of these belongs in docs/operations/runbooks/:

1. Fabric Manager / driver version mismatch. NVLink5 degrades to PCIe with no clear error. Detection: NVLink bandwidth scrape from DCGM; alert if reported topology bandwidth diverges from expected.
2. Old base image (CUDA 12.4) on B200. Kernels appear to load but fall back / fail at launch. Detection: container logs around model load.
3. NCCL 2.24 or earlier on Blackwell. Lacks the NVLink5 / NVSwitch5 topology support introduced in NCCL 2.25.1; results in small-message all-reduce regressions on multi-GPU. Detection: collective benchmark in the validation matrix.
4. Old OFED on Gen6. GPUDirect RDMA disabled silently, looks like generic network slowness. Detection: nvidia-smi topo --matrix + RDMA performance benchmark.

What this issue is NOT

• A CRD redesign for Blackwell. The existing Hardware.GPU.Count is sufficient; MIG profile selection is a follow-up if the validation matrix shows it's needed.
• A multi-node sharding plan (separate larger effort).
• A statement of when LLMKube is officially "Blackwell-certified." This is a validation roadmap, not a marketing claim.

Acceptance criteria

• All matrix rows land in docs/operations/b200-validation-matrix.md with placeholder pass/fail/blocked cells
• Each "concrete delta" section (1-7) lands as a separate scoped PR or sub-issue
• Hardware-blocked rows stay open until B200 access is reachable
• At least one runbook per silent-failure mode published under docs/operations/runbooks/
• Helm chart README documents tested driver / CUDA / GPU operator floors
• CRD GoDoc documents per-runtime FP8 / FP4 support

[Defilan]

Did a fact-check pass against authoritative sources (NVIDIA driver / CUDA / MIG docs, gpu-operator + k8s-device-plugin + dcgm-exporter + NCCL release notes, vllm-project/vllm RFC #18153, llama.cpp build docs, DGX OS 7 release notes). Six version-floor / runtime-support claims needed amending; three (CUDA 12.8 as first sm_100 toolkit, llama.cpp CMAKE_CUDA_ARCHITECTURES="90;100", MIG profile names) verified as written. Issue body updated in place; the companion validation matrix doc at PR #414 (docs/operations/b200-validation-matrix.md) carries the same corrections plus a quarterly re-verification cadence.

Net summary of the corrections:

• Driver floor moved from 560.35.03 → R570 (570.124.06 GA / 570.133.20 for HGX B200)
• vLLM Blackwell timeline: v0.15.x first claim per RFC #18153 (May 2025), not v0.7.x; practical floor v0.18+
• NCCL: 2.25.1 floor (Blackwell support), not 2.23
• k8s-device-plugin: v0.17.2 minimum (Blackwell-aware product labels), not v0.15
• dcgm-exporter recommended bumped to current line (4.5+); softened the named-counter list
• MLNX_OFED EOL-for-new-features as of Jan 2025; DOCA-OFED is the forward path; dropped unsourced kernel 6.5 floor