fix(gpu): run on Blackwell, and refuse a GPU this PyTorch build cannot use

[ncoevoet]

Summary

An RTX 50-series card ran facet.py, sailed through detection, was assigned the 16gb profile, and died on its first real tensor with "no kernel image is available for execution on the device." The published latest-cuda image was pinned to a torch that ships sm_50–sm_90 cubins; Blackwell is sm_120.

Three things had to change, and each uncovered the next.

The image. latest-cuda now builds on pytorch/pytorch:2.11.0-cuda12.8-cudnn9-runtime (sm_75–sm_120). That wheel drops sm_50/sm_60/sm_70, so Maxwell, Pascal and Volta cards get their own tag, latest-cuda-legacy, on the previous base. Two published images became three.

The detection. torch.cuda.is_available() answers "is there a driver and a device" and nothing else — get_arch_list() had zero occurrences in the tree, and no code allocated so much as a test tensor before committing a VRAM profile. Facet now compares the device's compute capability against the build's architecture list and launches one throwaway kernel first. A card the build cannot run falls back to CPU and names the image tag or wheel index to switch to.

The bare-metal installer. install.sh chose its wheel from the driver's CUDA version alone. A GTX 1080 behind a modern driver reports "CUDA Version: 12.8" exactly as an RTX 5090 does, so it installed cu128 and died the same death from the opposite direction.

Two unrelated regressions surfaced on the way and are fixed here. The published CPU image shipped a full CUDA build of PyTorch — measured at torch 2.6.0+cu124, 13 nvidia-* wheels, 8.81 GB — because a sympy==1.13.1 pin made pip discard the CPU-only torch to satisfy the resolver. Rebuilt: torch 2.13.0+cpu, zero nvidia-*, 3.34 GB. And the runtime UID silently moved from 1000 to 1001 on the new Ubuntu 24.04 base, which would have opened an existing install's SQLite database read-only on the next docker compose pull.

Root cause

is_available() conflates a device exists with this wheel can run on it. Those are different questions, and CUDA answers the second only when a kernel actually launches — which is why the gate reads the arch list and executes one, with the executed kernel winning.

The rule is deliberately an allow-list. Only a definite architecture mismatch, or a failure carrying CUDA's own no kernel image / invalid device function signature, marks a GPU unusable. An out-of-memory, a card held in exclusive mode, an unfamiliar CUDA error — all fail open and surface where they always did, at model load. A deny-list of the failures we happen to know about would demote a working GPU to CPU for the process lifetime and blame the install for it.

Splitting the dependency lock in two created an invariant nothing enforced: the files must stay identical apart from the sympy pin, because any other drift makes pip resolve the mismatch by replacing a base image's torch with a PyPI build — the 8.81 GB regression above, running backwards. Two CI gates now hold that line, plus the dependabot.yml ignore lists, where a dropped list dash merges an entry into its predecessor and yaml.safe_load accepts it.

Review pass

This branch was reviewed before opening, and roughly half the diff is what that found. The theme was one shape: the migration off is_available() was incomplete, and CI could not see the fix break.

• ModelManager.detect_vram() was left on the old predicate while its sibling ScoringConfig.detect_gpu_vram_gb was migrated. On a Blackwell box the two then disagreed — get_device() said cpu, detect_vram() reported the card's full VRAM — so the CPU fallback was budgeted as a 16 GB GPU run and skipped _apply_ram_safe_chunk_start, the guard against exactly that OOM.
• OCR built easyocr.Reader(gpu=True) on the same predicate. Moving weights is a memcpy and succeeds, so the missing cubin landed inside every readtext, which extract_text swallows — OCR returned nothing for an entire library where gpu=False would have worked.
• The /metrics gauges and the resource monitor published live VRAM for a card Facet had just refused to use.
• The two CI gates added for the lock split had dead failure paths: under GitHub's default bash -e {0}, count=$(grep -c ...) aborts the step before its own error message can print — the case the loop existed to catch — and the sympy comparison read a missing pin as "pinned to the same version ()".
• The ignore-list assertion covered pip only, while the entry pinning ARG BASE_IMAGE lives in the docker block.
• Publishing switched from build-push-action to a hand-rolled docker push, which silently drops the SLSA provenance attestation every Facet image currently carries. Restored, while keeping the smoke test a gate rather than a post-mortem.
• Five undici CVEs (one high) were closed with an overrides entry, as this repo already does for ws. npm audit fix --force wanted @angular/build 21.2.22, outside the exact 21.2.19 pinned here, and Dependabot build(deps): bump the angular group across 1 directory with 10 updates #109 only reaches 21.2.21 — neither would have closed them.

CodeQL alert #120 (py/command-line-injection, critical) was investigated and dismissed as a false positive: cmd is a list with no shell=True; the only request-derived argv value is the photo path, which is DB-validated, then os.path.realpath'd — always absolute, so it can never be parsed as an option flag (realpath('--core') → /abs/cwd/--core) — and scan-directory contained; profile/style are equality-matched against config allowlists; and the dangerous executable / extra_args fields live in scoring_config.json, which none of the four config-writing endpoints ever touch.

Verification

• Python 3648 passed, 7 skipped; client 1969 passed across 111 files. ruff (both the project config and CI's E,F,W), check_packaging.py, audit_i18n.py, ng lint, both tsconfig projects, ng build and gen:api all clean.
• The mypy ratchet improves by one: 161 against a clean-HEAD measurement of 162 in the same venv. Measured in a detached worktree rather than assumed, because the script's own docstring warns the count is environment-dependent. mypy-baseline.txt is left at 154, which tracks CI's dependency-free environment and cannot be measured locally.
• Every new test was confirmed able to fail. Breaking the digit split so sm_120 parses as major 1 minor 20 — the exact regression that would make the correct Blackwell image reject its own GPU — left the previous suite fully green; it now turns two tests red. Same for dropping the minor comparison, deleting the probe's synchronize, defeating the per-module memo key, blanking the mismatch reason, deleting the two lines that actually resolve RTX 50-series (Blackwell / sm_120): published CUDA image detects the GPU but cannot run kernels (torch 2.6+cu126) #119, and making the verbose branch raise. Each fault was reverted by retyping and the file confirmed byte-identical, never by git checkout.
• Both new CI gates were extracted from the YAML and executed verbatim against damaged fixtures: a dropped dash that yaml.safe_load accepts while pytorch/pytorch vanishes now fails the build, where before it printed "intact" and exited 0.
• The arch claim is backed by a real payload, not the advisory: torch._C._cuda_getArchFlags() on an installed cu128 wheel reports sm_70 sm_75 sm_80 sm_86 sm_90 sm_100 sm_120.
• The provenance regression was confirmed against live GHCR — both latest and latest-cuda currently return an OCI index carrying an attestation-manifest.
• Docs updated in all six languages, structural parity checked (34 headings, 42 code fences per file, unchanged from HEAD).

Note that docker-publish.yml runs on v* tags only, so the three-image matrix, the UID assertion and the arch smoke test do not execute on this PR — they first run at release, or via workflow_dispatch.

Closes #119.