Skip to content

Make the PyTorch install and model cache survive real-world Windows setups - #4

Merged
MarkDanielArndt merged 5 commits into
MarkDanielArndt:mainfrom
JonaRichter:fix/cuda-kernel-mismatch-fallback
Jul 30, 2026
Merged

Make the PyTorch install and model cache survive real-world Windows setups#4
MarkDanielArndt merged 5 commits into
MarkDanielArndt:mainfrom
JonaRichter:fix/cuda-kernel-mismatch-fallback

Conversation

@JonaRichter

@JonaRichter JonaRichter commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Summary

Four fixes for failures that made an analysis run impossible on some machines. Each was diagnosed from an actual failing install rather than from reading the code — one of them turned out to have a very different cause than its error message suggested.

What is in here

Fall back to CPU when CUDA's kernels do not match the GPU. torch.cuda.is_available() only confirms a CUDA runtime is present, not that the installed build's compiled kernels cover this GPU's compute capability. On a mismatch the first real kernel launch fails mid-analysis. A trivial canary op is not a reliable stand-in: observed on real hardware to succeed after a slow one-time JIT warmup while the model's own conv/batchnorm kernels still failed immediately afterwards. The probe therefore runs the actual model on a dummy input of its expected shape, and the model's final device is whatever that probe decided.

Separate a broken model load from a genuinely missing one. Exit code 2 previously covered both "the file is not downloaded" and "loading blew up for some other reason", and the UI always answered with "run the analysis again to trigger a download". For a broken torch install that advice is actively wrong — retrying just repeats the failure. Broken loads now get their own exit code and surface the real exception text instead of a generic message.

Pin torch below 2.13 on Windows. torch 2.13 moved to PEP 639 license-file collection, which copies its whole vendored third-party licence tree into torch-<version>.dist-info/licenses/. The deepest resulting path fails Windows installs with WinError 206 whenever site-packages is longer than roughly 86 characters — which a default Slicer installation already is. This is an open upstream regression in PyTorch itself, independently reported by users of other PyTorch-based Slicer extensions and confirmed by a Slicer core developer; the official PyTorch extension has a proposed but unmerged fix for it. PyTorchUtilsLogic.installTorch() already accepts a version requirement, so this passes <2.13.0 on Windows only and leaves macOS and Linux untouched.

Treat a checksum-mismatched cached model as missing. A truncated download left a 21-byte file where a 336 MB model belonged. Nothing checked content — the cache lookup only tested existence plus a zero-byte guard, so the file counted as cached and the failure surfaced deep inside torch.load() as a confusing unpickling error. The checks now use the verify_checksum() helper that already existed but had never been wired into this path, so a corrupted file routes through the normal download prompt instead.

A diagnosis that was initially wrong

The truncated-download problem was first written up as a checkpoint-format incompatibility with PyTorch's restricted unpickler, with several proposed options centred on relaxing weights_only. Measuring the file settled it differently: 21 bytes on disk against 352,517,483 expected, and weights_only=False failed earlier than weights_only=True — it was not a valid pickle stream at all. No security exception was needed, and the project's blanket rule forbidding weights_only=False in production code is untouched.

Verification

pytest tests/ — 521 passed, run against this branch on its own rather than only at the tip of the follow-up work.

Manually verified in Slicer: torch uninstalled and reinstalled from scratch through the real Run Analysis flow. The first attempt resolved 2.12.1+cpu; a second full uninstall/reinstall resolved 2.8.0+cu129 — a different build entirely, confirming the pin holds regardless of which sub-2.13 CPU or CUDA build gets selected. No WinError 206, no missing-torchgen error, and a full analysis completes.

Note on the version pin

The pin works around someone else's regression. Once the upstream fix reaches the released PyTorch extension it can be reverted; the version cap sits in a single place for exactly that reason.

@JonaRichter JonaRichter changed the title Fix CUDA/kernel mismatch fallback and clarify model-load failure messages Make the PyTorch install and model cache survive real-world Windows setups Jul 29, 2026
@MarkDanielArndt
MarkDanielArndt merged commit 0e98fed into MarkDanielArndt:main Jul 30, 2026
6 checks passed
@JonaRichter
JonaRichter deleted the fix/cuda-kernel-mismatch-fallback branch July 30, 2026 08:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants