8587 test erros on pytorch release 2508 on series 50 (#8770)

Fixes #8587.

### Description

On NVIDIA Blackwell GPUs (compute capability 12.x, sm_120), running
MONAI with `USE_COMPILED=True` produces **silently incorrect spatial
transform results**. The root cause is that the MONAI compiled C
extension (`monai._C`) is built at install time against a fixed set of
CUDA architectures (`TORCH_CUDA_ARCH_LIST`). Blackwell (sm_120) is not
included in the default build list (which tops out at sm_90, Hopper), so
`grid_pull` executes against a mismatched PTX or JIT path and silently
returns wrong values — there is no runtime error.

This affects two independent code paths:

**1. `spatial_resample` / `Resample`**
(`monai/transforms/spatial/functional.py`)
When `USE_COMPILED=True`, `spatial_resample` unconditionally calls
`grid_pull`. On Blackwell GPUs this produces incorrect resampling output
without raising any exception.

**2. `Warp`** (`monai/networks/blocks/warp.py`)
`Warp.__init__` stores interpolation and padding modes as **integers**
when `USE_COMPILED=True` (as required by `grid_pull`). The
PyTorch-native fallback path (`F.grid_sample`) requires **string**
modes. Without a Blackwell-aware fallback, there was no path to trigger
this mismatch — but once the device check forces the fallback, the
integer modes cause a type error at runtime.

#### Additional change to `runner.py` (⚠️not directly related to the
issue)

Add per-test timeout via --timeout flag

Tests that hang indefinitely (e.g. GPU ops stuck on Blackwell) block the
entire suite. Add a --timeout SECONDS option to tests/runner.py that
uses SIGALRM to interrupt any individual test that exceeds the limit;
the test is recorded as an error and the runner continues with the next
test.

- Default is 0 (disabled); SIGALRM support is checked at runtime so the
flag is silently ignored on Windows.
- runtests.sh gains a matching --timeout [secs] flag (default 180s when
the flag is given without a value) that is forwarded to runner.py for
unit tests.

```bash
#Usage:
  ./runtests.sh -u --timeout          # 3-minute per-test limit
  ./runtests.sh -u --timeout 60       # 1-minute per-test limit
  python tests/runner.py --timeout 180
```
**Fix**

A private helper `_compiled_unsupported(device: torch.device) -> bool`
is added to `monai/transforms/spatial/functional.py`. It returns `True`
for CUDA devices with compute capability major ≥ 12, and `False` for all
other devices (CPU, older GPUs).

- In `spatial_resample`, the compiled path is now gated on `USE_COMPILED
and not _compiled_unsupported(img.device)`, falling back to the
PyTorch-native `affine_grid + grid_sample` path on unsupported devices.
- In `Warp`, the same gate is applied in `forward()`. Additionally,
`__init__` now always stores **both** the compiled integer modes (for
`grid_pull`) and the native string modes (for `F.grid_sample`), ensuring
the fallback path has correctly-typed arguments regardless of how
`USE_COMPILED` was set at initialisation time.

Behaviour on all previously supported GPU architectures (sm_75 through
sm_90) is unchanged.

### Types of changes

- [x] Non-breaking change (fix or new feature that would not break
existing functionality).
- [ ] Breaking change (fix or new feature that would cause existing
functionality to change).
- [x] New tests added to cover the changes.
- [x] Integration tests passed locally by running `./runtests.sh -f -u
--net --coverage`.
- [x] Quick tests passed locally by running `./runtests.sh --quick
--unittests --disttests`.
- [x] In-line docstrings updated.
- [ ] Documentation updated, tested `make html` command in the `docs/`
folder.

---------

Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
Signed-off-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Kerfoot <17726042+ericspod@users.noreply.github.com>

Author: 4 people

Dockerfile.slim

@@ -18,7 +18,7 @@ ARG IMAGE=debian:12-slim
 
 FROM ${IMAGE} AS build
 
-ARG TORCH_CUDA_ARCH_LIST="7.5 8.0 8.6 8.9 9.0+PTX"
+ARG TORCH_CUDA_ARCH_LIST="7.5 8.0 8.6 8.9 9.0+PTX 12.0"
 
 ENV DEBIAN_FRONTEND=noninteractive
 ENV APT_INSTALL="apt install -y --no-install-recommends"
@@ -28,7 +28,7 @@ RUN apt update && apt upgrade -y && \
     wget https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64/cuda-keyring_1.1-1_all.deb && \
     dpkg -i cuda-keyring_1.1-1_all.deb && \
     apt update && \
-    ${APT_INSTALL} cuda-toolkit-12 && \
+    ${APT_INSTALL} cuda-toolkit-12-9 && \
     rm -rf /usr/lib/python*/EXTERNALLY-MANAGED /var/lib/apt/lists/* && \
     python -m pip install --upgrade --no-cache-dir --no-build-isolation pip
 
@@ -53,7 +53,8 @@ COPY tests ./tests
 COPY monai ./monai
 
 # install full deps
-RUN python -m pip install --no-cache-dir --no-build-isolation -r requirements-dev.txt
+RUN python -m pip install --no-cache-dir --no-build-isolation -U wheel wheel-stub
+RUN python -m pip install --no-cache-dir --no-build-isolation "torch>=2.8.0,<2.11" -r requirements-dev.txt
 
 # compile ext
 RUN CUDA_HOME=/usr/local/cuda FORCE_CUDA=1 USE_COMPILED=1 BUILD_MONAI=1 python setup.py develop

monai/apps/auto3dseg/bundle_gen.py

@@ -264,8 +264,7 @@ def _run_cmd(self, cmd: str, devices_info: str = "") -> subprocess.CompletedProc
                 look_up_option(self.device_setting["MN_START_METHOD"], ["bcprun"])
             except ValueError as err:
                 raise NotImplementedError(
-                    f"{self.device_setting['MN_START_METHOD']} is not supported yet."
-                    "Try modify BundleAlgo._run_cmd for your cluster."
+                    f"{self.device_setting['MN_START_METHOD']} is not supported yet. Try modify BundleAlgo._run_cmd for your cluster."
                 ) from err
 
             return _run_cmd_bcprun(cmd, n=self.device_setting["NUM_NODES"], p=self.device_setting["n_devices"])

monai/apps/detection/networks/retinanet_detector.py

@@ -342,8 +342,7 @@ def set_regular_matcher(
         """
         if fg_iou_thresh < bg_iou_thresh:
             raise ValueError(
-                "Require fg_iou_thresh >= bg_iou_thresh. "
-                f"Got fg_iou_thresh={fg_iou_thresh}, bg_iou_thresh={bg_iou_thresh}."
+                f"Required condition fg_iou_thresh >= bg_iou_thresh not met ({fg_iou_thresh=}, {bg_iou_thresh=})."
             )
         self.proposal_matcher = Matcher(
             fg_iou_thresh, bg_iou_thresh, allow_low_quality_matches=allow_low_quality_matches
@@ -519,7 +518,7 @@ def forward(
         else:
             if self.inferer is None:
                 raise ValueError(
-                    "`self.inferer` is not defined." "Please refer to function self.set_sliding_window_inferer(*)."
+                    "`self.inferer` is not defined. Please refer to function self.set_sliding_window_inferer(*)."
                 )
             head_outputs = predict_with_inferer(
                 images, self.network, keys=[self.cls_key, self.box_reg_key], inferer=self.inferer

monai/auto3dseg/analyzer.py

@@ -965,8 +965,7 @@ def __call__(self, data: dict) -> dict:
             self.hist_range = nr_channels * self.hist_range
         if len(self.hist_range) != nr_channels:
             raise ValueError(
-                f"There is a mismatch between the number of channels ({nr_channels}) "
-                f"and histogram ranges ({len(self.hist_range)})."
+                f"There is a mismatch between the number of channels ({nr_channels}) and histogram ranges ({len(self.hist_range)})."
             )
 
         # perform calculation

monai/csrc/ext.cpp

@@ -63,6 +63,15 @@ PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {
       .value("seventh", monai::InterpolationType::SeventhOrder)
       .export_values();
 
+  // build-time compile capability info
+  m.def("max_compute_capability", []() {
+#ifdef MONAI_MAX_COMPUTE_CAPABILITY
+    return MONAI_MAX_COMPUTE_CAPABILITY;
+#else
+    return 0;
+#endif
+  }, "Maximum compute capability (major*100+minor) the extension was compiled for, or 0 if unknown");
+
   // resample
   m.def("grid_pull", &monai::grid_pull, "GridPull");
   m.def("grid_pull_backward", &monai::grid_pull_backward, "GridPull backward");

monai/data/dataset.py

@@ -609,7 +609,7 @@ def __init__(
         # the cache is created without multi-threading
         self._read_env: Any | None = None
         # this runs on the primary thread/process
-        self._fill_cache_start_reader(show_progress=self.progress)
+        self._read_env = self._fill_cache_start_reader(show_progress=self.progress)
         print(f"Accessing lmdb file: {self.db_file.absolute()}.")
 
     def set_data(self, data: Sequence):
@@ -637,6 +637,12 @@ def _fill_cache_start_reader(self, show_progress=True):
         Args:
             show_progress: whether to show the progress bar if possible.
         """
+        # Close any open read environment before attempting write-mode access
+        # to prevent "environment already open" errors when multiple LMDBDataset
+        # instances target the same db file
+        if self._read_env is not None:
+            self._read_env.close()
+            self._read_env = None
         # create cache
         self.lmdb_kwargs["readonly"] = False
         env = lmdb.open(path=f"{self.db_file}", subdir=False, **self.lmdb_kwargs)
@@ -664,7 +670,7 @@ def _fill_cache_start_reader(self, show_progress=True):
                         size = env.info()["map_size"]
                         new_size = size * 2
                         warnings.warn(
-                            f"Resizing the cache database from {int(size) >> 20}MB" f" to {int(new_size) >> 20}MB."
+                            f"Resizing the cache database from {int(size) >> 20}MB to {int(new_size) >> 20}MB."
                         )
                         env.set_mapsize(new_size)
                     except lmdb.MapResizedError:

monai/data/wsi_reader.py

@@ -416,8 +416,7 @@ def get_data(
             # Check if there are three color channels for RGB
             elif mode in "RGB" and patch.shape[self.channel_dim] != 3:
                 raise ValueError(
-                    f"The image is expected to have three color channels in '{mode}' mode but has "
-                    f"{patch.shape[self.channel_dim]}. "
+                    f"The image is expected to have three color channels in '{mode}' mode but has {patch.shape[self.channel_dim]}. "
                 )
             # Get patch-related metadata
             metadata: dict = self._get_metadata(wsi=each_wsi, patch=patch, location=location, size=size, level=level)

monai/networks/blocks/warp.py

@@ -20,6 +20,7 @@
 from monai.config.deviceconfig import USE_COMPILED
 from monai.networks.layers.spatial_transforms import grid_pull
 from monai.networks.utils import meshgrid_ij
+from monai.transforms.spatial.functional import _compiled_unsupported
 from monai.utils import GridSampleMode, GridSamplePadMode, optional_import
 
 _C, _ = optional_import("monai._C")
@@ -63,6 +64,18 @@ def __init__(self, mode=GridSampleMode.BILINEAR.value, padding_mode=GridSamplePa
         super().__init__()
         # resolves _interp_mode for different methods
 
+        # Native string modes are always stored for the PyTorch fallback path.
+        # When USE_COMPILED=True but the device is unsupported at runtime (e.g. Blackwell GPU),
+        # forward() falls back to F.grid_sample which requires string, not integer, modes.
+        self._interp_mode_native = (
+            GridSampleMode(mode).value if mode in (m.value for m in GridSampleMode) else GridSampleMode.BILINEAR.value
+        )
+        self._padding_mode_native = (
+            GridSamplePadMode(padding_mode).value
+            if padding_mode in (p.value for p in GridSamplePadMode)
+            else GridSamplePadMode.BORDER.value
+        )
+
         if USE_COMPILED:
             if mode in (inter.value for inter in GridSampleMode):
                 mode = GridSampleMode(mode)
@@ -77,7 +90,7 @@ def __init__(self, mode=GridSampleMode.BILINEAR.value, padding_mode=GridSamplePa
             self._interp_mode = mode
         else:
             warnings.warn("monai.networks.blocks.Warp: Using PyTorch native grid_sample.")
-            self._interp_mode = GridSampleMode(mode).value
+            self._interp_mode = self._interp_mode_native
 
         # resolves _padding_mode for different methods
         if USE_COMPILED:
@@ -93,7 +106,7 @@ def __init__(self, mode=GridSampleMode.BILINEAR.value, padding_mode=GridSamplePa
                     padding_mode = 0  # default to nearest
             self._padding_mode = padding_mode
         else:
-            self._padding_mode = GridSamplePadMode(padding_mode).value
+            self._padding_mode = self._padding_mode_native
 
         self.ref_grid = None
         self.jitter = jitter
@@ -138,13 +151,15 @@ def forward(self, image: torch.Tensor, ddf: torch.Tensor):
         grid = self.get_reference_grid(ddf, jitter=self.jitter) + ddf
         grid = grid.permute([0] + list(range(2, 2 + spatial_dims)) + [1])  # (batch, ..., spatial_dims)
 
-        if not USE_COMPILED:  # pytorch native grid_sample
+        _use_compiled = USE_COMPILED and not _compiled_unsupported(image.device)
+
+        if not _use_compiled:  # pytorch native grid_sample
             for i, dim in enumerate(grid.shape[1:-1]):
                 grid[..., i] = grid[..., i] * 2 / (dim - 1) - 1
             index_ordering: list[int] = list(range(spatial_dims - 1, -1, -1))
             grid = grid[..., index_ordering]  # z, y, x -> x, y, z
             return F.grid_sample(
-                image, grid, mode=self._interp_mode, padding_mode=f"{self._padding_mode}", align_corners=True
+                image, grid, mode=self._interp_mode_native, padding_mode=self._padding_mode_native, align_corners=True
             )
 
         # using csrc resampling

monai/transforms/regularization/array.py

@@ -24,7 +24,6 @@
 
 
 class Mixer(RandomizableTransform):
-
     def __init__(self, batch_size: int, alpha: float = 1.0) -> None:
         """
         Mixer is a base class providing the basic logic for the mixup-class of

monai/transforms/spatial/array.py

@@ -34,6 +34,7 @@
 from monai.transforms.croppad.array import CenterSpatialCrop, ResizeWithPadOrCrop
 from monai.transforms.inverse import InvertibleTransform
 from monai.transforms.spatial.functional import (
+    _compiled_unsupported,
     affine_func,
     convert_box_to_points,
     convert_points_to_box,
@@ -2105,14 +2106,15 @@ def __call__(
         _align_corners = self.align_corners if align_corners is None else align_corners
         img_t, *_ = convert_data_type(img, torch.Tensor, dtype=_dtype, device=_device)
         sr = min(len(img_t.peek_pending_shape() if isinstance(img_t, MetaTensor) else img_t.shape[1:]), 3)
+        _use_compiled = USE_COMPILED and not _compiled_unsupported(img_t.device)
         backend, _interp_mode, _padding_mode, _ = resolves_modes(
             self.mode if mode is None else mode,
             self.padding_mode if padding_mode is None else padding_mode,
             backend=None,
-            use_compiled=USE_COMPILED,
+            use_compiled=_use_compiled,
         )
 
-        if USE_COMPILED or backend == TransformBackends.NUMPY:
+        if _use_compiled or backend == TransformBackends.NUMPY:
             grid_t, *_ = convert_to_dst_type(grid[:sr], img_t, dtype=grid.dtype, wrap_sequence=True)
             if isinstance(grid, torch.Tensor) and grid_t.data_ptr() == grid.data_ptr():
                 grid_t = grid_t.clone(memory_format=torch.contiguous_format)
@@ -2123,7 +2125,7 @@ def __call__(
                     grid_t[i] = ((_dim - 1) / _dim) * grid_t[i] + t if _align_corners else grid_t[i] + t
                 elif _align_corners:
                     grid_t[i] = ((_dim - 1) / _dim) * (grid_t[i] + 0.5)
-            if USE_COMPILED and backend == TransformBackends.TORCH:  # compiled is using torch backend param name
+            if _use_compiled and backend == TransformBackends.TORCH:  # compiled is using torch backend param name
                 grid_t = moveaxis(grid_t, 0, -1)  # type: ignore
                 out = grid_pull(
                     img_t.unsqueeze(0),
@@ -2141,6 +2143,20 @@ def __call__(
                     [_map_coord(c, grid_np, order=_interp_mode, mode=_padding_mode) for c in img_np]
                 )
                 out = convert_to_dst_type(out, img_t)[0]
+            else:
+                # Fallback to PyTorch grid_sample when compiled extension is unsupported.
+                # Convert grid coordinates from compiled convention [0, size-1] to PyTorch [-1, 1]
+                for i, dim in enumerate(img_t.shape[1 : 1 + sr]):
+                    _dim = max(2, dim)
+                    grid_t[i] = (grid_t[i] * 2.0 / _dim) - 1.0
+                grid_t = moveaxis(grid_t, 0, -1)  # type: ignore
+                out = torch.nn.functional.grid_sample(
+                    img_t.unsqueeze(0),
+                    grid_t.unsqueeze(0),
+                    mode=_interp_mode,
+                    padding_mode=_padding_mode,
+                    align_corners=None if _align_corners == TraceKeys.NONE else _align_corners,  # type: ignore
+                )[0]
         else:
             grid_t = moveaxis(grid[list(range(sr - 1, -1, -1))], 0, -1)  # type: ignore
             grid_t = convert_to_dst_type(grid_t, img_t, wrap_sequence=True)[0].unsqueeze(0)