Repository navigation
Description
What happens
On a CUDA build for an NVIDIA A40 (CUDA 12.8.1, AMPERE86, Kokkos 5.2.1), six tests in the GPU lane crash their process with a segmentation fault in the native session's advance on the radial route. Every stepper is affected, not only Adams:
tests/backends/test_monitor_reduction.py::test_a_device_s_split_removal_is_the_reference_s[radial-low_storage],[radial-multistep],[radial-runge_kutta]tests/sessions/test_native_session_device.py::test_a_spatial_route_repeats_and_meets_the_reference_on_the_device[radial]tests/sessions/test_native_session_device.py::test_a_collar_s_current_and_edge_meet_the_reference_on_the_device[adams]and[rk]
Fatal Python error: Segmentation fault
File ".../solprop_native/session.py", line 873 in advance
File "tests/monitor_cases.py", line 217 in record_after_march
File "tests/backends/test_monitor_reduction.py", line 161 in _check_split
A plain poe test-gpu run stops at the first crash (exit 139), at about 47 % of the lane. With -n 1 --max-worker-restart=20 -p no:randomly the lane finishes with 289 passed, 6 failed (these six) and 1 skipped. That is identical on main at a3ac8c44 and on #814's branch.
Reproduction
A CUDA build (-DSOLPROP_NATIVE_ENABLE_CUDA=ON -DSOLPROP_NATIVE_CUDA_ARCH=AMPERE86 -DSOLPROP_NATIVE_ENABLE_OPENMP=OFF -DSOLPROP_NATIVE_ENABLE_IPO=OFF), then:
python -m pytest -m gpu -p no:randomly tests/backends/test_monitor_reduction.pyIt crashes on [radial-low_storage]. The C stack had no symbols; a debug build is needed to locate it.
Why it matters
The radial route on the device crashes rather than refusing or running, and the crash takes down the whole GPU lane, so later GPU tests in the run go unexercised. Not checked on HIP.
Done when
The six tests pass, or refuse by name, on a CUDA build, and poe test-gpu completes without a crash.
Activity