Skip to content

Native radial route segfaults in session advance on a CUDA device #815

Open

Description

@jtravs

What happens

On a CUDA build for an NVIDIA A40 (CUDA 12.8.1, AMPERE86, Kokkos 5.2.1), six tests in the GPU lane crash their process with a segmentation fault in the native session's advance on the radial route. Every stepper is affected, not only Adams:

  • tests/backends/test_monitor_reduction.py::test_a_device_s_split_removal_is_the_reference_s[radial-low_storage], [radial-multistep], [radial-runge_kutta]
  • tests/sessions/test_native_session_device.py::test_a_spatial_route_repeats_and_meets_the_reference_on_the_device[radial]
  • tests/sessions/test_native_session_device.py::test_a_collar_s_current_and_edge_meet_the_reference_on_the_device[adams] and [rk]
Fatal Python error: Segmentation fault
  File ".../solprop_native/session.py", line 873 in advance
  File "tests/monitor_cases.py", line 217 in record_after_march
  File "tests/backends/test_monitor_reduction.py", line 161 in _check_split

A plain poe test-gpu run stops at the first crash (exit 139), at about 47 % of the lane. With -n 1 --max-worker-restart=20 -p no:randomly the lane finishes with 289 passed, 6 failed (these six) and 1 skipped. That is identical on main at a3ac8c44 and on #814's branch.

Reproduction

A CUDA build (-DSOLPROP_NATIVE_ENABLE_CUDA=ON -DSOLPROP_NATIVE_CUDA_ARCH=AMPERE86 -DSOLPROP_NATIVE_ENABLE_OPENMP=OFF -DSOLPROP_NATIVE_ENABLE_IPO=OFF), then:

python -m pytest -m gpu -p no:randomly tests/backends/test_monitor_reduction.py

It crashes on [radial-low_storage]. The C stack had no symbols; a debug build is needed to locate it.

Why it matters

The radial route on the device crashes rather than refusing or running, and the crash takes down the whole GPU lane, so later GPU tests in the run go unexercised. Not checked on HIP.

Done when

The six tests pass, or refuse by name, on a CUDA build, and poe test-gpu completes without a crash.

Activity

  1. added
    bugSomething isn't working
    priority: mediumReal work that should not wait long
    area: nativeThe native backend, threading and FFT planning
    area: gpuCUDA and HIP builds: qualification and performance
    area: free spaceRadial, Cartesian, line and planar free-space routes
    on Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: free spaceRadial, Cartesian, line and planar free-space routesarea: gpuCUDA and HIP builds: qualification and performancearea: nativeThe native backend, threading and FFT planningbugSomething isn't workingpriority: mediumReal work that should not wait long

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions