[BUG] FAILED: CUDSS call ended unsuccessfully with status = 3, details: "cudssSetThreadingLayer"

[j3soon]

Describe the bug

Nondeterministic behavior where either return optimal:

 | Explored | Unexplored |    Objective    |     Bound     | IntInf | Depth | Iter/Node | Gap    |  Time  |
Reduced costs: Found 1 improved bounds and 1 fixed variables
           0            0    -0.000000e+00    -0.000000e+00      170      0   1.1e+03      0.00%      0.53
Explored 0 nodes in 0.53s.
Absolute Gap 0.000000e+00 Objective -0.0000000000000000e+00 Upper Bound -0.0000000000000000e+00
Optimal solution found.
Solution objective: -0.000000 , relative_mip_gap 0.000000 solution_bound -0.000000 presolve_time 0.440575 total_solve_time 0.540780 max constraint violation 0.000000 max int violation 0.000000 max var bounds violation 0.000000 nodes 0 simplex_iterations 1072
CUOPT status= Optimal
Optimal

or with following segfault:

 |  Explored  | Unexplored |     Objective     |     Bound     | IntInf | Depth | Iter/Node |    Gap    |  Time  |
FAILED: CUDSS call ended unsuccessfully with status = 3, details: "cudssSetThreadingLayer"
Fatal Python error: Segmentation fault

Current thread 0x00007f20fd48c740 [python] (most recent call first):
  File "/usr/local/lib/python3.14/dist-packages/cuopt/linear_programming/solver/solver.py", line 102 in Solve
  File "/usr/local/lib/python3.14/dist-packages/cuopt/utilities/exception_handler.py", line 24 in func
  File "/app/repro.py", line 24 in <module>

Current thread's C stack trace (most recent call first):
Segmentation fault (core dumped)

or

Dual Simplex Phase 2
 Iter     Objective           Num Inf.  Sum Inf.     Perturb  Time
    0 -0.0000000000000000e+00     115 2.45300000e+03 0.00e+00 0.47
    1 -0.0000000000000000e+00     121 2.32400000e+03 0.00e+00 0.47
Removed perturbation of 8.17e-05.
FAILED: CUDSS call ended unsuccessfully with status = 3, details: "cudssSetThreadingLayer"
Fatal Python error: Segmentation fault

Thread 0x0000762f929c8740 [python] (most recent call first):
  File "/usr/local/lib/python3.14/dist-packages/cuopt/linear_programming/solver/solver.py", line 102 in Solve
  File "/usr/local/lib/python3.14/dist-packages/cuopt/utilities/exception_handler.py", line 24 in func
  File "/app/repro.py", line 24 in <module>

Current thread's C stack trace (most recent call first):
  Binary file "python", at _Py_DumpStack+0x4a [0x539cd6]
  Binary file "python" [0x54571e]
  Binary file "/usr/lib/x86_64-linux-gnu/libc.so.6", at +0x42520 [0x762f92a0f520]
  Binary file "/usr/local/lib/python3.14/dist-packages/rapids_logger/lib64/librapids_logger.so", at +0x16ec3 [0x762f91e7aec3]
  Binary file "/usr/local/lib/python3.14/dist-packages/rapids_logger/lib64/librapids_logger.so", at _ZN13rapids_logger6logger3logENS_10level_enumERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0xbe [0x762f91e744ee]
  Binary file "/usr/local/lib/python3.14/dist-packages/libcuopt/lib64/libcuopt.so", at _ZN5cuopt25mathematical_optimization7simplex8logger_t6printfEPKcz+0x1de [0x762f21e546be]
  Binary file "/usr/local/lib/python3.14/dist-packages/libcuopt/lib64/libcuopt.so", at _ZN5cuopt25mathematical_optimization3mip18branch_and_bound_tIidE5solveERNS0_7simplex14mip_solution_tIidEE+0x7f9 [0x762f2243c3b9]
  Binary file "/usr/local/lib/python3.14/dist-packages/libcuopt/lib64/libcuopt.so", at +0xf4a3d4 [0x762f21e4a3d4]
  Binary file "/usr/local/lib/python3.14/dist-packages/libcuopt/lib64/../../libcuopt_cu13.libs/libomp-8fe85495.so", at +0x66ba4 [0x762f8d330ba4]
  Binary file "/usr/local/lib/python3.14/dist-packages/libcuopt/lib64/../../libcuopt_cu13.libs/libomp-8fe85495.so", at +0x6a996 [0x762f8d334996]
  Binary file "/usr/local/lib/python3.14/dist-packages/libcuopt/lib64/../../libcuopt_cu13.libs/libomp-8fe85495.so", at +0x7c265 [0x762f8d346265]
  Binary file "/usr/local/lib/python3.14/dist-packages/libcuopt/lib64/../../libcuopt_cu13.libs/libomp-8fe85495.so", at +0x76874 [0x762f8d340874]
  Binary file "/usr/local/lib/python3.14/dist-packages/libcuopt/lib64/../../libcuopt_cu13.libs/libomp-8fe85495.so", at +0x7ae97 [0x762f8d344e97]
  Binary file "/usr/local/lib/python3.14/dist-packages/libcuopt/lib64/../../libcuopt_cu13.libs/libomp-8fe85495.so", at +0x47490 [0x762f8d311490]
  Binary file "/usr/local/lib/python3.14/dist-packages/libcuopt/lib64/../../libcuopt_cu13.libs/libomp-8fe85495.so", at +0xb826e [0x762f8d38226e]
  Binary file "/usr/lib/x86_64-linux-gnu/libc.so.6", at +0x94a83 [0x762f92a61a83]
  Binary file "/usr/lib/x86_64-linux-gnu/libc.so.6", at clone+0x44 [0x762f92af2a44]

Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, cuopt.linear_programming.internals.internals, cupy_backends.cuda._softlink, cupy_backends.cuda.api._runtime_enum, cupy_backends.cuda.api.runtime, cupy._util, cupy.cuda.device, cupy.cuda.memory_hook, cupy_backends.cuda.stream, cupy.cuda.graph, cupy.cuda.stream, cupy_backends.cuda.api._driver_enum, cupy_backends.cuda.api.driver, cupy.cuda.memory, cupy._core.internal, cupy._core._carray, cupy._core._dtype, cupy._core._scalar, cupy.cuda.texture, cupy.cuda.function, cupy_backends.cuda.libs.nvrtc, cupy.cuda.pinned_memory, cupy.cuda.common, cupy.cuda.cub, cupy_backends.cuda.libs.nvtx, cupy.cuda.thrust, cupy._core._accelerator, cupy._core._memory_range, cupy._core._fusion_thread_local, cupy._core._kernel, cupy._core._routines_manipulation, cupy._core._routines_binary, cupy._core._optimize_config, cupy._core._cub_reduction, cupy._core._reduction, cupy._core._routines_math, cupy._core._routines_indexing, cupy._core._routines_linalg, cupy._core._routines_logic, cupy._core._routines_sorting, cupy._core._routines_statistics, cupy._core.dlpack, cupy._core.flags, cupy._core.core, cupy._core._fusion_variable, cupy._core._fusion_trace, cupy._core._fusion_kernel, cupy._core.new_fusion, cupy._core.fusion, cupy._core.raw, cupy.fft._cache, cupy.fft._callback, cupy.random._generator_api, cupy.random._bit_generator, cupy.lib._polynomial, _cyutility, scipy._cyutility, scipy._lib._ccallback_c, yaml._yaml, numba.core.typeconv._typeconv, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._pcg64, numpy.random._generator, numpy.random._mt19937, numpy.random._philox, numpy.random._sfc64, numpy.random.mtrand, numba._helperlib, numba._dynfunc, numba._dispatcher, numba.core.runtime._nrt_python, numba.np.ufunc._internal, numba.experimental.jitclass._box, numba.cuda.types.abstract.itertools, numba.cuda.types.functions.itertools, numba.cuda.cext._typeconv, cuda.bindings._bindings.cydriver, cuda.bindings.cydriver, cuda.bindings.driver, cuda.bindings._bindings.cyruntime_ptds, cuda.bindings._bindings.cyruntime, cuda.bindings.cyruntime, cuda.bindings.runtime, cuda.bindings._internal.utils, cuda.bindings._internal.nvrtc, cuda.bindings.cynvrtc, cuda.bindings._internal.nvvm, cuda.bindings.cynvvm, cuda.bindings._internal.nvjitlink, cuda.bindings.cynvjitlink, cuda.bindings.nvrtc, cuda.bindings.nvvm, cuda.bindings.nvjitlink, cuda.core._utils.cuda_utils, cuda.core._utils.version, cuda.core._resource_handles, cuda.core._device_resources, cuda.core._event, cuda.core._stream, cuda.core._context, cuda.bindings._internal.nvml, cuda.bindings.cynvml, cuda.bindings.nvml, cuda.core.system._nvml_context, cuda.core.system._system, cuda.core.system._device, cuda.core.system._system_events, cuda.core.cu13.system._system, cuda.core.cu13.system._device, cuda.core.cu13.system._system_events, cuda.core._memory._ipc, cuda.core._memory._memory_pool, cuda.core._memory._peer_access_utils, cuda.core._memory._device_memory_resource, cuda.core._memory._pinned_memory_resource, cuda.core._dlpack, cuda.core._device, cuda.core._memory._buffer, cuda.core._memory._graph_memory_resource, cuda.core._memory._managed_memory_ops, cuda.core._memory._managed_memory_resource, cuda.core._layout, cuda.core._memoryview, cuda.core._launch_config, cuda.core._linker, cuda.core._program, cuda.core._module, cuda.core._graphics, cuda.core._tensor_map, cuda.core.graph._host_callback, cuda.core.graph._subclasses, cuda.core.graph._adjacency_set_proxy, cuda.core.graph._graph_node, cuda.core.graph._graph_definition, cuda.core.graph._graph_builder, cuda.core._kernel_arg_handler, cuda.core._launcher, cuda.core.texture._array, cuda.core.texture._mipmapped_array, cuda.core.texture._texture, cuda.core.texture._surface, numba.cuda.cext._helperlib, numba.cuda.cext.mviewbuf, numba.cuda.cext._dispatcher, rmm.pylibrmm.stream, rmm.pylibrmm.utils, rmm.pylibrmm.helper, rmm.pylibrmm.memory_resource._memory_resource, rmm.pylibrmm.memory_resource.experimental, rmm.pylibrmm.cuda_stream_pool, rmm.pylibrmm.device_buffer, rmm.librmm._logger, rmm.pylibrmm.logger, pyarrow.lib, pandas._libs._cyutility, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.properties, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pyarrow._compute, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, nvtx._lib.lib, nvtx._lib.profiler, pylibcudf.libcudf.types, pylibcudf.types, pylibcudf.libcudf.aggregation, pylibcudf.aggregation, pylibcudf.gpumemoryview, pylibcudf.utils, pylibcudf._interop_helpers, pylibcudf.table, pylibcudf.filling, pylibcudf.traits, pylibcudf.column, pylibcudf.scalar, pylibcudf.libcudf.binaryop, pylibcudf.binaryop, pylibcudf.column_factories, pylibcudf.concatenate, pylibcudf.context, pylibcudf.contiguous_split, pylibcudf.libcudf.copying, pylibcudf.copying, pylibcudf.libcudf.datetime, pylibcudf.datetime, pylibcudf.experimental._join_streams, pylibcudf.libcudf.expressions, pylibcudf.expressions, pylibcudf.groupby, pylibcudf.hashing, pylibcudf.interop, pylibcudf.io.datasource, pylibcudf.libcudf.io.json, pylibcudf.libcudf.io.types, pylibcudf.io.types, pylibcudf.io.avro, pylibcudf.io.csv, pylibcudf.io.parquet_metadata, pylibcudf.io.parquet, pylibcudf.io.text, pylibcudf.libcudf.io.hybrid_scan, pylibcudf.io.experimental.hybrid_scan, pylibcudf.io.json, pylibcudf.io.orc, pylibcudf.io.timezone, pylibcudf.join, pylibcudf.json, pylibcudf.libcudf.labeling, pylibcudf.labeling, pylibcudf.libcudf.lists.combine, pylibcudf.libcudf.lists.contains, pylibcudf.lists, pylibcudf.merge, pylibcudf.null_mask, pylibcudf.nvtext.byte_pair_encode, pylibcudf.nvtext.deduplicate, pylibcudf.nvtext.edit_distance, pylibcudf.nvtext.generate_ngrams, pylibcudf.nvtext.jaccard, pylibcudf.nvtext.minhash, pylibcudf.nvtext.ngrams_tokenize, pylibcudf.nvtext.normalize, pylibcudf.nvtext.replace, pylibcudf.libcudf.nvtext.stemmer, pylibcudf.nvtext.stemmer, pylibcudf.nvtext.tokenize, pylibcudf.nvtext.wordpiece_tokenize, pylibcudf.libcudf.partitioning, pylibcudf.partitioning, pylibcudf.prefetch, pylibcudf.quantiles, pylibcudf.libcudf.reduce, pylibcudf.reduce, pylibcudf.libcudf.replace, pylibcudf.replace, pylibcudf.reshape, pylibcudf.rolling, pylibcudf.libcudf.round, pylibcudf.round, pylibcudf.search, pylibcudf.sorting, pylibcudf.libcudf.stream_compaction, pylibcudf.stream_compaction, pylibcudf.strings.attributes, pylibcudf.libcudf.strings.char_types, pylibcudf.strings.capitalize, pylibcudf.strings.case, pylibcudf.strings.char_types, pylibcudf.libcudf.strings.combine, pylibcudf.strings.combine, pylibcudf.strings.regex_program, pylibcudf.strings.contains, pylibcudf.strings.convert.convert_booleans, pylibcudf.strings.convert.convert_datetime, pylibcudf.strings.convert.convert_durations, pylibcudf.strings.convert.convert_fixed_point, pylibcudf.strings.convert.convert_floats, pylibcudf.strings.convert.convert_integers, pylibcudf.strings.convert.convert_ipv4, pylibcudf.strings.convert.convert_lists, pylibcudf.strings.convert.convert_urls, pylibcudf.strings.extract, pylibcudf.strings.find, pylibcudf.strings.find_multiple, pylibcudf.strings.findall, pylibcudf.strings.padding, pylibcudf.libcudf.strings.regex_flags, pylibcudf.strings.regex_flags, pylibcudf.strings.repeat, pylibcudf.strings.replace, pylibcudf.strings.replace_re, pylibcudf.strings.reverse, pylibcudf.libcudf.strings.side_type, pylibcudf.strings.side_type, pylibcudf.strings.slice, pylibcudf.strings.split.partition, pylibcudf.strings.split.split, pylibcudf.strings.strip, pylibcudf.libcudf.strings.translate, pylibcudf.strings.translate, pylibcudf.strings.wrap, pylibcudf.table_equality, pylibcudf.transform, pylibcudf.transpose, pylibcudf.libcudf.unary, pylibcudf.unary, pylibcudf.utilities, numba.types.itertools, cudf._lib.strings_udf, pyarrow._feather, cuopt.linear_programming.data_model.data_model_wrapper, cuopt.linear_programming.io.parser_wrapper, scipy.sparse._sparsetools, charset_normalizer.md, charset_normalizer.cd, _csparsetools, scipy.sparse._csparsetools, pylibraft.common.cuda, pylibraft.common.handle, cuopt.linear_programming.solver_settings.solver_settings, cuopt.linear_programming.solver.solver_wrapper (total: 339)
Segmentation fault (core dumped)

Steps/Code to reproduce bug

Download two files:

• repro.py
• repro.mps.txt and rename to repro.mps (I cannot attach mps file here)

Note that these are minimal repro code generated by Codex extracted from a larger codebase.

Then run:

docker run --rm -it --entrypoint bash --gpus all -v $(pwd):/app -w /app nvidia/cuopt:26.8.0a-cuda13.3-py3.14
pip install pulp==3.3.2
PYTHONFAULTHANDLER=1 python repro.py

It should either return optimal or segfault within a few seconds.

Same in nvidia/cuopt:26.6.0-cuda13.2-py3.14 but seems harder to reproduce segfault (like a couple times per 100 runs), use following to repeat and grep:

for i in $(seq 1 100); do
  PYTHONFAULTHANDLER=1 python repro.py 2>&1 |
    grep -E 'FAILED: CUDSS call ended unsuccessfully|Fatal Python error: Segmentation fault' |
    sed "s/^/[run $i] /"
done

Above should take a couple of minutes to complete.

Expected behavior

Don't segfault even when ran 100 times.

Environment details (please complete the following information):

• Environment location: Docker
• Method of cuOpt install: Docker

Additional context

Tested on RTX PRO 6000 Blackwell Max-Q Workstation Edition. On 2 difference machines, one with driver 580.173.02 another with 595.84.

[hlinsen]

Thanks a ton for reporting @j3soon. We'll look into it.