Fix PDLP capture cublas error and hang on infeasible solutions

[akifcorduk]

Summary

Fixes two concurrency bugs in the MIP concurrent root-LP path that could crash or
hang the solver: (1) an intermittent CUBLAS_STATUS_INTERNAL_ERROR during PDLP
graph capture, and (2) a deadlock when the concurrent root LP fails to produce a
usable solution (e.g. on an infeasible model). Both are exercised by the same
flow — the heuristics' concurrent root LP (PDLP + cuDSS barrier) running alongside
B&B — and both reproduce independently of each other.

Problem

1. cuBLAS error during graph capture

PDLP captures its adaptive step into a CUDA graph (manual_cuda_graph_t,
cudaStreamCaptureModeThreadLocal) while the cuDSS barrier runs concurrently on a
separate handle. cuDSS's device-synchronizing, library-global work (handle
create/first factorization/destroy, allocations) invalidates PDLP's in-flight
capture. When that invalidation is observed inside a cuBLAS call (cuBLAS can't
return a CUDA error code), it surfaces as CUBLAS_STATUS_INTERNAL_ERROR thrown
from cublasdot in adaptive_step_size_strategy.cu.

manual_cuda_graph_t::run already recovered from this collision, but only when
cudaStreamEndCapture returned cudaErrorStreamCaptureInvalidated — not when a
library call threw mid-capture before EndCapture was reached. So that surface
escaped and aborted the solve.

2. Deadlock when the root LP fails

On an infeasible model the barrier's cuDSS factorization fails (NumericalError,
empty primal/dual). The heuristic then unconditionally raft::copy'd n elements
from the empty result, which threw. That exception unwound out of
dm.run_solver() and got parked at the MIP omp taskgroup join (it must wait for
child tasks before propagating). Meanwhile B&B's solve_root_relaxation spins in a
wait loop whose only exits are root_crossover_solution_set_ / root_concurrent_halt
— neither of which the (now-thrown) heuristic ever signals. Result: a hang that
ignores the time limit, with no Python traceback (exception stuck mid-unwind).

Solution

1. manual_cuda_graph.cuh — recover from thrown errors mid-capture

Wrap work() in try/catch. On a throw, end the capture and let its status
disambiguate: if the capture was invalidated, the recorded work was never issued —
drain the sticky error and re-run work eagerly (no capture, so the concurrent op
can't break it). Otherwise the error is genuine and is rethrown. This routes the
thrown-CUBLAS_STATUS_INTERNAL_ERROR surface into the same recovery the clean
cudaErrorStreamCaptureInvalidated path already used, without keying off cuBLAS's
ambiguous status code, and without masking real failures.

2. diversity_manager.cu — handle an unusable root LP and release B&B

After the concurrent root LP returns, check whether the result is usable
(status != NumericalError and primal/dual sizes match). When it isn't, skip the
copy/hand-off that previously threw and release B&B's root-relaxation wait via
branch_and_bound_ptr->set_root_concurrent_halt(1), so B&B falls back to its own
dual-simplex root instead of deadlocking. When the result is usable, behavior is
unchanged (copy + hand off via the root-relaxation callback).

Testing

• Reproduced the original crash signature and confirmed the hang is pre-existing on
  main (identical with/without the capture fix).
• Stress loop on the infeasible MILP: 62 iterations / 124 concurrent root-LP
  solves, 0 crashes, 0 hangs; every solve now terminates cleanly as
  Infeasible (B&B reports Root relaxation returned: INFEASIBLE).
• Tested on user provided infeasible mps file from issue [BUG] std::terminate (process abort) on an infeasible MILP in opportunistic (default) mode #1396

closes #1396

[nguidotti]

Thanks for the fix, Akif! I suggest to also add the failing example as a test case.

[coderabbitai]

📝 Walkthrough

Walkthrough

The PR hardens concurrent root LP solving in opportunistic MIP mode and improves CUDA graph robustness. Root LP results are now validated for usability (non-NumericalError status and matching solution sizes), with unusable results signaling B&B to halt its wait-release loop instead of causing deadlock or crash. Root-relaxation handoff and variable clamping are guarded by this usability flag. CUDA graph capture exception handling distinguishes and recovers from capture invalidation separately from other failures.

Changes

Concurrent Root LP Safety and CUDA Graph Robustness

Layer / File(s)	Summary
Root LP usability detection and validation cpp/src/mip_heuristics/diversity/diversity_manager.cu	Detects whether the concurrent root LP result is usable by checking PDLP termination status (rejecting NumericalError) and verifying that primal/dual solution sizes match expected LP buffer dimensions.
Unusable and usable root LP path handling with B&B signaling cpp/src/mip_heuristics/diversity/diversity_manager.cu	When root LP is unusable, the code avoids handing off invalid data, logs debug info, clears lp_optimal_exists, and signals B&B's root solve loop to halt via set_root_concurrent_halt(1) to prevent deadlock. When usable, performs standard termination-status-driven feasibility and bounds updates.
Root-relaxation handoff condition and bounds clamping guards cpp/src/mip_heuristics/diversity/diversity_manager.cu	Tightens the B&B root-relaxation handoff to require both root_lp_usable flag and callback presence. Restricts subsequent variable bounds clamping to occur only when root LP is usable and staged simplex is not used.
CUDA graph capture exception recovery with invalidation detection cpp/src/utilities/manual_cuda_graph.cuh	Expands documentation describing capture-hostile CUDA operations and invalidation detection mechanisms. Wraps work() execution in try/catch; on exception, explicitly ends capture and recovers specifically when capture was invalidated (draining sticky error and re-running work() eagerly), or destroys partially captured graph and rethrows on other failures.

---

🎯 4 (Complex) | ⏱️ ~45 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely summarizes the main fixes: PDLP capture cuBLAS error and infeasibility-induced hangs.
Linked Issues check	✅ Passed	Changes fully address linked issues #1021 and #1396 by preventing false infeasibility reports, eliminating hangs/deadlocks, and ensuring proper B&B signal handling on unusable root LP results.
Out of Scope Changes check	✅ Passed	All changes are strictly scoped to fixing the two identified concurrency bugs in the MIP root-LP path; no extraneous modifications detected.
Description check	✅ Passed	The pull request description thoroughly explains the two concurrency bugs being fixed, the root causes, the solutions implemented in each file, and testing validation.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[akifcorduk]

/merge