Index-based mlebabeclap_gsrb kernel

[ankithadas]

Summary

Rewrites mlebabeclap_gsrb (2D and 3D) from a Box-based kernel that loops internally
into a per-cell (i,j,k,n) kernel, and calls it from MLEBABecLap::Fsmooth through
AMREX_HOST_DEVICE_PARALLEL_FOR_4D instead of AMREX_LAUNCH_HOST_DEVICE_LAMBDA.

In 3D this also removes the hand-written quadruple for loop and the
// amrex::Loop here causes gcc 8 to crash. workaround that went with it, since there is
no longer a loop inside the kernel at all.

The arithmetic is untouched: only the loop wrapper is removed and the body re-indented.
The kernel already took its EB geometry as an EBData, so no argument rework was needed
beyond dropping Box const& box / int ncomp in favour of (i,j,k,n). Red/black
Gauss-Seidel only writes cells of one parity and only reads neighbours of the other, so
the switch from a sequential amrex::Loop to a concurrent ParallelFor introduces no
ordering dependence. Results are bitwise unchanged.

Additional background

This is part of the ongoing split of the stale WIP #4922 ("GPU specific kernels for
MLEBABecLap"), as requested there; it supersedes the mlebabeclap_gsrb part of #4922.
The templating on T that #4922 also introduced has been dropped, per the review comment
on that PR, and #4922's change of dhx from m_b_scalar/(h[0]*h[0]) to
m_b_scalar*dxinv[0]*dxinv[0] has deliberately been left out so that the answers stay
bit-for-bit the same.

This PR is independent of the mlebabeclap_adotx / mlebabeclap_adotx_centroid PRs
(different kernels, different function in AMReX_MLEBABecLap_F.cpp) and can be merged in
any order relative to them. The (i,j,k,n) form is the prerequisite for a later PR that
adds a fused (MultiFab-wide ParallelFor) GPU path for Fsmooth; that follow-up is not
included here.

Testing (all commands run from the repo root unless noted):

• cmake -S . -B build-split -DAMReX_SPACEDIM=3 -DAMReX_EB=ON -DAMReX_LINEAR_SOLVERS_EM=OFF -DAMReX_ENABLE_TESTS=ON -DAMReX_TEST_TYPE=Small -DAMReX_MPI=ON -DCMAKE_BUILD_TYPE=Release
  then cmake --build build-split -j8 and ctest --test-dir build-split --output-on-failure
  → builds clean, 9/9 tests pass. A 2D library build
  (-DAMReX_SPACEDIM=2 -DAMReX_EB=ON) also builds clean.

• Tests/LinearSolvers/CellEB, make -j8 COMP=llvm USE_MPI=FALSE DIM=3 and DIM=2,
  run over seven configurations (sphere, sphere + eb_is_dirichlet=1, rotated_box,
  two_spheres, flower, two-level sphere, periodic sphere; n_cell=64,
  verbose=2). This test drives the V-cycle, so Fsmooth is called on every level.
  The MLMG/BiCGStab residual histories and the initial/final max, 1- and 2-norm residuals
  are bitwise identical to development in both 2D and 3D.

• CUDA build on an NVIDIA RTX A5000 (CUDA 13.2, gcc 11.4):
  Tests/LinearSolvers/CellEB, make -j8 COMP=gnu USE_MPI=FALSE USE_CUDA=TRUE CUDA_ARCH=86 DIM=3,
  same seven configurations. MLMG iteration counts are identical to development
  (9, 12, 11, 3, 11, 28, 11); the residual values differ only at the level of the
  run-to-run nondeterminism of the unmodified binary (GPU reductions are not
  bit-reproducible), and all solves converge to resid/resid0 ~ 1e-13.

• The same CellEB matrix was re-run on Linux/gcc 11.4 (make -j8 COMP=gnu USE_MPI=FALSE,
  DIM=3 and DIM=2) against a development build in a sibling worktree: residual
  histories again bitwise identical in both dimensions.

Performance

Neither timing changed measurably; this PR is a refactor, not an optimisation.

CPU, Tests/LinearSolvers/CellEB main3d.gnu.TEST.ex
(make -j8 COMP=gnu USE_MPI=FALSE DIM=3, gcc 11.4, single rank),
inputs n_cell=128 eb2.geom_type=sphere eb_is_dirichlet=1 verbose=1.
All binaries were built first and then timed interleaved in the same session
(3 reps each) on a shared machine, so the absolute numbers are inflated but the
comparison is fair. Best of 3, MLMG Timers: Solve [s]:

grids	development	this PR
max_grid_size=32	4.284	4.280
max_grid_size=64	4.534	4.597

GPU (NVIDIA RTX A5000, CUDA 13.2,
make -j8 COMP=gnu USE_MPI=FALSE USE_CUDA=TRUE CUDA_ARCH=86 DIM=3), same test at
n_cell=256, again built up front and timed interleaved, best of 3:

grids	development	this PR
max_grid_size=32 (512 boxes)	1.085	1.062
max_grid_size=64 (64 boxes)	0.834	0.806

MLMG converged in the same number of iterations in every one of these runs.

Also worth noting for this one: the CPU path changes from a sequential amrex::Loop inside the kernel to AMREX_HOST_DEVICE_PARALLEL_FOR_4D, and vlo/vhi are now recomputed per cell rather than per box. Neither shows up in the timings above.

Checklist

The proposed changes:

• fix a bug or incorrect behavior in AMReX
• add new capabilities to AMReX
• changes answers in the test suite to more than roundoff level
• are likely to significantly affect the results of downstream AMReX users
• include documentation in the code and/or rst files, if appropriate

P.S Generated using Claude Code