Index-based mlebabeclap_adotx_centroid kernel - #5621
Open
ankithadas wants to merge 2 commits into
Open
ankithadas wants to merge 2 commits into
ankithadas wants to merge 2 commits into
Conversation
ankithadas
force-pushed
the
MLEBABecLap-AdotxCentroid-Index
branch
from
August 19, 2026 04:20
0ce7913 to
e28c136
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Rewrites
mlebabeclap_adotx_centroid(2D and 3D) from aBox-based kernel that loopsinternally (
amrex::Loop(box, ncomp, ...)) into a per-cell(i,j,k,n)kernel, andreplaces its eleven separate EB
Array4arguments (flag,vfrc,apx/apy/apz,fcx/fcy/fcz,ccent,ba,bcent) with the existingEBDataview.MLEBABecLap::Fapplynow calls it throughAMREX_HOST_DEVICE_PARALLEL_FOR_4Dand getsall EB geometry from
factory->getEBData(mfi), which lets thegetVolFrac()/getAreaFrac()/getFaceCent()/getBndryArea()/getBndryCent()/getCentroid()lookups at the top of
Fapplybe dropped entirely.The arithmetic inside the kernel is untouched: the loop wrapper is removed and the body is
only re-indented, and the EB arrays are re-bound to identically named local references
(
auto const& ccent = ebdata.get<EBData_t::centroid>();etc.), so the expressions arecharacter for character the same. Results are bitwise unchanged.
Additional background
This is part of the ongoing split of the stale WIP #4922 ("GPU specific kernels for
MLEBABecLap"), as requested there; it supersedes the
mlebabeclap_adotx_centroidpart of#4922. The templating on
Tthat #4922 also introduced has been dropped, per the reviewcomment on that PR.
This PR is stacked on top of the
mlebabeclap_adotxPR (branchankithadas:MLEBABecLap-Adotx-Index) because both edit the same block ofMLEBABecLap::Fapply; please merge that one first. The diff shown here againstdevelopmenttherefore contains that commit as well.Testing (all commands run from the repo root unless noted):
cmake -S . -B build-split -DAMReX_SPACEDIM=3 -DAMReX_EB=ON -DAMReX_LINEAR_SOLVERS_EM=OFF -DAMReX_ENABLE_TESTS=ON -DAMReX_TEST_TYPE=Small -DAMReX_MPI=ON -DCMAKE_BUILD_TYPE=Releasethen
cmake --build build-split -j8andctest --test-dir build-split --output-on-failure→ builds clean, 9/9 tests pass. A 2D library build
(
-DAMReX_SPACEDIM=2 -DAMReX_EB=ON) also builds clean.Tests/LinearSolvers/LeastSquaresis the test that actually exercises this kernel: itcalls
MLEBABecLap::setPhiOnCentroid()and thenMLMG::apply(). Built withmake -j8 COMP=llvm USE_MPI=FALSE DIM=2andDIM=3, and run oninputs.2d.base,inputs.2d.askew-x,inputs.2d.fullyrotated,inputs.2d.trianglewave,inputs.3d.poiseuille.askew-allandinputs.3d.poiseuille.aligned.xy-x. The resulting plotfiles are byte for byteidentical to those produced by
development(diff -rover all six).Tests/LinearSolvers/CellEB,make -j8 COMP=llvm USE_MPI=FALSE DIM=3andDIM=2,seven configurations (
sphere,sphere+eb_is_dirichlet=1,rotated_box,two_spheres,flower, two-levelsphere, periodicsphere;n_cell=64,verbose=2): MLMG/BiCGStab residual histories and final residual norms are bitwiseidentical to
development.CUDA build on an NVIDIA RTX A5000 (CUDA 13.2, gcc 11.4):
Tests/LinearSolvers/CellEB,make -j8 COMP=gnu USE_MPI=FALSE USE_CUDA=TRUE CUDA_ARCH=86 DIM=3.MLMG iteration counts are identical to
development(9, 12, 11, 3, 11, 28, 11); theresidual values differ only at the level of the run-to-run nondeterminism of the
unmodified binary (GPU reductions are not bit-reproducible).
The same CellEB matrix was re-run on Linux/gcc 11.4 (
make -j8 COMP=gnu USE_MPI=FALSE,DIM=3andDIM=2) against adevelopmentbuild in a sibling worktree: residualhistories again bitwise identical in both dimensions.
Performance
Neither timing changed measurably; this PR is a refactor, not an optimisation.
CPU,
Tests/LinearSolvers/CellEBmain3d.gnu.TEST.ex(
make -j8 COMP=gnu USE_MPI=FALSE DIM=3, gcc 11.4, single rank),inputs n_cell=128 eb2.geom_type=sphere eb_is_dirichlet=1 verbose=1.All binaries were built first and then timed interleaved in the same session
(3 reps each) on a shared machine, so the absolute numbers are inflated but the
comparison is fair. Best of 3, MLMG
Timers: Solve[s]:developmentmax_grid_size=32max_grid_size=64GPU (NVIDIA RTX A5000, CUDA 13.2,
make -j8 COMP=gnu USE_MPI=FALSE USE_CUDA=TRUE CUDA_ARCH=86 DIM=3), same test atn_cell=256, again built up front and timed interleaved, best of 3:developmentmax_grid_size=32(512 boxes)max_grid_size=64(64 boxes)MLMG converged in the same number of iterations in every one of these runs.
Checklist
The proposed changes:
P.S Generated using Claude Code