Add TIM::Runtime to initialize/finalize MPI and AMReX#23
Open
alperaltuntas wants to merge 4 commits into
Open
Conversation
The new Runtime class starts MPI and then AMReX when it is created, and shuts them down in the opposite order when it goes out of scope. The caller decides who owns MPI by choosing a constructor, instead of the runtime check FMS's mpp layer relied on. Only one Runtime is allowed per process.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces TIM::Runtime, a single-owner RAII runtime manager that centralizes MPI and AMReX initialization/finalization inside TIM, enabling downstream consumers (e.g., protoMOMxx) to rely on TIM for consistent startup/shutdown sequencing.
Changes:
- Added
TIM::Runtimewith explicit owner/guest MPI modes and ordered teardown (AMReX then MPI). - Integrated the new runtime into the TIM C++ library build/install.
- Added unit tests validating MPI collectives and AMReX
MultiFabreductions in both owner and guest modes, and updated test CMake wiring.
Reviewed changes
Copilot reviewed 6 out of 7 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
tim/cpp/core/tim_runtime.hpp |
Declares TIM::Runtime API, owner/guest constructors, communicator accessor, and one-per-process constraints. |
tim/cpp/core/tim_runtime.cpp |
Implements MPI/AMReX bring-up/tear-down and the single-runtime guard. |
tim/cpp/CMakeLists.txt |
Adds the runtime source to tim and installs the new header. |
test_tim/runtime/test_runtime_owner.cpp |
Adds owner-mode runtime test (Runtime owns MPI; AMReX runs on the owned comm). |
test_tim/runtime/test_runtime_guest.cpp |
Adds guest-mode runtime test (caller owns MPI; Runtime must not finalize MPI). |
test_tim/CMakeLists.txt |
Registers the new runtime test executables and CUDA language handling for them. |
.gitignore |
Un-ignores tim/cpp/core/ so the new core/ subtree is tracked despite a global core ignore pattern. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Add the TIM::abort function that aborts the run with a given message. Unlike amrex::abort, TIM::abort is safe in any state of MPI and AMReX.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
As a first step toward adding TIM in protoMOMxx as a submodule, this PR moves the direct
AMReX::initialize/finalizecalls from protoMOMxx to TIM.To do so, we add
TIM::Runtime, an RAII encapsulation of the infrastructure runtime: constructing it brings up MPI and then AMReX (as a guest on the resulting communicator), and destroying it tears them down in reverse order. This gives the TURBO stack one owned place for initialization and shutdown of dependencies like MPI, AMReX (and, in the future, PIO).Who owns MPI is an explicit caller decision made by choosing one of the two constructors of
TIM::Runtime. This replaces theMPI_INITIALIZEDruntime check FMS's mpp layer relied on.Runtime(int& argc, char**& argv): callsMPI_Initand, at destruction,MPI_Finalize(solo drivers). Aborts if MPI is already up.explicit Runtime(MPI_Comm): adopts a communicator from a caller that owns MPI (a coupler/cap) and never finalizes it. Aborts if MPI is not up.Either way AMReX runs as a guest on that communicator (
amrex::Initialize(comm)) — it never ownsargc/argvor MPI. Exactly oneRuntimemay exist per process (AMReX's runtime state is global).The following draft protoMOMxx PR is a preview of protoMOMxx will utilize
TIM::Runtime:. Simply,MOM::Infrabecomes a thin wrapper overTIM::Runtime(owner mode in solo_driver, guest mode in future NUOPC cap).TURBO-ESM/protoMOMxx#18
Changes:
tim/cpp/core/tim_runtime.{hpp,cpp}: theRuntimeclass: owner/guest constructors,comm()accessor, non-copyable/non-movable, one-per-process guard, ordered teardown (amrex::FinalizebeforeMPI_Finalize).test_tim/runtime/test_runtime_owner.cpp: with an owner-mode Runtime alive, direct MPI collectives oncomm()and AMReX MultiFab collective reductions work side by side.test_tim/runtime/test_runtime_guest.cpp:main()owns MPI like a coupler, handsMPI_COMM_WORLDto a guest-mode Runtime, runs the same coexistence checks, then verifies after Runtime's destruction that the guest did not finalize MPI and the caller's MPI is still usable.test_tim/CMakeLists.txt: registers both runtime test binaries (labels "unit"); the CUDA source-property handling is factored to cover them.