Skip to content

Add TIM::Runtime to initialize/finalize MPI and AMReX#23

Open
alperaltuntas wants to merge 4 commits into
mainfrom
tim_infra
Open

Add TIM::Runtime to initialize/finalize MPI and AMReX#23
alperaltuntas wants to merge 4 commits into
mainfrom
tim_infra

Conversation

@alperaltuntas

Copy link
Copy Markdown
Contributor

As a first step toward adding TIM in protoMOMxx as a submodule, this PR moves the direct AMReX::initialize/finalize calls from protoMOMxx to TIM.

To do so, we add TIM::Runtime, an RAII encapsulation of the infrastructure runtime: constructing it brings up MPI and then AMReX (as a guest on the resulting communicator), and destroying it tears them down in reverse order. This gives the TURBO stack one owned place for initialization and shutdown of dependencies like MPI, AMReX (and, in the future, PIO).

Who owns MPI is an explicit caller decision made by choosing one of the two constructors of TIM::Runtime. This replaces the MPI_INITIALIZED runtime check FMS's mpp layer relied on.

  • Owner modeRuntime(int& argc, char**& argv): calls MPI_Init and, at destruction, MPI_Finalize (solo drivers). Aborts if MPI is already up.
  • Guest modeexplicit Runtime(MPI_Comm): adopts a communicator from a caller that owns MPI (a coupler/cap) and never finalizes it. Aborts if MPI is not up.

Either way AMReX runs as a guest on that communicator (amrex::Initialize(comm)) — it never owns argc/argv or MPI. Exactly one Runtime may exist per process (AMReX's runtime state is global).

The following draft protoMOMxx PR is a preview of protoMOMxx will utilize TIM::Runtime:. Simply, MOM::Infra becomes a thin wrapper over TIM::Runtime (owner mode in solo_driver, guest mode in future NUOPC cap).
TURBO-ESM/protoMOMxx#18

Changes:

  • tim/cpp/core/tim_runtime.{hpp,cpp}: the Runtime class: owner/guest constructors, comm() accessor, non-copyable/non-movable, one-per-process guard, ordered teardown (amrex::Finalize before MPI_Finalize).
  • test_tim/runtime/test_runtime_owner.cpp: with an owner-mode Runtime alive, direct MPI collectives on comm() and AMReX MultiFab collective reductions work side by side.
  • test_tim/runtime/test_runtime_guest.cpp: main() owns MPI like a coupler, hands MPI_COMM_WORLD to a guest-mode Runtime, runs the same coexistence checks, then verifies after Runtime's destruction that the guest did not finalize MPI and the caller's MPI is still usable.
  • test_tim/CMakeLists.txt: registers both runtime test binaries (labels "unit"); the CUDA source-property handling is factored to cover them.

The new Runtime class starts MPI and then AMReX when it is created,
and shuts them down in the opposite order when it goes out of scope.
The caller decides who owns MPI by choosing a constructor, instead
of the runtime check FMS's mpp layer relied on. Only one Runtime is
allowed per process.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces TIM::Runtime, a single-owner RAII runtime manager that centralizes MPI and AMReX initialization/finalization inside TIM, enabling downstream consumers (e.g., protoMOMxx) to rely on TIM for consistent startup/shutdown sequencing.

Changes:

  • Added TIM::Runtime with explicit owner/guest MPI modes and ordered teardown (AMReX then MPI).
  • Integrated the new runtime into the TIM C++ library build/install.
  • Added unit tests validating MPI collectives and AMReX MultiFab reductions in both owner and guest modes, and updated test CMake wiring.

Reviewed changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
tim/cpp/core/tim_runtime.hpp Declares TIM::Runtime API, owner/guest constructors, communicator accessor, and one-per-process constraints.
tim/cpp/core/tim_runtime.cpp Implements MPI/AMReX bring-up/tear-down and the single-runtime guard.
tim/cpp/CMakeLists.txt Adds the runtime source to tim and installs the new header.
test_tim/runtime/test_runtime_owner.cpp Adds owner-mode runtime test (Runtime owns MPI; AMReX runs on the owned comm).
test_tim/runtime/test_runtime_guest.cpp Adds guest-mode runtime test (caller owns MPI; Runtime must not finalize MPI).
test_tim/CMakeLists.txt Registers the new runtime test executables and CUDA language handling for them.
.gitignore Un-ignores tim/cpp/core/ so the new core/ subtree is tracked despite a global core ignore pattern.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tim/cpp/core/tim_runtime.cpp
Comment thread test_tim/runtime/test_runtime_guest.cpp Outdated
Add the TIM::abort function that aborts the run with a given message.
Unlike amrex::abort, TIM::abort is safe in any state of MPI and AMReX.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants