GridapTrilinos.jl

GridapTrilinos is a Julia interface for using Trilinos linear solvers from Gridap/GridapDistributed workflows. The Julia side provides a TrilinosSolve linear solver; the Trilinos, Tpetra, Belos, Thyra, FROSch, and Kokkos calls live in a small C++ shared library exposed with CxxWrap.

Release notes are kept in NEWS.md.

Workflow

There are four parts to the workflow:

1. Install the Julia package dependencies.
2. Configure MPI.jl to use the same MPI implementation as Trilinos.
3. Build the C++ shared library against your local Trilinos installation.
4. Load GridapTrilinos and use TrilinosSolve as a Gridap linear solver.

Julia Setup

For a registered installation:

using Pkg
Pkg.add("GridapTrilinos")

For a source checkout, from the repository root:

using Pkg
Pkg.activate(".")
Pkg.instantiate()

This installs the Julia dependencies listed in Project.toml. The package can be installed without Trilinos. Without the optional C++ shared library, using GridapTrilinos and TrilinosSolve(...) work, while calls that need Trilinos throw an error explaining how to build the backend.

MPI Compatibility

The MPI used by Julia must be the same MPI implementation used to build Trilinos. This package uses Trilinos' MPI on the C++ side, while GridapDistributed reaches MPI through MPI.jl. If those are different MPI runtimes, the process can fail at load time or hang/crash during communication.

Configure MPI.jl through MPIPreferences before building this package. For a typical system MPI installation:

using Pkg
Pkg.add("MPIPreferences") # only needed if MPIPreferences is not already in your environment

using MPIPreferences
MPIPreferences.use_system_binary()

If libmpi is not in the dynamic linker search path, pass the library location:

using MPIPreferences
MPIPreferences.use_system_binary(;
    library_names = ["/path/to/mpi/lib/libmpi.so"],
    mpiexec = "/path/to/mpi/bin/mpiexec",
)

Restart Julia after changing MPIPreferences. Then instantiate/build the package. deps/build.jl checks that MPI.jl is configured for "system" MPI before compiling the C++ wrapper.

Building The C++ Library

The C++ wrapper must be built before calling the Trilinos solver. Each user builds it locally against their Julia, MPI, and Trilinos installation.

After installing the package, set TRILINOS_ROOT to the Trilinos installation prefix and build through Julia's package build step:

export TRILINOS_ROOT=/path/to/TrilinosInstall
julia -e 'using Pkg; Pkg.build("GridapTrilinos")'

For a source checkout with the repository environment activated, this is also fine:

export TRILINOS_ROOT=/path/to/TrilinosInstall
julia --project=. -e 'using Pkg; Pkg.build("GridapTrilinos")'

This calls deps/build.jl, which configures CMake with paths discovered from the active Julia package build environment. A checked-out repository can also call the developer convenience script directly:

export TRILINOS_ROOT=/path/to/TrilinosInstall
src/Sharedlib/configure.sh

The build configures CMake in a Julia scratch space and creates:

DEPOT_PATH[1]/scratchspaces/<GridapTrilinos uuid>/trilinos-backend/usr/lib/GridapTrilinos.so

This file is generated locally and is not stored inside the versioned package source directory, so it survives package source updates until the scratch space is removed by Julia's scratch-space cleanup. If it is missing, using GridapTrilinos still works, but any call into the Trilinos solver throws an error telling you to build the shared library first.

Custom Trilinos Solve Source

By default, the shared library builds the solver implementation in:

src/Sharedlib/TrilinosSolve.cpp

Advanced users can provide a different C++ source file at build time with GRIDAPTRILINOS_SOLVE_SOURCE:

export TRILINOS_ROOT=/path/to/TrilinosInstall
export GRIDAPTRILINOS_SOLVE_SOURCE=/path/to/MyTrilinosSolve.cpp
julia -e 'using Pkg; Pkg.build("GridapTrilinos")'

Relative paths are resolved from src/Sharedlib/, so this also works:

export TRILINOS_ROOT=/path/to/TrilinosInstall
export GRIDAPTRILINOS_SOLVE_SOURCE=MyTrilinosSolve.cpp
src/Sharedlib/configure.sh

The custom source must define the TrilinosSolve overloads declared in src/Sharedlib/TrilinosTypes.hpp. The setup overload returns a reusable cache:

TrilinosSolverCache TrilinosSolve(
  const RCP<crs_matrix_type>& A,
  const std::string& parameterFilePath,
  bool verbose);

TrilinosSolveData TrilinosSolve(
  const TrilinosSolverCache& solverCache,
  const RCP<vec_type>& b,
  bool verbose);

The one-shot overload can delegate through the cached path:

TrilinosSolveData TrilinosSolve(
  const RCP<crs_matrix_type>& A,
  const RCP<vec_type>& b,
  const std::string& parameterFilePath,
  bool verbose);

The Gridap/Tpetra construction and Julia wrapper code still comes from src/Sharedlib/TrilinosInterface.cpp; only the Trilinos solve implementation is replaced.

GRIDAPTRILINOS_SOLVE_SOURCE only swaps the C++ source file. If your custom solver needs additional CMake logic, include directories, libraries, compile definitions, or extra source files, edit src/Sharedlib/CMakeLists.txt or keep a project-specific copy of that CMake file. A second CMakeLists file is not loaded automatically.

Initialisation

Loading the package performs the CxxWrap and Kokkos setup:

using GridapTrilinos

When the scratch-space GridapTrilinos.so exists, the package:

• loads the C++ module with CxxWrap,
• calls KokkosInitialize() during Julia module initialisation,
• registers KokkosFinalize() with atexit.

You normally do not need to call KokkosInitialize() or KokkosFinalize() manually.

Usage

Create a solver from a Trilinos XML parameter file:

using GridapTrilinos

solver = TrilinosSolve("path/to/trilinos_parameters.xml")

TrilinosSolve implements the Gridap linear solver interface, so it is intended to be passed wherever a Gridap.Algebra.LinearSolver is expected. Internally, numerical setup builds the distributed Tpetra matrix and caches the Thyra LinearOpWithSolve. Repeated solve! calls with the same numerical setup reuse that solver state and only rebuild the Tpetra right-hand side and solution.

After a solve has run, inspect the recorded solver result:

result = solver.log

result.name
result.num_iters
result.residual
result.solve_time

Public API

The public API is:

• TrilinosSolve
• SolverResult
• name, num_iters, residual, solve_time, verbose, and depth

The lower-level C++ wrapper functions, Kokkos lifecycle hooks, setup structs, and Tpetra construction helpers are implementation details. Typical Gridap usage should go through TrilinosSolve. A solver's latest backend result is available as the solver.log property after a solve completes.

Repository Metadata

Suggested GitHub description:

Gridap linear solver interface for Trilinos through CxxWrap, Tpetra, Thyra,
Belos, and FROSch.

Suggested GitHub topics:

julia, gridap, trilinos, finite-element-method, linear-solvers, mpi, cxxwrap,
tpetra, thyra, belos, frosch

Development Checks

Run the package tests from the repository root:

julia --project=. -e 'using Pkg; Pkg.test("GridapTrilinos")'

The default run skips the MPI solves. To run both Poisson MPI tutorials:

GRIDAPTRILINOS_RUN_MPI_TESTS=true \
  mpiexecjl --project=. -n 4 julia --startup-file=no --color=yes \
    -e 'using Pkg; Pkg.test("GridapTrilinos")'

Or run one tutorial directly:

mpiexecjl --project=. -n 4 julia test/poisson_thyra.jl
mpiexecjl --project=. -n 4 julia test/poisson_frosch.jl
mpiexecjl --project=. -n 4 julia test/transient_cached.jl

Rebuild the C++ library after changing files in src/Sharedlib/:

export TRILINOS_ROOT=/path/to/TrilinosInstall
src/Sharedlib/configure.sh