Add GPU awareness to MPL using IGNORE_TKR directives by marsdeno · Pull Request #105 · ecmwf-ifs/fiat

marsdeno · 2026-03-10T07:50:15Z

Use IGNORE_TKR directive to ignore device residency status
of buffers, allowing use of GPU-aware MPI under MPL.

Copilot

Pull request overview

This PR introduces NVHPC-specific IGNORE_TKR-based directives to allow MPL MPI wrappers to accept device-resident buffers (for GPU-aware MPI), and refactors several MPL routines into Fortran submodules to host the implementations.

Changes:

Add IGNORE_DEVICE / IGNORE_CONTIG_DEVICE preprocessor definitions (NVHPC: !DIR$ IGNORE_TKR(...), otherwise comments) and apply them to MPL buffer dummy arguments.
Refactor MPL send/recv/broadcast/allreduce/alltoallv/gatherv/scatterv/allgatherv implementations into new *_implementation_mod.F90 submodules, leaving interface-only parent modules.
Update buffer dummy argument shapes (notably to assumed-rank (..) in several APIs) and adjust templates/calls accordingly.

Reviewed changes

Copilot reviewed 21 out of 22 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/fiat/CMakeLists.txt	Adds compiler-dependent preprocessor macros for IGNORE_TKR directives.
src/fiat/mpl/internal_f08/mpl_send_implementation_mod.F90	New submodule implementing MPL_SEND procedures.
src/fiat/mpl/internal_f08/mpl_recv_implementation_mod.F90	New submodule implementing MPL_RECV procedures.
src/fiat/mpl/internal_f08/mpl_broadcast_implementation_mod.F90	New submodule implementing MPL_BROADCAST procedures.
src/fiat/mpl/internal_f08/mpl_scatterv_mod.F90	Converts to interface-only module procedures; adds IGNORE_DEVICE and assumed-rank dummies.
src/fiat/mpl/internal_f08/mpl_scatterv_implementation_mod.F90	New submodule implementing MPL_SCATTERV procedures.
src/fiat/mpl/internal_f08/mpl_scatterv_realarray_tmpl.i90	New shared template for REAL scatterv implementation.
src/fiat/mpl/internal_f08/mpl_scatterv_integerarray_tmpl.i90	New shared template for INTEGER scatterv implementation.
src/fiat/mpl/internal_f08/mpl_gatherv_mod.F90	Converts to interface-only module procedures; adds IGNORE_DEVICE and assumed-rank dummies.
src/fiat/mpl/internal_f08/mpl_gatherv_implementation_mod.F90	New submodule implementing MPL_GATHERV procedures.
src/fiat/mpl/internal_f08/mpl_gatherv_realarray_tmpl.i90	Adjusts MPI_GATHERV/IGATHERV call to pass PRECVBUF directly.
src/fiat/mpl/internal_f08/mpl_gatherv_integerarray_tmpl.i90	New shared template for INTEGER gatherv implementation.
src/fiat/mpl/internal_f08/mpl_alltoallv_mod.F90	Converts to interface-only module procedures; adds IGNORE_DEVICE and assumed-rank dummies.
src/fiat/mpl/internal_f08/mpl_alltoallv_implementation_mod.F90	New submodule implementing MPL_ALLTOALLV procedures.
src/fiat/mpl/internal_f08/mpl_allreduce_mod.F90	Converts to interface-only module procedures; adds IGNORE_CONTIG_DEVICE markers.
src/fiat/mpl/internal_f08/mpl_allreduce_implementation_mod.F90	New submodule implementing MPL_ALLREDUCE procedures.
src/fiat/mpl/internal_f08/mpl_allgatherv_mod.F90	Converts to interface-only module procedures; adds IGNORE_DEVICE and assumed-rank dummies.
src/fiat/mpl/internal_f08/mpl_allgatherv_implementation_mod.F90	New submodule implementing MPL_ALLGATHERV procedures.
src/fiat/mpl/internal_f08/mpl_allgatherv_preamble.i90	Changes recv displacement handling when KRECVDISPL is present.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

Copilot · 2026-03-10T07:55:33Z

 IF(PRESENT(KRECVDISPL)) THEN
-  IRECVDISPL_PT => KRECVDISPL
+!!  IRECVDISPL_PT => KRECVDISPL
+  ALLOCATE(IRECVDISPL,SOURCE=KRECVDISPL)
+  IRECVDISPL_PT => IRECVDISPL


When KRECVDISPL is present, this preamble now allocates a local copy (IRECVDISPL) and points IRECVDISPL_PT at it. For non-blocking allgatherv, this local allocatable will be deallocated when the procedure returns, but MPI may still access the displacement array until the request completes, leading to use-after-free. Either keep pointing to the caller-provided KRECVDISPL in the non-blocking case, or register the allocated copy in YDDISPLS_LIST (and ensure it is freed only after request completion).

Copilot · 2026-03-10T07:55:33Z

+IF(LMPLSTATS) THEN
+  CALL MPL_SENDSTATS(ISENDCOUNT,MPI_INTEGER%MPI_VAL)
+  CALL MPL_RECVSTATS(SUM(KRECVCOUNTS),MPI_INTEGER%MPI_VAL)
+ENDIF


In MPL_ALLGATHERV_INT, KRECVCOUNTS is OPTIONAL (to support the scalar case), but stats currently call SUM(KRECVCOUNTS) unconditionally. This will be invalid when KRECVCOUNTS is not present; use SUM(IRECVCOUNTS) (the local array you already build) or guard with PRESENT(KRECVCOUNTS).

marsdeno · 2026-03-12T10:50:48Z

Code warnings thrown by intel compilers being investigated, as work-around for intel breaks NVHPC.

wdeconinck · 2026-03-13T16:08:50Z

Hi @marsdeno could I also draw your attention to missing target attributes in #108

wdeconinck · 2026-03-18T00:19:45Z

In the develop branch there are a lot of missing USE statements for e.g. MPI_ALLGATHER and MPI_IALLGATHER because they are guarded by ifdef "USE_8_BYTE_WORDS". In that case the functions are used with an implicit declaration and resolved by the MPI_F77 library. This is why we also get those gfortran warnings about mismatched MPI interfaces!

I think this PR fixes these issues from what I can tell, but I am not sure if this is done throughout and if we can somehow script this to be sure (or let a coding agent loose).

I don't think we are compiling fiat with USE_8_BYTE_WORDS at all. Do we foresee this will ever be needed?
Also from what I can tell the "mpi4to8" modules seem to be implemented using the MPI_F77 library still, so perhaps this is not going to be compatible at all. Perhaps we can remove this as well.

…implementation.

* interface module with implementing submodule * assumed rank for array arguments being communicated * IGNORE_TKR directive applied in compiler-flexible way

* interface module with implementing submodule * assumed rank for array arguments being communicated * IGNORE_TKR directive applied in compiler-flexible way * TEMPORARY : separation of gatherv template into integer template and real template to avoid calling ASSOCIATE on assumed-rank array (not allowed)

* interface module with implementing submodule * assumed rank for array arguments being communicated * IGNORE_TKR directive applied in compiler-flexible way * adjust call to MPL_BROADCAST in MPL_ALLREDUCE_MOD

* interface module with implementing submodule * assumed rank for array arguments being communicated * IGNORE_TKR directive applied in compiler-flexible way * TEMPORARY : separation of scatterv template into integer template and real template to avoid calling ASSOCIATE on assumed-rank array (not allowed)

* interface module with implementing submodule * assumed rank for array arguments being communicated * IGNORE_TKR directive applied in compiler-flexible way * TEMPORARY(?) work around Nvidia compiler bug affecting a pointer to an optional argument in a module procedure

* interface module with implementing submodule * assumed rank for array arguments being communicated * IGNORE_TKR directive applied in compiler-flexible way

* interface module with implementing submodule * assumed rank for array arguments being communicated * IGNORE_TKR directive applied in compiler-flexible way * NB : for MPL_ALLREDUCE we have used IGNORE_TKR(cd) instead of (d) to work around compiler bug

* symbols provided by mpi_serial library

…ith Intel / IntelLLVM. * the warning is believed to be unjustified in this case, a reproducer will be reported to Intel

* update ci to use nvhpc version 26.1 * switch to F77 MPL implementation for older compilers

…version

marsdeno requested review from Copilot and wdeconinck March 10, 2026 07:50

Copilot started reviewing on behalf of marsdeno March 10, 2026 07:51 View session

Copilot AI reviewed Mar 10, 2026

View reviewed changes

marsdeno force-pushed the feature/mpl_ignore_tkr branch from 3faa385 to 06d8991 Compare March 10, 2026 22:55

wdeconinck linked an issue Mar 16, 2026 that may be closed by this pull request

GPU-awaredness of MPL #90

Open

wdeconinck changed the title ~~Feature/mpl ignore tkr~~ Add GPU awareness to MPL using IGNORE_TKR directives Mar 18, 2026

marsdeno force-pushed the feature/mpl_ignore_tkr branch from 01d00b8 to cb8cbdd Compare March 18, 2026 07:50

marsdeno added 15 commits March 18, 2026 10:15

Transform MPL_SEND_MOD into separate interface module with submodule …

2daed20

…implementation.

Move to assumed rank syntax for send buffers.

38b920e

Add ignore_tkr directive for real and integer sends.

b61423f

Handle IGNORE_TKR directive in a compiler-dependent manner.

2501b8d

Follow recipe for MPL_RECV_MOD

4194f0b

* interface module with implementing submodule * assumed rank for array arguments being communicated * IGNORE_TKR directive applied in compiler-flexible way

Follow recipe for MPL_BROADCAST_MOD

7fca3c8

* interface module with implementing submodule * assumed rank for array arguments being communicated * IGNORE_TKR directive applied in compiler-flexible way * adjust call to MPL_BROADCAST in MPL_ALLREDUCE_MOD

Follow recipe for MPL_ALLTOALLV_MOD

4cfddd6

* interface module with implementing submodule * assumed rank for array arguments being communicated * IGNORE_TKR directive applied in compiler-flexible way

Tidy up development comments

fb2e5d6

Import MPI_xx routines as external symbols in mpi_f08_dummy_mod.

c70675e

* symbols provided by mpi_serial library

Disable warnings mentioning VOLATILE OR ASYNCHRONOUS when compiling w…

70da73f

…ith Intel / IntelLLVM. * the warning is believed to be unjustified in this case, a reproducer will be reported to Intel

Add LLVMFlang handling of IGNORE_TKR(d) directive

77bae3e

marsdeno force-pushed the feature/mpl_ignore_tkr branch from cb8cbdd to 77bae3e Compare March 18, 2026 11:28

marsdeno and others added 3 commits March 19, 2026 16:55

Handle bug in NVHPC versions older than 26.1

1332a9c

* update ci to use nvhpc version 26.1 * switch to F77 MPL implementation for older compilers

Remove use of implicit MPIF functions in the MPL_F08 version

059f971

Suppress warning with NVHPC about zero-sized array

704c30e

wdeconinck added 5 commits March 20, 2026 22:09

Extend mpi_f08_dummy_mod.F90

881b042

Fixup 1332a9c - Handle bug in NVHPC versions older than 26.1

3895cc7

Fixup 059f971 - Remove use of implicit MPIF functions in the MPL_F08 …

5d81b97

…version

Fixup 059f971 - Remove use of implicit MPIF functions in the MPL_F08 …

0b24fee

…version

Fixup 881b042 - Extend mpi_f08_dummy_mod.F90 ; Fixes ENABLE_MPI=OFF

bf68f47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add GPU awareness to MPL using IGNORE_TKR directives#105

Add GPU awareness to MPL using IGNORE_TKR directives#105
marsdeno wants to merge 23 commits into
ecmwf-ifs:developfrom
marsdeno:feature/mpl_ignore_tkr

marsdeno commented Mar 10, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Mar 10, 2026

Uh oh!

Copilot AI Mar 10, 2026

Uh oh!

marsdeno commented Mar 12, 2026

Uh oh!

wdeconinck commented Mar 13, 2026

Uh oh!

wdeconinck commented Mar 18, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

marsdeno commented Mar 10, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

marsdeno commented Mar 12, 2026

Uh oh!

wdeconinck commented Mar 13, 2026

Uh oh!

wdeconinck commented Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

wdeconinck commented Mar 18, 2026 •

edited

Loading