Issue detected by nvidia (nvhpc) compiler with -O2 or greater

[nmizukami]

nvhpc compiler option uses the following flags for optimization mode now:

FLAGS = -fast -O3 -Mdclchk

For some reason, -O3 or -O2 flag cause index error at line reachID(1:nRch) = reachID( ixRch_order ) in mpi_process.f90.

Looks like ixRch_order get zero values for some reason.

With this flag,

FLAGS = -O1 -Mdclchk

works fine. Also, MODE=debug works fine.

Will need to look into this more why ixRch_order gets zero values with higher optimization mode.

[scamp-nvidia]

Hi - I'm on the NVHPC compiler team (I was just at the NCAR hackathon and a teammate pointed this to me). If you decide this is a compiler issue and can carve out a smaller reproducing, I can help you in getting it reported and fixed. Or, we can help you identify any standard grey areas that are being exploited and are okay at low optimization, but might fail at higher one. Note that with reproducing examples of codes with larger code bases, I've had a lot of luck lately with asking AI bots for help in diagnosing and creating reproducing examples.

Depending on how hard a standalone run of this code is to build and run on a small system to recreate, I could also help carve out things. Please feel free to ping me if you decide you need some help!

[ekluzek]

@nmizukami I learned from an NVIDIA engineer at the hackathon that "-O" is somewhere between "-O1" and "-O2" so I suggest trying with it.

Note, also for CTSM we are using these options:

-O -Mnofma -i4 -gopt -Mextend -byteswapio -Mflushz -Kieee -tp=zen3 -Mstack_arrays -Mallocatable=03

You can also use "-Minfo=all -Mneginfo=all" to give you optimization information on what the compiler did to the code.

Note here's the definiton of some of those options:

../cmake_macros/nvhpc-prof.cmake:# -gopt Generate debug information without disabling optimizations
../cmake_macros/nvhpc-prof.cmake:# -Mnofma turns off Fused Multiply-Add (FMA) instructions
../cmake_macros/nvhpc-prof.cmake:  # -Kieee use IEEE division and enbale trps
../cmake_macros/nvhpc-prof.cmake:  # -MBounds check array bounds
../cmake_macros/nvhpc-prof.cmake:# -Mflushz does flush-to-zero for floating pointer operations
nvhpc-prof_derecho.cmake:  # -tp is the target processor
nvhpc-prof_derecho.cmake:  # -Mstack_arrays put automatic arrays on the stack
nvhpc-prof_derecho.cmake:  # -Mallocatable=O3 use the 2003 standard for allocatable arrays, which allows them
 to be used in more contexts and with more features than the older Fortran 90/95 standard

[ekluzek]

It looks like the extra option you use above "-Mdclchk" essentially ensures the same as adding "implicit none" everywhere in the code. Personally having "implicit none" in the code should be thought of as required, but I like making sure this is in the actual code rather than relying on the compiler option for it.

[nmizukami]

thanks Erik (@ekluzek), I tried CTSM flags, but -O does not work, but if I use -O1 works.

[scamp-nvidia]

You could play around with disabling things - like "-O2 -Mnovect" - I know vectorization gets turned on at O2. Inlining also may start getting turned on at that point, so "-O2 -Mnoinline". You could also disable some FMAs - "-O2 -Mnofma". If you can figure out what turns on at "-O2" that's not on at "-O1", it'll go a long way in helping you identify exactly where the problem could be getting introduced.

[scamp-nvidia]

I just remembered - another approach for resolving this sort of thing, assuming the issue is probably occurring in a single file, you can do a flag binary search. i.e. Compile half the files with one optimization flag and the other half with the lower optimization flag and see if you have the issue. If not - repeat with the unoptimized half, and so on. Then continuing on in that way until you isolate to a single file. If there's a couple files that are causing the issue, it's trickier, but also doable.

[nmizukami]

I just remembered - another approach for resolving this sort of thing, assuming the issue is probably occurring in a single file, you can do a flag binary search. i.e. Compile half the files with one optimization flag and the other half with the lower optimization flag and see if you have the issue. If not - repeat with the unoptimized half, and so on. Then continuing on in that way until you isolate to a single file. If there's a couple files that are causing the issue, it's trickier, but also doable.

Looks like this might be worth trying. I tried all the flag (changing globally== the same flag for all f90s), but does not resolve, and now it seems that the run failed at random locations in init_data under standalone/model_setup.f90. it seems that tasks are running at different speed and somehow failed. It is becoming more difficult to chase now. I have never run into this behavior in intel-oneapi (-O0 or -O3) and gcc (-O0 or -O3). and -O1 and -O0 in nvhpc is ok,