Simulations with FESOM 2.7 are significantly slower than FESOM 2.6 with output/restart enabled

[PeterRony]

FESOM2.7 simulations with a high-resolution mesh are significantly slower than FESOM2.6, when output and restart are enabled, whereas compute-only performance remains comparable.

Some FESOM-only experiments are carried out with a high-resolution mesh, with following cases:

1. No output, no restart
2. Restart only (yearly)
3. Output (yearly) + restart (yearly)

Model versions used:
Fesom 2.7: fesom2.7-recom3.1 (https://github.com/FESOM/fesom2/tree/fesom2.7-recom3.1/src), commit: 83d86d1
Fesom 2.6: f2.6r3.1 (https://github.com/FESOM/fesom2/tree/f2.6r3.1/src), commit: 26df848

---

Case 1: No output, no restart

Case 2: Only restart

Case 3: Output + restart

Observations:

Case	F2.6	F2.7
No output, no restart	5757 s	5750 s
Restart only	5824 s	8021 s (~1.38 times slower)
Output + restart	6276 s	12072 s (~1.92 times slower)

Detailed routine-wise timing shows I/O imbalance in F2.7:

For instance,

• In output: Mean values close to 235 s, Max values: 3792 s
• In restart: Mean values: ~47 s, max: 6295 s

Baseline (without output/restart) compute performance is similar between F2.6 and F2.7
The slowdown in F2.7 appears only when I/O is included.

[JanStreffing]

Hey @PeterRony. Thank you for turning this into an issue. I have a few questions:

• When you restart in FESOM2.7, is it from restart files written with 2.7, or from old restart files written with 2.6?
• Does the problem persist on the second leg of the run?
• What HPC system are you on?
• What mesh are use using?

[PeterRony]

Hi @JanStreffing, thank you for the quick follow up.
These experiments are carried out on albedo using a high-resolution coastal mesh with surface nodes close to 676000, and @svenharig will be testing the same experiments with CORE2 mesh.
All experiments shown here are the first year of cold starts (from 1958, using JRA-55 repeated year forcing), not restart runs. The values for next year run are also similar, for example, following is the routine-wise runtime for the year 1959, a continuation of simulation FESOM 2.7 with output and restart:

It also has similar anomalous high max values for output and restart runtime.

[JanStreffing]

Mhm, my last info from @mandresm and @pgierz, was that anything to do with file read/write on albedo could be hardware related. It seems to be somewhat reproducible though. Would this also happen, e.g. on levante@dkrz.de? In particular the fact that only some processes seem to have much longer execution times, is suspect.

Maybe someone from the HPC side can take a look? Other users have not reported such a slowdown, and 2.7. has been out for a while now.

[PeterRony]

Yes, noticed this issue on NHR@ZIB while trying to run FESOM2.7 with REcoM using same mesh. The run freezes while creating output/restart files and stops after time runs out. This is same as on albedo for REcoM setup but FESOM2.6 with REcoM runs fine. Also, tried FESOM-only setup on NHR@ZIB, simulation finished successfully, but got similar large run-time as on albedo.

[mueller79]

I also did a test run with FESOM2.7 using CORE2 mesh on albedo, using cold starts (from 1958, using JRA-55 repeated year forcing) like @PeterRony and it took ~ 3 h 30 min, while the same setup with FESOM2.6 takes ~1 h 10 min.

I noticed the following run time differences between fesom2.6 and fesom2.7:

[JanStreffing]

I think we are looking at two separate issues here. You @mueller79 have slow sea ice on every node. @PeterRony had normal sea ice integration speed, but slow restart writing, only on a select few nodes (mean is low, max is high)

@mueller79 Do you have the same evp options in the namelist.ice?

Both of you, do you have DISABLE_MULTITHREADING=OFF, or ASYNCHRONOUS_IO_THREADS=ON? If so, you can try DISABLE_MULTITHREADING=ON and/or ASYNCHRONOUS_IO_THREADS=OFF

[JanStreffing]

I just checked our 2.7. coupled high resolution run on levante: /work/bb1469/a270089/runtime/awiesm3-v3.4.1/AWI-ESM3-VEG-HR-CMIP7-Spinup_cont2/log

   0: ___MODEL RUNTIME per task [seconds]_____mean____________min____________max_
   0:   runtime ocean:                   14035.6455     13705.0693     14246.4541
   0:     > runtime oce. mix,pres. :      1785.4049      1699.1558      1870.0214
   0:     > runtime oce. dyn. u,v,w:      1533.8108      1347.8577      1677.5625
   0:     > runtime oce. dyn. ssh  :      6444.4336      6321.5610      6606.9727
   0:     > runtime oce. solve ssh :      6270.5503      6175.5049      6446.3306
   0:     > runtime oce. GM/Redi   :         0.1311         0.0353         0.3333
   0:     > runtime oce. tracer    :      4171.2671      3892.4893      4397.3101
   0:   runtime ice  :                    6558.2969      6467.7925      6611.8901
   0:     > runtime ice step :            6194.6133      6114.0854      6235.8311
   0:   runtime diag:                      328.3413       324.8765       330.6749
   0:   runtime output:                    345.3984       300.4418       382.4650
   0:   runtime restart:                    10.6007         5.5297        18.3677
   0:   runtime forcing:                   343.7672       318.6292       449.3000
   0:   runtime total (ice+oce):         20230.2598     19852.4590     20479.6348
   0:
   0:  ======================================================
   0:  ================ BENCHMARK RUNTIME ===================
   0:     Number of cores :                       5120
   0:     Max OpenMP threads :                       2
   0:     Runtime for all timesteps :       22515.0586 sec
   0:  ======================================================

We do not have this imbalance issue, nor the restart write time issue. Sea ice cost is high, but this is expected. In a none-RECOM setup sea ice is a significant fraction of overall compute.

[mueller79]

For both runs namelist.ice is the same.
In FESOM2.6 DISABLE_MULTITHREADING=ON and in FESOM2.7 ASYNCHRONOUS_IO_THREADS=OFF

These are my paths on albedo:

F2.6: namelists: /albedo/home/simuel001/model_code/F26R3P4_fromgit/fesom2/bin_4p3z2d
src: /albedo/home/simuel001/model_code/F26R3P4_fromgit/fesom2/src

F2.7: namelists: /albedo/home/simuel001/model_code/F27R31/fesom2/bin_merged_20260408
src: /albedo/home/simuel001/model_code/F27R31/fesom2/src

[PeterRony]

I also had similar large sea-ice integration speed in albedo and NHR@ZIB, with high resolution coastal mesh. It went away after albedo update (slowness with output/restart writing still exists), but, in NHR@ZIB, issue still remains. Sharing the screenshot of runtime for fesom2.7 with coastal mesh simulation in NHR@ZIB:

Paths in albedo:

MESH: /albedo/work/user/ropete001/MESHES_F2/OP_m4_IPsmooth_70layers/
FESOM 2.6: f2.6r3.1 (https://github.com/FESOM/fesom2/tree/f2.6r3.1/src)
namelists: /albedo/home/ropete001/F26/fesom2_4408f0e_test001_phy/bin_oceanpeak_m4_70levels_clim61_phy

FESOM 2.7: fesom2.7-recom3.1 (https://github.com/FESOM/fesom2/tree/fesom2.7-recom3.1/src)
namelists: /albedo/home/ropete001/F27/fesom2.7_physical/bin_OP_2p3z2d/

Other settings:

in FESOM 2.6: DISABLE_MULTITHREADING=ON
in FESOM 2.7: ASYNCHRONOUS_IO_THREADS=OFF

[svenharig]

Based on Rony's code and experiments I created a shorter test, that hopefully is reproducible and indeed they seem to confirm different behaviour of F2.6 and F2.7 on albedo. Simulation for 1 day and write output/restart of together about 10GB at the end.

For F2.6 the output performance is pretty consistent across the processes. Here showing average, min and max times of output in 5 experiments for each configuration. Mesh and amount of output is the same in all runs.

F2.6 on 480 Cores:
Experiment ---------------------------------------- Avg --------- Min -------- Max
fesom2.6_p0480_exp1: runtime output: 416.6660 416.3709 416.9671
fesom2.6_p0480_exp2: runtime output: 416.7315 416.4512 417.0600
fesom2.6_p0480_exp3: runtime output: 416.4597 416.1909 416.7851
fesom2.6_p0480_exp4: runtime output: 416.3543 416.0954 416.6980
fesom2.6_p0480_exp5: runtime output: 416.2463 415.9893 416.5368

For F2.7 timings are inconsistent across processes, but comparable in total for 480 cores

F2.7 on 480 Cores:
Experiment ---------------------------------------- Avg --------- Min -------- Max
fesom2.7_p0480_exp1: runtime output: 21.4810 3.8353 793.5604
fesom2.7_p0480_exp2: runtime output: 21.4941 3.7811 794.5897
fesom2.7_p0480_exp3: runtime output: 21.3941 3.7850 790.5161
fesom2.7_p0480_exp4: runtime output: 21.5055 3.7969 794.7192
fesom2.7_p0480_exp5: runtime output: 21.6043 3.7982 797.1719

...whereas for 2592 cores it stays inconsistent but gets considerably slower with a wider range between experiments

F2.7 on 2592 Cores:
Experiment ---------------------------------------- Avg --------- Min -------- Max
fesom2.7_p2592_exp1: runtime output: 15.4802 0.6510 3537.2437
fesom2.7_p2592_exp2: runtime output: 17.2738 0.6778 4036.9263
fesom2.7_p2592_exp3: runtime output: 17.2314 0.6702 3986.4106
fesom2.7_p2592_exp4: runtime output: 16.8015 0.7089 3855.5869
fesom2.7_p2592_exp5: runtime output: 17.1367 0.6772 3931.9253

Now Miguel spotted a difference in the albedo environment. In Fesom 2.6 (branch f2.6r3.1 used by Rony) there is an entry

export ENABLE_ALBEDO_INTELMPI_WORKAROUNDS=''

together with many more changes in src/CMakeLists.txt and the F90 code referring specifically to albedo that seem to have appeared in the branch

refactoring_albedo_env

(see pull request #400) that was probably not merged into the F2.7.

We're not sure if all or some of these changes are not included in the F2.7 tag, @patrickscholz, do you think it's worthwhile trying the setting for F2.7 as well? In this case I could give it a try and see if the inconsistent output becomes more regular.

[mandresm]

Hi @svenharig, thanks for the information. For your F2.7, what branch are you using?

[svenharig]

The branches that I'm using are

f2.6r3.1 for FESOM 2.6
fesom2.7-recom3.1_dev for FESOM 2.7 (so far identical to fesom2.7-recom3.1)

On albedo I'm running the tests in the folder

/albedo/work/user/sharig/f27_r31_dev/config/bin_fesom2.7_lrg/

/albedo/work/user/sharig/f26_r31/config/bin_fesom2.6_lrg/

I'm also running the same tests on levante, but since there I'm always seeing inconsistent output behavior (although in many cases considerably faster than on albedo) and @JanStreffing reported about nice and consistent output in his runs I'd like to double-check that before summarizing it all.

[JudithHauck]

Thanks, @svenharig .
Do you run FESOM or FESOM-REcoM in your experiments?
@chrisdane and me were surprised to see that F2.7 is faster than F2.6 in your tests, so showing the opposite behaviour to @PeterRony and @mueller79's reports.
If that has to do with albedo (and possibly NHR) environment that would explain why @JanStreffing can't reproduce it on DKRZ.