Change in pool's FESOM mesh leads to NaNs

[mandresm]

Hi @koldunovn,

@seb-wahl was running some simulations with FESOM in Levante that he cannot restart anymore. The suspects are a set of changes in the pool on the 2.1 mesh done by you:

ls -lrt /pool/data/AWICM//FESOM2/MESHES_FESOM2.1/core2/
...
drwxr-xr-x 2 a270088 ab0995     4096 May 28 09:32 dist_4
drwxr-xr-x 2 a270088 ab0995     4096 May 28 09:34 dist_2
drwxr-xr-x 2 a270088 ab0995    20480 May 28 09:42 dist_128
-rw-r--r-- 1 a270088 ab0995  3180567 May 28 22:19 elvls_raw.out
-rw-r--r-- 1 a270088 ab0995  9291100 May 28 22:19 edge_tri.out
-rw-r--r-- 1 a270088 ab0995  9291100 May 28 22:19 edges.out
-rw-r--r-- 1 a270088 ab0995       26 May 28 22:19 edgenum.out
-rw-r--r-- 1 a270088 ab0995  1649154 May 28 22:19 nlvls.out
-rw-r--r-- 1 a270088 ab0995  3180567 May 28 22:19 elvls.out
drwxr-xr-x 2 a270088 ab0995     4096 May 28 22:19 dist_12

@seb-wahl can give more details of what's going on, but he's interest right now I believe is on finding a set of mesh files that he can use to restart his simulation. Do you happen to have backups of those files?

Alternatively, @JanStreffing has also suggested that his input paths for AWICM3/AWIESM3 might be identical to the old ones, but might not be. Of course it would be better to have the old ones.

@koldunovn, do you think that any of the files changed on the 28th of May could lead to NaNs in the model?

@seb-wahl, how many nprocs did you use for FESOM (in your ESM-Tools runscript)? That would determine which dist directory you would be using.

[alxthor]

I could imagine this being related to #852

[seb-wahl]

Below is the summary (with massive help from claude) on what I did to finally think that the updated mesh could be the issue. The suggested code modification mentioned and which where necessary to find that the mesh might be the issue are not yet committed.
Side note: I have experience with other ocean and atmosphere models, but I'm are rather new to FESOM.

IF-SW003_branch_off_restart_NaN_investigation.pdf

[seb-wahl]

@seb-wahl, how many nprocs did you use for FESOM (in your ESM-Tools runscript)? That would determine which dist directory you would be using.

I tried at least nproc 72 and 5xx.

[seb-wahl]

I can now confirm that the change in meshes is indeed responsible for the NaNs. I made copy of /pool/data/AWICM//FESOM2/MESHES_FESOM2.1/core2/ at /work/bb0519/foci_input2/ICON-FESOM/FESOM2/MESHES_FESOM2.1/core2/ a while ago (and completely forgot about it). If I use the latter path as mesh_dir everything runs smooth.

[JanStreffing]

@koldunovn Could you check if the main core2 pool dir files need to be fixed from another version?

[JanStreffing]

@svenharig, Could this happen when we make a new partition? That seems to be the indication here, also from #852

[koldunovn]

This all is my fault. The #922 not only fixed the bug, but also change the nlvls.out, elvls.out, elvls_raw.out, edgenum.out, edges.out, edge_tri.out . The files are now fixed - the old variant is restored, and the #952 is submitted. Sorry about that :(

The fact I was able to do this in production folder probably calls for some protection mechanism.

[mandresm]

Yes, it is indeed worrying. I have write access to many things in pool/data/AWICM/FESOM2/FORCING/CORE2/, so you might want to correct that too:

$ find /pool/data/AWICM/FESOM2/ -type f -writable
/pool/data/AWICM/FESOM2/FORCING/CORE2/ncar_rad.1973.nc
/pool/data/AWICM/FESOM2/FORCING/CORE2/t_10.1957.nc
/pool/data/AWICM/FESOM2/FORCING/CORE2/q_10.1965.nc
/pool/data/AWICM/FESOM2/FORCING/CORE2/q_10.1967.nc
/pool/data/AWICM/FESOM2/FORCING/CORE2/slp.2004.nc
...