FESOM2 refuses to overwrite existing output files after restarting a failed simulation junk

[christian-stepanek]

I am using FESOM2.5 in a version where Lars put iceberg code into in a version of AWI-ESM2-wiso. I think though that this might be a general issue for FESOM2.

A run on albedo crashed during writing output because of disk quota. I noted that when simply resubmitting the simulation after solving the quota issue FESOM2 does crash when trying to write output files that are already there:

/albedo/work/user/stepanek/esm_experiments_6/piControl/log/piControl_awiesm_compute_20500101-20501231_7042474.log

  0:  piControl_205101.01_g3bid                        g3bid                 73      T     T
  0:  piControl_205101.01_jsbid                        jsbid                 71      T     T
  0:  piControl_205101.01_sp6h                         sp6h                  69      T     T
746:  error in line          424 io_netcdf_file_module.F90 NetCDF: File exists && NC_NOCLOBBER
792:  initializing I/O file for u
583:  associating mean I/O file /albedo/work/user/stepanek/esm_experiments_6/piControl/run_20500101-20501231/work/w.fesom.2050.nc
583:  w: current mean I/O counter =            1
583:  writing mean record for w; rec. count =            1
746: 1

Is there a good reason why FESOM2 behaves that way? Or could one change (compile-time) flags to make FESOM2 be less bothered by existing output files?

[trackow]

Hey Christian,

is this with the https://github.com/FESOM/fesom2/tree/wiso_add_icebergs branch? I can however only see the file https://github.com/FESOM/fesom2/blob/wiso_add_icebergs/src/io_netcdf_workaround_module.F90 there, not io_netcdf_file_module.F90.

In general, the NC_NOCLOBBER option means you want to create a new netCDF file only if the file does not already exist, so this does seem like expected behaviour with this setting to me and would need to be changed in the line 424. Would need to see the file though to have a better idea why that could have been chosen.

[christian-stepanek]

Dear Thomas,
thanks a lot for engaging with this problem.
Yes, it is a version of FESOM2.5 with icebergs that Lars has worked on to create. I do not know how close it is to wiso_add_icebergs. The last bits that you worked on are below, extracted from a git log - thereafter, lots of commits by Lars and others followed to merge fesom2.5 into wiso etc.

commit 3eb85156427d46f543e77ddce9b1da8cf266bef3
Author: Thomas Rackow <thomas.rackow@ecmwf.int>
Date:   Thu Dec 22 16:24:06 2022 +0000

    add 300m-average temp and salt to the exchanged fields

commit 565fbe2129bf8ff7e349a478a483026188897783
Author: Thomas Rackow <thomas.rackow@ecmwf.int>
Date:   Thu Dec 22 16:23:09 2022 +0000

    make tempzavg and saltzavg public

Not sure whether io_netcdf_file_module.F90 may be new in the version of FESOM2 I am using. Indeed, the source code explicitly defines use of the no_noclobber option:

   this%filepath = filepath

    cmode = ior(nf_noclobber, ior(nf_netcdf4, nf_classic_model))
    call assert_nc( nf_create(filepath, cmode, this%ncid) , __LINE__)
        
    ! create our dims in the file
    do i=1, size(this%dims)
      call assert_nc( nf_def_dim(this%ncid, this%dims(i)%name, this%dims(i)%len, this%dims(i)%ncid) , __LINE__)
    end do
    

This file is available here:
https://github.com/ackerlar/fesom2/blob/icb_into_v2.5/src/io_netcdf_file_module.F90

[trackow]

As a quick change, you could probably just write in line 423

cmode = ior(nf_clobber, ior(nf_netcdf4, nf_classic_model))

I have not seen this behaviour yet because for me, restarted runs always go to another run directory so there are never existing files... maybe something for @koldunovn to confirm this is indeed wanted like this

[JanStreffing]

Is this still an issue that needs attention @christian-stepanek, @trackow

[christian-stepanek]

I will check as soon as I rerun a period of simulation output again.