Only can restart from the last record in restart file

[koldunovn]

I am on refactoring branch.

Running simulation with monthly restarts, that, say, last for 3 months with monthly restarts. I have 3 restart records with time stamps:
time = 2675700, 5094900, 7773300

I would like to restart from the month 2, and chance the fesom.clock to:

   83700.0000000000               59        1958
   86400.0000000000               59        1958

I get the error:

raw restart time    7773300.00000000      does not match current clock time   5094900.00000000

So there is no way to start from anything but last record in the restart? If so, it's not a good behaviour, since in high resolution setups we restarts from couple of steps before the last one quite often.

@hegish @dsidoren

[hegish]

Looks like there is a fesom_raw_restart directory with an *.info file available (i.e. memory dump restart). If so, the memory dump is currently always preferred over the regular netcdf restart. You can remove the fesom_raw_restart directory and it should work as always.

[helgegoessling]

Speaking of FESOM restart files: It has always been a stumbling block that FESOM restart files contain multiple times rather than one file (or set of files) per restart date, where it would be straight forward to also name the exact restart date in the file name. Would also make the life for ESM tools easier; Miguel is working on an issues related to exactly this right now, if I understand correctly. Can we maybe consider changing this sometime?

[koldunovn]

@hegish I tried to remove the binary restarts, result is the same. Does this code imply we only look for the last record?

fesom2/src/io_restart.F90

Line 479 in 092a794

if(filegroup%files(i)%is_iorank()) then

[koldunovn]

Speaking of FESOM restart files: It has always been a stumbling block that FESOM restart files contain multiple times rather than on file (or set of files) per restart date, where it would be straight forward to also name the exact restart date in the file name. Would also make the life for ESM tools easier; Miguel is working on an issues related to exactly this right now, if I understand correctly. Can we maybe consider changing this sometime?

Moving around individual files would be easier of course. @hegish implemented the new scheme, where each year get it's folder and then each variable.

Before, when the grass was greener and the files much smaller having everything in one file was reasonable, now, when I have restarts of several TB per year, having them splitted to individual time steps make sense. @dsidoren ?

[koldunovn]

@dsidoren says he is OK with that :) We can discuss details in a separate thread. @helgegoessling can you maybe create an issue and list what would be your dream file organisational structure for restarts? @mandresm you probably have an opinion on this as well?

[hegish]

@hegish I tried to remove the binary restarts, result is the same. Does this code imply we only look for the last record?

If it does not work to load older restarts, then it probably never worked before, e.g. before the refactoring branch?

Does this code imply we only look for the last record?

fesom2/src/io_restart.F90

Line 479 in 092a794

if(filegroup%files(i)%is_iorank()) then

No, this only makes sure the following part is executed from a single process only.

[koldunovn]

@hegish I tried to remove the binary restarts, result is the same. Does this code imply we only look for the last record?

If it does not work to load older restarts, then it probably never worked before, e.g. before the refactoring branch?

I used this functionality almost every week with FESOM2.1, so no, it worked.

[hegish]

@dsidoren says he is OK with that :) We can discuss details in a separate thread. @helgegoessling can you maybe create an issue and list what would be your dream file organisational structure for restarts? @mandresm you probably have an opinion on this as well?

Currently we have one restart directory per year. In the directory we have a single file per restart field/variable. Before that, we had a single file per year, containing all time steps for all variables.

If we consider a separate set of files for each restart step, this would lead to a bunch of differently named files or directories, as the restart frequency can be anything from steply to yearly.
How about a meeting and someone prepares a draft, then we can discuss it further if you like.

[hegish]

@hegish I tried to remove the binary restarts, result is the same. Does this code imply we only look for the last record?

If it does not work to load older restarts, then it probably never worked before, e.g. before the refactoring branch?

I used this functionality almost every week with FESOM2.1, so no, it worked.

Do you know if it ever worked with the refactoring branch?

[koldunovn]

Do you know if it ever worked with the refactoring branch?

I don't know, but why it wouldn't if it works with FESOM2.1?

[hegish]

I will look into it (:

[helgegoessling]

@dsidoren says he is OK with that :) We can discuss details in a separate thread. @helgegoessling can you maybe create an issue and list what would be your dream file organisational structure for restarts? @mandresm you probably have an opinion on this as well?

To be honest, I think @mandresm is best situated to judge how an ideal restart-file organization would look like. (I'm also wondering: are any of the other models collected in ESM-Tools combining multiple restart states in one file?)

Regarding @hegish 's comment

the restart frequency can be anything from steply to yearly"

I'm wondering whether we need to distinguish between files actually meant for restarts, and files that store instantaneous fields ("snapshots"), which, if I'm not mistaken, in FESOM happen to be the same (but would not e.g. in models that need two previous timesteps for a clean restart). So, in special cases one might want to store instantaneous values at high frequency, up to stepwise, but I can hardly think of a case where one would need to be able to actually restart stepwise.

[mandresm]

Hi all, sorry about my late reply.

To be honest, I think @mandresm is best situated to judge how an ideal restart-file organization would look like. (I'm also wondering: are any of the other models collected in ESM-Tools combining multiple restart states in one file?)

I think FESOM is the only model having this behavior. EDIT: @JanStreffing , what about OpenIFS? Does it keep data from previous runs on its restarts?

I think the question here is not necessarily what is best in terms of ESM-Tools, since ESM-Tools is conceived precisely for dealing with the rarities of the different models. I'm currently working on something to avoid this problem with FESOM-2.0, and the main difficulty is a bug on ESM-Tools, that I am fixing.

However, in my opinion, it makes sense to have restart files that correspond exclusively to the duration of a given run, and that are time-stamped with the final date of that run. Having restart files time-stamped for the current year and keeping adding restarts to them with successive runs it seems to me not super intuitive and adds an additional consideration when you are restarting a simulation.

ESM-Tools thinks of isolated runs, and keep them in separated folders. At the end of each run it copies/moves the restart into the general restart folder and if there is already a restart with the same name (same year), it time stamps both files, to don't overwrite them. This leads to an unnecessary consumption of disk space, that could be avoided by telling ESM-Tools to clean restart files at the end of each year, and only keep the last one. Or we could choose to move the restart files between runs until a year is completed. So, IMO if there are good technical reasons to keep all restarts inside a yearly file (i.e. I/O performance) then ESM-Tools can and should be able to deal with it.

What I just said above applies to the netCDF parallel restarts (one restart per field in the restart folders) and to the old restart files (.oce.restart.nc and .ice.restart.nc). The only thing I would change about the fesom_raw_restarts (it might be there already, in fact, I'm pretty sure that's the case) is that you can control through the namelist if you want to use them to restart or not.

[JanStreffing]

I assume we will be writing only one file per MPI thread and variable. Since this refactored FESOM2 will have strong OMP scaling, that should reduce the number of restart files by a factor 36 or so. I think investing the file count back into one file per restart timestep might be a productive endeavor saving time later.

I think FESOM is the only model having this behavior. EDIT: @JanStreffing , what about OpenIFS? Does it keep data from previous runs on its restarts?

OpenIFS creates one restart file per MPI thread and per restart time. The names are build by counting up the time from initial date to the first timestep of the new run:

srfDDDDDDDDHHMM.NPROC
srf000073050000.0001

A different option is to do what the wave model WAM (included in OpenIFS) does. It too creates a restart file per MPI thread and per restart time, but the naming goes:

LAWYYYYMMDDHHMMSS_DDDDDDHHMMSS.NPROC_NPROC_total
LAW18500101000000_007304230000.85_384

where both the initial time and the last step of the previous simulation is stored in the name. I consider this inferior, as it leads to trouble when branching off and changing the timestep length. Such information can best be stored in a restart control file like fesom.clock