runoff output from fesom2.7-recom3.1 with AWICM forcing substantially higher than expected

[tanvinagwekar]

When running FESOM 2.7 with AWICM forcing, the runoff output is unexpectedly high compared to (1) a run with FESOM2.1 using the same AWICM forcing and (2) the original AWICM runoff forcing. Figure attached shows the difference.

What could be a possible reason for this and places in the code to inspect the same?

[JanStreffing]

Thanks for the report, @tanvinagwekar. Before I dig into the code I'd like to reproduce your exact setup, so could you share a few details?

• Is this a standalone FESOM run forced by AWICM output, or a coupled run? If standalone, what is your runoff_data_source and the relevant runoff lines from namelist.forcing?

• Which runoff variable name and units are in your forcing file?

• Where is the FESOM2.7-REcoM3.1 code you built from (path on the machine, plus the git branch and commit)?

• Same for the FESOM2.1 code used in the comparison run.

• Path to the FESOM2.7 experiment directory.

• Path to the FESOM2.1 experiment directory.

• Path to the original AWICM runoff forcing file you compared against.

With those I can line up the two runoff code paths and check the unit handling against your actual files rather than guessing. Which machine are these on?

[tanvinagwekar]

Hi @JanStreffing thanks for looking into this. Here is the information you asked for:

• yes, this is FESOM-RecoM standalone run forced by AWICM. runoff_data_source = AWICM.

• This is what I have in namelist.forcing regarding runoff:
  nm_runoff_file = '/albedo/work/projects/p_retake/runoff_spinup/runoff_fesom_'
  runoff_data_source ='AWICM' !Dai09, CORE2
  use_runoff_mapper = .FALSE.
  runoff_basins_file = 'runoff_maps_regular.nc'
  runoff_radius = 500000.

• FESOM2.7-REcoM3.1 code I run is on albedo (branch= fesom2.7-recom3.1), the path is: /albedo/home/tnagweka/fesom2.7_new/

• In the original forcing file the variable name is "runoff" and it is "m/s".

• FESOM2.1-REcoM3.1 is here: /albedo/home/tnagweka/fesom2.1_recom/

• FESOM2.7 experiment directory: /albedo/home/tnagweka/fesom2.7_new/fesom2/test_run2/

• FESOM2.1 experiment directory: /albedo/home/tnagweka/fesom2.1_recom/fesom2/hist/

• Path for the original AWICM runoff forcing file: /albedo/work/user/tnagweka/forcing/runoff_data/CORE2_hist/ (note: although I specified a different path here than the one in namelist.forcing for nm_runoff_file as mentioned above, same files exists at both locations). All of it is on albedo.

[JanStreffing]

Thanks @tanvinagwekar, that's everything I needed. I think this is the same root cause as #821 (fill values in 2D fields), introduced by PR #799 (merged Oct 28, 2025).

What PR #799 did: every netCDF output stream now declares _FillValue = 9.96921e+36, and zero-valued grid points are written as that fill value instead of as 0.0. The intent was to mask 3D points below bathymetry, but the check is value-based with no wet-cell test, so any 2D field that has legitimate zeros over most of the domain gets the same treatment. #821 reports this for a_ice/m_ice. River runoff has the same shape of field, with sparse nonzero values at outlet nodes and zero elsewhere, so it is affected too.

Quick check from your runoff.fesom.*.nc:

• ~48,358 nodes carry real runoff values (matches the ~48,619 nonzero nodes in the runoff_fesom_*_redistributed.nc input)
• ~78,500 nodes per timestep hold 9.96921e+36, the _FillValue
• the spatial sum across nodes matches your FESOM 2.1 output and the input file, so total freshwater is conserved
• the spatial mean is inflated by a factor 126858 / 48358 ≈ 2.62, which matches the ~2.8× factor you applied in Check_forcings.ipynb

xarray's mean() defaults to skipna=True, so runoff.mean(dim='nod2') ignores the FillValue nodes and divides by ~48k instead of ~127k. FESOM 2.1 wrote 0.0 at those nodes, so the same mean operation divided by ~127k there. The model physics is fine — only the diagnostic mean is distorted by how the zeros are encoded.

Two ways forward:

1. Workaround now: in your notebook, either replace fills with zero before averaging via runoff = runoff.fillna(0), or compute area-weighted integrals (which are unaffected, since FillValue nodes contribute nothing).
2. Real fix: PR mask netcdf output by wet-cell topography instead of value==0 #929 (draft) replaces the value-based test with a wet-cell topology check (ulevels_nod2D / nlevels_nod2D), which writes 0.0 at non-outlet wet nodes and _FillValue only at land or cavity. That fixes this case too. I'll prioritize it for 2.7.8.

[tanvinagwekar]

@JanStreffing, thanks for the help.
I would have one more issue, model (fesom2.7-recom3.1) simulation with AWICM runoff forcing crashes with a error of temperature exceeding 60degC. this is when I use runoff_data_source=AWICM. It was able to simulate over 120 years correctly and then it started to crash. Now for a test I am currently running with runoff_data_source=CORE2, with which it appears to work fine for the moment.

Here are the experiment folder:
with AWICM runoff forcing: /albedo/home/tnagweka/fesom2.7_new/fesom2/test_run2/
with CORE2 runoff climatology: /albedo/home/tnagweka/fesom2.7_new/fesom2/bin_core_runoff/

I am not sure if it is the forcing or something else that results in a crash with AWICM runoff. Would be great if you can take a look.