runtime error with Izumi_nag -- doesn't like NaN's in _FillValue?

[ekluzek]

The following test fails at runtime with the NAG compiler on Izumi. It seems to dislike _FillValue being set to NaN.

SMS_D_Ld5.5x5_amazon_rHDMA.I2000Clm60SpMizGs.izumi_nag.mizuroute-default

cesm.log:

 Attribute logunit not set for ROF
 ROF: PIO numiotasks= 1
 ROF: PIO stride= 24
 ROF: PIO rearranger= 2
 ROF: PIO root= 1
 shr_file_mod.F90 913 This routine is depricated - use shr_log_setLogUnit instead -2134114300
ERROR: 0 NetCDF: Not a valid data type or _FillValue type mismatch err_num = -45 fname = /fs/cgd/data0/erik/ctsm_worktree/answer_changes/libraries/parallelio/src/clib/pioc_support.c line = 2457
[cli_0]: aborting job:

The mizuRoute gets to the middle of the initialization as follows.

rof.log:

---- mis. routing options --- 
  Lateral flow drains at the top of headwater reaches only for IRF, KWE, MC, DW
 mizuRoute initialization
  mizuRoute npes =  24
  mizuRoute iam  =  0
  mizuRoute cal  = noleap
  mizuRoute time = days since 2000-01-01 00:00:00
  inst_name = ROF
mizuRoute tracers= LIQ: ICE:
(route_ini) mizuRoute run at   86400.0000000    [sec] step   1 per a coupling
 define run:
    run type              = initial
    coupling_frequency    =    8.6400000000000000E+04 [day]
    mizuRoute timestep    =    8.6400000000000000E+04 [sec]

[ekluzek]

OK, I tried with swapping out the ntopo file and mesh file for ones that have _FillValue set to -9999. It get's further, but still dies in handling the mesh file. As seen here...

shell_commands:

./xmlchange ROF_DOMAIN_MESH=/fs/cgd/csm/inputdata/rof/mizuRoute/meshes/HDMAmz_5x5_amazon_ctrcrd_cdf5_ESMFmesh_c20260610.nc

user_nl_mizuroute_control:

<fname_ntopOld>              ntopo_HDMAmz_5x5_amazon_cdf5_c20260610.nc      ! filename

PET00.ESMF_LogFile:

20260610 125857.659 INFO             PET00   (lnd_comp_nuopc):(InitializeRealize)  done
20260610 125857.659 INFO             PET00   (rof_comp_nuopc):(InitializeRealize)  called
20260610 125858.337 ERROR            PET00 /home/jedwards/src/ESMF/src/Infrastructure/Mesh/src/ESMCI_Mesh_FileIO.C:428 ESMCI_mesh_create_from_ESMFMesh_   The NetCDF Library returned an error  -  Unable to open existing file: /fs/cgd/csm/inputdata/rof/mizuRoute/meshes/HDMAmz_5x5_amazon_ctrcrd_cdf5_ESMFmesh_c20260610.nc, (PIO/NetCDF error = NetCDF: Not a valid data type or _FillValue type mismatch)
20260610 125858.490 ERROR            PET00 /home/jedwards/src/ESMF/src/Infrastructure/Mesh/src/ESMCI_Mesh_FileIO.C:670 ESMCI_mesh_create_from_ESMFMesh_   The NetCDF Library returned an error  - Internal subroutine call returned Error
20260610 125858.490 ERROR            PET00 /home/jedwards/src/ESMF/src/Infrastructure/Mesh/src/ESMCI_Mesh_FileIO.C:284 ESMCI_mesh_create_from_file()   The NetCDF Library returned an error  - Internal subroutine call returned Error
20260610 125858.490 ERROR            PET00 ESMCI_MeshCap.C:2605 MeshCap::meshcreatefromfilenew()   The NetCDF Library returned an error  - Internal subroutine call returned Error
20260610 125858.490 ERROR            PET00 ESMF_Mesh.F90:1970 ESMF_MeshCreateFromFile()   The NetCDF Library returned an error  - Internal subroutine call returned Error
20260610 125858.490 ERROR            PET00 rof_comp_nuopc.F90:594   The NetCDF Library returned an error  - Passing error in return code
20260610 125858.490 ERROR            PET00 ESM0001:src/addon/NUOPC/src/NUOPC_Driver.F90:2918   The NetCDF Library returned an error  - Phase 'IPDv01p3' Initialize for modelComp 4: ROF did not return ESMF_SUCCESS
20260610 125858.490 ERROR            PET00 ESM0001:src/addon/NUOPC/src/NUOPC_Driver.F90:2001   The NetCDF Library returned an error  - Passing error in return code
20260610 125858.490 ERROR            PET00 ensemble:src/addon/NUOPC/src/NUOPC_Driver.F90:2918   The NetCDF Library returned an error  - Phase 'IPDv02p3' Initialize for modelComp 1: ESM0001 did not return ESMF_SUCCESS
20260610 125858.490 ERROR            PET00 ensemble:src/addon/NUOPC/src/NUOPC_Driver.F90:2006   The NetCDF Library returned an error  - Passing error in return code
20260610 125858.490 ERROR            PET00 ensemble:src/addon/NUOPC/src/NUOPC_Driver.F90:489   The NetCDF Library returned an error  - Passing error in return code
20260610 125858.490 ERROR            PET00 esmApp.F90:134   The NetCDF Library returned an error  - Passing error in return code
20260610 125858.490 INFO             PET00    Finalizing ESMF with endflag==ESMF_END_ABORT

I upped the ESMF and PIO debug levels to show that the problem is in the mesh file:

   252  12:51   ./xmlchange ESMF_VERBOSITY_LEVEL=low
   254  12:52   ./xmlchange PIO_DEBUG_LEVEL=1

However the new mesh file doesn't have a NaN on the file at all. And it fails with the _FillValue message whether the attribute is deleted or set to -9999.

[ekluzek]

I get a similar problem with these as well:

SMS_D_Ld5_PS.nldas2_nldas2_rUSGS_mnldas2.I2000Clm60SpMizGs.izumi_nag.mizuroute-default
SMS_D_P48x1_Ld5.f19_f19_mt232.I2000Clm60SpMizGs.izumi_nag.mizuroute-default