NorESM - how to generate and validate cmor output

[mvertens]

Getting the code on NIRD (obtaining the pythone code and the cmor tables)

> git clone https://github.com/ESMCI/cmip7-prep
> cd cmip7-prep
> git submodule init
> git submodule update 

Activating the conda and python environment

> conda activate /projects/NS9560K/diagnostics/cmordev_env_312
> export PYTHONPATH=</path/to/MY/cmip7-prep/src>:$PYTHONPATH

step 0: (optional) generating the input yaml file(s) - convert_csv_to_yaml.py

Starting from the NorEsm data request google sheet - https://docs.google.com/spreadsheets/d/1XUdCTl1zKsWi_yTvMZqsMtBnnUPIhIeTPVSLHgL0Hdo/edit?gid=1341553017#gid=1341553017
To general all the will generate the required input yaml files:

> python convert_csv_to_yaml.py [-h] [--model {noresm,cesm}] [--input INPUT]

The following columns are required by convert_csv_to_yaml.py and must be filled out correctly in the spread sheet order to generate the input yaml file (all other columns are ignored)

• Branded variable Name
• Modelling Realm - Primary
• Description
• Units (from Physical Parameter)
• Dimensions
• NorESM3 name (dependency)

Note - the yaml files are part of the repository. If the spreadsheet is updated the yaml files must be regenerated and a PR to "main" must be issued to updated the yaml files.

step 1: generating time series from time slice files - gen_timeseries.py

As an example we will use the data from NorESMhub/noresm3_dev_simulations#458
data is at /nird/datalake/NS9560K/noresm3/cases/nhist.ne16pg3_tn14.noresm3_0_beta20.cmip7workflow.2026-06-23
Several notes:

• You need to be on NIRD IPCC for this. You can still use the same directory but you need to reactivate the conda environment.
• --years 1950:1969:5 below creates time series output in 5 year increments.
• the following commands took about 1/2 hour on NIRD to create time series for the atm, lnd and ice history output

> ./gen_timeseries.py --help

> ./gen_timeseries.py --realm atmos  --inputdir /nird/datalake/NS9560K/noresm3/cases/nhist.ne16pg3_tn14.noresm3_0_beta20.cmip7workflow.2026-06-23/atm/hist/ --outputdir /nird/datalake/NS9560K/noresm3/test_cmip7prep/nhist.ne16pg3_tn14.noresm3_0_beta20.cmip7workflow.2026-06-23/atm/timeseries --years 1950:1969:5 --model noresm --workers 16

> ./gen_timeseries.py --realm land   --inputdir /nird/datalake/NS9560K/noresm3/cases/nhist.ne16pg3_tn14.noresm3_0_beta20.cmip7workflow.2026-06-23/lnd/hist/ --outputdir /nird/datalake/NS9560K/noresm3/test_cmip7prep/nhist.ne16pg3_tn14.noresm3_0_beta20.cmip7workflow.2026-06-23/lnd/timeseries --years 1950:1969:5 --model noresm --workers 16

> ./gen_timeseries.py --realm seaIce --inputdir /nird/datalake/NS9560K/noresm3/cases/nhist.ne16pg3_tn14.noresm3_0_beta20.cmip7workflow.2026-06-23/ice/hist/ --outputdir /nird/datalake/NS9560K/noresm3/test_cmip7prep/nhist.ne16pg3_tn14.noresm3_0_beta20.cmip7workflow.2026-06-23/ice/timeseries --years 1950:1969:5 --model noresm --workers 16

step 2: generating cmorized output - cmor_driver.py

In cmor_driver.py the following is hard-wired. This maps cam, clm and cice history output to the --realm and --frequency input variables (which is mon by default). For example, if cam.h2a are not produced - no 6 hour cmor output can be produced.

 "noresm": {
       "atmos": {
           "mon": ["cam.h0a"],
           "day": ["cam.h1a"],
           "6hr": ["cam.h2a"],
           "3hr": ["cam.h4a"],
       },
       "atmosChem": {
           "mon": ["cam.h0a"],
           "day": ["cam.h1a"],
           "6hr": ["cam.h2a"],
           "3hr": ["cam.h4a"],
       },
       "aerosol": {
           "mon": ["cam.h0a"],
           "day": ["cam.h1a"],
           "6hr": ["cam.h2a"],
           "3hr": ["cam.h4a"],
       },
       "land": {
           "mon": ["clm2.h0a"],
           "day": ["clm2.h1a"],
           "3hr": ["clm2.h2a"],
           "yr":  ["clm2.h2a"] # Temporary change for WIEMIP TODO to change back ["clm2.h3a"],
       },
       "seaIce": {
           "mon": ["cice.h."],
           "day": ["cice.h1."],
       },

The following will produce cmor output given the history files that were created for the example run (note that only 2 daily output cmor files where obtained from the camh1a files.

> ./cmor_driver.py --help

> ./cmor_driver.py --realm atmos --tsdir /nird/datalake/NS9560K/noresm3/test_cmip7prep/nhist.ne16pg3_tn14.noresm3_0_beta20.cmip7workflow.2026-06-23/atm/timeseries/ --outputdir /nird/datalake/NS9560K/noresm3/test_cmip7prep/nhist.ne16pg3_tn14.noresm3_0_beta20.cmip7workflow.2026-06-23/atm/cmor --workers 1 --resolution ne16 --model noresm &> output.atmos.mon &

> ./cmor_driver.py --realm atmos --tsdir /nird/datalake/NS9560K/noresm3/test_cmip7prep/nhist.ne16pg3_tn14.noresm3_0_beta20.cmip7workflow.2026-06-23/atm/timeseries/ --outdir /nird/datalake/NS9560K/noresm3/test_cmip7prep/nhist.ne16pg3_tn14.noresm3_0_beta20.cmip7workflow.2026-06-23/atm/cmor --workers 1 --resolution ne16 --model noresm --frequency day &> output.atmos.day &

> ./cmor_driver.py --realm aerosol --tsdir /nird/datalake/NS9560K/noresm3/test_cmip7prep/nhist.ne16pg3_tn14.noresm3_0_beta20.cmip7workflow.2026-06-23/atm/timeseries/ --outdir /nird/datalake/NS9560K/noresm3/test_cmip7prep/nhist.ne16pg3_tn14.noresm3_0_beta20.cmip7workflow.2026-06-23/aerosol/cmor --workers 1 --resolution ne16 --model noresm &> output.aerosol.mon &

> ./cmor_driver.py --realm atmosChem --tsdir /nird/datalake/NS9560K/noresm3/test_cmip7prep/nhist.ne16pg3_tn14.noresm3_0_beta20.cmip7workflow.2026-06-23/atm/timeseries/ --outdir /nird/datalake/NS9560K/noresm3/test_cmip7prep/nhist.ne16pg3_tn14.noresm3_0_beta20.cmip7workflow.2026-06-23/atmosChem/cmor --workers 1 --resolution ne16 --model noresm &> output.atmosChem.mon &

> ./cmor_driver.py --realm land --tsdir
/nird/datalake/NS9560K/noresm3/test_cmip7prep/nhist.ne16pg3_tn14.noresm3_0_beta20.cmip7workflow.2026-06-23/lnd/timeseries/ --outputdir /nird/datalake/NS9560K/noresm3/test_cmip7prep/nhist.ne16pg3_tn14.noresm3_0_beta20.cmip7workflow.2026-06-23/lnd/cmor --workers 1 --resolution ne16 --model noresm &> output.land.mon &

NOTE: to have the appropriate cice namelists used for cmorization - user_nl_cice needs to currently contain the contents of user_nl_cice in NorESMhub/noresm3_dev_simulations#437. This was not the case with the example simulation and as a result the following command did not generate any cmorized files for the seaIce realm:

> ./cmor_driver.py --realm seaIce --tsdir /nird/datalake/NS9560K/noresm3/test_cmip7prep/nhist.ne16pg3_tn14.noresm3_0_beta20.cmip7workflow.2026-06-23/ice/timeseries/ --outputdir /nird/datalake/NS9560K/noresm3/test_cmip7prep/nhist.ne16pg3_tn14.noresm3_0_beta20.cmip7workflow.2026-06-23/ice/cmor --workers 1 --resolution ne16 --model noresm &> output.seaIce.mon &

NOTE: If you grep for 'Finished' in the output log - you can see how many variables were successfully processed.
If you grep for 'Exception' in the output log - you can see which variables resulted in an error.

step 3: validate the output - validate_cmor_output.py

python validate_cmor_output.py --model noresm --realm atmos --frequency mon --root-output-path /nird/datalake/NS9560K/noresm3/test_cmip7prep/nhist.ne16pg3_tn14.noresm3_0_beta20.cmip7workflow.2026-06-23/atm/cmor --plot-maps --plot-timeseries