new end-to-end workflow and better handling of input arguments to gen_timeseries and cmor_driver

[mvertens]

new end-to-end workflow scripts/run_reference_case.py

With this new script a output from a coupled run can be processed and checked end-to-end. The script runs gen_timeseries, cmor_driver and validate_cmor_output for every realm of a target archived case. A table is then printed with a summary of what worked.

python scripts/run_reference_case.py --model noresm --atmos-res ne16 \
  --experiment piControl --case-dir <archive> --outdir <where> \
  --years 1526:1527:2

This script does the following:

• plans the run first (--dry-run prints the commands without running them)
• checks the experiment name against the controlled vocabulary before starting
• runs time series generation, CMORization and validation for each realm
• logs every step separately
• reports which succeeded and which failed.

restartability

One requirement of an end-to-end workflow is that if you interrupt it or there is an error - you can resume without starting back from the beginning. Repeating the same command now continues where it stopped via the following:

• across steps. Each step that succeeds is recorded in <outdir>/completed_steps.json, and a repeated run skips those. So if a run dies partway through, rerunning it picks up at the next unfinished step.
• from time series generation. gen_timeseries.py now keeps existing output and produces only the timesteps that are missing, rather than rewriting everything. So even a step that was interrupted midway resumes rather than
  repeating its work.
• --force runs everything again and rewrites existing time series.
• A run with nothing left to do is now reported as success

updated arguments to gen_timeseries.py and cmor_driver.py

Both of these scripts had long flat option lists with defaults on most options. Options are now grouped in --help under "required arguments" and a few plainer headings, and the ones describing the case must be given.

gen_timeseries.py

	before	now
--model	defaulted to cesm	required
--realm	four realms accepted	all eight that the tables declare

cmor_driver.py

	before	now
--model, --realm, --frequency, --experiment	all defaulted	required
--tsdir	optional, otherwise guessed from a CIME case	required
--resolution	any of five grids, for any realm	--atmos-res, and only where it applies
--caseroot, --cimeroot	used to locate time series	removed
--version	printed a version number	removed

The input grid is now worked out from the model and the realm, so sea-ice data
can no longer be put through atmosphere weight files by mistake. --atmos-res
is only needed for the atmospheric and land realms; the others derive their own.

other additions

• New tests covering every realm and frequency for both models, which run in about two seconds and need no model output
• --ocn-static-file now applies only where it is used, and says when it is being ignored
• A corrected licence identifier that was stopping three tests from running - Installation notes for setting up on a laptop

What you will need to change

Existing commands that relied on the defaults need the values spelled out:

# before
python scripts/cmor_driver.py --realm atmos --tsdir /path/to/tseries

# now
python scripts/cmor_driver.py --model noresm --realm atmos --frequency mon \
  --experiment piControl --atmos-res ne16 --tsdir /path/to/tseries

Anyone who used --caseroot/--cimeroot to locate time series now passes
--tsdir directly. That path was <DOUT_S_ROOT>/<component>/proc/tseries/<frequency>.

Drafted with Claude Code.

[maritsandstad]

All good just have a couple of questions, probably this can just go in.

[maritsandstad]

Eventually, I think we would want to send the
case.add_argument(
"--realization-initialization-physics-forcing",
type=str,
default="r1i1p1f1",
help="Variant indices in the format rXiYpZfW (default: r1i1p1f1)",
)
which cmor driver adds here, so you could also use this harness to run the full cmorisation (not just for testing). There might be other options like that which we want to include for that, but that is at least one.

[mvertens]

I have added this to the workflow - and modified the cmor_driver argument:

  --variant-label rXiYpZfW, --realization-initialization-physics-forcing rXiYpZfW
             Ensemble member this output belongs to: realization, initialization, physics and forcing indices, e.g. r1i1p1f1. This is the CMIP variant_label, and appears in the output paths and file names. The longer spelling is accepted as an alias. (default: r1i1p1f1)

I think we want this capability to not only do testing by also the full cmorisation workflow - which should be in place with this PR.

[maritsandstad]

This is very good, but I wonder how much it will conflict with my current PR for the grids, might need to rehash that for this...

[maritsandstad]

Maybe it actually matches up quite well...don't know yet...

[mvertens]

@maritsandstad - I am trying to test this on the current run n1850GaxgGHG.LM.nor30b25.528.20260928 - using 2 years of data from it. I want to make sure everything works before I modify the draft status.