new end-to-end workflow and better handling of input arguments to gen_timeseries and cmor_driver

.github/workflows/pytest.yaml

@@ -12,12 +12,15 @@ jobs:
     steps:
       - name: Checkout
         uses: actions/checkout@v4
-      - name: Checkout cmip7-cmor-tables from CESM-Development fork and cesm-dev branch
-        uses: actions/checkout@v4
         with:
-          repository: CESM-Development/cmip7-cmor-tables
-          ref: cesm-dev
-          path: cmip7-cmor-tables
+          # The CMOR tables come from the submodule.
+          submodules: recursive
+      - name: Show which CMOR tables are in use
+        run: |
+          git -C cmip7-cmor-tables log -1 --format='%H %s' \
+            || echo "cmip7-cmor-tables is not a git checkout"
+          grep -o '"CC-BY-4[^"]*"' cmip7-cmor-tables/tables-cvs/cmor-cvs.json \
+            | sort -u
       - name: Set up Python
         uses: actions/setup-python@v5
         with:
@@ -48,14 +51,14 @@ jobs:
             /home/runner/cmor
             /home/runner/cmor3-source
             /home/runner/.cache/pip
-          key: cmor-${{ runner.os }}-${{ hashFiles('requirements.txt') }}-${{ hashFiles('**/cmor_version.txt') }}-cmor3.14.0
+          key: cmor-${{ runner.os }}-${{ hashFiles('requirements.txt') }}-${{ hashFiles('**/cmor_version.txt') }}-cmor3.15.3
           restore-keys: |
-            cmor-${{ runner.os }}-3.14.0
+            cmor-${{ runner.os }}-3.15.3
 
       - name: build and install cmor
         run: |
           if [ ! -d "$HOME/cmor" ]; then
-            git clone https://github.com/PCMDI/cmor.git $HOME/cmor3-source -b 3.14.0
+            git clone https://github.com/PCMDI/cmor.git $HOME/cmor3-source -b 3.15.3
             cd $HOME/cmor3-source
             ./configure --prefix=$HOME/cmor --with-netcdf=/usr/local --with-udunits2=/usr/local
             make

.pylintrc

@@ -17,5 +17,5 @@ disable=
 max-line-length=100
 
 [TYPECHECK]
-ignored-modules=xarray,xesmf,cmor,numpy,geocat.comp,yaml,cftime,pytest,dask
+ignored-modules=xarray,xesmf,cmor,numpy,geocat.comp,yaml,cftime,pytest,dask,gents
 ignored-classes=cmor

INSTALL.md

@@ -49,6 +49,40 @@ pip install gents dulwich cmip7-data-request-api
 
 These three are not on conda-forge, so they must come from pip.
 
+Note that `pip install <pkg>` does nothing when the package is already present.
+To move to a newer release, ask for it explicitly:
+
+```bash
+pip install -U gents dulwich cmip7-data-request-api
+```
+
+## Local development environment (macOS)
+
+The steps above create the shared analysis environment on NIRD.  For running the
+test suite and editing the code on a laptop, create a private environment
+instead.  Everything the tests need, including CMOR and ESMF, has an
+`osx-arm64` build on conda-forge, so no part of the stack has to be skipped.
+
+```bash
+conda create -n cmip7-dev -c conda-forge python=3.12 cmor=3.15 \
+  numpy xarray netcdf4 cftime pyyaml pytest pytest-cov \
+  xesmf=0.8.7 esmpy=8.9.0 geocat-comp=2025.10.01
+```
+
+```bash
+conda activate cmip7-dev
+pip install -e . --no-deps
+pip install gents
+```
+
+```bash
+pytest -q
+```
+
+Use **Miniforge** rather than Anaconda for this (see the BLAS entry under
+Troubleshooting).  `dulwich` and `cmip7-data-request-api` are only needed for
+the data-request and CV tooling, so a test-only environment can leave them out.
+
 ## CMIP7 CMOR tables
 
 This recipe only provides the CMOR **library** (3.15). The matching **tables**
@@ -67,3 +101,28 @@ labels). Keep that checkout up to date, or CMOR will reject otherwise-valid runs
 - **`license_id "..." could not be found` / `grid_label "..." is invalid`:**
   the `cmip7-cmor-tables` checkout is out of date. Update it to the version
   matching CMOR 3.15 / the CMIP7 controlled vocabulary.
+- **`zero-size array to reduction operation minimum which has no identity`,
+  logged with a traceback while pulling metadata:** one or more history files
+  hold no time records, which happens when a run segment opens a history stream
+  but ends before a sample is written.  GenTS takes a minimum over each file's
+  time coordinate, so an empty file fails.  This is **not fatal** --
+  `HFCollection.pull_metadata` defaults to `raise_errors=False`, logs the
+  failure with `exc_info=True` and drops the file, so the run continues.
+  `gen_timeseries.py` screens such files out up front so they appear as one
+  warning each instead of a traceback.
+- **`Error importing numpy: you should not try to import numpy from its source
+  directory`:** misleading message; numpy prints it whenever its compiled
+  extension fails to load for any reason.  Get the real cause with
+  ```bash
+  python -c "import traceback
+  try: import numpy
+  except ImportError as e: traceback.print_exception(e.__cause__)"
+  ```
+  A mixed-channel Anaconda install can leave `libcblas`/`libblas` symlinked to
+  an OpenBLAS build that was never installed, in which case the cause names a
+  missing `.dylib`.  A conda-forge-only base (Miniforge) avoids this; repairing
+  a broken one needs `conda install --force-reinstall openblas blas`.
+- **`environment.yml` disagrees with these instructions:** it is an export of
+  the older `cmordev_env`, not the `cmordev_env_312` built above, so its pins
+  (including `cmor` and `gents`) are not the versions this recipe installs.
+  Treat the steps here as authoritative.

data/cmor_dataset.json

@@ -31,7 +31,7 @@
   "sub_experiment": "none",
     "archive_id": "WCRP",
     "area_label" : "u",
-    "license_id": "CC-BY-4-0",
+    "license_id": "CC-BY-4.0",
   "realization_index": "r1",
   "initialization_index": "i1",
   "physics_index": "p1",

pytest.ini

@@ -1,4 +1,4 @@
 [pytest]
 addopts = --doctest-modules
-norecursedirs = scripts
+norecursedirs = scripts cmip7-cmor-tables
 pythonpath = src .