v0.15.0

Changelog

Breaking Changes

• The hash that identifies a diagnostic's input datasets is now computed deterministically across pandas versions and platforms.
  It previously relied on pandas.util.hash_pandas_object, whose output changes between pandas releases and platforms.

  Because the underlying hash values change, the committed regression catalog_hash values have been regenerated,
  and existing databases will re-run each execution once on first use after upgrading. (#741)

• The minimum supported Python version is now 3.12; Python 3.11 is no longer supported.
  Update your environment to Python 3.12 or newer before upgrading. (#745)

• Removed the legacy central regression-testing framework in favour of the native baselines managed by ref test-cases.
  The DiagnosticValidator and ExecutionRegression helpers, the diagnostic_validation, execution_regression, and regression_data_dir pytest fixtures (from climate_ref.conftest_plugin), the central tests/test-data/regression/ baseline tree, and the per-provider test_diagnostics / test_build_results tests have all been removed.

  Provider regression baselines are now recorded and verified entirely through the ref test-cases CLI and the per-package committed bundles (see Regression baselines). (#757)

Features

• Added ref test-cases mint, replay, and sync commands for managing native regression baselines.
  mint records a test case's native outputs into the baseline store,
  replay regenerates a bundle from the stored outputs and checks it against the committed copy,
  and sync downloads the native outputs referenced by committed baselines into the local cache. (#724)

• Added the ref test-cases ci-gate command, which decides how CI should verify each
  test case's regression baseline against the base branch: skip it, replay the cached
  native baseline, re-run the diagnostic in full, or fail an unauthorised change.
  Baselines are now coupled to their input catalogs, so changing a test case's inputs
  without regenerating its baseline is reported as a failure rather than passing silently. (#727)

• Added a Cloudflare R2 object-store backend for diagnostic regression baselines, so ref test-cases mint can publish a test case's native outputs to the shared baseline store rather than only a local directory.
  The store endpoint and bucket are configured through the REF_NATIVE_STORE_* settings (defaulting to the project's public baseline bucket), and write credentials are supplied via the environment or a named AWS/R2 profile, never the persisted configuration.

  Minting is also easier to get right.
  ref test-cases mint now checks the store credentials and bucket up front and stops with a clear message when they are misconfigured, the new ref test-cases check-store command verifies store connectivity without minting, ref test-cases mint --dry-run previews what would be minted, and ref test-cases fetch restores a missing local paths file on its own so a fresh checkout no longer needs --force. (#732)

• Regression baselines are now verified automatically in CI.
  Every pull request runs the coupling gate and replays each test case's cached native baseline, a nightly job re-checks every baseline for silent drift, and a manually gated workflow mints new baselines to the shared object store behind required reviewers — so write credentials never run on untrusted pull-request code.

  The diagnostic testing guide also gained a "pull request workflow" section explaining what the gate decides for each change and how to publish a new baseline. (#733)

• ILAMB regression baselines are migrated to the per-package Framework B layout, starting with the mrsos-wangmao, gpp-fluxnet2015 and lai-avh15c1 cmip6 test cases. Each carries a committed CMEC bundle and a manifest; native blobs are published separately through the gated mint workflow. ILAMB executions now also declare their scalar CSV and netCDF outputs in the CMEC output bundle so they are persisted with the results and can be replayed. (#738)

• Extended the ILAMB cmip6 regression-baseline migration to the per-package Framework B layout with nine more test cases:
  gpp-wecann, mrro-lora, csoil-hwsd2, nbp-hoffman, snc-esacci, burntfractionall-gfed, emp-gleamgpcp2.3, thetao-woa2023-surface and so-woa2023-surface. (#743)

• ref test-cases runs now materialise their outputs into an inspectable, gitignored output slot
  (<case>/output/<label>/) holding the curated native files, the rebuilt committed bundle, and a source stamp,
  so you can see and diff exactly what a run produced.
  Added a build verb that rebuilds the committed bundle from an existing slot without re-executing the diagnostic,
  a mint --from-replay option that re-authors a baseline from the stored native instead of re-running it,
  and a --label option that keeps several runs side by side for comparison. (#748)

• Added a --format json option to ref datasets list, ref providers list, ref executions list-groups, and ref config list for machine-readable output with full, untruncated identifiers and paths. (#765)

• Added ref config init, get, set, unset, edit, and validate commands to make configuration onboarding and management easier from the CLI. (#768)

• Added a five-minute Quick Start guide
  and a new model-vs-observation diagnostic to the example provider,
  global-sst-bias,
  which compares modelled global-mean sea surface temperature (tos, area-weighted by areacello)
  against the HadISST-1-1 observations and emits scalar metrics, the model/reference/bias time series, and figures.
  A curated quickstart data registry provides the single reference dataset it needs,
  avoiding the multi-gigabyte obs4REF download. (#769)

• The CI coupling gate now couples each regression baseline to its diagnostic's Diagnostic.version.
  A test case fails the gate when the diagnostic's in-code version no longer matches the version recorded
  when its baseline was minted, so an execution-affecting change can no longer slip through as a green replay
  against a stale committed bundle.
  The new author-declared diagnostic_version field lives in the test case manifest.json (schema version 2),
  and the new ref test-cases migrate-manifests command backfills it across existing baselines.
  A legitimate revert is still permitted: lower the diagnostic version and re-mint the baseline. (#770)

Improvements

• ILAMB reference (observational) series are now tagged with their source via the existing
  reference_source_id dimension (for example reference_source_id="WangMao"),
  so a reference series is self-identifying without adding a new schema field. (#743)
• Committed regression baseline bundles (series.json and diagnostic.json) now round floating-point values to seven significant figures when written, giving stable, reviewable bytes that no longer churn between local and CI runs while staying well within the regression comparison tolerance. (#744)
• Reduced database size by deduplicating the index of series metric values into a shared index_axis table, referenced by each series rather than stored inline on every row.
  A database migration backfills existing data losslessly; on SQLite, run VACUUM afterwards to release the freed space on disk. (#747)
• Sped up ref executions reingest by eager-loading datasets and batching the database writes, so bulk reingests now scale with the number of executions rather than the number of data files. (#750)

Bug Fixes

• Regression series.json baselines are now written with a stable, dimension-sorted order. Previously a diagnostic could emit its series in a platform-dependent order, causing a committed bundle minted on one platform to falsely fail the regression gate on another even when every value matched within tolerance. (#738)
• Selecting the latest dataset version now compares version numbers numerically, so versions like v10 are correctly preferred over v2 (previously the older version could be chosen). (#739)
• Fixed a bug where a diagnostic whose results failed to ingest was incorrectly recorded as successful with no metric values; such executions are now marked failed and retried on the next solve. (#740)
• ILAMB regression output.json bundles are now written with a stable, sorted block order
  (the plot and data sections are sorted by filename),
  so a baseline regenerated on one machine reproduces byte-for-byte on another
  instead of reordering and producing a spurious diff. (#743)
• Fixed ref executions reingest reporting a reingest as...

v0.14.7

Changelog

Bug Fixes

• Fixed ref solve --timeout 0 discarding completed work instead of waiting for it.
  A timeout of 0 (or any non-positive value) now means "wait with no time limit",
  so executions are collected, copied to the results directory, and ingested as expected;
  previously they ran but their outputs were left in the scratch directory.
  Use --no-wait to queue executions and exit immediately,
  and recover any orphaned scratch outputs from an earlier run with ref executions reingest --include-failed. (#735)
• Fixed ref executions reingest failing for ESMValTool diagnostics.
  Reingest copies an execution's outputs to a new directory, but the absolute paths recorded in ESMValTool's provenance files still pointed at the original location,
  which caused an error that silently rolled back the reingest.
  These embedded paths are now re-pointed at the new output directory so the results are reingested as expected. (#736)

Changes

• 5b239c3 Bump version: 0.14.6 → 0.14.7
• 12be1fa refactor(reingest): reuse core rewrite_tree for path rewriting
• 3807459 docs: add changelog for #736
• 6f35062 fix(reingest): rewrite embedded absolute paths when copying execution outputs
• 6eba410 chore: clean up docs
• 08a55e0 docs: add changelog for #735
• e4ada47 fix(solve): wait and collect results when --timeout 0
• 66acade chore(deps): bump bleach from 6.3.0 to 6.4.0
• d04fb98 chore(deps): bump cryptography from 46.0.7 to 48.0.1
• 0f8f00d chore(deps): bump pyjwt from 2.12.0 to 2.13.0

v0.14.6

Changelog

Bug Fixes

• Fixed loading the default configuration in environments where the user cache directory cannot be written (for example, read-only or restricted HPC systems).
  Previously, running a ref command without an existing configuration file could fail with Error loading configuration because the default ignore-datasets file could not be cached.
  The REF now degrades gracefully and continues without the cached ignore-datasets file. (#734)

Changes

• 306f5ea Bump version: 0.14.5 → 0.14.6
• fd95be2 docs: add changelog for #734
• 0142b9f fix(config): don't fail config load when ignore-datasets cache is unwritable

v0.14.5

Changelog

Bug Fixes

• Made ESMValTool regression baselines deterministic by giving each captured run a stable execution directory instead of a timestamped one, so re-running a test case no longer churns the committed output. (#714)
• Fixed fetching datasets from ESGF failing with Must have equal len keys and value when setting with an iterable under pandas 3.0,
  by running the intake-esgf catalogue build with the legacy object-string dtype. (#726)

Trivial/Internal Changes

• #719, #720, #723, #725

Changes

• da4461c Bump version: 0.14.4 → 0.14.5
• 27cdeb4 docs: cleanup
• 31eca3d ci: stop per-job coverage xml from enforcing the fail_under gate
• 8633217 docs: update changelog for #725
• 999ff88 ci: split integration tests into a parallel job and cap PR concurrency
• 250d55d fix(esgf): build intake-esgf catalogue with legacy object-string dtype
• a9f3df2 docs: add changelog entry for #725
• da3f60c ci: parallelise tests-providers into a per-provider matrix
• f2be958 chore: cleanup
• cfea48b docs: add changelog entry for #723
• 48bd0b5 test: silence deprecation warnings and speed up the climate-ref suite
• a75c835 fix(core): address PR review comments on regression primitives
• 468f823 chore: remove unneeded annotations
• d2e336c chore: remove unneeded annotations
• 1bd11e3 chore: pull out the extra layer of tolerance redirection
• 0733172 chore(deps): bump tornado from 6.5.5 to 6.5.6
• 6a66af6 fix(core): apply review fixes to regression-baseline primitives
• 9455f7b test(core): expand output-file coverage and apply review fixes
• c83e316 chore(deps): bump codecov/codecov-action in the github-actions group
• c2a7ccc docs(changelog): add fragment for native regression-baseline primitives
• 9cabe73 feat(core): add native regression-baseline primitives (RFC 0005)
• 3a0ebd4 docs(changelog): add fragment for output-file operations refactor
• 6936a27 refactor(core): extract shared output-file operations into climate_ref_core.output_files
• 4765fcf chore: clean ups
• 835a371 docs(changelog): add fragment for #714
• cacd200 fix(esmvaltool): stabilise execution dir for regression capture
• 6ddfa89 chore(deps): bump gitpython from 3.1.47 to 3.1.50
• 8065f73 chore(deps): bump astral-sh/setup-uv in the github-actions group

v0.14.4

Changelog

Improvements

• Added GPCP dataset version to 3.3 for the PMP precipitation climatology reference dataset. (#672)
• Updated GPCP dataset version to 3.3 for the PMP precipitation climatology reference dataset. (#684)
• Removed the pandas upper bound and fixed pandas 3 compatibility in catalog and ILAMB result handling. (#708)

Bug Fixes

• Updates the PMP provider data directory (#685)
• Fixed a KeyError: 'branded_variable' when solving diagnostics against CMIP7 datasets.
  The branded_variable facet is now reconstructed when a data catalog is loaded from the database,
  so it is available to data requirement filters. (#712)

Improved Documentation

• Added a Zenodo DOI badge to the README and documentation landing page so users can easily find how to cite the REF. (#634)
• Added a getting-started guide for modelling centres that explains how to evaluate local or pre-publication CMOR-compliant model output with Climate-REF, including deployment options, accessing results, and a suggested first run. (#709)

Trivial/Internal Changes

• #706

Changes

• d26bdce Bump version: 0.14.3 → 0.14.4
• dfee0e3 refactor: declare derived catalog columns on the adapter
• b191910 docs: add changelog fragment for PR 712
• 9d43fee fix: derive branded_variable when loading CMIP7 catalog from DB
• baa358b docs: add Zenodo DOI badge to README and docs landing page
• 673b6fa docs: add changelog fragment for modelling centre guide
• 5206586 docs: add modelling centre onboarding guide
• cf96a14 test: Add tests for the helper
• 42cea87 chore: rename changelog fragment to PR 708
• 182f088 chore: remove pandas upper bound and fix pandas 3 compatibility
• 3b9973c docs(changelog): add fragment for series regression check
• 4995b17 test(core): add series.json regression comparison
• ef7ab30 chore: update the regression.
• aebe346 chore: increment version and fix test
• 22dfaeb feat: add example package to deployment
• b2e1a76 fix: tests
• 7b76f57 fix: test
• 2f41764 docs: Changelog
• 1d17468 fix: correct the data path for the new pmp cache
• 9ab3365 Update GPCP dataset version in changelog
• a69c4c0 Update reference dataset from GPCP-Monthly-3-2 to GPCP-3-3
• 3b6d2e1 chore(deps): bump pymdown-extensions from 10.21 to 10.21.3
• 368334f chore(deps): bump idna from 3.11 to 3.15
• 5734953 Update 672.improvement.md
• 5a5812a add changelog
• 1f72461 Add new climatology dataset entry
• cbbd022 chore(deps-dev): bump the python-dependencies group with 4 updates

v0.14.3

Changelog

Bug Fixes

• Fixed a race condition when trying to create the same ExecutionGroup concurrently.
  Database.get_or_create and Database.update_or_create now wrap the INSERT in a SAVEPOINT
  and re-fetch the winning row on conflict instead of aborting the transaction. (#679)

Changes

• c07e141 Bump version: 0.14.2 → 0.14.3
• 10c9dd0 fix(database): address PR review comments
• 3aae695 docs: changelog
• 9f6d74c docs: changelog for #679
• c60b8ed fix(database): make get_or_create / update_or_create race-safe

v0.14.2

Changelog

Bug Fixes

• Changed the behaviour for register_dataset treatment of files absent from the current ingest slice as kept-in-place with a warning, instead of raising NotImplementedError. (#677)

Changes

• c1d9ce2 Bump version: 0.14.1 → 0.14.2
• 787e62b docs: changelog
• 82fbd00 fix(ingest): keep files absent from current ingest, warn instead of abort

v0.14.1

Changelog

Features

• Added a --chunk-size option to ref datasets ingest (CMIP6 and CMIP7) that streams the catalog in directory-aligned batches instead of loading the whole archive into memory at once.
  Peak memory is now bounded by chunk_size rather than by the total number of files in the input tree. (#674)

Improvements

• Skip the redundant per-dataset validate_data_catalog call inside register_dataset. The production ingest path already validates the catalog (and each streamed chunk) once up-front, so the inner re-validation only duplicated work. Cuts ~20% off ingest wall time. (#675)

Bug Fixes

• Stopped re-ingesting unchanged CMIP6 files on every run.
  Previously, ref datasets ingest reported every file as updated and emitted a "Updating file metadata" warning per file even when nothing on disk had changed, because the file-metadata comparison treated a str loaded from the database as unequal to a freshly parsed cftime.datetime. Re-ingesting an unchanged directory is now a no-op. (#673)

Changes

• aa18732 Bump version: 0.14.0 → 0.14.1
• 761a673 refactor(ingest): raise RefException from the register_dataset guard
• 8f86f10 perf(ingest): keep cheap per-slice metadata-consistency guard
• 3b06df2 feat(ingest): extend streaming ingest to CMIP7
• 190dd2b chore: cleanup
• 29c2013 chore(changelog): rename news fragment to PR number
• 7133baf perf(ingest): drop redundant per-dataset validate_data_catalog call
• e78a3ab test: add test coverage
• f4ea29a docs: address PR review comments
• 8042735 docs: reformat
• baded9b chore(changelog): rename news fragment to PR number
• 9b2438d feat(ingest): stream catalog in chunks to bound ingest memory
• cd00b64 docs: add changelog for PR #673
• 0823e3c fix(ingest): stop re-ingesting unchanged CMIP6 files every run

v0.14.0

Changelog

Features

• Write the diagnostic version to the database.

  Bumping a diagnostic's version now creates a fresh execution group
  (preserving the prior version's group and results) instead of overwriting it,
  and each execution is stamped with the provider version that produced it. (#667)

Improvements

• Reworked the layout of execution output directories.
  New executions now write to <provider>/<diagnostic>/<group_short>/<execution_id>/
  instead of <provider>/<diagnostic>/<dataset_hash>/,
  so reruns of the same diagnostic group no longer overwrite earlier outputs.
  Existing rows on disk continue to resolve through their stored Execution.output_fragment. (#655)
• Added diagnostic versioning read-path foundations.
  New database columns track the diagnostic version for each execution group,
  the promoted version for each diagnostic, and the provider version for each execution.
  Default queries now return only results at the promoted version, and the ref executions stats command reflects the same filter. (#665)

Bug Fixes

• ref datasets ingest now exits with a non-zero status when one or more input directories fail to ingest,
  so cron- and Kubernetes-based deployments can detect failures.
  The CMIP6, CMIP7, and obs4MIPs adapters also now tolerate individual files with missing DRS components:
  those files are logged and skipped instead of aborting the whole batch. (#668)

Changes

• 1d1bfd4 Bump version: 0.13.2 → 0.14.0
• d018f9a fix(ingest): exit non-zero on failure and tolerate per-file DRS errors
• fa30b96 chore: tweak changelog
• ef3eaaf docs: add changelog for PR #667
• 6da4003 feat(versioning): activate diagnostic versioning write path
• 22ed810 chore(deps): bump urllib3 from 2.6.3 to 2.7.0
• ff35ab7 chore: clean up comments
• cc21064 test(versioning): tighten stats filter assertion
• 0fffe26 chore(changelog): add news fragment for PR #665 (diagnostic versioning read-path)
• dfcb0a3 feat(versioning): diagnostic versioning read-path foundations
• 22b3d79 fix(fragment): address PR review feedback on output-fragment helpers
• eebc320 docs(fragment): rename _TEMP_DIAGNOSTIC_VERSION and drop future-PR comment
• 64e2c71 refactor(solver): extract assign_execution_fragment helper and add safety cap
• 6da6249 docs: rename changelog placeholder to 655.improvement.md
• acc8972 refactor(solver): switch execution output layout to //<group_short>/<execution_id>

v0.13.2

Changelog

Features

• Added support for Python 3.14. (#625)

Improvements

• Add extra options for users to ingest options other than base to parsl functions (#651)
• Updated ESMValCore to v2.14.0 and ESMValTool to 2.15.0.dev15+gdead90ca8. (#652)

Bug Fixes

• Fixed ref test-cases fetch aborting the entire run when a single test case could not be parsed (for example because of a PermissionError on a cached CMIP6 file); the failing test case is now logged as a warning and the loop continues. (#639)
• Fixed ref solve periodically reporting executions as never finishing. Stuck executions left in the in-progress state by a crashed worker, walltime kill, or hung diagnostic are now reaped and retried on the next solve, the CLI --timeout default has been raised to 6 hours to match the worker time limit, and per-task timeouts in the local executor cancel hung diagnostics instead of blocking the whole pool. (#641)

Improved Documentation

• Updated CITATION.cff with extra author names and orcid ids. (#636)

Trivial/Internal Changes

• #619, #637, #638

Changes

• e916682 Bump version: 0.13.1 → 0.13.2
• a604ada Revert "Merge pull request #654 from Climate-REF/ci/ilamb-debug-workflow"
• bbd7360 chore(deps): bump mistune from 3.2.0 to 3.2.1
• f6bfd45 chore(deps): bump mako from 1.3.11 to 1.3.12
• 9567a9f chore(deps-dev): bump pip from 26.0.1 to 26.1
• 557738a chore(deps): bump jupyter-server from 2.17.0 to 2.18.0
• 369f4bf docs: add changelog for #654
• 411e3e1 ci: add standalone ILAMB debug workflow
• 97126a8 Add changelog
• 5e04639 Update ESMValCore to v2.14.0 and ESMValTool to 2.15.0.dev15+gdead90ca8
• d9d02ea add changelog
• a654697 add executor.config.extra_* so options other than base can be injected to parsl functions
• 06a745f chore(deps-dev): bump jupyterlab from 4.5.6 to 4.5.7
• 42fd0e8 chore(deps-dev): bump notebook from 7.5.5 to 7.5.6
• dcdef80 Update CITATION.cff
• 8e1cf40 chore: fix trailing line break
• 87ad343 Update CITATION.cff with additional authors and ORCID IDs
• 8d3bd97 docs: address PR review comments
• a69a7ec fix(solver): detach execution before commit to prevent autobegin clash
• eb60774 docs: add changelog entry for #641
• 7ac8da1 refactor(executor): consolidate failure-recording helpers and cache solver lookups
• dab0174 test(executor): add e2e failure-mode coverage for LocalExecutor
• 7df2750 fix(executor): make solve resilient to stuck executions
• 26296ac chore: pin climate_ref_client as known-third-party for ruff isort
• ed4654c docs(changelog): add 639.fix.md
• 8deaf46 fix(test-cases): don't abort fetch loop when one test case fails
• f8f04c8 chore: Use new docker image tags
• d3748d9 docs(changelog): add 638.trivial.md
• f1ba085 test(dataset_registry): isolate cache-dir tests from REF_DATASET_CACHE_DIR
• 332f281 docs: Changelog
• 09b1007 chore: update GHA hooks
• 8843752 chore: update pre-commit versions
• 972c270 Update CITATION.cff to version 1.2.1
• 52525fe chore(deps): bump gitpython from 3.1.46 to 3.1.47
• 739f61a chore(deps): bump python-dotenv from 1.2.1 to 1.2.2
• 5b5f624 chore(deps): bump nbconvert from 7.17.0 to 7.17.1
• 88b67c6 chore(deps-dev): bump the python-dependencies group across 1 directory with 4 updates
• 86f74ef chore(deps): bump mako from 1.3.10 to 1.3.11
• 2d5d9bf chore(deps): bump softprops/action-gh-release
• ef5e3de docs: add changelog for #619
• d8d1885 build(docs): swap mkdocs-typer for mkdocs-typer2
• 0b92945 chore(deps): bump pygments from 2.19.2 to 2.20.0
• 6082685 chore(deps): bump pillow from 12.1.1 to 12.2.0
• 5c7aea6 fix(types): annotate esgf_data_catalog_trimmed result dict
• 020f928 chore(deps-dev): bump the python-dependencies group across 1 directory with 7 updates
• 6eb413a chore: update lock
• 7833bc4 Revert "chore: add Python 3.14 support across packages and CI"
• 576e86b docs: add changelog entry for PR #625
• a086282 chore: add Python 3.14 support across packages and CI
• dd29e46 chore: remove windows from the pypi test