Add remaining ILAMB cmip6 regression baselines; stabilise output.json and tag reference series#743
Merged
Merged
Conversation
Completes the cmip6 ILAMB regression-baseline batch (RFC 0005 PR-6a),
adding committed bundles for the diagnostics beyond the initial
mrsos/gpp-fluxnet2015/lai batch:
gpp-wecann, mrro-lora, csoil-hwsd2, nbp-hoffman, snc-esacci,
burntfractionall-gfed, thetao-woa2023-surface, and so-woa2023-surface.
Generated locally via `ref test-cases run --fetch`.
Native blobs are not yet minted (native={});
the gated regression-mint workflow populates them.
amoc-rapid and emp-gleamgpcp2.3 are intentionally excluded:
their cmip6 runs fail for data/diagnostic reasons
(missing GPCP-2-3 obs reference; an amoc corrcoef shape mismatch)
unrelated to the baseline machinery.
Codecov Report✅ All modified and coverable lines are covered by tests.
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
lewisjared
temporarily deployed
to
native-baselines
GitHub Actions
Inactive
Its earlier run failed only because the GPCP-2-3 obs4REF reference was not present locally, not because of a diagnostic defect; with the reference fetched the case now passes and seeds a baseline. Updates the changelog fragment to include it and drops the committed-bundle/native-layer note.
lewisjared
changed the title
Sort the plot/data blocks of the ILAMB CMEC output bundle by filename so the committed output.json reproduces byte-for-byte across machines. The *.png glob previously leaked filesystem order into the bundle, so a baseline regenerated locally diffed against the same one minted in CI. Tag ILAMB reference (observational) series with reference_source_id (the obs source already in scope, e.g. "WangMao"), so a reference series self-identifies via the existing schema rather than only as source_id="Reference". Bump ILAMBStandard.version 1 -> 2 so the solver creates a fresh execution group for the changed results rather than reusing v1. Regenerate all committed cmip6 ILAMB baselines under the new code and bump test_case_version 1 -> 2 for the three already-minted cases (gpp-fluxnet2015, mrsos-wangmao, lai-avh15c1). The coupling gate replays them against their native baselines and they reproduce within tolerance.
lewisjared
changed the title
Re-mint all 12 runnable cmip6 ILAMB cases so the native snapshot is regenerated under the deterministic output.json ordering and reference_source_id tagging. The committed bundles are unchanged (the fresh runs reproduce them byte-for-byte), so only the manifests' native blocks are updated. The coupling gate replays all 12 cases against the new blobs and they reproduce (0 drifted).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Completes the cmip6 ILAMB regression-baseline migration begun in #738 (which seeded the first three cases —
mrsos-wangmao,gpp-fluxnet2015,lai-avh15c1), and stabilises the ILAMB output bundle so baselines reproduce across machines.Baselines. Adds committed baselines for the remaining nine runnable cmip6 ILAMB diagnostics:
gpp-wecann,mrro-lora,csoil-hwsd2,nbp-hoffman,snc-esacci,burntfractionall-gfed,emp-gleamgpcp2.3,thetao-woa2023-surface, andso-woa2023-surface. Each consists of the committed CMEC bundle (series.json,diagnostic.json,output.json), the inputcatalog.yaml, and amanifest.json.Determinism fix. The CMEC
output.jsonplot/data blocks were written in filesystem-glob order, so a baseline regenerated locally diffed against the same one minted in CI. They are now sorted by filename, so the bundle reproduces byte-for-byte regardless of platform.Self-identifying reference series. ILAMB reference (observational) series are now tagged with
reference_source_id(e.g.reference_source_id="WangMao") via the existing schema, so a reference series is identifiable beyondsource_id="Reference".Versioning.
ILAMBStandard.versionis bumped 1 → 2 (the results changed shape). All committed cmip6 baselines were regenerated under the new code; the three already-minted cases (gpp-fluxnet2015,mrsos-wangmao,lai-avh15c1) carry atest_case_version1 → 2 bump, and the coupling gate replays them against their native baselines and they reproduce within tolerance.Only
amoc-rapidis left out of the cmip6 batch: it hits anumpy.corrcoefshape mismatch (model 130 vs RAPID reference 129 timesteps) inside the ILAMB amoc analysis, unrelated to the baseline machinery. The cmip7 cases remain deferred to the in-flight CMIP7 compatibility work (#671, #686, #702, #704).Checklist
Please confirm that this pull request has done the following:
test_validate_test_case_regressionand the ILAMB unit testschangelog/—743.feature.md,743.fix.md,743.improvement.md