Add netCDF to CF-compliant Zarr data pipeline - #13
Open
annefou wants to merge 3 commits into
Open
Conversation
Adds the scripts that turn the seven NorESM2/BLOM monthly-mean netCDF files into a single cloud-optimised Zarr store for the FAIR2Adapt dashboard. - sharepoint_login.py / sharepoint_fetch.py: retrieve the source netCDF from the NERSC share, which is reachable only via email one-time passcode - fetch_blom_grid.py: recover the 2-D tripolar coordinates missing from the source files, verified against a local field before use - make_zarr.py: merge the seven files, chunk one map per chunk, compress with zstd, and correct the CF metadata The source standard_name values 'Ocean temperature' and 'Ocean salinity' are not in the CF standard name table; sst, sss and the U-236 tracers carried none. cell_methods and time_bnds are added since these are monthly means. No credentials are included; tokens are passed via CLI or environment.
Publishes the dataset as a FAIR Digital Object in ROHUB, described with the Climate Change Adaptation RO-Crate profile and I-ADOPT variable decomposition. - create_rohub_ro.py: creates or repairs the Research Object, idempotent and self-verifying; re-running converges on the same result - load_rohub_triples.py: loads a TTL into a ROHUB annotation triple by triple - metadata/: the CCA profile crate, plus the three annotation TTLs actually loaded into ROHUB (profile conformance, I-ADOPT decomposition, AIDA claims) The I-ADOPT decomposition follows Barbara Magagna's definitions in mabablue/I-ADOPT-Variables, using her vocabulary URIs (NERC S06 for practical salinity, QUDT for thermodynamic temperature, EnvThes for water). Four ROHUB API behaviours are documented in create_rohub_ro.py because each costs an hour to rediscover: annotations do not ingest uploaded TTL, ros_create returns an object rather than a dict, folder names are template-specific, and reads are eventually consistent. No credentials are included; ROHUB login and the dashboard token both come from the environment.
Barbara Magagna's suggestion: potential temperature is vocab.nerc.ac.uk/collection/P14/current/GVAR0653/ (prefLabel 'Potential Temperature'), which matches temp's CF standard name sea_water_potential_temperature. sst keeps QUDT thermodynamic temperature, following her Ocean/SST.ttl. Worth revisiting: BLOM is isopycnic, so its surface field is potential temperature too, while her SST.ttl describes radiative skin temperature from satellites. Applied to the live Research Object by deleting and reloading the I-ADOPT annotation -- the ROHUB client has no per-triple delete.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the scripts that turn the seven NorESM2/BLOM monthly-mean netCDF files into a single cloud-optimised Zarr store for the FAIR2Adapt dashboard, extending the published store from 2 variables (SST, temperature) to all 7.
Scripts
sharepoint_login.pysharepoint_fetch.pyfetch_blom_grid.pymake_zarr.pyNo credentials are included — tokens are passed by CLI argument or environment variable, and session state is written outside the working tree.
What the conversion fixes
Geolocation. The source files carry no coordinates for their 385×360 grid:
latis a 1-D zonal-diagnostic axis, andcell_measures: "area: parea"points at apareavariable absent from every file. The 2-Dlongitude/latitudeare recovered from the published store and verified bit-identical against a local field before use —fetch_blom_grid.py --verifyrefuses to write a grid that does not match.CF compliance. Checked against CF standard name table v94 (5071 names):
tempOcean temperature❌ not in CFsea_water_potential_temperaturesalnOcean salinity❌ not in CFsea_water_salinitysstsea_surface_temperaturessssea_surface_salinityU236GF/SF/LHcell_methods = "time: mean"is set on every variable, andtime_bndsspans each averaging month, as CF requires bounds oncecell_methodsdeclares an average.Metadata. Only
tempandsstcarried global attributes; the other five files had none. The ACDD block is promoted to dataset level and applied uniformly, withgeospatial_*computed from the grid.Chunking. Source chunks
(1, 27, 193, 180)mean one rendered map costs ~16 chunk reads. The store uses(time=1, sigma=1, y=385, x=360)— one chunk per map, ~554 KB, against ~55 MB for the equivalent read in the current published store. Compression is zstd-5 + shuffle, replacing zlib-1.Verification
All 7 variables were checked bit-identical to the source netCDF across sampled timesteps (t=0, 53, 107) after conversion. Store is 6.1 GB, 28,852 objects.
Needs confirmation before a DOI is minted
tempis mapped tosea_water_potential_temperatureon the basis that BLOM is isopycnic and carries potential temperature. If it is conservative temperature, this standard name is wrong — worth confirming with Yanchun He. Flagged in the code and inarctic-isotopes/README.md.The U-236 tracers deliberately carry no
standard_name: CF has no term for them and inventing one is worse than omitting it. They should be described through I-ADOPT in the RO-Crate, per the CCA-Crate profile.