Skip to content

feat(esm_catalog): PR-A1b-1 — scan ingest + NetCDF#1485

Open
siligam wants to merge 9 commits into
esm-catalog/pr-a1a-stac-foundationfrom
esm-catalog/pr-a1b1-scan-netcdf
Open

feat(esm_catalog): PR-A1b-1 — scan ingest + NetCDF#1485
siligam wants to merge 9 commits into
esm-catalog/pr-a1a-stac-foundationfrom
esm-catalog/pr-a1b1-scan-netcdf

Conversation

@siligam

@siligam siligam commented Jun 13, 2026

Copy link
Copy Markdown
Contributor

PR-A1b-1 — scan ingest + NetCDF (Miguel's "scan run dirs" unit)

Provenance: scan/context.py, scan/detect.py, scan/namelist.py, scan/netcdf.py, and scan/upath.py are ported from Paul Gierz's esm-tools-plus/simcat/pgierz-workbench prototype branch; scan/ingest.py is new for this decomposition.

Third slice of #1473's decomposition. Adds the scan/ingest layer that turns files into in-memory STAC (collections + items), and completes the cross-layer rewiring that the STAC foundation (#1484) deferred.

Stacked on #1484 (esm-catalog/pr-a1a-stac-foundation). Review #1483#1484 → this. Retargets to release as the stack merges.

What's here (+1,814 lines)

  • scan/upath.py — remote-FS helpers; re-exports parse_uri/to_uri from esm_catalog.uri
  • scan/namelist.py — Fortran namelist parsing (+ get_namelist_config_path, moved here from the stac layer)
  • scan/netcdf.py — NetCDF scanner (xarray) → STAC metadata
  • scan/detect.py — format dispatch by suffix + magic bytes (GRIB/ECHAM lazy — they land in PR-A1b-2)
  • scan/context.py — collection-context resolution using the neutral CollectionContext; scan_all_namelists
  • scan/ingest.pyscan_tree(root): the storage-free pipeline (list → detect/scan → resolve → STAC)
  • esm-catalog scan <path> [--output FILE] CLI command
  • focused unit tests (27 total in the suite, all green on 3.9 + 3.12)

Rewiring completed (deferred from #1484)

  • scan/context.py uses from esm_catalog.context import CollectionContext (no local dataclass)
  • CollectionContext.collection_title re-added and now honored by make_collection
  • get_namelist_config_path reintroduced in the scan layer
  • make_collection(ctx, namelists=, fesom_info=) called with the new pure signature; ctx.namelists_by_component populated before item-building

Scope notes

  • Storage-free: scan_tree builds STAC in memory; DuckDB persistence is PR-A2.
  • Cycle stays broken: scan → stac only; the A1a guard test still passes.
  • GRIB/ECHAM scanners follow in PR-A1b-2.

Plan: PR-A1b plan

@siligam
siligam force-pushed the esm-catalog/pr-a1a-stac-foundation branch from 8d58172 to 3b5b80c Compare June 17, 2026 12:12
@siligam
siligam force-pushed the esm-catalog/pr-a1b1-scan-netcdf branch from 3cb83f3 to e853cac Compare June 17, 2026 12:12
@siligam
siligam force-pushed the esm-catalog/pr-a1a-stac-foundation branch from 3b5b80c to 541592a Compare June 17, 2026 15:53
@siligam
siligam force-pushed the esm-catalog/pr-a1b1-scan-netcdf branch from e853cac to bdc93bd Compare June 17, 2026 15:54
@siligam
siligam force-pushed the esm-catalog/pr-a1a-stac-foundation branch from 541592a to 7e5af57 Compare June 17, 2026 16:34
@siligam
siligam force-pushed the esm-catalog/pr-a1b1-scan-netcdf branch from bdc93bd to 0b538f0 Compare June 17, 2026 16:34
@siligam

siligam commented Jun 17, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto the corrected PR-0 base (#1483) — test relocation to tests/test_esm_catalog/, restored dedicated CI workflow at the new path, and a couple of CI fixes (doctest-collection exclusion, eval_type_backport for pydantic on Python 3.9). No change to this PR's own diff/scope. CI is green.

@siligam
siligam force-pushed the esm-catalog/pr-a1a-stac-foundation branch from 7e5af57 to d995632 Compare June 18, 2026 12:42
@siligam
siligam force-pushed the esm-catalog/pr-a1b1-scan-netcdf branch from 0b538f0 to 9be7194 Compare June 24, 2026 13:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant