feat(esm_catalog): PR-A1a — STAC data-model foundation

[siligam]

PR-A1a — STAC data-model foundation (+ scan⇄stac cycle-break)

Provenance: hpc/, the STAC extensions (paleo, datacube, contacts, registry, hpc), and stac/item.py are ported from Paul Gierz's esm-tools-plus/simcat/pgierz-workbench prototype branch; context.py/uri.py (breaking the scan⇔stac import cycle) are new for this decomposition.

Second slice of #1473's decomposition. Lands the STAC data model so it stands alone — reviewable and testable without the scan, storage, or api layers present.

Stacked on 1483 (esm-catalog/pr-0-scaffold). Review that first; this PR's diff shows only the STAC foundation. Will be retargeted to release once PR-0 merges.

What's here (+1,502 lines, 21 files)

• stac/ — collection.py, item.py, and extensions (datacube, contacts, hpc, paleo, namelist, registry)
• hpc/ — dependency leaf (storage-state detection)
• uri.py (new, neutral) — parse_uri/to_uri, moved out of scan/upath.py
• context.py (new, neutral) — CollectionContext, carrying pre-scanned namelists_by_component
• focused unit tests + a cycle-guard test

The cycle-break (the point of this PR)

stac/ previously reached into scan/ through three lazy imports — a hidden dependency cycle that hurts reviewability/maintainability. This PR removes all of it:

• URI helpers → neutral esm_catalog.uri
• CollectionContext → neutral esm_catalog.context
• make_collection and add_namelist_item_extension are now pure — they receive pre-scanned namelist data instead of scanning. The scanning itself moves to the scan layer (next PR, A1b).

tests/test_no_scan_dependency.py keeps it broken: an AST test fails if any stac/ file imports esm_catalog.scan, plus a runtime test that reloads the stac modules with scan imports blocked.

Tests

15/15 pass (pytest src/esm_catalog/tests), Python 3.12.

Context

Full plan: decomposition design. Follows: PR-A1b (scanners), which reintroduces the namelist scanning and scan→stac (one-directional) usage.

[siligam]

Rebased onto the corrected PR-0 base (#1483) — test relocation to tests/test_esm_catalog/, restored dedicated CI workflow at the new path, and a couple of CI fixes (doctest-collection exclusion, eval_type_backport for pydantic on Python 3.9). No change to this PR's own diff/scope. CI is green.

[mandresm]

Hi @siligam,

I just reviewed hpc/detect.py. I am suggesting some design changes to reuse some of our detect infrastructure and also to make the functions location-agnostic.

I think I'll do more of these "incomplete" reviews, so that you can start working on certain things.

[mandresm]

This belongs to the configs/machine/albedo.yaml

[mandresm]

This belongs to the configs/machine/levante.yaml

[mandresm]

You can detect the machine and which yaml file to load with:

https://github.com/esm-tools/esm_tools/blob/72019da9a0cddf4fa22a1a94e8577145dbbdf2bc/src/esm_parser/esm_parser.py#L2554-2609

[mandresm]

We need an strategy for tape archives that are not registered in the config/machine yamls. One could simply have a storage dir in the configs with 1 yaml file per storage with the following info:

configs/storage/hpss.yaml

storge:
    storage_type: "hpss"
    state: offline
    recall_time_estimate: 300
    path_str: "/arch/|/hpss/"

[pgierz]

I agree. How about attaching it to the computer and having some useful objects for Storage with attributes and methods like get/put?

computer:
    name: albedo
    storage:
        - cls: disk
          name: albedo-work
          type: pfs
          ... : lustre  # I don't know lhs here. Maybe 'provider' (others would be gpfs, gifs, bgfs, and one base class generic-parallel-filesystem)
          state: online
          path_prefix: "/albedo/work"
          comment: "... something we can use in functions consuming a SimulationStorage object)

Then we have objects in Python like:

@dataclass
class DiskSimulationStorage:
    name: str
    ...

class ScoutFSTapeSimulationStorage:

    def get(p: UPath) -> UPath: ...

This idea could be expanded...

[pgierz]

Sorry, my example above is missing the specific part that makes this interesting: computer gets an attribute called storage, which is a list of things. One of those things is an object representing the tape system that this particular HPC can communicate with.

[mandresm]

I agree. How about attaching it to the computer and having some useful objects for Storage with attributes and methods like get/put?

computer:
    name: albedo
    storage:
        - cls: disk
          name: albedo-work
          type: pfs
          ... : lustre  # I don't know lhs here. Maybe 'provider' (others would be gpfs, gifs, bgfs, and one base class generic-parallel-filesystem)
          state: online
          path_prefix: "/albedo/work"
          comment: "... something we can use in functions consuming a SimulationStorage object)

Then we have objects in Python like:

@dataclass
class DiskSimulationStorage:
    name: str
    ...

class ScoutFSTapeSimulationStorage:

    def get(p: UPath) -> UPath: ...

This idea could be expanded...

I find this idea a bit over engineer for what we currently need. Additional features can always be added in the future but we need a minimal product before the end of the year and we are very delayed.

I will tweak @pgierz's approach to the following:

machines/albedo.yaml

computer: 
    name: albedo
    [ ... ]

storage:
    albedo-gpfs: # Not a list but a dictionary
        type: gpfs
        state: online
        as_few_parameters_as_possible: we can add more parameters later on when needed for other functionalities from the requirements

machines/awi-hsm.yaml

storage:
    awi-hsm:
        type: tape
        state: offline

Consider which parameters need to be set dynamically (to be extracted via functions in the catalog module), and which ones can be static (defined in yaml files). My guess is that almost everything can static.

[mandresm]

Sorry, my example above is missing the specific part that makes this interesting: computer gets an attribute called storage, which is a list of things.

I don't like this. Computer should no have storage inside, as storage can be an independent entity from the HPC (e.g. tape).

One of those things is an object representing the tape system that this particular HPC can communicate with.

Yes, but you can also communicate with a tape storage system from somewhere else right? I wouldn't make it an object for now anyway.

[mandresm]

Can you make this function machine agnostic? Specific details can be stored in config/machine/ yamls. See my comments below :)

[siligam]

Hi Miguel, thanks for the review — agreed on the core idea: the facility/storage metadata shouldn't be hardcoded path-string checks in Python, and your configs/storage/hpss.yaml shape is the right pattern for that.

One thing I want to flag before implementing: hooking into esm_parser.determine_computer_and_node_from_hostname() would pull esm_parser in as a runtime dependency of esm_catalog. Part of the goal of this decomposition has been to keep esm_catalog installable and testable standalone (separate setup.cfg, dedicated CI, no esm_parser/esm_tools imports so far). Also, that function answers "what machine is this process running on" via socket.gethostname(), whereas detect_hpc_storage(path) needs to classify an arbitrary path string that may belong to a different host or facility entirely (our current tests exercise paths that don't exist on disk and aren't tied to the runner's hostname) — so it's solving a slightly different problem than what we need here.

What I'd like to do instead: keep the matching path-pattern-based (as it is today), but move the per-machine/per-storage metadata out of Python and into small yaml files that esm_catalog owns and reads directly (plain yaml.safe_load, no esm_parser import) — e.g. esm_catalog/configs/machines/albedo.yaml, esm_catalog/configs/storage/hpss.yaml, matching your proposed shape (path_str/pattern, storage_type, state, recall_time_estimate). That gets us the "machine-agnostic Python, data lives in yaml" outcome you're after without re-coupling esm_catalog to esm_parser.

Does that work for you, or did you have a reason for wanting this to live in the shared configs/machines/ tree specifically (e.g. so the data is shared with the rest of esm-tools)? Happy to go that route too if so, but then we'd want a dependency-free way to read it (plain yaml read with a configurable path, rather than via esm_parser).

[pgierz]

Hi @siligam,

I have a feeling this is going straight to @claude with some additions of your own, so here are instructions you can add to your own prompts:

One thing I want to flag before implementing: hooking into esm_parser.determine_computer_and_node_from_hostname() would pull esm_parser in as a runtime dependency of esm_catalog. Part of the goal of this decomposition has been to keep esm_catalog installable and testable standalone (separate setup.cfg, dedicated CI, no esm_parser/esm_tools imports so far). Also, that function answers "what machine is this process running on" via socket.gethostname(), whereas detect_hpc_storage(path) needs to classify an arbitrary path string that may belong to a different host or facility entirely (our current tests exercise paths that don't exist on disk and aren't tied to the runner's hostname) — so it's solving a slightly different problem than what we need here.

Mixing together esm-parser as an import dependency is fine, the entire system is designed to be together anyway.

I am not sure where @claude got the idea that this needs to be entirely separate and modular, but, it doesn't. @mandresm will remember the nightmare it was trying to keep separate packages for esm-environment, esm-master, etc. I at least do not want to have the AI make us repeat that.

What I'd like to do instead: keep the matching path-pattern-based (as it is today), but move the per-machine/per-storage metadata out of Python and into small yaml files

OK, but...

that esm_catalog owns and reads directly (plain yaml.safe_load, no esm_parser import) — e.g. esm_catalog/configs/machines/albedo.yaml, esm_catalog/configs/storage/hpss.yaml,

...very strong NO!!. We have esm-parser specifically for opening and interpreting our config files. Do not reinvent that.

matching your proposed shape (path_str/pattern, storage_type, state, recall_time_estimate). That gets us the "machine-agnostic Python, data lives in yaml" outcome you're after without re-coupling esm_catalog to esm_parser.

See the comment above. This needs serious human discussion first, and if we go in the direction I imagined, we might need a whiteboard session with the rest of the core developers (@mandresm especially, but I would be interested in seeing @JanStreffing and @seb-wahl's input if they have time)

Does that work for you, or did you have a reason for wanting this to live in the shared configs/machines/ tree specifically (e.g. so the data is shared with the rest of esm-tools)?

It should be in the global data tree because (and I would hope that this should be obvious) we do not want to maintain more than one source of truth. The chance of drift (especially in the age of AI coding) is far too large if we do not have a strict N=1 policy for where program state comes from.

[siligam]

@pgierz Got it, thanks. I'll hold off on the detect_hpc_storage() redesign until after the core-dev discussion, but will go ahead and drop the esm_parser-coupling concern / stop reinventing yaml loading in the meantime.

[mandresm]

Hi @siligam,

I am breaking down this PR in even more pieces. I have created this plan with claude-code: https://github.com/esm-tools/esm_tools/blob/22519379dc96e17e65d8d302c6d969375f0ec139/src/esm_catalog/PR-A1A-STAC-FOUNDATION-PLAN.md

And I have already start implementing that plan, see #1504. This PR would be close without merging once all the steps are completed. I think this is best as we remove the back and forth of me asking for changes to follow the ESM-Tools software conventions and you having to implement them. The good news is that you can also review my PRs so that you get an idea of what my suggestions would be anyway.

[mandresm]

@siligam, replying to some of your comments here:

One thing I want to flag before implementing: hooking into esm_parser.determine_computer_and_node_from_hostname() would pull esm_parser in as a runtime dependency of esm_catalog. Part of the goal of this decomposition has been to keep esm_catalog installable and testable standalone (separate setup.cfg, dedicated CI, no esm_parser/esm_tools imports so far). Also, that function answers "what machine is this process running on" via socket.gethostname(), whereas detect_hpc_storage(path) needs to classify an arbitrary path string that may belong to a different host or facility entirely (our current tests exercise paths that don't exist on disk and aren't tied to the runner's hostname) — so it's solving a slightly different problem than what we need here.

What I'd like to do instead: keep the matching path-pattern-based (as it is today), but move the per-machine/per-storage metadata out of Python and into small yaml files that esm_catalog owns and reads directly (plain yaml.safe_load, no esm_parser import) — e.g. esm_catalog/configs/machines/albedo.yaml, esm_catalog/configs/storage/hpss.yaml, matching your proposed shape (path_str/pattern, storage_type, state, recall_time_estimate). That gets us the "machine-agnostic Python, data lives in yaml" outcome you're after without re-coupling esm_catalog to esm_parser.

Those are all very good points and the solution you are proposing makes sense to me.

Does that work for you, or did you have a reason for wanting this to live in the shared configs/machines/ tree specifically

Yeah, all configs should live in the same configs directory. This is a ESM-Tools convention and is where users expect to have things defined. Also, in some point we will separate configs into a different repo for AWI, so that the backend becomes independent of the AWI configs.

Happy to go that route too if so, but then we'd want a dependency-free way to read it (plain yaml read with a configurable path, rather than via esm_parser).

Dependency-free is not a priority for me. But I don't think we need the esm_parser so we can avoid using the esm_parser here for sure. I can guess we are going to need esm_parser to generate the config from a given runscript in the future, but for now, we can live in an "easier" world where we can specify which yaml we read.

[mandresm]

This is about what @pgierz said:

Mixing together esm-parser as an import dependency is fine, the entire system is designed to be together anyway.

I am not sure where @claude got the idea that this needs to be entirely separate and modular, but, it doesn't. @mandresm will remember the nightmare it was trying to keep separate packages for esm-environment, esm-master, etc. I at least do not want to have the AI make us repeat that.

Similar to what I said. Keep things separated if easy, possible and no duplication.

See the #1484 (comment). This needs serious human discussion first, and if we go in the direction I imagined, we might need a whiteboard session with the rest of the core developers (@mandresm especially, but I would be interested in seeing @JanStreffing and @seb-wahl's input if they have time)

I don't see why this is so crucial to discuss now. I would choose for now static definitions, create a new root key storage and see how far we get with that. I described this already in a comment above.

[siligam]

Thanks Miguel — that all makes sense, and dict-not-list for storage at the config-root level is cleaner than what Paul and I were sketching. Concretely, here's what I'll implement:

configs/machines/albedo.yaml:

storage:
  albedo-work:
    storage_type: lustre
    state: online

configs/storage/hpss.yaml (for archives not tied to a specific machine):

storage:
  hpss:
    storage_type: hpss
    state: offline
    recall_time_estimate: 300
    path_str: "/arch/ |/hpss/"

detect_hpc_storage(path) in hpc/detect.py keeps doing what it does today — matching an arbitrary path string against known prefixes — except the prefixes and metadata now come from loading all configs/machines/*.yaml + configs/storage/*.yaml entries under storage: instead of being hardcoded in Python.

For loading, I'll use esm_parser.yaml_file_to_dict() directly against esm_parser.CONFIG_PATH (glob machines/*.yaml and storage/*.yaml, read each, collect the storage dict) — no need for hostname resolution here since we're matching a path string, not "what machine am I on". This sidesteps the esm_parser-coupling question entirely: it's the same yaml-loading utility esm_master's config already goes through (e.g. the cli_overrides_to_dict path in #1505), just without any hostname/computer lookup layered on top.

Will start on this unless you see a problem with reading configs that way.

[siligam]

Pushed (61b1fbf): configs/machines/albedo.yaml and configs/machines/levante.yaml get a new storage: root key (dict, per your suggestion), plus a new configs/storage/hpss.yaml for the archive/tape entry not tied to a machine. detect_hpc_storage() in hpc/detect.py now loads these via esm_parser.yaml_file_to_dict()/CONFIG_PATH and matches each entry's path_str regex against the given path (falls back to statvfs probing if nothing matches, same as before).

One consequence worth flagging: this is the first time esm_catalog imports esm_parser. Given the broader push to integrate rather than keep it standalone, I went with it — but it means esm_catalog's CI can't install with --no-deps anymore, so I dropped the Python 3.12 leg from esm-catalog-tests.yml (matches the rest of esm-tools, which only tests 3.9 for the same ruamel.yaml.clib==0.2.7 reason).

Added tests/test_esm_catalog/test_hpc_detect.py — hermetic tests against fixture yaml plus regression tests against the real configs/ files (including the /work/ vs /levante exclusion, now expressed as a regex). Full esm_catalog suite (23 tests) green.

[siligam]

Had a second look at 61b1fbf and evaluated a few follow-up findings:

1. Levante shadowing — real, fixed in 71ca077: path_str tightened from the broad ^(?!.*/levante).*/work/ to ^/work/[a-z]{2}[0-9]{4}/, matching the actual DKRZ project-code convention (verified against 15 real /work/<code> paths already in this repo's configs/runscripts — ab0246, bb0519, mh0033, etc., all 2 letters + 4 digits). Added a regression test that fails against the old pattern and passes against the new one. Also makes the negative lookahead redundant (levante has no digits, so it can never fill the project-code slot) — dropped it.
2. HPSS /arch/ false-positive — same class of issue, but I couldn't find any real archive-path convention in the repo to safely tighten against without guessing, so left it as originally proposed (matches what you suggested upthread). Flagging as known residual risk rather than silently "fixing" it on an assumption.
3. Test coverage gaps (remote-path fallback, real unmocked statvfs) — valid and added, independent of everything else.
4. One suggestion was to drop esm_parser in favor of plain yaml.safe_load — rejected. That reverses the explicit call already made in this thread (Paul: "We have esm-parser specifically for opening and interpreting our config files. Do not reinvent that... very strong NO!!"), and would also revert the Python-3.9-only CI decision from 61b1fbf back to a --no-deps/3.12 setup. Keeping esm_parser and the 3.9-only CI as-is.

Note 1 and 2 weren't regressions introduced by 61b1fbf — both existing heuristics were carried over faithfully from the original hardcoded Python. Worth tightening regardless, done for Levante.

[siligam]

This PR is now fully superseded and can be closed: everything the plan carved out of it exists as its own PR — A1b (#1504) and the test coverage (#1508) are merged, and A1c (#1511), A1d (#1507), A1e (#1512), A1f (#1513), A1g (#1506) are open. The plan file on this branch is ticked through A1g. @mandresm you said you'd close this once the steps were done — I think we're there, so feel free to close from my side whenever you are.

One loose end for the record: the recent commits here (61b1fbf, 71ca077) added storage: sections to configs/machines/*.yaml plus regex-based path matching in detect_hpc_storage(). The A1c that actually landed (#1511) went with the plan's design instead — the caller passes its machine config down, no path matching — so that approach dies with this PR. The storage: yaml layout is still in this branch's history if we ever need path-based classification for the tape-archive case.