feat(esm_catalog): PR-A4 — esm-tools integration layer

[siligam]

Summary

Provenance: this implementation is substantially Paul Gierz's work, carried over from his esm-tools-plus/simcat/pgierz-workbench prototype branch (commits Mar–Apr 2026). This PR ports it, largely as-is, into the reviewable stack.

• Adds integration/config.py: helpers to load/parse ESM-Tools config files — load_config (via ruamel.yaml), find_finished_configs, get_outdata_files, find_file_operations_log, get_outdata_from_file_operations (extracts outdata entries + MD5 checksums from the tidy log), extract_stac_metadata.
• Adds integration/esm_tools.py: add_files(db, files, config, checksums=None) and add_run(db, experiment_dir, component, run_datestamp, config). add_run() implements the source priority chain — tidy log (with checksums) → finished_config outdata_targets (paths only). add_files() guards against missing/zero-byte files, deduplicates by symlink-resolved path, and injects file:checksum into item assets when checksums are provided.
• Adds ruamel.yaml>=0.17 to extras_require["catalog"] and CI.
• 12 new tests covering all config helpers, YAML parsing, add_files round-trip, zero-byte/missing-file skip, and checksum injection.

Full ingest pipeline (Phase A complete):

# Live tidy-phase hook
add_run(db=exp/"catalog.duckdb", experiment_dir=exp, component="echam",
        run_datestamp="20000101-20001231", experiment_config=config)

# Or batch scan + merge
esm-catalog scan /work/experiments/exp-alpha --db exp-alpha/catalog.duckdb
esm-catalog scan /work/experiments/exp-beta  --db exp-beta/catalog.duckdb
esm-catalog merge exp-alpha/catalog.duckdb exp-beta/catalog.duckdb --output global.duckdb

Test plan

• pytest src/esm_catalog/tests/test_integration.py -v — 12 tests pass locally
• Full suite pytest src/esm_catalog/tests — 74 tests, 0 failures
• CI green on Python 3.9 + 3.12

🤖 Generated with Claude Code

[siligam]

Rebased onto the corrected PR-0 base (#1483) — test relocation to tests/test_esm_catalog/, restored dedicated CI workflow at the new path, and a couple of CI fixes (doctest-collection exclusion, eval_type_backport for pydantic on Python 3.9). No change to this PR's own diff/scope. CI is green.

[siligam]

Heads-up on the vcs_info.yaml loader (find_vcs_info/load_vcs_info, intended to surface model branch/git-hash/binary-dir): checked a real Albedo experiment and found it's currently inert.

add_vcs_info() / check_vcs_info_against_last_run() (src/esm_runscripts/prepare.py) are defined but never called anywhere in the esm-tools Python codebase, and are commented out in every single recipe file across the entire configs/ tree (awiesm, focioifs, oifs, pism, ...). Confirmed no *_vcs_info.yaml exists anywhere in /albedo/work/user/pgierz/SciComp/Tutorials/AWIESM_Basics/experiments/basic-001 (checked the archived log/ and all run_*/log/ subdirs).

The loader itself is harmless — find_vcs_info() returns None gracefully when the file is absent — but it currently has nothing to find in any real setup. Two things worth knowing:

1. model_dir (the source/binary directory) is already available directly in finished_config.yaml per component, no vcs_info.yaml needed for that part.
2. On Albedo, echam-6.3.05p2's model_dir isn't even a git checkout (no .git/), so hash/branch_name wouldn't resolve there regardless of whether add_vcs_info were enabled.

Not blocking — opened #1503 to track it. Miguel confirmed these were briefly live in production, broke esm_runscripts, and got commented out; never revisited due to other priorities. Now that catalog/provenance metadata needs it, it's on the "less pressing" list to fix properly.