feat(esm-tools-plus/simcat): add esm_catalog — STAC-based experiment catalog

[siligam]

Context

Part of the ESM-Tools-plus/simcat initiative, which adds a STAC-based experiment catalog to ESM-Tools for indexing, querying and browsing climate model output.

This PR introduces esm_catalog — the catalog API, scanner, and CLI.
A companion PR adds esm_viz — the visualization service.

Reviewer tip: this is a new top-level package. All files are additions. Start with src/esm_catalog/ARCHITECTURE.md for an overview, then src/esm_catalog/cli.py for the user-facing entry points.

---

What's included

src/esm_catalog/ — the package (108 files, ~32k lines)

Sub-package	Purpose
api/	FastAPI STAC server (stac-fastapi-api backed by DuckDB)
scan/	NetCDF / GRIB / namelist scanner; writes STAC items to DuckDB
stac/	STAC Collection/Item builders + ESM-specific extensions
storage/	DuckDB storage layer + personal collections
hpc/	HPC system detection (Albedo, Levante, …)
integration/	ESM-Tools runscript hooks for auto-scanning
mcp/	Model Context Protocol server for LLM tool access
cli.py	esm-catalog CLI (scan, serve, mcp, reindex, …)

STAC extensions defined:

• hpc — HPC system name on assets (hpc:system)
• namelist — F90 namelist parameters (nml:*)
• paleo — deep-time metadata (paleo:years_bp)
• datacube — variable/dimension axes
• contacts — experiment owner info

API highlights:

• GET /collections — all experiments, filterable via CQL2
• GET /experiments — experiment-level search with CQL2 (component=, variable=, nml:radctl.co2vmr > 284)
• GET /search — STAC item search
• GET /queryables — OGC queryables for browser filter UI
• GET /paleo-presets — named paleo time periods (LGM, mid-Holocene, …)
• GET /collections/{id}/nml-parameters — namelist params for an experiment
• Personal collections CRUD (/personal/…)
• Catalog registry API (/catalogs) + web admin UI at /ui

Option A catalog layout (current): one STAC Collection per experiment, components stored as item properties (properties.component), enabling cross-component queries without per-component collection proliferation.

tests/test_esm_catalog/ — test suite

~4000 lines covering API, CLI, storage, scanning, personal collections, integration, HPC detection, and STAC object construction.

Shared infrastructure (also in this PR)

File	Purpose
pixi.toml / pixi.lock	Pixi environment for local dev
setup.py	Editable install entry point
.github/workflows/docker-esm-catalog.yml	Docker image CI
docs/esm_catalog_*.rst	User guide, architecture ref, VirtualiZarr workflow
examples/fesom_stac/	Catalog builder examples for FESOM experiments
examples/echam_grib/	ECHAM GRIB reading examples
examples/notebooks/	Jupyter notebook walkthrough
utils/	See esm_viz PR

---

Key design decisions

• DuckDB as storage: single-file, zero-infrastructure, fast JSON/array queries via SQL. No Postgres required on HPC.
• STAC as the wire format: interoperable with existing STAC clients (stac-browser, pystac, intake-stac).
• CQL2 filtering: full OGC CQL2-text/JSON support, including namelist parameter range queries.
• Option A layout: one collection per experiment (not one per component) keeps the browser grid clean and allows multi-component queries.

---

Test plan

• pixi run pytest tests/test_esm_catalog/ passes
• esm-catalog scan --help and esm-catalog serve --help run without error
• esm-catalog scan <experiment_path> produces a catalog.duckdb
• esm-catalog serve --catalog catalog.duckdb serves GET /collections and GET /search
• GET /queryables returns variables, components, experiment types
• CQL2 filter ?filter=variable='sst' returns only SST items

🤖 Generated with Claude Code

[mandresm]

I don't think I can review 32k lines of code. ESM-Tools is a ~85k line code itself. @siligam, does it really need to be so huge? The issue is not only review time but also maintainability.

I would expect that this PR does "just" a few things:

1. Scans experiments or run dirs
2. Constructs shards for the DB
3. Attempt to sync the shard with the global DB

Can you attempt to break it down into much smaller units? That would help me learning the code but also would probably also improve integration and structure.

[siligam]

@mandresm — you're right, and thanks for the push-back. 32k lines in one commit isn't a fair ask.

We've stepped back and written a decomposition plan that reintroduces esm_catalog as a dependency-ordered stack of small, individually reviewable PRs against release. The opening PRs follow exactly the three units you described:

1. scan run dirs → PR-A1b (with PR-A1a landing the STAC data model first, which also breaks a scan⇄stac import cycle)
2. construct DB shards → PR-A2 (per-experiment .duckdb)
3. sync shard → global DB → PR-A3 (federation)

…then the STAC API serve layer (B1–B4), and optional layers last. The MCP/LLM layer is being dropped (no deliverable backing, redundant with the OGC-standard API, and run_python is a security liability); its code is archived on tag archive/mcp-llm so nothing is lost.

Full plan: 2026-06-13-esm-catalog-pr-decomposition-design.md

Converting this PR to draft — it'll be superseded by the sliced stack, which I'll open shortly, and closed once the first sliced PR is up.

[siligam]

First two slices are up, @mandresm:

• feat(esm_catalog): PR-0 — package scaffold #1483 — PR-0: package scaffold (+102 lines)
• feat(esm_catalog): PR-A1a — STAC data-model foundation #1484 — PR-A1a: STAC data-model foundation + scan⇄stac cycle-break (+~1.5k lines, stacked on feat(esm_catalog): PR-0 — package scaffold #1483)

Suggest reviewing #1483 first (trivial), then #1484. This PR stays as a draft reference until the sliced stack lands, then I'll close it.