[catalog] Feature checklist: stock vs. custom STAC Browser decision

[siligam]

Context

ESM STAC Catalog has grown a rich set of features. With limited funding, the team needs to decide which features to keep and whether maintaining a custom STAC Browser fork is justified.

This issue collects all implemented and planned features as a decision-aid checklist for the upcoming team discussion. Features are grouped by:

• Maintenance tier: 🟢 works with stock browser / 🟡 custom API only (no browser changes) / 🔴 requires custom browser fork
• Priority: ⭐ must-have / 💡 nice-to-have / ❓ undecided
• Status: ✅ implemented / 📋 planned
• Iteration: 1️⃣ / 2️⃣ / 3️⃣

The full checklist lives in esm-tools-plus-simcat-features-checklist.md on branch esm-tools-plus/simcat/llm-mcp.

---

Central Trade-off

Everything in the API/backend layer — scanning, DuckDB, STAC API, CQL2 filtering, NML/paleo/HPC metadata, MCP/LLM tools — works with the stock STAC Browser or no browser at all.

The custom browser fork adds only:

Browser-only feature	Workaround without fork?
Namelist (NML) search UI	Yes – MCP search_collections tool or direct CQL2 API
Visualization panel (click-to-plot)	Yes – MCP preview_item tool
Python code snippets panel	Yes – MCP open_and_run / manual scripting
CQL2 NOT operator fix (upstream bug)	Submit PR to upstream STAC Browser repo
Personal collections UI	No UI workaround; backend API exists
Collection badges on item cards	Minor UX loss only

---

Decision Scenarios

Option	Browser maintenance effort	What you get
A: Stock browser only	Zero	All backend/API/MCP features; NML search via API/MCP only
B: Stock browser + upstream NOT-fix PR	Minimal (one-time PR)	Same as A, plus CQL2 NOT queries working in browser
C: Maintain custom browser fork	High (JS/Vue + tracking upstream releases)	All currently implemented features
D: API + MCP only, no browser	Zero	Full API + LLM/MCP access; no click-to-explore browser UI

---

Feature Checklist (abbreviated)

Core Cataloging — all 🟢 stock browser compatible

• ✅ ⭐ 1️⃣ Scan NetCDF / GRIB / extension-less files
• ✅ ❓Remote filesystem scanning (SSH, ScoutFS/tape, S3)
• ✅ ⭐ 1️⃣ Parallel + SLURM batch scanning
• ✅ ⭐ 1️⃣ DuckDB storage (per-experiment, portable)
• ✅ ⭐ 1️⃣ ESM-Tools live integration (add_files() from tidy phase)
• ✅ ⭐ 1️⃣ Catalog federation (multiple experiments as one API)

Standard STAC API — all 🟢 stock browser compatible

• ✅ ⭐ 1️⃣ Item search, CQL2 filtering, queryables
• ✅ ⭐ 1️⃣ Collection browsing and filtering
• ✅ ⭐ 3️⃣ Health/readiness probes -> Kubernetes -> needed for load balancing if a lot of people start accessing/modifying

Science-Specific Search

• ✅ ⭐ 1️⃣ 🔴 Namelist (NML) parameter search — API works stock; browser UI needs fork
• ✅ ⭐ 1️⃣ 🟡 Paleo time search (paleo:year, presets: LGM, Eemian, Mid-Holocene)
• ✅ ⭐ 1️⃣ 🟡 HPC storage state (online/nearline/offline, tape recall time) -> Location (DKRZ, AWI, ...) and state (levante, albedo, tape...) (static)
• ✅ ⭐ 1️⃣ 🟢 FESOM unstructured grid metadata

Visualization

• ✅ 💡2️⃣/3️⃣ 🔴 ESM Visualization Service browser panel — needs fork; MCP path works without it -> most expensive, requires a server in location, up to the user/location admins
• ✅ 💡2️⃣/3️⃣ 🟡 FESOM triangular mesh rendering (via MCP preview_item) -> depends on the previous item

LLM / MCP Tools — all 🟡, zero browser dependency

• ✅ ⭐ 2️⃣/3️⃣ list_collections, get_collection_info, search_collections, search_items
• ✅ 💡 2️⃣/3️⃣ preview_item, open_and_run, run_python, compare_collections
• ✅ 💡2️⃣/3️⃣ Multiple transports: stdio, streamable-HTTP (Open WebUI), SSE, OpenAPI
• Make an issue to discuss options and requirements, run locally (in the HPCs) vs run globally in the Browser GUI

Access Control

• ✅ ⭐ 2️⃣ 🟡 JupyterHub authentication + group-based write access
• 📋 ⭐ 2️⃣ 🟡 Read-only public metadata tier for Helmholtz (metadata only, no file paths)
• 1️⃣ First approach with some access control general assumptions (at AWI) -> in 2️⃣ more granular access (at Helmholz level) control will be enabled and run a workshop for showing how to manage access to already submitted simulations

Browser Fork–Specific (lost if switching to stock)

• ✅ ⭐ 1️⃣ 🔴 CQL2 NOT operator fix (upstream bug — candidate for upstream PR)
• ✅ 💡 🔴 Visualization panel
• ✅ 💡 1️⃣ 🔴 Python code snippets
• ✅ ⭐ 1️⃣ 🔴 collection badges

Catalogue Server

• ✅ ⭐ 1️⃣ Virtual machine for hosting the catalogue, browser, etc, services

---

Questions for the Team

• Who are the primary users — Python scripters or non-programmers needing a browser UI?
• Is maintaining a JS/Vue browser fork sustainable with current funding?
• Is the CQL2 NOT fix worth submitting as a PR to the upstream STAC Browser project?
• Is click-to-visualize in the browser a must-have, or is the MCP/LLM path sufficient?
• Does the Helmholtz public metadata tier require any browser changes, or is it purely API-side?
• Should personal collections / run quality annotations be prioritised enough to justify fork maintenance?

[mandresm]

23.04.2026 Meeting

TODOs

• @siligam will add labels to the features for "remote connection needed" and "netcdf reading needed"
• Miguel will add a label for the iteration

Decisions made

• In "Decision Scenarios" option C is chosen: "Maintain custom browser fork"
• ✅ ⭐ Remote filesystem scanning (SSH, ScoutFS/tape, S3) -> this is a question mark
  • Won't do as scanning
  • When data is missing -> notify the user about the owner of the simulation missing data
  • Delegate in users
• All the other features in Core Cataloging have been accepted

[siligam]

Section 9 · Multi-Site Catalog Serving & Data Locality

This section captures two related concerns raised ahead of the team meeting: (a) hosting catalogs from different HPC sites (AWI albedo + DKRZ levante) under a single API, and (b) what happens to catalog entries when data migrates between storage tiers (Lustre → tape → S3).

---

9.1 Cross-Site Catalog Serving (multi-site)

The DuckDB file is portable — you can scan an experiment on levante, copy the .duckdb to AWI, and esm-catalog register it into the AWI STAC API. The federation layer was designed for exactly this. Metadata browsing works fully.

The split picture:

Capability	Status	Notes
Copy DuckDB from DKRZ → AWI, serve metadata	✅ works today	Just rsync + register
hpc:facility / hpc:system records data origin	✅ works today	Set at scan time on source machine
STAC Browser discovery, filtering, collection browsing	✅ works today	All metadata operations are local to the DB
Asset links in browser pointing to DKRZ paths	⚠️ stale	file:///work/... URIs are inaccessible from AWI
MCP open_and_run / run_python / preview_item	❌ breaks	These open paths directly — DKRZ paths unreachable from AWI server
MCP list_collections / search_items / get_collection_info	✅ works	Metadata-only tools, no filesystem access

Root cause: asset hrefs are stored as file:///absolute/path. When the catalog is served from a different site, these URIs are valid only on the source machine. The hpc:facility field records the origin but nothing currently uses it to warn or reroute.

Possible paths forward (not yet implemented):

Option	Effort	Tradeoff
Scan-over-SSH from AWI (ssh://levante.dkrz.de/...)	Medium	hrefs become ssh:// URIs; xarray+fsspec can open them but needs credentials and is slow
S3 mirror — copy DKRZ data to shared S3	High	hrefs become s3://; fully accessible; significant storage cost
Compute-at-source — MCP server runs on levante, catalog API at AWI	Medium	Clean split; discovery at AWI, execution at DKRZ; needs routing layer
Metadata-only mode — AWI serves DKRZ catalogs as read-only metadata, no data access tools	Low	Works today; just document the limitation clearly

Questions for the team:

• Is metadata-only cross-site serving sufficient for Helmholtz public access, or do we need data access too?
• Is there a shared S3 between AWI and DKRZ available or planned?
• Should the MCP tools detect hpc:facility != local and return a clear "data not accessible from this site" message instead of silently failing?

---

9.2 Data Locality & Storage Tier Staleness

At AWI, data moves between storage tiers over its lifetime: Lustre (hot) → ScoutFS/tape (nearline) → HPSS archive (cold). The catalog records state at scan time only — it does not track subsequent migrations.

Two distinct cases:

Case A — Transparent HSM migration (path stays the same)
ScoutFS/Lustre HSM on albedo keeps the path (/albedo/work/.../output.nc) unchanged; the file is staged to tape transparently. The catalog path is still valid — the OS recalls it on access. Only hpc:state goes stale ("online" in catalog, true state: "nearline").

Case B — Hard archive move (path changes)
File is copied/moved from /albedo/work/... to /arch/... or /hpss/.... The catalog item's href now points to a non-existent path. Because item IDs include a hash of the original path, re-scanning the new archive path creates a new item — the old one becomes a ghost entry with a dead link.

Current implementation gaps:

Gap	Impact
No bulk hpc:state refresh command	States go stale silently after HSM migration
Item ID includes path hash → moving a file creates orphan items	Dead hrefs accumulate; no duplicate detection
No orphan detection / dead-link sweep	Users may try to access files that no longer exist at that path
hpc:last_access set only at scan time, never updated on access	Stale recency information
Remote HSM state detection disabled for ssh:// paths	Cannot check tape state of DKRZ files from AWI

What exists today:

• hpc:state detection at scan time via dmattr (GPFS) and lfs hsm_state (Lustre HSM), with path-prefix heuristics as fallback
• hpc:storage_tier (hot/warm/cold) derived from storage type and state
• Upsert on re-scan: if the file is at the same path, re-scanning overwrites the item cleanly

Possible paths forward (not yet implemented):

Feature	Effort	Notes
esm-catalog update-state --db X — sweep all items, query current HSM state, update in-place	Medium	Only updates hpc:state/hpc:storage_tier fields, no full rescan
Orphan detection — flag items whose href no longer resolves	Low	Read-only sweep; marks items hpc:state: "missing"
Path redirect / alias — record old path → new archive path mapping	Medium	Allows browser/MCP to follow the move without re-scanning
Periodic cron-based state refresh (SLURM job)	Low	Automates update-state on a schedule

Questions for the team:

• Is stale hpc:state a real problem in practice, or do users already know what's on tape?
• Should orphaned items be deleted or just flagged? (Deletion is irreversible if the file comes back online)
• Is a lightweight update-state sweep command worth implementing, or is periodic re-scan sufficient?
• For Case B (hard archive moves): is this common at AWI/DKRZ, or do HSM setups keep paths stable?

[siligam]

Decision (2026-04-24 team meeting)

Option C selected: maintain the custom STAC Browser fork.

The team agreed that the features requiring browser modifications are necessary:

• NML (namelist) parameter search UI in the browser
• CQL2 NOT operator fix
• ESM Visualization Service panel
• Python code snippets panel

The custom fork will be maintained going forward as part of the WP SimCat scope.