Skip to content

feat(esm-tools-plus/simcat): add esm_viz — visualization service for catalog items#1474

Draft
siligam wants to merge 6 commits into
releasefrom
esm-tools-plus/simcat/pr-esm-viz
Draft

feat(esm-tools-plus/simcat): add esm_viz — visualization service for catalog items#1474
siligam wants to merge 6 commits into
releasefrom
esm-tools-plus/simcat/pr-esm-viz

Conversation

@siligam

@siligam siligam commented May 18, 2026

Copy link
Copy Markdown
Contributor

Context

Provenance: this implementation is substantially Paul Gierz's work, carried over from his esm-tools-plus/simcat/pgierz-workbench prototype branch (commits Mar–Apr 2026). This PR ports it, largely as-is, into the reviewable stack.

Part of the ESM-Tools-plus/simcat initiative.

This PR introduces esm_viz — the visualization service that renders STAC item data as interactive previews and static plots.
The companion PR #1473 introduces esm_catalog, the catalog API that esm_viz integrates with.

Reviewer tip: esm_viz is an independent FastAPI service with no imports from esm_catalog. It reads STAC item URLs and renders data. Start with src/esm_viz/app.py for the API surface, then src/esm_viz/interactive.py for the panel-based preview logic.


What's included

src/esm_viz/ — the package (10 modules, ~4400 lines)

Module Purpose
app.py FastAPI application factory and route registration
cli.py esm-viz serve CLI entry point
interactive.py Panel-based interactive previews embedded in the browser
static_preview.py Static PNG/SVG plot generation (matplotlib)
collection.py Collection-level overview plots
fesom.py FESOM ocean model data rendering (mesh-aware)
compute.py Dask/distributed compute coordination
compute_routes.py REST routes for compute job management
readers.py NetCDF / GRIB data readers via xarray

API endpoints:

  • GET /preview/{item_id} — static preview image for a STAC item
  • GET /preview/{item_id}/panel — interactive Panel app embedded in the browser
  • GET /collection/{collection_id}/overview — collection-level overview plot
  • GET /health — health check
  • Compute routes for Dask cluster management

tests/test_esm_viz/ — contract tests

~500 lines of contract tests verifying the API shape and compute contract independently of a running service.

Deployment files

File Purpose
.github/workflows/docker-esm-viz.yml Docker image CI
utils/docker-compose.yml Compose file for running catalog API + viz service together
utils/hpc-deploy.sh HPC deployment helper (Albedo/Levante)

Integration with esm_catalog

esm_viz is configured via the vizServer URL in the STAC Browser config and linked from catalog item assets. It is not imported by esm_catalog — the two services communicate only over HTTP. Either can be deployed independently.

Typical deployment:

esm-catalog serve --port 23006   # STAC API
esm-viz serve --port 23007       # Viz service
# STAC Browser configured with vizServer: "http://host:23007"

Test plan

  • pytest tests/test_esm_viz/ passes
  • esm-viz serve --help runs without error
  • GET /health returns {"status": "healthy"}
  • GET /preview/{item_id} returns an image for a valid NetCDF item
  • GET /preview/{item_id}/panel returns an embeddable Panel app

🤖 Generated with Claude Code

siligam added a commit that referenced this pull request Jun 13, 2026
Design for reintroducing esm_catalog to release as a dependency-ordered
stack of small, reviewable PRs (responding to review feedback on #1473).
Rebuild-from-end-state; opening PRs mirror scan->shard->sync; MCP dropped.

Covers all three parallel workstreams: esm_catalog (#1473), esm_viz (#1474),
and the stac-browser fork. esm_catalog API core is the critical path; esm_viz
and the browser UI run as parallel tracks off it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@siligam

siligam commented Jun 13, 2026

Copy link
Copy Markdown
Contributor Author

Flagging before review: this PR (~5.7k lines, one commit) is getting the same treatment as #1473.

esm_viz is an independent service (imports no esm_catalog, talks over HTTP), so it becomes its own small stack: V1 service skeleton + render core → V2 FESOM mesh rendering → V3 land-sea mask → V4 deploy/CI. It's nice-to-have priority and sequences after the esm_catalog API core, in parallel with the browser work (it feeds the browser viz panel).

Converting to draft pending the sliced stack. Plan (§7–8): 2026-06-13-esm-catalog-pr-decomposition-design.md

@siligam
siligam marked this pull request as draft June 13, 2026 11:04
siligam and others added 6 commits June 17, 2026 14:10
…talog items

Introduces the esm_viz package as part of the ESM-Tools-plus/simcat
initiative. Provides a FastAPI-based visualization server for rendering
STAC item data (NetCDF, GRIB) with interactive previews and static plots.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Paul Gierz <pgierz@awi.de>
The main esm-tools CI runs pytest --doctest-modules src, which picks up
esm_catalog's test suite requiring optional deps (fastapi, duckdb, etc.).
Excluding via norecursedirs keeps CI green without those deps installed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…rom pytest

The CI also runs pytest -v . which discovers tests/ subdirs.
Extend norecursedirs to cover all esm_catalog/viz test directories.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
esm_viz uses Python 3.10+ union syntax (X | None) which fails at
collection time on Python 3.8. Exclude it from the base CI sweep
alongside esm_catalog.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The call with an empty string fails for non-editable or modern pip
editable installs that don't create esm-tools.egg-link. Catch
FileNotFoundError and omit the cd step from the upgrade message
rather than crashing esm_master.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Python 3.8 does not support X | None union syntax or built-in generics
as runtime annotations. Deferring annotation evaluation restores
compatibility.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@siligam
siligam force-pushed the esm-tools-plus/simcat/pr-esm-viz branch from 20cb83e to 7b4bf66 Compare June 17, 2026 12:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant