Skip to content

feat(esm_catalog): PR-C1 — middleware + export utilities#1494

Open
siligam wants to merge 1 commit into
esm-catalog/pr-b4-federation-apifrom
esm-catalog/pr-c1-middleware-export
Open

feat(esm_catalog): PR-C1 — middleware + export utilities#1494
siligam wants to merge 1 commit into
esm-catalog/pr-b4-federation-apifrom
esm-catalog/pr-c1-middleware-export

Conversation

@siligam

@siligam siligam commented Jun 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Provenance: this implementation is substantially Paul Gierz's work, carried over from his esm-tools-plus/simcat/pgierz-workbench prototype branch (commits Mar–Apr 2026). This PR ports it, largely as-is, into the reviewable stack.

  • Adds api/middleware.py: CorrelationIdMiddleware (sets request.state.correlation_id to 8-char UUID) + RequestLoggingMiddleware (logs incoming method/path and outgoing status/latency; handles X-Forwarded-For)
  • Adds storage/export.py: export_parquet() + export_json() for writing STAC items to disk; import_parquet() for the SLURM parallel-scan merge workflow; pyarrow is a lazy import (only needed at call time, not install time)
  • Wires RequestLoggingMiddleware into create_app() middleware stack (before CORS)
  • 8 new tests (137 total green); 2 pyarrow tests auto-skip when pyarrow isn't installed

Test plan

  • pytest src/esm_catalog/tests/test_middleware_export.py -v — 8 pass (2 pyarrow skipped)
  • pytest src/esm_catalog/tests/ — 137 total green
  • CI esm-catalog-tests passes on 3.9 + 3.12

Stack position

release → PR-0 → A1a → A1b-1 → A1b-2 → A2 → A3 → A4 → B1 → B2 → B3 → B4 → C1 (this PR)

🤖 Generated with Claude Code

@siligam
siligam force-pushed the esm-catalog/pr-b4-federation-api branch from 840c483 to 16a3cdd Compare June 14, 2026 00:12
@siligam
siligam force-pushed the esm-catalog/pr-c1-middleware-export branch from 5f8f856 to c06cb4f Compare June 14, 2026 00:12
@siligam
siligam force-pushed the esm-catalog/pr-b4-federation-api branch from 16a3cdd to ed75fed Compare June 17, 2026 12:13
@siligam
siligam force-pushed the esm-catalog/pr-c1-middleware-export branch from c06cb4f to a095781 Compare June 17, 2026 12:14
@siligam
siligam force-pushed the esm-catalog/pr-b4-federation-api branch from ed75fed to 1f401c8 Compare June 17, 2026 16:00
@siligam
siligam force-pushed the esm-catalog/pr-c1-middleware-export branch from a095781 to 73dbc73 Compare June 17, 2026 16:01
@siligam
siligam force-pushed the esm-catalog/pr-b4-federation-api branch from 1f401c8 to 952b8d2 Compare June 17, 2026 16:34
@siligam
siligam force-pushed the esm-catalog/pr-c1-middleware-export branch from 73dbc73 to 96e0047 Compare June 17, 2026 16:34
Adds RequestLoggingMiddleware/CorrelationIdMiddleware (starlette-based
request tracing) and export.py (Parquet/JSON export via pyarrow). Wires
RequestLoggingMiddleware into create_app(). 8 new tests (137 total).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@siligam
siligam force-pushed the esm-catalog/pr-b4-federation-api branch from 952b8d2 to 673be4b Compare June 17, 2026 17:03
@siligam
siligam force-pushed the esm-catalog/pr-c1-middleware-export branch from 96e0047 to 5f389e0 Compare June 17, 2026 17:03
@siligam

siligam commented Jun 17, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto the corrected PR-0 base (#1483) — test relocation to tests/test_esm_catalog/, restored dedicated CI workflow at the new path, and a couple of CI fixes (doctest-collection exclusion, eval_type_backport for pydantic on Python 3.9). No change to this PR's own diff/scope. CI is green.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant