Add THREDDS catalog connector to the acquire stage by Hackshaven · Pull Request #227 · zyra-project/zyra

Hackshaven · 2026-06-18T06:24:08Z

Summary

Adds a dedicated zyra acquire thredds connector that reads a THREDDS Data Server catalog.xml, maps each dataset's urlPath to its HTTPServer (fileServer) download URL, and lists, syncs, or fetches matching datasets. Closes the workflow gap in NOAA-GSL/zyra#283 — previously acquire http only handled individual URLs, with no bulk catalog enumeration like S3/FTP offer.

zyra acquire thredds https://gsl.noaa.gov/thredds/catalog/fv3-chem-0p25deg-grib2/catalog.xml \
  --sync-dir ./frames --pattern '\.grib2$' --since-period P1D

Note: this supersedes #226 (same work). It's a single squashed, signed-off commit on a fresh branch — #226 hit a branch-ruleset deadlock on follow-up pushes; this avoids it.

What's included

Area	File
Backend	`src/zyra/connectors/backends/thredds.py` — `parse_catalog`, `enumerate_datasets`, `list_files`, `sync_directory`
CLI	`src/zyra/connectors/ingest/__init__.py` — `thredds` subcommand (registered for `acquire` and `import`)
API	`AcquireThreddsArgs`, `AcquireThreddsRun`, wired into the `POST /acquire` discriminated union
Tests	`tests/connectors/test_thredds_backend.py`, `tests/cli/test_acquire_thredds.py` (offline fixtures)
Docs / sample	ingest `README.md` THREDDS section; `samples/pipelines/thredds_to_local.yaml`
Generated	capabilities manifest + OpenAPI `sha256` snapshot

Design notes

Dedicated subcommand (not an acquire http flag) — THREDDS concepts (services, catalogRef recursion, urlPath→fileServer mapping) don't map onto the HTTP anchor-scraper; matches the s3/ftp precedent.
Opt-in recursion — single catalog by default; --recursive follows nested catalogRef entries up to --max-depth (default 3), with a cycle guard. Non-XML responses (HTML error pages / auth redirects) are logged and skipped so one bad catalog doesn't abort traversal.
--output works for single matches; --output-dir required only for multi-file fetches (clear error otherwise).
Consistent vocabulary — --pattern/--since/--until/--since-period/--date-format (strftime tokens) and the sync replacement options reuse the existing connector surface.
No new dependencies — stdlib xml.etree.ElementTree + existing requests. Tests are hermetic via a fetcher/catalog_xml injection.

Validation

ruff format/check clean; backend + CLI + OpenAPI snapshot tests passing
Regenerated zyra generate-manifest and the OpenAPI snapshot

Addresses NOAA-GSL/zyra#283

🤖 Generated with Claude Code

Generated by Claude Code

Implements `zyra acquire thredds` to enumerate datasets from a THREDDS catalog.xml, map each dataset's urlPath to its HTTPServer (fileServer) download URL, and list, sync, or fetch matching datasets. Supports opt-in recursion into nested catalogRef entries (with a depth cap and cycle guard), pattern/date filtering, and sync replacement options that mirror the existing FTP connector vocabulary. - New backend: connectors/backends/thredds.py (hermetic; supports a fetcher/catalog_xml injection so tests need no network). Malformed, non-XML catalog responses are logged and skipped during recursion. - CLI: acquire/import `thredds` subcommand. `--output` fetches a single matched dataset (default stdout); `--output-dir` is required only for multi-file fetches. `--date-format` documents strftime tokens. - API: AcquireThreddsArgs/AcquireThreddsRun wired into POST /acquire. - Tests: backend + CLI tests using offline catalog fixtures. - Docs: ingest README THREDDS section; sample pipeline thredds_to_local.yaml. - Generated capabilities manifest and OpenAPI sha256 snapshot. Addresses NOAA-GSL/zyra#283 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vfz8KKCKV7AGZ9RSPCiRoH Signed-off-by: Claude <noreply@anthropic.com>

github-actions · 2026-06-18T06:24:27Z

🔁 Relay created/updated upstream PR: NOAA-GSL/zyra#286

cloudflare-workers-and-pages · 2026-06-18T06:26:51Z

Deploying zyra with Cloudflare Pages

Latest commit:	`a3d52c6`
Status:	✅ Deploy successful!
Preview URL:	https://1b138c32.zyra.pages.dev
Branch Preview URL:	https://claude-thredds-acquire-283.zyra.pages.dev

View logs

Copilot

Pull request overview

Adds a new THREDDS “catalog.xml” acquisition connector to Zyra’s ingest/acquire layer, including CLI and API wiring, so users can enumerate and bulk-fetch datasets from THREDDS Data Server catalogs (similar to existing S3/FTP directory-style workflows).

Changes:

Introduces zyra acquire thredds / zyra import thredds backed by a new THREDDS backend that parses catalogs, optionally recurses catalogRefs, and supports list/sync/fetch flows.
Wires THREDDS into the API acquire discriminated union + schemas, and updates wizard capabilities manifests/OpenAPI snapshot hash.
Adds hermetic backend + CLI tests and a sample pipeline YAML.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/snapshots/openapi_sha256.txt	Updates OpenAPI snapshot hash to reflect new acquire tool.
tests/connectors/test_thredds_backend.py	Adds offline unit tests for catalog parsing, filtering, recursion, and sync behavior.
tests/cli/test_acquire_thredds.py	Adds hermetic CLI-level tests for `zyra acquire thredds` list/sync/output behaviors.
src/zyra/wizard/zyra_capabilities/acquire.json	Adds `acquire thredds` and `import thredds` command metadata for the wizard/capabilities system.
src/zyra/wizard/zyra_capabilities.json	Same capabilities updates in the consolidated wizard capabilities file.
src/zyra/connectors/ingest/README.md	Documents THREDDS usage patterns and options alongside other ingest connectors.
src/zyra/connectors/ingest/init.py	Registers the new `thredds` subcommand and implements the CLI handler.
src/zyra/connectors/backends/thredds.py	Implements THREDDS catalog parsing, dataset enumeration, and sync/fetch helpers.
src/zyra/api/schemas/domain_args.py	Adds `AcquireThreddsArgs` and header normalization for acquire thredds.
src/zyra/api/routers/domain_acquire.py	Extends the acquire request union to include `AcquireThreddsRun`.
src/zyra/api/models/domain_api.py	Adds `AcquireThreddsRun` model for the API discriminator.
samples/pipelines/thredds_to_local.yaml	Adds a sample pipeline demonstrating THREDDS → transform → visualize → export.

- _build_download_url: support an absolute HTTPServer service base (with scheme/netloc) by using it as the download root directly, while relative bases continue to resolve against the catalog's host. Uses urllib.parse urljoin/urlparse instead of manual concatenation. Adds a unit test asserting an absolute-base catalog points at the declared host. - _cmd_thredds: warn (not error) when more than one of --list/--sync-dir/ --output-dir is passed, naming the mode that actually runs, keeping the existing precedence (--list > --sync-dir > fetch). - Document basename-collision behavior on sync_directory and the --output-dir help text (datasets sharing a basename overwrite, as with the s3/ftp backends). - Lift the in-function `from pathlib import Path` in _cmd_thredds to the module-level imports. Regenerate capabilities manifest for the updated --output-dir help text; OpenAPI snapshot is unchanged (internal/help-only changes). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vfz8KKCKV7AGZ9RSPCiRoH Signed-off-by: Claude <noreply@anthropic.com>

github-actions · 2026-06-18T18:17:35Z

🔁 Relay created/updated upstream PR: NOAA-GSL/zyra#286

Recursion across catalogRef entries that point at a different host (absolute xlink:href) or a catalog with a different HTTPServer base can produce distinct datasets that share the same urlPath. De-duplicating on urlPath alone would drop one of them, so key the seen-set on the fully resolved download_url instead. Adds a cross-host recursion test. Pairs with the absolute fileServer base handling added in the previous commit. Addresses Copilot review feedback on NOAA-GSL/zyra#283. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vfz8KKCKV7AGZ9RSPCiRoH Signed-off-by: Claude <noreply@anthropic.com>

github-actions · 2026-06-18T18:20:39Z

🔁 Relay created/updated upstream PR: NOAA-GSL/zyra#286

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.

In _should_download, when --recheck-existing is set but the server omits or blocks Content-Length (http_backend.get_size returns None), the previous logic forced a re-download every run. Keep the local copy instead, matching the FTP backend's recheck_existing semantics (re-download only on a known size mismatch). Avoids unnecessary network/mirror load against servers that don't expose HEAD/Content-Length. Adds a regression test. Addresses Copilot review feedback on NOAA-GSL/zyra#283. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vfz8KKCKV7AGZ9RSPCiRoH Signed-off-by: Claude <noreply@anthropic.com>

github-actions · 2026-06-18T18:27:03Z

🔁 Relay created/updated upstream PR: NOAA-GSL/zyra#286

Hackshaven mentioned this pull request Jun 18, 2026

Add THREDDS catalog connector to the acquire stage #226

Closed

Hackshaven mentioned this pull request Jun 18, 2026

Codex Relay: zyra-project/zyra PR #227 — Add THREDDS catalog connector to the acquire stage NOAA-GSL/zyra#286

Open

Hackshaven requested a review from Copilot June 18, 2026 06:26

Copilot started reviewing on behalf of Hackshaven June 18, 2026 06:27 View session

Copilot AI reviewed Jun 18, 2026

View reviewed changes

Comment thread src/zyra/connectors/backends/thredds.py

Hackshaven self-assigned this Jun 18, 2026

Hackshaven requested a review from Copilot June 18, 2026 18:21

Copilot started reviewing on behalf of Hackshaven June 18, 2026 18:21 View session

Copilot AI reviewed Jun 18, 2026

View reviewed changes

Comment thread src/zyra/connectors/backends/thredds.py Outdated

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add THREDDS catalog connector to the acquire stage#227

Add THREDDS catalog connector to the acquire stage#227
Hackshaven wants to merge 4 commits into
mirror/stagingfrom
claude/thredds-acquire-283

Hackshaven commented Jun 18, 2026

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

cloudflare-workers-and-pages Bot commented Jun 18, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Hackshaven commented Jun 18, 2026

Summary

What's included

Design notes

Validation

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

cloudflare-workers-and-pages Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying zyra with Cloudflare Pages

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cloudflare-workers-and-pages Bot commented Jun 18, 2026 •

edited

Loading