Automate monthly incremental catalog updates via GitHub Actions

[NewGraphEnvironment]

Problem

The catalog is five months stale: the source URL inventory (data/urls_list.txt, 60,126 URLs) was last refreshed 2026-02-18, the S3 catalog last built 2026-02-11, and pgstac last registered 2026-02-13 (58,019 items). The BC objectstore has a history of substantial undocumented growth (22,548 → 58,109 files discovered in a single change-detection run), so unprocessed additions accumulate quickly.

Automation was scoped as Phases 3–4 of #5, but #5 auto-closed when PR #9 merged Phases 1–2, and the remaining phases were never re-tracked. The original design (cron on a standing VM) also predates the pattern now proven elsewhere: water-temp-bc runs its monthly snapshot as a GitHub Actions cron with OIDC-scoped S3 access and no standing machine (decision record: NewGraphEnvironment/water-temp-bc#17; reference implementation: its .github/workflows/snapshot.yml).

Proposed Solution

Adopt the water-temp-bc pattern: .github/workflows/update.yml on a monthly cron + workflow_dispatch, OIDC role provisioned in rtj, incremental pipeline run on the runner, catalog synced to S3, refreshed caches committed back to main.

Prerequisites

• OIDC role with least-privilege access to s3://stac-dem-bc, provisioned via rtj's modules/gha_s3_role (companion issue in tracking links below); role ARN posted back here
• Migrate from conda to uv for Python dependency management #16 — uv migration, so CI Python setup is seconds instead of a conda solve

CI-readiness fixes

• scripts/stac_utils.py — replace the hardcoded local output root with an environment-variable override so the pipeline can write inside a runner workspace
• Add a CI-safe S3 sync path: scripts/s3_sync.R currently runs aws s3 sync --delete with a named profile. A stateless runner holds only collection.json plus the new items, so --delete would remove every existing item object from the bucket, and the named profile bypasses OIDC credentials. CI needs a no-delete sync (or targeted copy) on the default credential chain
• Regenerate data/urls_new.txt from change detection at the start of every run — the committed copy is a stale artifact and must never be trusted as input
• R toolchain in CI for urls_fetch.R / detect_changes.R (needs ngr), via r-lib/actions as in the water-temp-bc workflow

Workflow shape

• Triggers: monthly cron + workflow_dispatch; permissions: id-token: write, contents: write
• Steps: checkout (validation caches are committed, so the runner starts warm) → set up R + Python → aws-actions/configure-aws-credentials with the provisioned role ARN → fetch fresh listing + detect changes → exit early when nothing changed → access checks → fetch current collection.json from S3 (~11 MB; the 58k item JSONs are not needed locally) → item_create.py --incremental → item_validate.py --incremental → CI-safe sync → commit refreshed caches (urls_list.txt, stac_geotiff_checks.csv, stac_item_validation.csv) back to main
• Timeout sized for catch-up runs: typical incremental is 5–15 min for 50–100 new files (metadata reads run ~3 files/s with 32 workers)

Registration

pgstac registration stays a separate step on the pgstac host (geoserv) for v1 — one command, full delete-and-reload ~46 min (S3 item download dominates; the DB load itself is ~10 s). Follow-up once the workflow is stable: incremental pypgstac load --method upsert of only the new items (seconds), with a periodic full reload to reconcile.

Rollout

• One workflow_dispatch catch-up run (five months of growth) verified end to end: items on S3, caches committed back, collection queryable through the API after registration
• Enable the cron

Out of scope

• Full rebuilds (~5.5 h) — too close to the 6 h Actions job limit; DO-infrastructure build capability is tracked in NewGraphEnvironment/rtj#49
• Source-file deletions — urls_deleted.txt keeps the audit trail and the role is provisioned without s3:DeleteObject; deletion handling lands with the registration follow-up
• Structured logging/benchmarking — Add structured logging and performance benchmarking #6 complements unattended runs but does not block this

Relates to #5
Relates to #6
Relates to #16
Relates to NewGraphEnvironment/rtj#49
Relates to NewGraphEnvironment/rtj#184

[NewGraphEnvironment]

OIDC role provisioned and applied via NewGraphEnvironment/rtj#189 (closes rtj#184):

arn:aws:iam::414155577829:role/role_gha_stac_dem_bc

Trust is StringEquals-scoped to repo:NewGraphEnvironment/stac_dem_bc:ref:refs/heads/main; permissions are ListBucket/GetBucketLocation + GetObject/PutObject — no delete. Bucket versioning is now enabled on stac-dem-bc (rollback safety for the in-place collection.json rewrite each run; noncurrent-expiry deliberately unset, documented at the bucket entry in rtj).

This matches the ARN already referenced in .github/workflows/update.yml, so no workflow change is needed.