From c72ecf0a1ac1a1d6ad42a66fe92eff3975cb037b Mon Sep 17 00:00:00 2001
From: almac2022 <al@newgraphenvironment.com>
Date: Sat, 18 Jul 2026 15:51:22 -0700
Subject: [PATCH 1/7] Archive issue #13 PWF (work landed, closed 2026-02-17)

Move stale planning/active files to planning/archive/2026-02-issue-13-url-accessibility/
with outcome README. Clears active/ for issue #23.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HN1tMTLezPVkv9eT86FLSH
---
 .../2026-02-issue-13-url-accessibility/README.md      | 11 +++++++++++
 .../2026-02-issue-13-url-accessibility}/progress.md   |  0
 .../2026-02-issue-13-url-accessibility}/task_plan.md  |  0
 3 files changed, 11 insertions(+)
 create mode 100644 planning/archive/2026-02-issue-13-url-accessibility/README.md
 rename planning/{active => archive/2026-02-issue-13-url-accessibility}/progress.md (100%)
 rename planning/{active => archive/2026-02-issue-13-url-accessibility}/task_plan.md (100%)

diff --git a/planning/archive/2026-02-issue-13-url-accessibility/README.md b/planning/archive/2026-02-issue-13-url-accessibility/README.md
new file mode 100644
index 0000000..50acd3f
--- /dev/null
+++ b/planning/archive/2026-02-issue-13-url-accessibility/README.md
@@ -0,0 +1,11 @@
+# Issue #13 — Source URL Accessibility Validation
+
+**Closed:** 2026-02-17 · **Branch:** `13-fix-s3-permissions`
+
+## Outcome
+
+Added source-URL accessibility checking to the pipeline: `check_url_accessible()` in `scripts/stac_utils.py` and `scripts/urls_check_access.py` (parallel HTTP HEAD checks with an incremental CSV cache at `data/urls_access_checks.csv`, shareable with GeoBC). Integrated into `scripts/build_safe.sh` as Step 3.5, warn-only — inaccessible sources are logged but don't block the build, since GeoTIFF validation skips unreadable files downstream.
+
+The original trigger (6 items under `092p045` returning 403 from the BC objectstore) resolved upstream during the work — GeoBC fixed the permissions and all known-bad URLs returned 200 by close. The script remains as ongoing monitoring for new URLs.
+
+Archived 2026-07-18 while initializing issue #23 (monthly automation), which wires `urls_check_access.py` into the GitHub Actions workflow as a warn-only step — the same semantics established here.
diff --git a/planning/active/progress.md b/planning/archive/2026-02-issue-13-url-accessibility/progress.md
similarity index 100%
rename from planning/active/progress.md
rename to planning/archive/2026-02-issue-13-url-accessibility/progress.md
diff --git a/planning/active/task_plan.md b/planning/archive/2026-02-issue-13-url-accessibility/task_plan.md
similarity index 100%
rename from planning/active/task_plan.md
rename to planning/archive/2026-02-issue-13-url-accessibility/task_plan.md

From 2984555b99b050ece165157569d775a526d5070e Mon Sep 17 00:00:00 2001
From: almac2022 <al@newgraphenvironment.com>
Date: Sat, 18 Jul 2026 15:52:32 -0700
Subject: [PATCH 2/7] Initialize PWF baseline for #23

Approved phases: CI-safe pipeline fixes, rtj#184 companion infra,
monthly update workflow, post-merge catch-up run. Plan reviewed by
Plan agent before baseline (2 blockers + 8 gaps absorbed).

Relates to #23

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HN1tMTLezPVkv9eT86FLSH
---
 planning/active/findings.md  | 29 +++++++++++++++++++++++++
 planning/active/progress.md  | 12 +++++++++++
 planning/active/task_plan.md | 41 ++++++++++++++++++++++++++++++++++++
 3 files changed, 82 insertions(+)
 create mode 100644 planning/active/findings.md
 create mode 100644 planning/active/progress.md
 create mode 100644 planning/active/task_plan.md

diff --git a/planning/active/findings.md b/planning/active/findings.md
new file mode 100644
index 0000000..a5436e5
--- /dev/null
+++ b/planning/active/findings.md
@@ -0,0 +1,29 @@
+# Findings — Automate monthly incremental catalog updates via GitHub Actions (#23)
+
+## Issue context
+
+Issue #23 (filed 2026-07-18): catalog five months stale (inventory 2026-02-18, S3 build 2026-02-11, pgstac registration 2026-02-13 with 58,019 items). Phase 3 automation from #5 never re-tracked after PR #9's "Closes #5" auto-closed the umbrella. Adopt the water-temp-bc pattern (monthly GHA cron + OIDC, decision record NewGraphEnvironment/water-temp-bc#17, reference workflow `.github/workflows/snapshot.yml`). Companion infra: rtj#184 (`modules/gha_s3_role` consumer). Registration stays manual-on-geoserv for v1; incremental pypgstac upsert is a named follow-up. Out of scope: full rebuilds (rtj#49), source-file deletions, #16 closure.
+
+## Verified pipeline contracts (2026-07-18 exploration)
+
+- `scripts/detect_changes.R` — self-contained: fetches fresh listing (`ngr::ngr_s3_keys_get`), diffs against `data/urls_list.txt`, writes/deletes `urls_new.txt` + `urls_deleted.txt`, overwrites the cache, sink()-logs to `logs/`. Exit 0 = no changes, 1 = changes; **R errors also exit 1** (ambiguous — fix in Phase 1) and a **deletions-only month exits 1 with no urls_new.txt** (workflow must branch on file presence). `urls_fetch.R` is redundant in CI.
+- `scripts/item_create.py --incremental` — reads `data/urls_new.txt`, needs only `$OUTPUT_DIR/collection.json` locally (10.9 MB from S3; item JSONs not required), appends links with duplicate prevention, saves collection. `get_output_dir()` (`scripts/stac_utils.py:39`) hardcodes `/Users/airvine/...` — used by item_create, collection_create, item_reprocess.
+- `scripts/item_validate.py` — already parameterized (`--items-dir`, `--incremental`) and already exits non-zero when any item invalid (verified `return 0 if invalid == 0 else 1`). Caveat: the count spans history + new, so one bad item blocks the whole batch and re-fails monthly — v1 fail-loud by design; remediation via `urls_invalid_items.txt` + `item_reprocess.py` (document in README triage).
+- `scripts/urls_check_access.py` — **hard-exits 1 on any inaccessible URL** (even from cache). `build_safe.sh` Step 3.5 treats it warn-only; the workflow must too. `data/urls_access_checks.csv` is currently untracked — commit-back uses `git add -A data/` and the CSV gets committed (GeoBC-report purpose from #13).
+- `scripts/s3_sync.R` — laptop-only (`--delete --profile airvine`, hardcoded path). Never run in CI: `--delete` from a stateless runner would wipe the 58k items. **S3 is the only catalog copy** (local prod dir verified empty; CLAUDE.md's "Local STAC output" note is stale).
+- `environment.yml` — pure pip under conda → `uv pip install` works today; #16 not a blocker. Missing `jsonschema` (via `pystac[validation]` — the validate gate depends on it) and `requests`.
+- **~2,098 URLs in the cache have no catalog item** (60,126 URLs vs 58,028 validated items), including the 90 parenthesized files added post-build (commit 0d5ab5c, 2026-02-18 — after the Feb 11 S3 build). Being cached, detect_changes will never re-flag them → one-time reconciliation (`urls_reconcile.py`) trims the cache to item-backed URLs so the catch-up run picks them up naturally.
+- State atomicity on a runner comes free for the linear path (caches persist only via end-of-job commit; failures self-heal) — but a truncated listing would poison the cache silently, hence the <90% plausibility guard.
+
+## Infra (rtj + water-temp-bc)
+
+- `modules/gha_s3_role` (rtj#147): role name derives to `role_gha_stac_dem_bc`; ListBucket + GetBucketLocation on bucket, Get/PutObject on objects, **no DeleteObject by default** — sufficient for no-delete sync and the backstop against the `--delete` wipe scenario. Account OIDC provider already provisioned. ARN pattern: `arn:aws:iam::414155577829:role/role_gha_stac_dem_bc`.
+- Bucket `stac-dem-bc`: public + CORS, **no versioning** (water-temp-bc has it) → optional versioning rider in Phase 2 since collection.json is overwritten in place every run.
+- STAC API = stac-fastapi-pgstac + TiTiler + Caddy on **geoserv** (`images.a11s.one`); DB `stac` holds stac-dem-bc. Registration: `rtj/scripts/geoserv/stac_register-pypgstac.sh` on-host — full delete-and-reload, 46 min for 58k items (S3 download dominates; DB load 10 s).
+- rtj has an in-flight branch (`172-stac-floodplains-bc-bucket`) → Phase 2 branches off rtj main. No branch protection on stac_dem_bc main; `claude.yml` has no push trigger; GITHUB_TOKEN pushes don't retrigger workflows.
+- Public repo → GitHub disables cron after 60 days without repo activity; no-change months produce no commits (document in triage).
+- Side finding (out of scope, flagged to user): `rtj/scripts/geoserv/stac_register-pypgstac.sh:82` still has the parallel-append interleave pattern from the conventions — harmless for KB-scale DEM items, trips on large payloads.
+
+## Plan-review disposition (adversarial Plan-agent pass, 2026-07-18)
+
+Absorbed into phases: warn-only access check (B1), deletions-only branch + `git add -A data/` commit semantics (B2, G4), listing plausibility guard (G5), items-before-collection sync order (G6), orphaned-URL reconciliation (G7), rebase-before-push (G8), explicit exit-code capture (G9), cron auto-disable doc (G10), oversized-batch fallback doc (A12 — a naive "process first N" cap would orphan the remainder because the cache updates eagerly), import smoke check + Python ≥3.10 pin (A13, A14), ngr SHA pin in DESCRIPTION Remotes (S18), Phase 4 count math (A21). Dropped as verified-unnecessary: "add exit code to item_validate" (already correct). Catch-up sizing note: at ~6,450 items/hour, a 35k-scale surprise busts the 330-min timeout non-convergently — log `wc -l urls_new.txt` early; fallback is a manual local run.
diff --git a/planning/active/progress.md b/planning/active/progress.md
new file mode 100644
index 0000000..2a395b6
--- /dev/null
+++ b/planning/active/progress.md
@@ -0,0 +1,12 @@
+# Progress — Automate monthly incremental catalog updates via GitHub Actions (#23)
+
+## Session 2026-07-18
+
+- Investigated why #5 closed (PR #9 "Closes #5" auto-close with Phases 3–4 unbuilt); filed #23 + companion rtj#184
+- Plan-mode exploration of pipeline contracts, water-temp-bc reference workflow, rtj infra (subagent survey)
+- Adversarial Plan-agent review: 2 blockers + 8 gaps, all absorbed (see findings.md disposition)
+- Phases approved by user via plan mode
+- Pushed main (CLAUDE.md sync commit), created branch `23-automate-monthly-incremental-catalog-upd`
+- Archived issue #13 PWF → `planning/archive/2026-02-issue-13-url-accessibility/` (commit c72ecf0)
+- Scaffolded PWF baseline from issue #23 with approved phases
+- Next: Phase 1 (CI-safe pipeline fixes)
diff --git a/planning/active/task_plan.md b/planning/active/task_plan.md
new file mode 100644
index 0000000..7ed9491
--- /dev/null
+++ b/planning/active/task_plan.md
@@ -0,0 +1,41 @@
+# Task: Automate monthly incremental catalog updates via GitHub Actions (#23)
+
+The catalog is five months stale: the source URL inventory was last refreshed 2026-02-18, the S3 catalog last built 2026-02-11, and pgstac last registered 2026-02-13. Automation was scoped as Phases 3–4 of #5 but never re-tracked after that issue auto-closed, and its standing-VM premise no longer matches infrastructure. Adopt the water-temp-bc pattern: monthly GitHub Actions cron + `workflow_dispatch`, OIDC role provisioned in rtj (rtj#184), incremental pipeline run on the runner, catalog synced to S3, refreshed caches committed back to `main`.
+
+## Phase 1: CI-safe pipeline fixes
+
+- [ ] `scripts/stac_utils.py`: `get_output_dir()` honors `STAC_OUTPUT_DIR` env override before mode defaults
+- [ ] `scripts/detect_changes.R`: tryCatch wrapper — R errors exit 2 (0 = no changes, 1 = changes, 2 = error); plausibility guard: fresh listing < 90% of cached → exit 2 without touching cache or outputs
+- [ ] `environment.yml`: add `pystac[validation]` and `requests` so the CI dep list and conda env stay consistent
+- [ ] New `scripts/s3_sync-ci.sh`: no-delete sync on default credential chain; items first (`--exclude "collection.json"`), `aws s3 cp` collection.json last; `--dryrun` passthrough; guards (`STAC_OUTPUT_DIR` set, dir non-empty, collection.json present)
+- [ ] New `scripts/urls_reconcile.py` (one-off helper): compute item-backed URLs from the validation CSV via the url→id mapping, rewrite `urls_list.txt` to that subset (`--dry-run` default) so detect_changes re-flags the ~2,098 never-built URLs on the catch-up run
+- [ ] Cold-path rehearsal (local): temp `STAC_OUTPUT_DIR` with only S3-fetched collection.json, synthetic 2–3-URL urls_new.txt → item_create → item_validate → `s3_sync-ci.sh --dryrun`; assert zero deletions planned, items-before-collection order, only new files uploaded
+
+## Phase 2: Companion infra — rtj#184 (in ~/Projects/repo/rtj, fresh branch off rtj main)
+
+- [ ] Add `module "stac_dem_bc_update"` + ARN output to `env/prod/main.tf` (block as written in rtj#184)
+- [ ] Optional rider (user call): enable versioning on `stac-dem-bc` in `var.s3_buckets` — no versioning today and collection.json is overwritten in place every run
+- [ ] `tofu plan` / `apply` in env/prod (apply is collaborative — user credentials)
+- [ ] Post role ARN to #23; commit in rtj closes rtj#184
+
+## Phase 3: Workflow
+
+- [ ] Minimal `DESCRIPTION` for R CI deps (ngr pinned to SHA via Remotes, readr, fs), mirroring water-temp-bc
+- [ ] `.github/workflows/update.yml`: monthly cron (`23 9 3 * *`) + `workflow_dispatch`; `permissions: id-token: write, contents: write`; concurrency group (no cancel); `timeout-minutes: 330`; Python ≥3.10 pin; steps: checkout → setup R + deps → setup uv + `uv pip install` (incl. `pystac[validation]`, requests) → import smoke check (`python -c "import rasterio, rio_stac"`) → `configure-aws-credentials` (role_gha_stac_dem_bc) → detect changes with explicit exit-code capture (0 → early-exit success; 2 → fail; 1 → continue) → log `wc -l urls_new.txt` → branch: item steps only if urls_new.txt exists & non-empty (deletions-only month still reaches commit-back) → `urls_check_access.py --urls-file data/urls_new.txt` as warn-only (`continue-on-error`, CSV surfaced via artifact) → fetch collection.json from S3 → `item_create.py --incremental` → `item_validate.py --incremental --items-dir` (hard gate) → `s3_sync-ci.sh` → commit-back: `git add -A data/` (covers modifications, deletions, new access CSV), `git pull --rebase origin main`, push as bot → upload `logs/` artifact (always)
+- [ ] `scripts/README.md` Automation section: schedule, manual dispatch, failure triage (invalid-item-blocks-batch → `urls_invalid_items.txt` + `item_reprocess.py`; oversized-batch fallback = manual local run; public-repo 60-day cron auto-disable), registration step
+- [ ] CLAUDE.md: fix stale "Local STAC output" path note (S3 is the sole catalog copy)
+
+## Phase 4: Catch-up run + verification (post-merge)
+
+- [ ] PR via `/gh-pr-push` (body: Relates to #23, why urls_fetch.R is bypassed in CI), merge
+- [ ] Seed catch-up: run `urls_reconcile.py` for real, commit trimmed `urls_list.txt` to main — next run re-detects the ~2,098 never-built URLs plus 5 months of growth
+- [ ] `workflow_dispatch` from main; watch timing; verify cache commit lands and count math holds: S3 objects ≈ valid-item count + collection.json; residual URL-vs-item gap fully explained by known-invalid entries in `stac_geotiff_checks.csv`
+- [ ] Register on geoserv (`stac_register-pypgstac.sh stac-dem-bc ...`); verify pgstac count + API query at images.a11s.one returns a new item
+- [ ] Confirm cron live; close #23 via docs commit ("monthly automation live; Fixes #23")
+
+## Validation
+
+- [ ] Cold-path rehearsal passed (dry-run: zero deletions, correct order)
+- [ ] `/code-check` clean on each commit
+- [ ] Catch-up run: new items on S3 + caches committed by bot + API returns a new item + count math documented in run log
+- [ ] PWF checkboxes match landed work; `/planning-archive` on completion

From 80bb5254948b8457f14f08241aedb76643dc360f Mon Sep 17 00:00:00 2001
From: almac2022 <al@newgraphenvironment.com>
Date: Sat, 18 Jul 2026 16:20:47 -0700
Subject: [PATCH 3/7] Phase 1: make incremental pipeline CI-safe for stateless
 runners

- stac_utils.py: STAC_OUTPUT_DIR env override for get_output_dir()
- detect_changes.R: exit contract 0/1/2 (no changes/changes/error) via
  tryCatch; plausibility guard refuses cache update when fresh listing
  < 90% of cached (truncated-fetch poisoning); no library() calls so
  missing packages exit 2, not a false "changes detected"
- scripts/s3_sync-ci.sh: no-delete sync on default credential chain,
  item JSONs before collection.json (no dangling links on failure),
  --dryrun passthrough, trailing-slash bucket normalization
- scripts/urls_reconcile.py: one-off cache trim to item-backed URLs so
  detection re-flags 2,107 never-built URLs (90 parenthesized + post-
  build refresh remainder)
- environment.yml: pystac[validation] (jsonschema for the validate
  gate) + requests

Cold-path rehearsal passed on a stateless workspace (S3-fetched
collection.json only): dedupe, validate skip, dry-run sync with zero
deletions. Live ngr fetch shows objectstore at 98,039 URLs (+37,900
since Feb) - catch-up run moves to the documented local fallback path;
task_plan Phase 4 amended.

Relates to #23

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HN1tMTLezPVkv9eT86FLSH
---
 environment.yml              |   3 +-
 planning/active/findings.md  |  19 +++
 planning/active/progress.md  |   4 +-
 planning/active/task_plan.md |  17 +--
 scripts/detect_changes.R     | 230 +++++++++++++++++++----------------
 scripts/s3_sync-ci.sh        |  62 ++++++++++
 scripts/stac_utils.py        |  11 +-
 scripts/urls_reconcile.py    |  71 +++++++++++
 8 files changed, 303 insertions(+), 114 deletions(-)
 create mode 100755 scripts/s3_sync-ci.sh
 create mode 100755 scripts/urls_reconcile.py

diff --git a/environment.yml b/environment.yml
index e0347ce..81be418 100644
--- a/environment.yml
+++ b/environment.yml
@@ -7,7 +7,7 @@ dependencies:
   - pip
   - pip:
       # Core STAC packages
-      - pystac>=1.12.0
+      - pystac[validation]>=1.12.0  # validation extra = jsonschema, required by item_validate.py
       - pystac-client>=0.8.0
       - rio-stac>=0.11.0
       # Geospatial packages
@@ -17,5 +17,6 @@ dependencies:
       # Data processing
       - pandas
       # Utilities
+      - requests  # HTTP checks (stac_utils, urls_check_access.py)
       - tqdm
       - deepdiff  # JSON/dict comparison for QA and debugging
diff --git a/planning/active/findings.md b/planning/active/findings.md
index a5436e5..9b6cfee 100644
--- a/planning/active/findings.md
+++ b/planning/active/findings.md
@@ -24,6 +24,25 @@ Issue #23 (filed 2026-07-18): catalog five months stale (inventory 2026-02-18, S
 - Public repo → GitHub disables cron after 60 days without repo activity; no-change months produce no commits (document in triage).
 - Side finding (out of scope, flagged to user): `rtj/scripts/geoserv/stac_register-pypgstac.sh:82` still has the parallel-append interleave pattern from the conventions — harmless for KB-scale DEM items, trips on large payloads.
 
+## Phase 1 cold-path rehearsal (2026-07-18)
+
+Stateless-runner path exercised locally with a uv venv (Python 3.12, exact CI dep list — the install itself doubles as the CI-install rehearsal; `import rasterio, rio_stac, pystac, jsonschema` smoke passed, rasterio 1.5.0 manylinux/macos wheels bundle GDAL+PROJ):
+
+- Workspace = temp `STAC_OUTPUT_DIR` holding only the S3-fetched `collection.json` (10.9 MB); synthetic 3-URL `urls_new.txt` from existing catalog URLs.
+- `item_create.py --incremental`: loaded 58,019 links, extracted metadata remotely (~2.3 s/file), wrote 3 item JSONs, added 0 links / skipped 3 duplicates (dedupe ✓), saved collection into the workspace ✓.
+- `item_validate.py --items-dir "$STAC_OUTPUT_DIR" --incremental`: found 3 files, all already in the committed CSV → "Validating 0 new items", exit 0 ✓ (gate semantics for genuinely-new items verified by code review).
+- `s3_sync-ci.sh --dryrun`: 3 item uploads then collection.json last, **zero delete operations** ✓; empty-array expansion works on macOS bash 3.2.
+- `urls_reconcile.py` dry-run: 60,126 cached / 58,019 item-backed / **2,107 never built** (parenthesized "(2)" files lead the list). Item-backed count exactly matches the Feb pgstac registration (58,019).
+- Rehearsal side-effects on tracked `data/` files restored via git checkout (the run's `stac_geotiff_checks.csv` rewrite also shrank it 60,325→60,307 rows — pre-existing item_create cache-merge behavior, not investigated here).
+
+**URL-scheme quirk (load-bearing):** every URL in the cache, the checks CSV, and ngr's live output uses a single-slash scheme (`https:/nrs...`, 0 double-slash of 98,039 fetched). Format is uniform across all three sources, so diffs are sane — but any future ngr "fix" to emit `https://` would make one detection run flag everything new+deleted. The plausibility guard doesn't catch same-size format flips; the massive-new-count log line is the tell.
+
+**Catch-up sizing (changes Phase 4):** live fetch 2026-07-18 returned **98,039 URLs vs 60,126 cached → ~37,900 new in 5 months (+63%)**, echoing the Feb discovery (+35,569). With the 2,107 reconciled, catch-up ≈ 40k items ≈ 6+ h at the documented ~6,450 items/h — beyond the 330-min GHA timeout and non-convergent on retry. The plan's documented oversized-batch fallback (manual local run) is therefore the catch-up path; the dispatch run verifies steady state instead. Steady-state months at the observed growth rate (~7,600 files/month ≈ 75 min) fit the timeout comfortably.
+
+## Code-check (3 fresh-eyes rounds, 2026-07-18)
+
+Round 1: `library(ngr)` sat before the tryCatch — a missing package on a fresh runner would exit 1, which the workflow reads as "changes detected" against the stale tracked urls_new.txt. Fixed by removing library() calls entirely (namespaced calls fail inside tryCatch → exit 2). Round 2: a trailing slash in `STAC_S3_BUCKET` would make the collection.json `cp` write a hidden `/collection.json` key — sync fine, cp exit 0, live collection silently never updates. Fixed with `BUCKET="${BUCKET%/}"`. Round 3 clean; also verified the 8 validation-CSV ids absent from the cache are the 8 known upstream-deleted files, and that the exit-0 path deletes stale tracked urls_new/urls_deleted from the checkout (stateless runner can't reprocess stale change files).
+
 ## Plan-review disposition (adversarial Plan-agent pass, 2026-07-18)
 
 Absorbed into phases: warn-only access check (B1), deletions-only branch + `git add -A data/` commit semantics (B2, G4), listing plausibility guard (G5), items-before-collection sync order (G6), orphaned-URL reconciliation (G7), rebase-before-push (G8), explicit exit-code capture (G9), cron auto-disable doc (G10), oversized-batch fallback doc (A12 — a naive "process first N" cap would orphan the remainder because the cache updates eagerly), import smoke check + Python ≥3.10 pin (A13, A14), ngr SHA pin in DESCRIPTION Remotes (S18), Phase 4 count math (A21). Dropped as verified-unnecessary: "add exit code to item_validate" (already correct). Catch-up sizing note: at ~6,450 items/hour, a 35k-scale surprise busts the 330-min timeout non-convergently — log `wc -l urls_new.txt` early; fallback is a manual local run.
diff --git a/planning/active/progress.md b/planning/active/progress.md
index 2a395b6..985959f 100644
--- a/planning/active/progress.md
+++ b/planning/active/progress.md
@@ -9,4 +9,6 @@
 - Pushed main (CLAUDE.md sync commit), created branch `23-automate-monthly-incremental-catalog-upd`
 - Archived issue #13 PWF → `planning/archive/2026-02-issue-13-url-accessibility/` (commit c72ecf0)
 - Scaffolded PWF baseline from issue #23 with approved phases
-- Next: Phase 1 (CI-safe pipeline fixes)
+- Phase 1 complete: STAC_OUTPUT_DIR override, detect_changes exit contract + plausibility guard, environment.yml deps, s3_sync-ci.sh, urls_reconcile.py; cold-path rehearsal passed (zero deletes, correct order, dedupe verified)
+- Live ngr fetch: objectstore at 98,039 URLs (+37,900 since Feb) → catch-up moved to local run, dispatch verifies steady state (task_plan Phase 4 amended; findings.md has the numbers)
+- Next: Phase 2 (rtj#184 role) and Phase 3 (workflow)
diff --git a/planning/active/task_plan.md b/planning/active/task_plan.md
index 7ed9491..4678a3b 100644
--- a/planning/active/task_plan.md
+++ b/planning/active/task_plan.md
@@ -4,12 +4,12 @@ The catalog is five months stale: the source URL inventory was last refreshed 20
 
 ## Phase 1: CI-safe pipeline fixes
 
-- [ ] `scripts/stac_utils.py`: `get_output_dir()` honors `STAC_OUTPUT_DIR` env override before mode defaults
-- [ ] `scripts/detect_changes.R`: tryCatch wrapper — R errors exit 2 (0 = no changes, 1 = changes, 2 = error); plausibility guard: fresh listing < 90% of cached → exit 2 without touching cache or outputs
-- [ ] `environment.yml`: add `pystac[validation]` and `requests` so the CI dep list and conda env stay consistent
-- [ ] New `scripts/s3_sync-ci.sh`: no-delete sync on default credential chain; items first (`--exclude "collection.json"`), `aws s3 cp` collection.json last; `--dryrun` passthrough; guards (`STAC_OUTPUT_DIR` set, dir non-empty, collection.json present)
-- [ ] New `scripts/urls_reconcile.py` (one-off helper): compute item-backed URLs from the validation CSV via the url→id mapping, rewrite `urls_list.txt` to that subset (`--dry-run` default) so detect_changes re-flags the ~2,098 never-built URLs on the catch-up run
-- [ ] Cold-path rehearsal (local): temp `STAC_OUTPUT_DIR` with only S3-fetched collection.json, synthetic 2–3-URL urls_new.txt → item_create → item_validate → `s3_sync-ci.sh --dryrun`; assert zero deletions planned, items-before-collection order, only new files uploaded
+- [x] `scripts/stac_utils.py`: `get_output_dir()` honors `STAC_OUTPUT_DIR` env override before mode defaults
+- [x] `scripts/detect_changes.R`: tryCatch wrapper — R errors exit 2 (0 = no changes, 1 = changes, 2 = error); plausibility guard: fresh listing < 90% of cached → exit 2 without touching cache or outputs
+- [x] `environment.yml`: add `pystac[validation]` and `requests` so the CI dep list and conda env stay consistent
+- [x] New `scripts/s3_sync-ci.sh`: no-delete sync on default credential chain; items first (`--exclude "collection.json"`), `aws s3 cp` collection.json last; `--dryrun` passthrough; guards (`STAC_OUTPUT_DIR` set, dir non-empty, collection.json present)
+- [x] New `scripts/urls_reconcile.py` (one-off helper): compute item-backed URLs from the validation CSV via the url→id mapping, rewrite `urls_list.txt` to that subset (dry-run default, `--apply` to rewrite) — re-flags the 2,107 never-built URLs on the catch-up run
+- [x] Cold-path rehearsal (local): temp `STAC_OUTPUT_DIR` with only S3-fetched collection.json, synthetic 3-URL urls_new.txt → item_create → item_validate → `s3_sync-ci.sh --dryrun`; verified zero deletions planned, items-before-collection order, only new files uploaded (see findings.md 2026-07-18 rehearsal)
 
 ## Phase 2: Companion infra — rtj#184 (in ~/Projects/repo/rtj, fresh branch off rtj main)
 
@@ -28,8 +28,9 @@ The catalog is five months stale: the source URL inventory was last refreshed 20
 ## Phase 4: Catch-up run + verification (post-merge)
 
 - [ ] PR via `/gh-pr-push` (body: Relates to #23, why urls_fetch.R is bypassed in CI), merge
-- [ ] Seed catch-up: run `urls_reconcile.py` for real, commit trimmed `urls_list.txt` to main — next run re-detects the ~2,098 never-built URLs plus 5 months of growth
-- [ ] `workflow_dispatch` from main; watch timing; verify cache commit lands and count math holds: S3 objects ≈ valid-item count + collection.json; residual URL-vs-item gap fully explained by known-invalid entries in `stac_geotiff_checks.csv`
+- [ ] Seed catch-up: run `urls_reconcile.py --apply`, commit trimmed `urls_list.txt` to main — next detection re-flags the 2,107 never-built URLs plus 5 months of growth
+- [ ] Catch-up build runs LOCALLY, not via dispatch (live fetch 2026-07-18: objectstore now at 98,039 URLs → ~40k new+reconciled items ≈ 6+ h at Feb rates, beyond the GHA timeout — this is the plan's documented oversized-batch path): detect_changes → incremental pipeline with `STAC_OUTPUT_DIR` workspace → `s3_sync-ci.sh` → commit caches
+- [ ] `workflow_dispatch` from main verifies the steady-state path end to end (post-catch-up delta or clean no-change exit); count math: S3 objects ≈ valid-item count + collection.json; residual URL-vs-item gap fully explained by known-invalid entries in `stac_geotiff_checks.csv`
 - [ ] Register on geoserv (`stac_register-pypgstac.sh stac-dem-bc ...`); verify pgstac count + API query at images.a11s.one returns a new item
 - [ ] Confirm cron live; close #23 via docs commit ("monthly automation live; Fixes #23")
 
diff --git a/scripts/detect_changes.R b/scripts/detect_changes.R
index 345a8b7..3ee6386 100644
--- a/scripts/detect_changes.R
+++ b/scripts/detect_changes.R
@@ -2,127 +2,151 @@
 # Change Detection: Compare BC DEM objectstore with cached URL list
 # Outputs: urls_new.txt, urls_deleted.txt
 # Updates: urls_list.txt
-# Exit: 0 if no changes, 1 if changes detected
+# Exit: 0 = no changes, 1 = changes detected, 2 = error (including the
+#       plausibility guard — a suspiciously short listing refuses to touch
+#       the cache or outputs, since a truncated fetch committed into
+#       urls_list.txt would mark everything deleted and poison the next run)
 
-library(ngr)
+# NOTE: no library() calls here — everything is namespaced (ngr::, readr::,
+# fs::) so a missing package fails inside the tryCatch and exits 2 (error),
+# not 1 (which the workflow reads as "changes detected").
 
 # Setup logging
 timestamp <- format(Sys.time(), "%Y%m%d_%H%M%S")
 log_file <- sprintf("logs/%s_change_detection.log", timestamp)
+fs::dir_create("logs")
 sink(log_file, split = TRUE)
 
-cat("=== STAC DEM BC Change Detection ===\n")
-cat(sprintf("Started: %s\n\n", Sys.time()))
-
-# Paths
-url_bucket <- "https://nrs.objectstore.gov.bc.ca/gdwuts"
-cache_file <- "data/urls_list.txt"
-new_file <- "data/urls_new.txt"
-deleted_file <- "data/urls_deleted.txt"
-
-# Step 1: Fetch fresh URL list from objectstore
-cat("Fetching fresh URL list from BC objectstore...\n")
-cat(sprintf("  Bucket: %s\n", url_bucket))
-cat(sprintf("  Pattern: dem + *.tif\n\n"))
-
-start_time <- Sys.time()
-
-fresh_urls <- ngr::ngr_s3_keys_get(
-  url_bucket = url_bucket,
-  prefix = "",
-  pattern = c("dem", "*.tif")
-)
-
-fetch_time <- as.numeric(difftime(Sys.time(), start_time, units = "secs"))
-
-cat(sprintf("Fetched %d URLs in %.1f seconds (%.1f URLs/sec)\n\n",
-            length(fresh_urls), fetch_time, length(fresh_urls) / fetch_time))
-
-# Step 2: Load cached URL list
-if (file.exists(cache_file)) {
-  cat(sprintf("Loading cached URL list from %s...\n", cache_file))
-  cached_urls <- readr::read_lines(cache_file)
-  cat(sprintf("Cached: %d URLs\n\n", length(cached_urls)))
-} else {
-  cat(sprintf("No cache found at %s - treating all URLs as new\n\n", cache_file))
-  cached_urls <- character(0)
-}
-
-# Step 3: Compare to find changes
-cat("Comparing cached vs. fresh URLs...\n")
-
-# Convert to character vectors for comparison
-fresh_set <- as.character(fresh_urls)
-cached_set <- as.character(cached_urls)
-
-# Find new and deleted URLs
-new_urls <- setdiff(fresh_set, cached_set)
-deleted_urls <- setdiff(cached_set, fresh_set)
-
-cat(sprintf("  New URLs: %d\n", length(new_urls)))
-cat(sprintf("  Deleted URLs: %d\n", length(deleted_urls)))
-cat(sprintf("  Unchanged URLs: %d\n\n", length(intersect(fresh_set, cached_set))))
-
-# Step 4: Write output files
-fs::dir_create("data")
+result <- tryCatch({
+  cat("=== STAC DEM BC Change Detection ===\n")
+  cat(sprintf("Started: %s\n\n", Sys.time()))
+
+  # Paths
+  url_bucket <- "https://nrs.objectstore.gov.bc.ca/gdwuts"
+  cache_file <- "data/urls_list.txt"
+  new_file <- "data/urls_new.txt"
+  deleted_file <- "data/urls_deleted.txt"
+
+  # Step 1: Fetch fresh URL list from objectstore
+  cat("Fetching fresh URL list from BC objectstore...\n")
+  cat(sprintf("  Bucket: %s\n", url_bucket))
+  cat(sprintf("  Pattern: dem + *.tif\n\n"))
+
+  start_time <- Sys.time()
+
+  fresh_urls <- ngr::ngr_s3_keys_get(
+    url_bucket = url_bucket,
+    prefix = "",
+    pattern = c("dem", "*.tif")
+  )
+
+  fetch_time <- as.numeric(difftime(Sys.time(), start_time, units = "secs"))
+
+  cat(sprintf("Fetched %d URLs in %.1f seconds (%.1f URLs/sec)\n\n",
+              length(fresh_urls), fetch_time, length(fresh_urls) / fetch_time))
+
+  # Step 2: Load cached URL list
+  if (file.exists(cache_file)) {
+    cat(sprintf("Loading cached URL list from %s...\n", cache_file))
+    cached_urls <- readr::read_lines(cache_file)
+    cat(sprintf("Cached: %d URLs\n\n", length(cached_urls)))
+  } else {
+    cat(sprintf("No cache found at %s - treating all URLs as new\n\n", cache_file))
+    cached_urls <- character(0)
+  }
 
-if (length(new_urls) > 0) {
-  cat(sprintf("Writing new URLs to %s...\n", new_file))
-  readr::write_lines(new_urls, new_file)
-  cat(sprintf("  Wrote %d URLs\n", length(new_urls)))
-} else {
-  cat("No new URLs - not creating urls_new.txt\n")
-  if (file.exists(new_file)) {
-    file.remove(new_file)
-    cat("  Removed old urls_new.txt\n")
+  # Convert to character vectors for comparison
+  fresh_set <- as.character(fresh_urls)
+  cached_set <- as.character(cached_urls)
+
+  # Plausibility guard: a truncated or empty listing must not poison the
+  # cache. The dataset has only ever grown; a >10% shrink is a fetch
+  # problem until proven otherwise.
+  if (length(cached_set) > 0 && length(fresh_set) < 0.9 * length(cached_set)) {
+    stop(sprintf(
+      "fresh listing (%d URLs) is < 90%% of cached (%d) - refusing to update cache (truncated fetch?)",
+      length(fresh_set), length(cached_set)
+    ))
   }
-}
 
-if (length(deleted_urls) > 0) {
-  cat(sprintf("Writing deleted URLs to %s...\n", deleted_file))
-  readr::write_lines(deleted_urls, deleted_file)
-  cat(sprintf("  Wrote %d URLs\n", length(deleted_urls)))
-} else {
-  cat("No deleted URLs - not creating urls_deleted.txt\n")
-  if (file.exists(deleted_file)) {
-    file.remove(deleted_file)
-    cat("  Removed old urls_deleted.txt\n")
+  # Step 3: Compare to find changes
+  cat("Comparing cached vs. fresh URLs...\n")
+
+  # Find new and deleted URLs
+  new_urls <- setdiff(fresh_set, cached_set)
+  deleted_urls <- setdiff(cached_set, fresh_set)
+
+  cat(sprintf("  New URLs: %d\n", length(new_urls)))
+  cat(sprintf("  Deleted URLs: %d\n", length(deleted_urls)))
+  cat(sprintf("  Unchanged URLs: %d\n\n", length(intersect(fresh_set, cached_set))))
+
+  # Step 4: Write output files
+  fs::dir_create("data")
+
+  if (length(new_urls) > 0) {
+    cat(sprintf("Writing new URLs to %s...\n", new_file))
+    readr::write_lines(new_urls, new_file)
+    cat(sprintf("  Wrote %d URLs\n", length(new_urls)))
+  } else {
+    cat("No new URLs - not creating urls_new.txt\n")
+    if (file.exists(new_file)) {
+      file.remove(new_file)
+      cat("  Removed old urls_new.txt\n")
+    }
   }
-}
 
-# Step 5: Update cache with fresh URLs
-cat(sprintf("\nUpdating cache (%s) with fresh URL list...\n", cache_file))
-readr::write_lines(fresh_set, cache_file)
-cat(sprintf("  Wrote %d URLs to cache\n", length(fresh_set)))
-
-# Summary
-cat("\n=== SUMMARY ===\n")
-cat(sprintf("Fresh URLs: %d\n", length(fresh_set)))
-cat(sprintf("Cached URLs: %d\n", length(cached_set)))
-cat(sprintf("New: %d\n", length(new_urls)))
-cat(sprintf("Deleted: %d\n", length(deleted_urls)))
-cat(sprintf("Total changes: %d\n", length(new_urls) + length(deleted_urls)))
-
-# Show sample of new URLs if any
-if (length(new_urls) > 0) {
-  cat("\nSample new URLs (first 5):\n")
-  for (url in head(new_urls, 5)) {
-    cat(sprintf("  %s\n", url))
+  if (length(deleted_urls) > 0) {
+    cat(sprintf("Writing deleted URLs to %s...\n", deleted_file))
+    readr::write_lines(deleted_urls, deleted_file)
+    cat(sprintf("  Wrote %d URLs\n", length(deleted_urls)))
+  } else {
+    cat("No deleted URLs - not creating urls_deleted.txt\n")
+    if (file.exists(deleted_file)) {
+      file.remove(deleted_file)
+      cat("  Removed old urls_deleted.txt\n")
+    }
   }
-  if (length(new_urls) > 5) {
-    cat(sprintf("  ... and %d more\n", length(new_urls) - 5))
+
+  # Step 5: Update cache with fresh URLs
+  cat(sprintf("\nUpdating cache (%s) with fresh URL list...\n", cache_file))
+  readr::write_lines(fresh_set, cache_file)
+  cat(sprintf("  Wrote %d URLs to cache\n", length(fresh_set)))
+
+  # Summary
+  cat("\n=== SUMMARY ===\n")
+  cat(sprintf("Fresh URLs: %d\n", length(fresh_set)))
+  cat(sprintf("Cached URLs: %d\n", length(cached_set)))
+  cat(sprintf("New: %d\n", length(new_urls)))
+  cat(sprintf("Deleted: %d\n", length(deleted_urls)))
+  cat(sprintf("Total changes: %d\n", length(new_urls) + length(deleted_urls)))
+
+  # Show sample of new URLs if any
+  if (length(new_urls) > 0) {
+    cat("\nSample new URLs (first 5):\n")
+    for (url in head(new_urls, 5)) {
+      cat(sprintf("  %s\n", url))
+    }
+    if (length(new_urls) > 5) {
+      cat(sprintf("  ... and %d more\n", length(new_urls) - 5))
+    }
   }
-}
 
-cat(sprintf("\nCompleted: %s\n", Sys.time()))
-cat(sprintf("Log saved to: %s\n", log_file))
+  cat(sprintf("\nCompleted: %s\n", Sys.time()))
+  cat(sprintf("Log saved to: %s\n", log_file))
 
-sink()
+  length(new_urls) > 0 || length(deleted_urls) > 0
+}, error = function(e) {
+  cat(sprintf("\nERROR: %s\n", conditionMessage(e)))
+  "error"
+})
 
-# Exit code: 0 if no changes, 1 if changes detected
-changes_detected <- length(new_urls) > 0 || length(deleted_urls) > 0
+sink()
 
-if (changes_detected) {
+# Exit code: 0 = no changes, 1 = changes detected, 2 = error
+if (identical(result, "error")) {
+  cat("Error during change detection - exit code 2\n")
+  quit(status = 2)
+} else if (isTRUE(result)) {
   cat("Changes detected - exit code 1\n")
   quit(status = 1)
 } else {
diff --git a/scripts/s3_sync-ci.sh b/scripts/s3_sync-ci.sh
new file mode 100755
index 0000000..ad633dd
--- /dev/null
+++ b/scripts/s3_sync-ci.sh
@@ -0,0 +1,62 @@
+#!/bin/bash
+# CI-safe S3 sync for the STAC catalog: item JSONs first, collection.json last.
+#
+# Built for a stateless runner whose $STAC_OUTPUT_DIR holds ONLY
+# collection.json plus the newly created item JSONs. Two hard rules:
+#
+#   1. Never --delete. S3 is the only complete copy of the catalog
+#      (58k+ items); syncing a partial local tree with --delete would
+#      remove every object not present locally. (scripts/s3_sync.R is the
+#      laptop full-catalog tool and keeps its --delete.)
+#   2. Items before collection. collection.json links the new items, so it
+#      uploads last — a failure mid-run then leaves unreferenced items
+#      (harmless) rather than dangling links on the live collection.
+#
+# Credentials come from the default provider chain (OIDC in CI, local
+# profile via AWS_PROFILE when run by hand).
+#
+# Usage: ./scripts/s3_sync-ci.sh [--dryrun]
+# Env:   STAC_OUTPUT_DIR  local catalog dir (required)
+#        STAC_S3_BUCKET   target (default: s3://stac-dem-bc)
+
+set -euo pipefail
+
+EXTRA_ARGS=()
+if [ "${1:-}" = "--dryrun" ]; then
+  EXTRA_ARGS=(--dryrun)
+elif [ -n "${1:-}" ]; then
+  echo "Usage: $0 [--dryrun]" >&2
+  exit 1
+fi
+
+BUCKET="${STAC_S3_BUCKET:-s3://stac-dem-bc}"
+# strip any trailing slash: "s3://bucket/" + "/collection.json" would write a
+# hidden "/collection.json" key — cp succeeds, live collection never updates
+BUCKET="${BUCKET%/}"
+
+if [ -z "${STAC_OUTPUT_DIR:-}" ]; then
+  echo "ERROR: STAC_OUTPUT_DIR is not set" >&2
+  exit 1
+fi
+if [ ! -d "$STAC_OUTPUT_DIR" ]; then
+  echo "ERROR: STAC_OUTPUT_DIR is not a directory: $STAC_OUTPUT_DIR" >&2
+  exit 1
+fi
+if [ ! -s "$STAC_OUTPUT_DIR/collection.json" ]; then
+  echo "ERROR: collection.json missing or empty in $STAC_OUTPUT_DIR" >&2
+  exit 1
+fi
+
+ITEM_COUNT=$(find "$STAC_OUTPUT_DIR" -maxdepth 1 -type f -name "*.json" ! -name "collection.json" | wc -l | tr -d ' ')
+echo "Uploading $ITEM_COUNT item JSON(s), then collection.json: $STAC_OUTPUT_DIR -> $BUCKET"
+
+# ${EXTRA_ARGS[@]+...} keeps set -u happy on bash 3.2 (macOS) when the array is empty
+aws s3 sync "$STAC_OUTPUT_DIR" "$BUCKET" \
+  --exclude "collection.json" --exclude "*/.*" --exclude ".*" \
+  ${EXTRA_ARGS[@]+"${EXTRA_ARGS[@]}"}
+
+aws s3 cp "$STAC_OUTPUT_DIR/collection.json" "$BUCKET/collection.json" \
+  --content-type application/json \
+  ${EXTRA_ARGS[@]+"${EXTRA_ARGS[@]}"}
+
+echo "Sync complete: $ITEM_COUNT item(s) + collection.json"
diff --git a/scripts/stac_utils.py b/scripts/stac_utils.py
index 913cbe6..8ed06d3 100644
--- a/scripts/stac_utils.py
+++ b/scripts/stac_utils.py
@@ -10,6 +10,7 @@
 
 import json
 import logging
+import os
 import re
 from datetime import datetime, timezone
 
@@ -37,7 +38,15 @@
 
 
 def get_output_dir(test_only: bool) -> str:
-    """Return the local output directory based on mode."""
+    """Return the local output directory based on mode.
+
+    STAC_OUTPUT_DIR overrides both modes — set by the CI workflow, where
+    the catalog is assembled in a runner workspace instead of the local
+    gis directory.
+    """
+    env_dir = os.environ.get("STAC_OUTPUT_DIR")
+    if env_dir:
+        return env_dir
     if test_only:
         return "/Users/airvine/Projects/gis/stac_dem_bc/stac/dev/stac_dem_bc"
     return "/Users/airvine/Projects/gis/stac_dem_bc/stac/prod/stac_dem_bc"
diff --git a/scripts/urls_reconcile.py b/scripts/urls_reconcile.py
new file mode 100755
index 0000000..d8bdc68
--- /dev/null
+++ b/scripts/urls_reconcile.py
@@ -0,0 +1,71 @@
+#!/usr/bin/env python3
+"""One-off cache reconciliation: trim data/urls_list.txt to item-backed URLs.
+
+detect_changes.R only flags URLs that are absent from the cache. URLs that
+entered the cache without ever producing a catalog item (the 90
+parenthesized files added after the Feb 2026 build, plus ~2k others from a
+post-build refresh) are therefore invisible to change detection forever.
+
+Rewriting the cache to just the item-backed subset makes the next
+detection run treat every never-built URL as new and process it through
+the normal incremental path.
+
+Usage:
+    python scripts/urls_reconcile.py            # dry-run: report only
+    python scripts/urls_reconcile.py --apply    # rewrite data/urls_list.txt
+"""
+
+import argparse
+import csv
+import sys
+
+from stac_utils import url_to_item_id
+
+URLS_FILE = "data/urls_list.txt"
+VALIDATION_CSV = "data/stac_item_validation.csv"
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Trim urls_list.txt to URLs backed by a catalog item"
+    )
+    parser.add_argument(
+        "--apply", action="store_true",
+        help="Rewrite data/urls_list.txt (default: dry-run report)"
+    )
+    args = parser.parse_args()
+
+    with open(URLS_FILE) as f:
+        urls = [line for line in f.read().splitlines() if line.strip()]
+
+    with open(VALIDATION_CSV, newline="") as f:
+        item_ids = {row["item_id"] for row in csv.DictReader(f)}
+
+    backed = [url for url in urls if url_to_item_id(url) in item_ids]
+    orphaned = [url for url in urls if url_to_item_id(url) not in item_ids]
+
+    print(f"URLs in cache:      {len(urls)}")
+    print(f"Item-backed:        {len(backed)}")
+    print(f"Never built:        {len(orphaned)}")
+    for url in orphaned[:10]:
+        print(f"  {url}")
+    if len(orphaned) > 10:
+        print(f"  ... and {len(orphaned) - 10} more")
+
+    if not orphaned:
+        print("Cache already reconciled - nothing to do")
+        return 0
+
+    if args.apply:
+        with open(URLS_FILE, "w") as f:
+            f.write("\n".join(backed) + "\n")
+        print(f"Rewrote {URLS_FILE} with {len(backed)} item-backed URLs")
+        print("Next detect_changes.R run will re-flag the never-built URLs as new")
+    else:
+        print("Dry-run - pass --apply to rewrite data/urls_list.txt")
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

From 3704cb2397d459c30b1966a2c06d4ac877f0e8ae Mon Sep 17 00:00:00 2001
From: almac2022 <al@newgraphenvironment.com>
Date: Sat, 18 Jul 2026 16:43:51 -0700
Subject: [PATCH 4/7] Phase 3: monthly update workflow, R dep manifest,
 automation docs

- .github/workflows/update.yml: monthly cron (3rd, 09:23 UTC) +
  workflow_dispatch; OIDC via role_gha_stac_dem_bc; detect step maps the
  0/1/2 exit contract to outputs; item steps gated on urls_new.txt
  presence (deletions-only months still commit caches) and on a nonzero
  created count (an all-invalid batch stays green with a shortfall
  warning instead of wedging); access check warn-only; sync via
  s3_sync-ci.sh; caches committed back with rebase-before-push; logs
  uploaded as artifact (overwrite-safe for re-runs)
- DESCRIPTION: R dep manifest for setup-r-dependencies (fs, ngr@519c03b
  - the locally verified build - readr), water-temp-bc pattern
- scripts/README.md: Automation section with failure triage (shortfall
  remediation, oversized-batch local fallback, 60-day cron auto-disable)
- CLAUDE.md: S3 is the only complete catalog copy; local builds use the
  STAC_OUTPUT_DIR override

Relates to #23

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HN1tMTLezPVkv9eT86FLSH
---
 .github/workflows/update.yml | 160 +++++++++++++++++++++++++++++++++++
 CLAUDE.md                    |   6 +-
 DESCRIPTION                  |  16 ++++
 planning/active/findings.md  |   4 +
 planning/active/progress.md  |   3 +-
 planning/active/task_plan.md |   8 +-
 scripts/README.md            |  23 ++++-
 7 files changed, 208 insertions(+), 12 deletions(-)
 create mode 100644 .github/workflows/update.yml
 create mode 100644 DESCRIPTION

diff --git a/.github/workflows/update.yml b/.github/workflows/update.yml
new file mode 100644
index 0000000..ea61074
--- /dev/null
+++ b/.github/workflows/update.yml
@@ -0,0 +1,160 @@
+name: Monthly incremental update
+
+# Detect new DEM GeoTIFFs on the BC objectstore, build STAC items for them,
+# sync to s3://stac-dem-bc, and commit refreshed data/ caches back to main.
+# Modeled on water-temp-bc's snapshot.yml: monthly cron on a GitHub-hosted
+# runner, AWS auth via OIDC against role_gha_stac_dem_bc (provisioned by
+# NewGraphEnvironment/rtj#184, trust scoped to main).
+#
+# detect step exit contract: 0 = no changes (skip rest, succeed),
+# 1 = changes detected (continue), 2 = error (fail). A deletions-only month
+# exits 1 with no urls_new.txt - item steps are skipped but the cache commit
+# still records the deletions.
+#
+# State model: data/ caches persist only via the end-of-job commit, so a
+# failed run discards its partial state and the next run re-detects cleanly.
+# pgstac registration is a separate manual step on geoserv - see
+# scripts/README.md.
+
+on:
+  schedule:
+    # 3rd of the month, 09:23 UTC - staggered from water-temp-bc (1st) and
+    # off the top of the hour. GitHub may delay scheduled runs under load.
+    - cron: '23 9 3 * *'
+  workflow_dispatch:
+
+permissions:
+  id-token: write   # OIDC token for AWS
+  contents: write   # commit refreshed data/ caches back to main
+
+concurrency:
+  group: stac-update
+  cancel-in-progress: false
+
+env:
+  STAC_OUTPUT_DIR: ${{ github.workspace }}/stac_out
+
+jobs:
+  update:
+    runs-on: ubuntu-latest
+    # Recent growth averages ~7.6k files/month ~ 75-90 min at observed rates.
+    # Oversized batches (>~35k) cannot fit any timeout - see README triage.
+    timeout-minutes: 330
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: r-lib/actions/setup-r@v2
+        with:
+          use-public-rspm: true
+
+      - uses: r-lib/actions/setup-r-dependencies@v2
+        with:
+          extra-packages: any::sessioninfo
+
+      - uses: astral-sh/setup-uv@v5
+
+      - name: Install Python dependencies
+        run: |
+          uv venv --python 3.12 .venv
+          uv pip install --python .venv/bin/python \
+            "pystac[validation]>=1.12.0" "pystac-client>=0.8.0" \
+            "rio-stac>=0.11.0" "rasterio>=1.4.0" rio-cogeo shapely \
+            pandas requests tqdm deepdiff
+          .venv/bin/python -c "import rasterio, rio_stac, pystac, jsonschema; print('imports OK, rasterio', rasterio.__version__)"
+
+      - uses: aws-actions/configure-aws-credentials@v4
+        with:
+          role-to-assume: arn:aws:iam::414155577829:role/role_gha_stac_dem_bc
+          aws-region: us-west-2
+
+      - name: Detect changes
+        id: detect
+        shell: bash
+        run: |
+          set +e
+          Rscript scripts/detect_changes.R
+          code=$?
+          set -e
+          echo "detect exit code: $code"
+          case "$code" in
+            0) echo "changes=false" >> "$GITHUB_OUTPUT" ;;
+            1) echo "changes=true"  >> "$GITHUB_OUTPUT" ;;
+            *) exit "$code" ;;
+          esac
+          if [ -s data/urls_new.txt ]; then
+            echo "new URLs: $(wc -l < data/urls_new.txt)"
+            echo "new_urls=true" >> "$GITHUB_OUTPUT"
+          else
+            echo "no new URLs (no changes, or deletions only)"
+            echo "new_urls=false" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: Check source URL access (warn-only)
+        if: steps.detect.outputs.new_urls == 'true'
+        continue-on-error: true
+        run: .venv/bin/python scripts/urls_check_access.py --urls-file data/urls_new.txt
+
+      - name: Fetch current collection from S3
+        if: steps.detect.outputs.new_urls == 'true'
+        run: |
+          mkdir -p "$STAC_OUTPUT_DIR"
+          curl -fsSL https://stac-dem-bc.s3.amazonaws.com/collection.json \
+            -o "$STAC_OUTPUT_DIR/collection.json"
+
+      - name: Create STAC items (incremental)
+        if: steps.detect.outputs.new_urls == 'true'
+        run: .venv/bin/python scripts/item_create.py --incremental
+
+      - name: Count created items (warn on shortfall)
+        id: created
+        if: steps.detect.outputs.new_urls == 'true'
+        run: |
+          NEW=$(wc -l < data/urls_new.txt | tr -d ' ')
+          CREATED=$(find "$STAC_OUTPUT_DIR" -maxdepth 1 -type f -name "*.json" ! -name "collection.json" | wc -l | tr -d ' ')
+          echo "expected $NEW new items, created $CREATED"
+          echo "count=$CREATED" >> "$GITHUB_OUTPUT"
+          if [ "$CREATED" -lt "$NEW" ]; then
+            echo "::warning title=Item shortfall::created $CREATED of $NEW new items - some source URLs failed metadata extraction or were invalid; see run-logs artifact and the triage notes in scripts/README.md"
+          fi
+
+      # created == 0 (an all-invalid batch, precedent: the 90 parenthesized
+      # files arrived as one such delta) must not wedge the month: skip
+      # validate/sync, stay green with the shortfall warning, and let the
+      # cache commit record the batch as attempted.
+      - name: Validate new items (gate)
+        if: steps.detect.outputs.new_urls == 'true' && steps.created.outputs.count != '0'
+        run: .venv/bin/python scripts/item_validate.py --items-dir "$STAC_OUTPUT_DIR" --incremental
+
+      - name: Sync catalog to S3
+        if: steps.detect.outputs.new_urls == 'true' && steps.created.outputs.count != '0'
+        run: bash scripts/s3_sync-ci.sh
+
+      - name: Commit refreshed caches
+        if: steps.detect.outputs.changes == 'true'
+        run: |
+          git config user.name "github-actions[bot]"
+          git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
+          git add -A data/
+          if git diff --cached --quiet; then
+            echo "no cache changes to commit"
+            exit 0
+          fi
+          git commit -m "Monthly incremental update: refresh caches ($(date -u +%Y-%m))"
+          git pull --rebase origin main
+          git push origin HEAD:main
+
+      - name: Upload run logs
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: run-logs
+          overwrite: true   # re-runs of failed jobs otherwise 409 on the existing artifact
+          path: |
+            logs/*.log
+            data/urls_access_checks.csv
+          if-no-files-found: ignore
+
+      - name: Session info
+        if: always()
+        run: Rscript -e 'sessioninfo::session_info()'
diff --git a/CLAUDE.md b/CLAUDE.md
index 3b90817..1d19c6c 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -220,10 +220,8 @@ WHY: Reprocessing same URLs (e.g., after failures, testing) would create duplica
 ### File Locations
 - **Main repo:** `/Users/airvine/Projects/repo/stac_dem_bc`
 - **Phase 1-2 worktree:** `/Users/airvine/Projects/repo/stac_dem_bc-phase1-2-modernization`
-- **Infrastructure repo:** `/Users/airvine/Projects/repo/awshak` (future migration)
-- **Local STAC output:** `/Users/airvine/Projects/gis/stac_dem_bc/stac/prod/stac_dem_bc`
-- **S3 bucket:** `s3://stac-dem-bc/`
-- **VM path:** `/home/airvine/stac_dem_bc/`
+- **Infrastructure repo:** `/Users/airvine/Projects/repo/rtj` (formerly awshak; provisions the bucket, OIDC role, and geoserv STAC host)
+- **STAC catalog:** `s3://stac-dem-bc/` is the only complete copy. Local builds write to a scratch workspace via the `STAC_OUTPUT_DIR` env override in `scripts/stac_utils.py` (the old `/Users/airvine/Projects/gis/.../stac/prod` dir is empty/historical)
 
 <\!-- BEGIN SOUL CONVENTIONS — DO NOT EDIT BELOW THIS LINE -->
 
diff --git a/DESCRIPTION b/DESCRIPTION
new file mode 100644
index 0000000..c7b00d1
--- /dev/null
+++ b/DESCRIPTION
@@ -0,0 +1,16 @@
+Package: stac.dem.bc
+Type: Project
+Title: Manifest for stac_dem_bc dependency resolution
+Version: 0.0.0.9000
+Description: Not an R package. This DESCRIPTION exists so that
+    r-lib/actions/setup-r-dependencies can resolve runtime deps for the
+    monthly update GitHub Actions workflow (change detection via ngr).
+    Mirrors the water-temp-bc pattern.
+License: MIT
+Encoding: UTF-8
+Imports:
+    fs,
+    ngr,
+    readr
+Remotes:
+    NewGraphEnvironment/ngr@519c03baa8554b9a82ff995dbe510bfb923a624d
diff --git a/planning/active/findings.md b/planning/active/findings.md
index 9b6cfee..b004b50 100644
--- a/planning/active/findings.md
+++ b/planning/active/findings.md
@@ -43,6 +43,10 @@ Stateless-runner path exercised locally with a uv venv (Python 3.12, exact CI de
 
 Round 1: `library(ngr)` sat before the tryCatch — a missing package on a fresh runner would exit 1, which the workflow reads as "changes detected" against the stale tracked urls_new.txt. Fixed by removing library() calls entirely (namespaced calls fail inside tryCatch → exit 2). Round 2: a trailing slash in `STAC_S3_BUCKET` would make the collection.json `cp` write a hidden `/collection.json` key — sync fine, cp exit 0, live collection silently never updates. Fixed with `BUCKET="${BUCKET%/}"`. Round 3 clean; also verified the 8 validation-CSV ids absent from the cache are the 8 known upstream-deleted files, and that the exit-0 path deletes stale tracked urls_new/urls_deleted from the checkout (stateless runner can't reprocess stale change files).
 
+## Code-check Phase 3 (2 fresh-eyes rounds, 2026-07-18)
+
+Round 1: (a) partial item-build failures are silent-permanent — item_create exits 0 on per-item failures, the full fresh list commits, and failed reads get cached as not-a-GeoTIFF (never retried); mitigated with a warn-only shortfall step + README remediation recipe, deeper failure accounting in item_create deferred to a follow-up issue. (b) upload-artifact v4 without `overwrite: true` 409s on re-runs of failed jobs — fixed. (c) README catch-up tense implied already-done — fixed. Round 2 verified all three fixes and found the zero-created corner: an all-invalid batch (precedent: the 90 parenthesized files arrived as one such delta) would hard-fail item_validate ("no item JSONs found" → exit 1) and repeat monthly; fixed by exporting the created count and gating validate/sync on it — the all-invalid month stays green with the warning and the cache commit records the batch as attempted. Also traced and confirmed the README remediation recipe end-to-end (CSV row delete → reconcile --apply → re-detect → rebuild).
+
 ## Plan-review disposition (adversarial Plan-agent pass, 2026-07-18)
 
 Absorbed into phases: warn-only access check (B1), deletions-only branch + `git add -A data/` commit semantics (B2, G4), listing plausibility guard (G5), items-before-collection sync order (G6), orphaned-URL reconciliation (G7), rebase-before-push (G8), explicit exit-code capture (G9), cron auto-disable doc (G10), oversized-batch fallback doc (A12 — a naive "process first N" cap would orphan the remainder because the cache updates eagerly), import smoke check + Python ≥3.10 pin (A13, A14), ngr SHA pin in DESCRIPTION Remotes (S18), Phase 4 count math (A21). Dropped as verified-unnecessary: "add exit code to item_validate" (already correct). Catch-up sizing note: at ~6,450 items/hour, a 35k-scale surprise busts the 330-min timeout non-convergently — log `wc -l urls_new.txt` early; fallback is a manual local run.
diff --git a/planning/active/progress.md b/planning/active/progress.md
index 985959f..e6c9db2 100644
--- a/planning/active/progress.md
+++ b/planning/active/progress.md
@@ -11,4 +11,5 @@
 - Scaffolded PWF baseline from issue #23 with approved phases
 - Phase 1 complete: STAC_OUTPUT_DIR override, detect_changes exit contract + plausibility guard, environment.yml deps, s3_sync-ci.sh, urls_reconcile.py; cold-path rehearsal passed (zero deletes, correct order, dedupe verified)
 - Live ngr fetch: objectstore at 98,039 URLs (+37,900 since Feb) → catch-up moved to local run, dispatch verifies steady state (task_plan Phase 4 amended; findings.md has the numbers)
-- Next: Phase 2 (rtj#184 role) and Phase 3 (workflow)
+- Phase 3 complete: DESCRIPTION (ngr@519c03b), update.yml (15 steps, ASCII-clean, YAML-validated), README Automation section, CLAUDE.md path fix
+- Next: Phase 2 (rtj#184 role edit + user-run tofu apply), then PR
diff --git a/planning/active/task_plan.md b/planning/active/task_plan.md
index 4678a3b..19aa5fa 100644
--- a/planning/active/task_plan.md
+++ b/planning/active/task_plan.md
@@ -20,10 +20,10 @@ The catalog is five months stale: the source URL inventory was last refreshed 20
 
 ## Phase 3: Workflow
 
-- [ ] Minimal `DESCRIPTION` for R CI deps (ngr pinned to SHA via Remotes, readr, fs), mirroring water-temp-bc
-- [ ] `.github/workflows/update.yml`: monthly cron (`23 9 3 * *`) + `workflow_dispatch`; `permissions: id-token: write, contents: write`; concurrency group (no cancel); `timeout-minutes: 330`; Python ≥3.10 pin; steps: checkout → setup R + deps → setup uv + `uv pip install` (incl. `pystac[validation]`, requests) → import smoke check (`python -c "import rasterio, rio_stac"`) → `configure-aws-credentials` (role_gha_stac_dem_bc) → detect changes with explicit exit-code capture (0 → early-exit success; 2 → fail; 1 → continue) → log `wc -l urls_new.txt` → branch: item steps only if urls_new.txt exists & non-empty (deletions-only month still reaches commit-back) → `urls_check_access.py --urls-file data/urls_new.txt` as warn-only (`continue-on-error`, CSV surfaced via artifact) → fetch collection.json from S3 → `item_create.py --incremental` → `item_validate.py --incremental --items-dir` (hard gate) → `s3_sync-ci.sh` → commit-back: `git add -A data/` (covers modifications, deletions, new access CSV), `git pull --rebase origin main`, push as bot → upload `logs/` artifact (always)
-- [ ] `scripts/README.md` Automation section: schedule, manual dispatch, failure triage (invalid-item-blocks-batch → `urls_invalid_items.txt` + `item_reprocess.py`; oversized-batch fallback = manual local run; public-repo 60-day cron auto-disable), registration step
-- [ ] CLAUDE.md: fix stale "Local STAC output" path note (S3 is the sole catalog copy)
+- [x] Minimal `DESCRIPTION` for R CI deps (ngr pinned to SHA via Remotes, readr, fs), mirroring water-temp-bc — pinned 519c03b, the locally verified build
+- [x] `.github/workflows/update.yml`: monthly cron (`23 9 3 * *`) + `workflow_dispatch`; `permissions: id-token: write, contents: write`; concurrency group (no cancel); `timeout-minutes: 330`; Python ≥3.10 pin; steps: checkout → setup R + deps → setup uv + `uv pip install` (incl. `pystac[validation]`, requests) → import smoke check (`python -c "import rasterio, rio_stac"`) → `configure-aws-credentials` (role_gha_stac_dem_bc) → detect changes with explicit exit-code capture (0 → early-exit success; 2 → fail; 1 → continue) → log `wc -l urls_new.txt` → branch: item steps only if urls_new.txt exists & non-empty (deletions-only month still reaches commit-back) → `urls_check_access.py --urls-file data/urls_new.txt` as warn-only (`continue-on-error`, CSV surfaced via artifact) → fetch collection.json from S3 → `item_create.py --incremental` → `item_validate.py --incremental --items-dir` (hard gate) → `s3_sync-ci.sh` → commit-back: `git add -A data/` (covers modifications, deletions, new access CSV), `git pull --rebase origin main`, push as bot → upload `logs/` artifact (always)
+- [x] `scripts/README.md` Automation section: schedule, manual dispatch, failure triage (invalid-item-blocks-batch → `urls_invalid_items.txt` + `item_reprocess.py`; oversized-batch fallback = manual local run; public-repo 60-day cron auto-disable), registration step
+- [x] CLAUDE.md: fix stale "Local STAC output" path note (S3 is the sole catalog copy)
 
 ## Phase 4: Catch-up run + verification (post-merge)
 
diff --git a/scripts/README.md b/scripts/README.md
index 4027a8b..59cff47 100644
--- a/scripts/README.md
+++ b/scripts/README.md
@@ -147,12 +147,29 @@ The bottleneck is network: each GeoTIFF must be partially read over HTTP to extr
 | AWS CLI | Configured with write access to `s3://stac-dem-bc` |
 | System | `rio` CLI tools (installed with rasterio) |
 
+## Automation
+
+`.github/workflows/update.yml` runs the incremental pipeline monthly (3rd of the month, 09:23 UTC) on a GitHub-hosted runner, and can be run on demand from the Actions tab (`workflow_dispatch`). It authenticates to AWS via OIDC (`role_gha_stac_dem_bc`, provisioned in the rtj infrastructure repo — no stored keys) and:
+
+1. Detects changes against the committed `data/urls_list.txt` cache (exit 0 = no changes → clean early exit; 1 = changes; 2 = error)
+2. Builds and validates STAC items for new URLs only, in a runner workspace (`STAC_OUTPUT_DIR`) seeded with the live `collection.json` from S3
+3. Syncs item JSONs then `collection.json` (in that order, never `--delete`) via `s3_sync-ci.sh`
+4. Commits the refreshed `data/` caches back to `main` — a failed run therefore persists nothing and the next run re-detects cleanly
+
+**Failure triage:**
+
+- **One invalid item blocks the whole batch** (the validate step is a deliberate hard gate, and it re-fails monthly until fixed). Remediate with `item_extract_invalid.py` → `item_reprocess.py`, or investigate via the run's `run-logs` artifact.
+- **Inaccessible source URLs do not block** — the access check is warn-only (matching `build_safe.sh`); results land in `data/urls_access_checks.csv` for reporting to GeoBC.
+- **Item shortfall warning**: the run annotates a warning when fewer items were created than URLs detected (an all-invalid batch stays green — validate/sync are skipped and the batch is recorded as attempted). Individual metadata reads can fail transiently, and a failed read is cached in `data/stac_geotiff_checks.csv` as not-a-GeoTIFF — so those URLs are not retried automatically. To recover: delete the affected rows from `stac_geotiff_checks.csv`, run `urls_reconcile.py --apply`, commit both files, and the next run rebuilds them.
+- **Oversized batches**: a month with more than ~35k new files cannot fit the job timeout, and re-running does not help (the run commits nothing, so it repeats identically). Run the pipeline locally instead (the initial 2026 catch-up follows this same local path), then let the cron resume.
+- **Cron auto-disable**: GitHub disables scheduled workflows in public repos after ~60 days without repository activity. No-change months produce no commits, so after a quiet stretch check the Actions tab and re-enable/dispatch.
+
 ## After the Pipeline
 
-Once the catalog is on S3, register it in pgstac to make it searchable:
+Once the catalog is on S3, register it in pgstac to make it searchable. Registration runs on the STAC host (geoserv) using the scripts in the rtj repo (`scripts/geoserv/`):
 
 ```bash
-ssh root@<VM_IP> "bash /tmp/stac_register-pypgstac.sh stac-dem-bc https://stac-dem-bc.s3.amazonaws.com"
+ssh <geoserv> "bash stac_register-pypgstac.sh stac-dem-bc https://stac-dem-bc.s3.amazonaws.com"
 ```
 
-This loads the STAC records into PostgreSQL, powering the search API at `images.a11s.one`. Once registered, the collection is browsable in QGIS (STAC Data Source Manager), through the API directly, or any STAC-compatible client.
+This loads the STAC records into PostgreSQL, powering the search API at `images.a11s.one`. Once registered, the collection is browsable in QGIS (STAC Data Source Manager), through the API directly, or any STAC-compatible client. A full reload takes ~46 minutes (dominated by downloading item JSONs from S3; the database load itself is seconds) — an incremental `pypgstac` upsert path is a planned follow-up.

From 6279ad8e628a01695ca826fc888988dc56a2164e Mon Sep 17 00:00:00 2001
From: almac2022 <al@newgraphenvironment.com>
Date: Sat, 18 Jul 2026 16:45:49 -0700
Subject: [PATCH 5/7] Sync PWF: Phase 2 rtj branch pushed, Phase 3 complete

Relates to #23

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HN1tMTLezPVkv9eT86FLSH
---
 planning/active/progress.md  | 6 ++++--
 planning/active/task_plan.md | 6 +++---
 2 files changed, 7 insertions(+), 5 deletions(-)

diff --git a/planning/active/progress.md b/planning/active/progress.md
index e6c9db2..aefbb69 100644
--- a/planning/active/progress.md
+++ b/planning/active/progress.md
@@ -11,5 +11,7 @@
 - Scaffolded PWF baseline from issue #23 with approved phases
 - Phase 1 complete: STAC_OUTPUT_DIR override, detect_changes exit contract + plausibility guard, environment.yml deps, s3_sync-ci.sh, urls_reconcile.py; cold-path rehearsal passed (zero deletes, correct order, dedupe verified)
 - Live ngr fetch: objectstore at 98,039 URLs (+37,900 since Feb) → catch-up moved to local run, dispatch verifies steady state (task_plan Phase 4 amended; findings.md has the numbers)
-- Phase 3 complete: DESCRIPTION (ngr@519c03b), update.yml (15 steps, ASCII-clean, YAML-validated), README Automation section, CLAUDE.md path fix
-- Next: Phase 2 (rtj#184 role edit + user-run tofu apply), then PR
+- Phase 3 complete: DESCRIPTION (ngr@519c03b), update.yml (16 steps, ASCII-clean, YAML-validated, 2 code-check rounds — zero-created gate, artifact overwrite, shortfall warning), README Automation section, CLAUDE.md path fix
+- Phase 2 code done: rtj branch `184-gha-s3-role-stac-dem-bc` pushed (984d00b role module; c81bcf4 optional versioning rider as droppable commit); rtj checkout returned to main
+- Blocked on user: tofu plan/apply in rtj env/prod, then post role ARN to #23 and merge rtj branch
+- Next after apply: /gh-pr-push here, merge, then Phase 4 (reconcile seed + LOCAL catch-up ~40k items, dispatch verifies steady state, geoserv registration)
diff --git a/planning/active/task_plan.md b/planning/active/task_plan.md
index 19aa5fa..30d2989 100644
--- a/planning/active/task_plan.md
+++ b/planning/active/task_plan.md
@@ -13,10 +13,10 @@ The catalog is five months stale: the source URL inventory was last refreshed 20
 
 ## Phase 2: Companion infra — rtj#184 (in ~/Projects/repo/rtj, fresh branch off rtj main)
 
-- [ ] Add `module "stac_dem_bc_update"` + ARN output to `env/prod/main.tf` (block as written in rtj#184)
-- [ ] Optional rider (user call): enable versioning on `stac-dem-bc` in `var.s3_buckets` — no versioning today and collection.json is overwritten in place every run
+- [x] Add `module "stac_dem_bc_update"` + ARN output to `env/prod/main.tf` (rtj branch `184-gha-s3-role-stac-dem-bc`, commit 984d00b, pushed)
+- [x] Optional rider (user call): enable versioning on `stac-dem-bc` — committed separately (c81bcf4) so it can be dropped before apply if unwanted
 - [ ] `tofu plan` / `apply` in env/prod (apply is collaborative — user credentials)
-- [ ] Post role ARN to #23; commit in rtj closes rtj#184
+- [ ] Post role ARN to #23; merge rtj branch closing rtj#184
 
 ## Phase 3: Workflow
 

From dc778e9f5175446014f5fca12d5218383741cc3e Mon Sep 17 00:00:00 2001
From: almac2022 <al@newgraphenvironment.com>
Date: Sat, 18 Jul 2026 17:58:38 -0700
Subject: [PATCH 6/7] Sync PWF: Phase 2 complete (rtj PR #189 merged, ARN
 posted)

Relates to #23

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HN1tMTLezPVkv9eT86FLSH
---
 planning/active/progress.md  | 4 ++--
 planning/active/task_plan.md | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/planning/active/progress.md b/planning/active/progress.md
index aefbb69..f16a838 100644
--- a/planning/active/progress.md
+++ b/planning/active/progress.md
@@ -13,5 +13,5 @@
 - Live ngr fetch: objectstore at 98,039 URLs (+37,900 since Feb) → catch-up moved to local run, dispatch verifies steady state (task_plan Phase 4 amended; findings.md has the numbers)
 - Phase 3 complete: DESCRIPTION (ngr@519c03b), update.yml (16 steps, ASCII-clean, YAML-validated, 2 code-check rounds — zero-created gate, artifact overwrite, shortfall warning), README Automation section, CLAUDE.md path fix
 - Phase 2 code done: rtj branch `184-gha-s3-role-stac-dem-bc` pushed (984d00b role module; c81bcf4 optional versioning rider as droppable commit); rtj checkout returned to main
-- Blocked on user: tofu plan/apply in rtj env/prod, then post role ARN to #23 and merge rtj branch
-- Next after apply: /gh-pr-push here, merge, then Phase 4 (reconcile seed + LOCAL catch-up ~40k items, dispatch verifies steady state, geoserv registration)
+- Phase 2 complete: user applied on M1; merged as rtj PR #189, rtj#184 closed, versioning live (no-expiry documented), ARN posted to #23
+- Next: /gh-pr-push here, merge, then Phase 4 (reconcile seed + LOCAL catch-up ~40k items, dispatch verifies steady state, geoserv registration)
diff --git a/planning/active/task_plan.md b/planning/active/task_plan.md
index 30d2989..3156486 100644
--- a/planning/active/task_plan.md
+++ b/planning/active/task_plan.md
@@ -15,8 +15,8 @@ The catalog is five months stale: the source URL inventory was last refreshed 20
 
 - [x] Add `module "stac_dem_bc_update"` + ARN output to `env/prod/main.tf` (rtj branch `184-gha-s3-role-stac-dem-bc`, commit 984d00b, pushed)
 - [x] Optional rider (user call): enable versioning on `stac-dem-bc` — committed separately (c81bcf4) so it can be dropped before apply if unwanted
-- [ ] `tofu plan` / `apply` in env/prod (apply is collaborative — user credentials)
-- [ ] Post role ARN to #23; merge rtj branch closing rtj#184
+- [x] `tofu plan` / `apply` in env/prod — user applied; merged as rtj PR #189 (with review commit 15e717f documenting the deliberate no-expiry-on-versioning decision); rtj#184 closed
+- [x] Post role ARN to #23 (issuecomment-5013597947) — matches the ARN already in update.yml
 
 ## Phase 3: Workflow
 

From d4724ab7f377cd24bad0f98040d53bc277149b21 Mon Sep 17 00:00:00 2001
From: almac2022 <al@newgraphenvironment.com>
Date: Sat, 18 Jul 2026 18:00:50 -0700
Subject: [PATCH 7/7] Point SRED tracking at the sred repo (sred#8)

The sred-2025-2026 repo refs are superseded.

Relates to #23

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HN1tMTLezPVkv9eT86FLSH
---
 CLAUDE.md | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/CLAUDE.md b/CLAUDE.md
index 1d19c6c..34df40a 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -115,10 +115,8 @@ Source URLs → GeoTIFF Validation → Item Creation → JSON Validation → Reg
 - No R dependency for core workflows
 
 ### SRED Tracking
-- Primary: https://github.com/NewGraphEnvironment/sred-2025-2026/issues/8
-- Secondary: https://github.com/NewGraphEnvironment/sred-2025-2026/issues/3
-- Repo issue: https://github.com/NewGraphEnvironment/stac_dem_bc/issues/3
-- Milestone: https://github.com/NewGraphEnvironment/sred-2025-2026/milestone/1
+- Primary: https://github.com/NewGraphEnvironment/sred/issues/8 — PR-body xref form: `Relates to NewGraphEnvironment/sred#8`
+- (The old `sred-2025-2026` repo refs are superseded — SRED tracking lives in the `sred` repo)
 
 ---