PDG wind-down — migration to `abruptthawmapping` (Sept 2026)

Status: IN PROGRESS — DEADLINE MOVED IN. Runbook, decision record, and the parity evidence that gates deletion. Started 2026-08-27.

The deadline is now Monday 2026-08-31, not 09-07

All PDG access closes 2026-08-31 (user, 2026-08-28) — a week earlier than the funding cliff this plan was built around, and storage bills against PDG until then. The §4 phase dates (B 28 Aug–1 Sep, C 1–3 Sep, … G 6 Sep) are void; everything compresses into ~72 hours. Two consequences that change the method, not just the dates:

The bulk copy moves to Storage Transfer Service. §4's "Copy method" specifies gcloud storage cp/rsync from rts-ops — a 2-vCPU box driving ~47 M object operations, chosen when there was a week. It also assumed we hold no setIamPolicy in the org-managed PDG project. That is true at project level and false at bucket level: yyang@ and rtsmapping@ both hold storage.buckets.setIamPolicy on all four PDG source buckets (verified 2026-08-28), which is exactly what STS needs. STS is managed, parallel, resumable, survives a VM reboot, and takes rts-ops off the critical path entirely — including its outstanding gcloud auth login (§5c). Grants: computing/grant_sts_access.ps1.

Scope must be triaged, not just accelerated. The S2 exports cannot finish regardless — 22.9 M EECU-s per year against a 3.6 M/month allowance — so what matters by Monday is that delivered cells are copied and the driver resumes against the new bucket, not that any year completes.

Measured 2026-08-28: pdg-planet-data 36.5 TB, rts-mapping-v2-usw1 20.8 TB, rts-mapping-v2-usc1 0.28 TB, rts-mapping-v2 0.27 TB — ~57.6 TB total.

GCP funding for pdg-project-406720 ends 2026-09-07, but access is being cut on 08-31. Everything this project runs on lives there. This document records what moved, where it went, why, and what was checked before anything was deleted.

Abort position: every step is a copy, never a move. Until §6 runs, the PDG originals are intact and the migration can be abandoned with no loss. Only §6 is irreversible.

1. What was there

Measured 2026-08-27 (storage.googleapis.com/storage/v2/total_bytes, live-object series).

Source	Size	Objects	Fate
`pdg-planet-data/global_quarterly/{2019,2022,2025}`	34.5 TB	~5.5 M	copy → Coldline
`rts-mapping-v2-usw1/S2_RGB/`	18.7 TiB	~9 k	copy → Coldline (`2024_train` Standard)
`rts-mapping-v2-usw1/inference/`	~210 GB	~41.7 M	copy → Standard
`rts-mapping-v2`	254 GB	—	copy → Standard, restructured
`rts-mapping-v2-usc1/ee_mirror/`	13 GB	~1.6 k	copy → us-central1 only
`rts-mapping-v2-usc1/staging/`	247 GB	—	not copied — stale Banks slice, disposable
`a100-8x-train` local disks	741 GB + 21 GB	—	audit, upload remainder, delete VM
`pdg-artifact-registry` (`rts-train:v2`, `rts-infer:v1`, `rts-review:v1`)	10.8 GB	—	push to new repo
EE assets `projects/pdg-project-406720/assets/*`	small	9 (not 4)	re-ingest

Not ours, untouched: download-vm, gke-water-cluster-*, pdg-storage-default, argo-filestore, the lake_drainage_test image.

This table has been wrong three times — verify against live listings, not against it. Found 2026-08-28 while scoping the transfer jobs:

Two prefixes missing entirely — rts-mapping-v2-usw1/ee_staging/ and .../interannual_inference/. Both are now copied and parity-verified.

Nine EE assets, not four: RTS_Sentinel_ROI, south_density_10km, south_likelihood_95m, south_mask, south_rts, south_rts_candidates, south_rts_centroids, south_rts_high_confidence, south_rts_t65. All nine are public and behind the published map, and all nine die with the project.

pdg-planet-data measures 36.5 TB live, against the 34.5 TB recorded here.

scripts/verify_migration_parity.py therefore counts what is in the buckets rather than checking off this table. Anything driven from the table alone would have silently dropped two prefixes and five map assets.

infrastructure.md used to list /mnt/argo_filestore (2 TB PDG Filestore) as our persistent shared storage. Checked 2026-08-27: not mounted on the master, holds nothing of ours. Nothing was stranded there; the doc was stale.

2. Decisions

Compute pauses. abruptthawmapping has zero A100/L4 quota and no credit, and the interannual campaign was 9 % into 2022's S2 export and 3 % into 2019's — it could not finish before the cliff. No GPU quota requested. Inference is paused indefinitely. Acquisition (Planet + S2 export) continues: it is server-side work costing almost nothing in GCP, and it is the campaign's critical path.
Storage tiered by lifecycle, not kept uniformly hot. ~$230–300/mo instead of ~$1,100.
The rating campaign ran to the last moment and cut over at the end, rather than pausing for the migration.
All 41.5 M per-tile probability COGs migrated — no dropping, no tarballs.
The office PC (ARCHITECTURE) becomes the control node; unattended work moves to a small always-on rts-ops VM. See README.md §2.

Two findings worth keeping

Do not tier the per-tile probs. Nearline/Coldline/Archive bill a 128 KiB minimum per object. 41.5 M objects averaging ~2.5 KB would bill as 5.3 TB rather than 100 GB — Coldline would cost roughly 10× Standard. The obvious optimisation is backwards here.
ee_mirror must stay in us-central1. Earth Engine's loadGeoTIFF reads US-CENTRAL1 buckets only. That bucket exists for that reason and nothing else; folding it into the us-west1 bucket would silently break the published map.

3. Path history — old → new

Old paths are embedded in MLflow run metadata and in archived run configs, and are not rewritten. This table is how you read them.

The rule is: the bucket changes, the prefix never does. Substitute the bucket name and every path below it is unchanged — verified against the live buckets 2026-09-01.

Old	New
`gs://rts-mapping-v2/…`	`gs://rts-arctic-us/…`
`gs://rts-mapping-v2/training/…`	`gs://rts-arctic-us/training/…`
`gs://rts-mapping-v2/label_sources/…`	`gs://rts-arctic-us/label_sources/…`
`gs://rts-mapping-v2/RTS_MODEL_V2/…`	`gs://rts-arctic-us/RTS_MODEL_V2/…`
`gs://rts-mapping-v2/RTS_MODEL_V1_1/…`	`gs://rts-arctic-us/RTS_MODEL_V1_1/…`
`gs://rts-mapping-v2/RTS_MODEL_V21/…`	`gs://rts-arctic-us/RTS_MODEL_V21/…`
`gs://rts-mapping-v2/RTS_MODEL_V2_scale05/…`	`gs://rts-arctic-us/RTS_MODEL_V2_scale05/…`
`gs://rts-mapping-v2/runs/…`	`gs://rts-arctic-us/runs/…`
`gs://pdg-planet-data/global_quarterly/<y>/q3/…`	`gs://rts-arctic-usw1/global_quarterly/<y>/q3/…` (prefix unchanged — see below)
`gs://rts-mapping-v2-usw1/S2_RGB/<y>_<region>/…`	`gs://rts-arctic-usw1/S2_RGB/<y>_<region>/…` (prefix unchanged — see below)
`gs://rts-mapping-v2-usw1/inference/…`	`gs://rts-arctic-usw1/inference/…` (prefixes unchanged below the bucket)
`gs://rts-mapping-v2-usc1/ee_mirror/…`	`gs://rts-arctic-usc1/ee_mirror/…` (unchanged below the bucket)

REVISED 2026-08-28 — nothing is renamed now; only the bucket changes. The original map renamed Planet to imagery/planet_q3/, S2 to imagery/s2_composites/, and the model trees to experiments/v1.0/ etc. With the deadline at 08-31 that rename cost more than it bought:

The table above was left describing the abandoned rename until 2026-09-01 — it still sent readers to gs://rts-arctic-us/experiments/v1.0/ and …/labels/, neither of which exists. A path map is the one thing in this document that gets used after everyone has forgotten the migration, so it is now stated as a rule ("bucket changes, prefix does not") and checked against the live bucket rather than against the plan.

Planet's prefix is baked into the order CSV, not just read from it — planetscope-download/filter_to_domain.py writes delivery_location as global_quarterly/<y>/q3/<col>/<row>/, Planet's servers deliver to exactly that, and list_delivered() lists it to skip already-ordered quads. Renaming means regenerating the CSV and re-verifying the skip logic on Heidi's running multi-day loop, inside 72 hours. As a bonus, Heidi's change becomes a single --bucket flag — no new key, no new CSV, no code.
S2's prefix is mid-export. 2022 is 10 % delivered and 2019 3 %; the driver resumes by listing what is already there.
This is the same reasoning §3 already used to leave inference/ alone, applied consistently.

The one argument for renaming was the Coldline lifecycle rule, which was scoped to imagery/. That is solved by widening the rule rather than moving 57 TB into a new shape: it now matches imagery/, global_quarterly/, S2_RGB/ (applied 2026-08-28). inference/ remains excluded — verified — because those 41.5 M ~2.5 KB objects would bill a 128 KiB minimum each.

inference/ keeps its internal layout deliberately. Those prefixes are read by claim.py, the crop server's prefix check, chip_index, the batch manifest and the shard queue; renaming them would mean code changes plus a full re-verification of the review stack during the tightest week of the deadline, in exchange for tidiness. The bucket rename alone already touches every one of those paths.

3b. Day-1 audit — what on the master is genuinely local-only

Run 2026-08-27, read-only. The headline "741 GB + 21 GB on local disks" is misleading: most of it is either mirrored already or regenerable by an existing convention. Comparing each local tree against its GCS mirror:

Local tree	Local	GCS mirror	On GCS	Reading
`v1.0/runs`	223 GB	`RTS_MODEL_V2/runs/`	50 GB	Expected — the mirror is deliberately slim (`best_deployment.pth` + config + summary + log + figures). The gap is `resume_latest-*.pth`, local-only by convention
`v2.1/PRETRAIN_CORPUS`	189 GB	—	—	The one real decision. The 295,429-tile MAE corpus. The v2.1 programme is closed negative, and the corpus is rebuildable from v1.0 data + the corpus builder
`v2.1/runs`	45 GB	`RTS_MODEL_V21/`	5.7 GB	12 × `resume_latest-*.pth` = 48 GB, regenerable training state
`inference/south/review`	68 GB	`inference/2025q3_south/internal/`	67 GiB	Mirrored — the crop/chip archive
`inference/south/products_local`	15 GB	`inference/2025q3_south/products/`	25 GiB	Mirrored
`inference/tiles_2022q3_*.csv`	7 GB	—	—	Not mirrored. The 2022 tile grid; small, regenerable in ~25 min, but cheap to keep
`v1.0/data_local`	100 GB	`training/v1.0/`	44 GiB	Staged working copy; the mirror is the SSoT
`nvme_scratch/south_t65`	21 GB	`inference/2025q3_south/products/`	—	The 0.65-contour build; confirm object-by-object before deleting — no audit has ever covered this disk

So Phase E's real payload is ~7 GB that must be uploaded (the 2022 tile lists), one 189 GB judgement call (the MAE corpus), and a verification pass over everything else. That is a very different afternoon from "drain 762 GB".

A gap in this table, found 2026-08-28. It walks the big trees and misses a small one: /mnt/outputs/planetscope-download/ — Heidi's acquisition state (step-1/2 grid geojson, per-year status/*.json progress, logs/). Tens of MB, so it never showed up in a du-driven audit, but it is her resume state and the step-1 geojson costs Planet API listings to regenerate. Now mirrored to gs://rts-arctic-usw1/planetscope-download/ and restored onto rts-ops. The lesson is that "what is large" and "what is unreproducible" are different questions, and only the second one gates deletion.

The MAE corpus decision. Precedent exists: the five 26 GB sat-7B dead-arm checkpoints were deliberately dropped in the 2026-07-16 cleanup, with the rationale recorded in artifact_inventory.md. The corpus is the same shape of decision at 189 GB — a closed programme's input, reproducible from data we are keeping. Coldline would cost ~$0.76/month, so this is not really about money; it is about whether a rebuild is ever wanted. DECIDED 2026-08-28: dropped. Not copied; it dies with PDG access on 08-31. Rationale and the rebuild path are recorded alongside the sat-7B precedent in artifact_inventory.md. The epoch-20/40/60 MAE checkpoints stay on GCS, so a length-ablation revisit remains possible without the corpus.

4. Runbook

Detailed phases, dates and rationale: the approved migration plan. Condensed here as the operational sequence.

A (27–28 Aug) — request the human-blocked items (Planet delivery SA key, roles/iap.tunnelResourceAccessor — for rtsmapping@, the account that SSHes from ARCHITECTURE, not yyang@ as first written; roles/editor for yyang@ if the ADC is ever to provision, which it currently is not; licence confirmation); notify PDG; set up ARCHITECTURE as the control node; then, from there, run §4b to create the three buckets, the rts Artifact Registry repo and rts-ops. Copy the deployment packages first and verify their MD5s; run the throughput probe. (The read-only local-disk audit is done — §3b.)

Provisioning deliberately waits for ARCHITECTURE (user decision, 2026-08-27): the machines that outlive this migration should be created from the host that outlives it, not from the master being deleted.
B (28 Aug – 1 Sep) — bulk copy, largest first, from rts-ops.
C (1–3 Sep) — per producer: stop submitting → drain → freeze → delta-sync → verify parity on the frozen source → repoint → restart → watch it work. One at a time.
D (2–4 Sep) — rts-review; push the four images.
E (3–5 Sep) — drain the master's disks; re-ingest the EE assets and republish the app.
F (5 Sep) — the §5 gate. 7 Sep held as slack.
G (6 Sep) — cut reviewers over; delete PDG VMs, IP, Cloud Run, images, our buckets.

Copy method

gcloud storage cp -r -n / rsync run from rts-ops under yyang@'s ADC — that identity already has read on the PDG buckets and write on abruptthawmapping, so no IAM change is needed, which matters in an org-managed project where we hold no setIamPolicy. Copies are server-side; the VM only orchestrates. The two high-object-count legs are chunked by natural sub-prefix (shard id, quad column) at ~32-way concurrency rather than run as one giant rsync, so they are resumable and independently verifiable.

Same-location pairs throughout (US→US, us-west1→us-west1, us-central1→us-central1), so there is no network egress charge. Expect ~$235 of Class-A operations for ~47 M object writes.

4a. Operational hazard found the hard way: the checkout is live infrastructure

2026-08-27. Creating the migration branch off main removed interannual_inference/ from the working tree for ~40 minutes — because that package had not yet been merged to main, and this checkout is the one the cron alerters and the S2 export driver run from (-v /home/…/RTSmappingDL:/app). Consequences, all now closed:

The interannual alerter failed four consecutive cron runs (15:10–15:40). The first run after the tree was restored (15:50) announced normally, so the alert was delayed ~23 minutes, not lost. alerts_seen.json is announce-once state and was never corrupted.
The 2019 S2 export died at 15:26 on a transient Earth Engine 403 Not signed up for Earth Engine. This was not caused by the branch switch — a file swap cannot produce a server-side 403, and EE was verified healthy immediately after (2,220 operations visible, 1,998 PENDING, 2 RUNNING). Restarted at 15:52; it resumed correctly, skipping the 55 already-delivered cells and queueing the remaining 1,744.
Heidi's Planet acquisition was never affected.

The traceback from the dying export had line numbers that did not match its own source — the signature of Python re-reading a file that changed on disk under a running process. That is the real lesson:

rts-ops gets a dedicated checkout that is never branch-switched. Development happens on ARCHITECTURE; the ops host tracks one branch and is updated by an explicit git pull at a moment of your choosing, not as a side effect of someone else's work. Anything long and unattended is reading those files the whole time it runs.

4b. Provisioning — run these from `ARCHITECTURE`

Status 2026-08-27: §4b COMPLETE — all six steps done. rts-ops exists and is reachable. ARCHITECTURE is now set up as the control node (control_node.md §1) — gcloud on the new project/zone, the CLI/ADC identity split settled, the SSH block for rts-ops written and parsing. Provisioning waited for that, so that the machines are created from the host that will outlive the migration rather than from the master we are deleting.

Pre-flight checks that passed on ARCHITECTURE:

Check	Result
`rtsmapping@` can create buckets / VMs / AR repos / enable services	all GRANTED (`testIamPermissions`)
`yyang@` ADC reads all four PDG source buckets	200 on `pdg-planet-data`, `rts-mapping-v2`, `-usw1`, `-usc1`
Destination buckets absent (nothing half-created)	404 × 3, as expected
`default-allow-ssh` covers the IAP range `35.235.240.0/20`	yes — `0.0.0.0/0 tcp:22`, no firewall work needed
VPC	single auto-mode `default` network

One check failed on the day and has since been cleared: the IAP grant. It is worth reading control_node.md §1a before touching SSH on this project — roles/editor does not carry IAP tunnel access while roles/owner does, and the OS Login username differs between PDG and abruptthawmapping.

§4b step	State
1. APIs	done — `storagetransfer` + `iap` enabled, `artifactregistry` already on
2. Three buckets	done — `rts-arctic-us` (US), `-usw1` (US-WEST1), `-usc1` (US-CENTRAL1), UBLA on all three, all empty
3. Lifecycle	done — Coldline@30d scoped to `imagery/` on `-usw1` only; no lifecycle on `-us`/`-usc1`, confirmed
4. Artifact Registry	done — `rts` (DOCKER, us-west1)
5. `rts-ops`	done — `e2-standard-2`, no external IP, Ubuntu 22.04.5, 2 vCPU / 7 GB / 194 GB disk; IAP SSH verified
6. Write verification	passed — see below

Step 6 ran early and passed. The runbook has it running from rts-ops, but the point of the check is the identity, and that identity (yyang@'s ADC) lives on ARCHITECTURE too — so it was run from the desk instead of waiting for a box that is blocked. normalization_stats.json copied rts-mapping-v2-usw1 → rts-arctic-usw1 and the MD5 matched (yC/MH+H/nMKn7WrfhDJ26w==) on both sides; the test object was deleted. Cross-project read plus destination write both work under the copy identity, which was the thing worth knowing before a 34.5 TB leg rather than four days into one.

Run as rtsmapping@woodwellclimate.org — that account holds roles/editor; yyang@ has only viewer + storage.objectUser and will be refused.

gcloud config set account rtsmapping@woodwellclimate.org
gcloud config set project abruptthawmapping

1. APIs — DONE 2026-08-27

gcloud services enable storagetransfer.googleapis.com artifactregistry.googleapis.com iap.googleapis.com

artifactregistry was already on; storagetransfer and iap were enabled by this run. Enabling iap.googleapis.com does not grant tunnel access — that is IAM, and it is missing (step 5).

2. The three buckets

Locations match their sources, so every copy is same-location and pays no network egress. Uniform bucket-level access: no per-object ACLs to reason about.

gcloud storage buckets create gs://rts-arctic-us   --location=US          --uniform-bucket-level-access
gcloud storage buckets create gs://rts-arctic-usw1 --location=US-WEST1    --uniform-bucket-level-access
gcloud storage buckets create gs://rts-arctic-usc1 --location=US-CENTRAL1 --uniform-bucket-level-access

rts-arctic-usc1 exists for one reason: Earth Engine's loadGeoTIFF reads US-CENTRAL1 buckets only. Do not "tidy it up" into the us-west1 bucket later — that silently breaks the published map.

3. Lifecycle — imagery ages into Coldline

Applied to the bulk bucket only, and scoped to imagery/, so future years inherit the policy without anyone remembering to set it. Nothing under inference/: those objects average ~2.5 KB and the colder classes bill a 128 KiB minimum per object, which would make Coldline roughly 10× the cost of Standard.

@'
{"lifecycle": {"rule": [
  {"action": {"type": "SetStorageClass", "storageClass": "COLDLINE"},
   "condition": {"age": 30, "matchesPrefix": ["imagery/"], "matchesStorageClass": ["STANDARD"]}}
]}}
'@ | Out-File -Encoding ascii lifecycle.json
gcloud storage buckets update gs://rts-arctic-usw1 --lifecycle-file=lifecycle.json

imagery/s2_composites/2024_train is read by training and should stay hot; after the copy, put it back with gcloud storage objects update "gs://rts-arctic-usw1/imagery/s2_composites/2024_train/**" --storage-class=STANDARD.

4. Artifact Registry

gcloud artifacts repositories create rts --repository-format=docker --location=us-west1 `
    --description="RTS mapping images (migrated from pdg-artifact-registry)"

5. `rts-ops` — the migration workhorse

No external IP: it is reached over IAP. Created before the bulk copy because the copy runs from it, and it must outlive a100-8x-train.

Was blocked; cleared 2026-08-27. roles/editor does not carry iap.tunnelInstances.accessViaIAP, so this VM would have been created unreachable. yyang@ was granted roles/owner and used it to give rtsmapping@ roles/iap.tunnelResourceAccessor; the VM was then created --no-address as designed and IAP SSH confirmed. Full write-up, and the OS Login username trap that follows it, in control_node.md §1a.

gcloud compute instances create rts-ops `
    --zone=us-west1-a --machine-type=e2-standard-2 `
    --image-family=ubuntu-2204-lts --image-project=ubuntu-os-cloud `
    --boot-disk-size=200GB --boot-disk-type=pd-balanced `
    --no-address `
    --scopes=https://www.googleapis.com/auth/cloud-platform `
    --metadata=enable-oslogin=TRUE

# IAP needs ingress from Google's tunnel range; the default network's
# default-allow-ssh (0.0.0.0/0 on tcp:22) already covers it. If that rule has been
# tightened, add the narrower one instead:
gcloud compute firewall-rules create allow-iap-ssh `
    --network=default --direction=INGRESS --action=allow `
    --rules=tcp:22 --source-ranges=35.235.240.0/20

5a. Egress — a `--no-address` VM has none, and this step was missing

Found 2026-08-27, the hard way. A VM created --no-address on the stock default subnet can reach nothing — not GitHub, not PyPI, not apt, and not storage.googleapis.com. The first three block provisioning; the fourth would have stopped the Phase-B copy dead, because rts-ops must reach the GCS API to orchestrate it even though the bytes move server-side. Measured on the fresh box: all four hosts refused, only the metadata server answered.

Two independent layers, and they fix different things:

Layer	Covers	Cost	State
Private Google Access (subnet flag)	`.googleapis.com`, `.pkg.dev` — GCS, OAuth, Artifact Registry	free	enabled 2026-08-27
Cloud NAT (router + gateway)	everything else — GitHub, PyPI, apt, Docker	~$32/mo + $0.045/GB	created — `rts-ops-router` / `rts-ops-nat`, us-west1, `AUTO_ONLY` over all subnet ranges; verified live 2026-09-01

# Layer 1 — free, and the one the copy actually depends on. Already applied.
gcloud compute networks subnets update default --region=us-west1 --enable-private-ip-google-access

# Layer 2 — required for provisioning and for the `git pull` workflow in §4a.
gcloud compute routers create rts-ops-router --network=default --region=us-west1
gcloud compute routers nats create rts-ops-nat --router=rts-ops-router --region=us-west1 `
    --auto-allocate-nat-external-ips --nat-all-subnet-ip-ranges

PGA alone is not enough to build the box, but it is enough to run the copy. After enabling it, storage.googleapis.com, oauth2.googleapis.com and us-west1-docker.pkg.dev all answered while github.com, pypi.org and download.docker.com stayed dark. So if NAT is ever unwanted, the copy still works — only maintenance breaks.

NAT does not weaken the no-inbound posture. It is egress-only; nothing becomes reachable from the internet, and the reason for --no-address is unaffected. Note also that GCS traffic prefers PGA once enabled, so the 34.5 TB copy generates no NAT data charges — the gateway's hourly rate is essentially the whole cost.

5b. On-box provisioning — state 2026-08-27

Item	Where	State
Repo checkout	`/opt/rts/RTSmapping_v2`	done — tracks `main` @ `9d438db`, never branch-switch it (§4a)
Shared group	`rts` (gid 1002)	done — OS Login gives each user a private uid/gid and puts nobody in `google-sudoers`, so a shared box needs an explicit group. Heidi was never added, and does not need to be — see the note below
Docker	apt `docker.io` 29.1.3	done
Shared venv	`/opt/rts/venv` (Python 3.10.12)	done — same dependency floors as `requirements.txt`; no torch, the box has no GPU
`rts-dataprep:v1`	local image, 1.14 GB	done — built from `computing/Dockerfile.dataprep`, EE + geo imports smoke-tested
Work dirs	`/mnt/outputs/{interannual_inference,planetscope-download}`	done
User ADC	`~/.config/gcloud` per user	done 2026-08-28 — `yyang@`, mode 600, quota project `abruptthawmapping`, reads all three PDG buckets
Slack webhook	`/mnt/outputs/planetscope-download/slack_webhook` (mode 600)	done 2026-08-28 — moved host-to-host from the master through a pipe, never written to a bucket or displayed
Cron entries	`/etc/cron.d/planetscope-acquisition`	done 2026-08-28 — acquisition alerter installed and dry-run clean. The interannual one is deliberately not installed: the S2 export is abandoned (cutover §2) and the campaign has nothing to alert on yet
Acquisition venv	`/mnt/outputs/planetscope-venv`	done 2026-08-28 — geopandas 1.1.4 / shapely 2.1.2; the box had none, so Heidi could not have run there
Acquisition work dir	`/mnt/outputs/planetscope-download/{data,status,logs}`	done 2026-08-28 — restored from the master (see §3b)

Audit 2026-09-01 — three drifts between this table and the live box, none breaking.

The rts group has one member (rtsmapping@), and Heidi is not in it. She has nonetheless been running 2019 for 24 h. The acquisition work dirs are drwxrwsrwt — 1777 plus setgid — so anyone can create files, the group stays rts, and the sticky bit stops users deleting each other's. status/2019.json is mode 666 and still owned by rtsmapping@ because she rewrites it in place rather than recreating it. The usermod -aG rts step above is therefore not a prerequisite for acquisition, and /opt/rts/RTSmapping_v2 stays rwxrwsr-x root:rts — she can read the checkout and cannot write it, which is the arrangement you want.
The acquisition cron runs as root, not as $U. Harmless here: alert_if_stopped.py reads local status files and posts to Slack, and touches no GCS. It would matter for the interannual alerter, which does need an ADC — install that one as $U.
The per-year ADC scheme in interannual_inference/config.yaml is not provisioned. accounts.2019 points at /mnt/outputs/adc-rtsmapping/application_default_credentials.json, which does not exist; /mnt/outputs/adc-heidi/ exists but holds only logs/, active_config and configurations/ — an application_default_login that was started 2026-08-28 08:29 and never completed. Only the default ADC (~/.config/gcloud/, yyang@) is real, and it works — it lists the four state objects under gs://rts-arctic-usw1/interannual_inference/state/. This is a Phase-0 prerequisite, not a migration gap: the S2 export is abandoned and no year needs a second account until the campaign restarts. Create them with the CLOUDSDK_CONFIG= form above before it does.

5c. `gcloud` CLI and ADC are different credentials — the copy needs both

Found 2026-08-28. §4's copy method says the copy runs "from rts-ops under yyang@'s ADC". That is true of Python, and false of the gcloud CLI. On a GCE VM the CLI defaults to the attached service account and ignores ADC entirely, so with ADC correctly installed:

gcloud config get-value account   ->  801926669176-compute@developer.gserviceaccount.com
gcloud storage ls gs://rts-mapping-v2-usw1/…   ->  403
python -c "from google.cloud import storage; …"  ->  200   (same box, same moment)

So gcloud storage cp/rsync — the tool the runbook actually specifies for the 34.5 TB — would have run as the wrong identity and failed on every PDG read. The box needs a second login:

gcloud auth login                 # CLI credential, separate from ADC
gcloud config set account yyang@woodwellclimate.org

Do not paper over this with CLOUDSDK_AUTH_ACCESS_TOKEN=$(gcloud auth application-default print-access-token). It works, and the token expires in an hour — fine for a probe, useless for a multi-day copy. gcloud auth login stores a refresh token that renews itself.

The VM service account is not a substitute for the user ADC. 801926669176-compute@… returns 403 on both pdg-planet-data and rts-mapping-v2-usw1 — it belongs to abruptthawmapping and has no standing on PDG. So the copy, the state mirror and Earth Engine all need yyang@'s ADC created on the box. Never copy the credential file; on a headless host use the device flow:

gcloud auth application-default login --no-browser

Per-user ADCs for a shared box, per README.md §6: CLOUDSDK_CONFIG=/mnt/outputs/adc-<name> gcloud auth application-default login --no-browser.

Verified working without ADC: status.py renders the year × stage matrix, and both alerters run clean under --dry-run. The matrix reads all-pending because local state is empty — the real state is in gs://…/interannual_inference/state/, so §5 gate row 5 cannot pass until the ADC exists. That is the check, not a fault.

Cron content, ready to install ($U = the OS Login user that owns the ADC):

# /etc/cron.d/rts-interannual-inference
*/10 * * * * $U cd /opt/rts/RTSmapping_v2 && /opt/rts/venv/bin/python interannual_inference/alert.py >> /mnt/outputs/interannual_inference/logs/alert.log 2>&1

# /etc/cron.d/rts-acquisition-alert
*/10 * * * * $U cd /opt/rts/RTSmapping_v2 && /opt/rts/venv/bin/python planetscope-download/alert_if_stopped.py >> /mnt/outputs/planetscope-download/alert.log 2>&1

Do not start cron on rts-ops while the master's cron is still running. Both alerters are announce-once against their own alerts_seen.json, and rts-ops has an empty one — so two live alerters means every open incident is announced twice, and at cutover a second drive.py could act on a stage the master already owns. Install the cron entries as part of the Phase-C cutover for that producer, not before. The state seeded here is a snapshot for verification; re-sync it from the mirror at cutover.

6. Verify before copying anything

# From rts-ops, as yyang@ — read on the PDG side, write on the destination.
gcloud storage cp gs://rts-mapping-v2-usw1/inference/2025q3_south/packages/seed42/normalization_stats.json \
                  gs://rts-arctic-usw1/_writetest.json
gcloud storage rm gs://rts-arctic-usw1/_writetest.json

A refusal here means the identity is wrong, and is much cheaper to discover now than four days into a 34.5 TB copy.

4c. Earth Engine assets — migrated 2026-08-28

All nine assets (not the four §1 recorded) now live under projects/abruptthawmapping/assets/.

Use ee.data.copyAsset, not re-ingestion. The runbook says "re-ingest", which would mean rebuilding each asset from its shapefile/GeoTIFF in GCS. copyAsset duplicates within Earth Engine's own storage instead: no GCS source, no ingestion task per asset, and the copy is byte-faithful rather than a re-derivation. All eight remaining assets copied in one pass.

Asset	Type	Bytes	Public
`south_likelihood_95m`	IMAGE	138,985,514	yes
`south_mask`	IMAGE	103,130,889	yes
`south_rts_candidates`	TABLE	64,443,796	yes
`south_rts_high_confidence`	TABLE	38,401,207	yes
`south_rts_t65`	TABLE	25,951,444	yes
`south_rts`	TABLE	20,044,157	yes
`south_rts_centroids`	TABLE	6,055,213	yes
`south_density_10km`	IMAGE	5,574,368	yes
`RTS_Sentinel_ROI`	TABLE	5,363	no — private in PDG, kept private

ACLs were read from each PDG source and mirrored, rather than blanket-set: eight were public and are public again; RTS_Sentinel_ROI was private and stays private.

"Restricted mode" does not block ingestion or copying. abruptthawmapping has exceeded its noncommercial EECU allowance and every EE call warns about it, which looked like it might sink the map. It does not: a test INGEST_TABLE submitted and reached SUCCEEDED, and all eight copies went through. The restriction is on batch compute (the S2 exports), not asset operations.

The legacy task API lies about ingestion. ee.data.getTaskStatus() reported UNKNOWN indefinitely for a job that had already succeeded. Use ee.data.listOperations() — it showed INGEST_TABLE SUCCEEDED correctly. Do not conclude an ingest has stalled from getTaskStatus.

Repointed to match: ee_south_app.js, ee_south_viewer.js, ee_qc_rater.js, build_ee_qc_rater.py, ingest_ee_app_assets.py and test_build_ee_qc_rater.py — asset paths to abruptthawmapping, and the loadGeoTIFF chip/probability prefixes to gs://rts-arctic-usc1/. That last one still matters for the reason §2 gives: loadGeoTIFF reads US-CENTRAL1 only.

4d. The inventory was wrong a fourth time — `usc1/staging/`, caught during teardown

2026-08-29. §1's inventory was already noted as wrong three times. It was wrong a fourth, and this one was found with the delete already running.

gs://rts-mapping-v2-usc1/staging/ held 2,661 objects / 265 GB of Banks Island Planet quads (staging/banks/quads/global_quarterly_2025q3_mosaic_*.tif). The destination held zero. The seven-leg pair list only ever named ee_mirror/ for that bucket, so every parity run — all of which passed — was answering a question that never included this prefix. A prefix-scoped check cannot find data outside its own prefixes, and nothing in the gate compensated for that.

It survived because gcloud storage rm --recursive works through buckets in the order the script lists them and was still grinding through usw1's 42 M objects. Copied out (23.5 GiB/s, same-region rewrite) and verified: 2,661 = 2,661 objects, 265,077,801,178 = 265,077,801,178 bytes.

The fix for the class of error, not just the instance: re-ran parity with no prefix at all, comparing whole bucket to whole bucket —

python scripts/gcs_parity.py --src gs://rts-mapping-v2-usc1 --dst gs://rts-arctic-usc1
#   source 6,722 objects  279,068,274,113 bytes
#   dest   6,722 objects  279,068,274,113 bytes   PARITY PASS

rts-mapping-v2 was always compared this way (its pair used an empty prefix), which is why it needed no rescue.

`usw1` could no longer be enumerated — so it was confirmed a different way

S2_RGB/ and ee_staging/ were already deleted, so a whole-bucket comparison was no longer possible. Cloud Monitoring keeps the history the bucket no longer can:

`storage.googleapis.com/storage/v2/total_bytes`	bytes
`type: live-object`	20,875,696,884,709
`type: soft-deleted-object`	359,324,933,372
sum of the four migrated legs	20,876,507,422,915

Live bytes match the migrated legs to 810 MB — 0.004 % — and that residual is the S2 export rewriting composites between the hourly metric sample and the parity run, the same drift measured directly at 823 MB. There was no fifth prefix. The 359 GB of soft-deleted objects is what that same rewriting left behind, and is exactly what disabling soft-delete before the deletes disposed of rather than billing for another week.

5. Verification — the gate before deletion

Nothing in §6 runs until every row passes. (To be filled in as the migration proceeds.)

#	Check	Result
1	Parity per prefix — object count and total bytes, source vs destination. Acquisition prefixes must use the frozen Phase-C measurement, not the provisional Phase-B one	PASS 2026-08-28 — all 7 legs, every object compared by name, size and checksum, never sampled. `planet` 5,000,891 obj / 39.46 TB and `s2` 14,780 / 20.72 TB identical, both against genuinely frozen sources (export abandoned + EE queue cancelled, order loop stopped, delivery tail drained). `experiments` 253,035 / 272.81 GB, `ee_mirror` 4,061 / 13.99 GB, `ee_staging` 3, `interannual` 4 — exact. `inference`: src 42,342,548 obj (exactly the `inference-cogs` job's `copied`) vs dst 42,347,929 — `missing = 0` everywhere, and the 5,381-object / 28,918,093,888-byte surplus reconciles to the byte: `t65_build/` 5,374 obj / 21,480,253,328 B and the two 2022 tile lists 2 obj / 7,437,655,168 B, both drained from the master's local disks so they never existed in PDG; 5 obj / 27,943 B of new-app review activity; and a 157,449 B residual that is precisely the size delta of the single `differing` object, `internal/deploy/rts-review.tar.gz`, rebuilt 08-28 for the new VM against the 08-05 original. Nothing in PDG is absent from the new project — the only property that gates deletion. Walked in 66 min as 2,161 parallel prefix chunks (~7,600 obj/s) rather than the ~22 h a single process needed — chosen for reaction time, not speed: a FAIL found Saturday midday would leave no room to re-sync and re-verify before Monday. The single-process cross-check was abandoned at 9.8 h once teardown began deleting `rts-mapping-v2-usw1`, its source side: it could only have returned a spurious FAIL as objects vanished under the walk. The parallel result stands as the measurement, taken before any deletion started.
2	~~`gcloud storage hash` on 200 random objects per leg~~, plus all of `packages/seed{42,43,44}` and `normalization_stats.json`	SUPERSEDED — folded into row 1. The sampling half of this row is obsolete: `gcs_parity.py` compares name, size and MD5 for every object, which is what a 200-object sample was a cheap approximation of. Sampling a 42 M-object prefix was never going to find a single bad object anyway — the tested reason the sampler was rejected when it was first written. The exhaustive half stands and passed: the 18 deployment-package objects are MD5-identical, including all three seed `weights.pth` and the shared `normalization_stats.json` (`yC/MH+H/nMKn7WrfhDJ26w==`). Row 3 then anchors those same weights to production.
3	Frozen-model reload — migrated packages reproduce the recorded 3-seed anchor	PASS 2026-08-28 — `scripts/verify_frozen_model.py`. Anchored on something stronger than the plan assumed: every shard manifest written during the production run records `model_checkpoint_sha`, the SHA256 of each seed's `weights.pth` as loaded on 2026-07-07. All three migrated packages match it — seed42 `e0ffdbff…`, seed43 `335aeedd…`, seed44 `fa4866cd…` — so the migrated weights are tied to the delivered map through a hash neither copy could have influenced, not merely to whatever sat in PDG. 25 manifests spread across the 1,000-shard run agree on one ensemble triple (a disagreement would have meant the map was not the product of a single model), and each package's `deployment_config.yaml` still carries the delivered calibration: threshold 0.65, temperature 0.512321, matching the manifests.
4	No dead references — repo grep for old bucket/project names returns only historical prose	PASS 2026-08-28 — every live default repointed across 24 files: argparse defaults, shell `${VAR:-…}` fallbacks, module constants, `cloudbuild.yaml`, and the EE `--project` defaults. Remaining hits are docstrings, comments, `configs/*.yaml` (kept per §3) and test fixtures — i.e. exactly the historical prose this row allows. `stages.py` was already safe: it passes every bucket explicitly from `cfg["paths"]`, so the live campaign path never used those defaults — but anything run by hand after 08-31 would have silently targeted a dead project. 71 tests pass over the changed code.
5	Cold start on `rts-ops` over IAP; `status.py` reproduces the campaign grid; a dry-run stage resolves; survives a reboot	PASS 2026-08-28 — grid matches (2022 `s2_export` ▶10 %, 2019 ▶3 %); `run_stage.py` refused `s2_index` naming `s2_export`, `drive.py` stopped at the same point and recognised the detached export; rebooted 08:45:23, Docker/ADC/venv/state/IAP all intact
6	Live acquisition unbroken — `check_status.py` advancing into the new bucket, `ord/min` back at 38–39	PASS 2026-09-01 — Heidi restarted on `rts-ops` 2026-08-31 16:39. The log names the right target on its own: `Ordering 308686 quads for 2019q3 -> gs://rts-arctic-usw1/...`, with `Prior delivery: 215263 quads already present` — it listed the new bucket, so nothing was re-ordered and no Planet quota was re-spent. Now 253,145 / 308,686 (82.0 %), 0 failed, 37.4 ord/min, heartbeat fresh, alerts log empty. Delivery confirmed by counting rather than inferring: `list_delivered()` returns 253,163 quads, +37,900 since the restart, matching the 37,882 ordered. The four `could not verify delivery credentials: pdg-planet-data` lines in the log are all dated 08-27/28 — the old run on the master, not this one.
7	Review app end-to-end on the new host — rater page, 301 batches / 60,167 items, a crop served, manifest still 404, claim → submit → idempotent retry → 409	PASS 2026-08-28. Old app stopped first (the freeze), then the two stores diffed with `gcs_parity.py`: nothing missing, nothing differing, 5 objects extra at the destination — `claims/b00043-45`, `done/b00044`, `verdicts/b00044.jsonl`. The new deployment is strictly ahead: reviewers were cut over and one has already submitted a batch there. So the final sync was correctly a no-op, and a blind `--overwrite-when=different` would have been the wrong instinct — see the corrected cutover §4. Live checks against the new bucket: claim `b00046` (201 items) with the marker verified in `gs://rts-arctic-usw1/...`, reopen-submitted 409, unknown batch 404, reopen-held 200, crop 200 / 28,856 B JPEG. Claim marker then deleted so the batch returned to the queue rather than sitting blocked for `STALE_AFTER_S` = one week. No verdict was fabricated — `merge_review_verdicts.py` pools every `verdicts/.jsonl` into the verified inventory, so a synthetic submission would have contaminated the science. Submit idempotency is covered by `test_submit_persists_and_is_idempotent`; what the migration needed to prove was that this* deployment reads and writes this bucket, which the claim marker does.
8	Public EE map renders from the new assets and the new usc1 mirror	PASS 2026-08-28 — verified against the published bundle, not the editor: `https://abruptthawmapping.projects.earthengine.app/javascript/south-rts-map-modules.json` now reads all four layers from `projects/abruptthawmapping/assets/{south_rts_t65,south_rts_centroids,south_rts_candidates,south_density_10km}`, carries `rts-arctic-usw1` and zero references to `rts-mapping-v2-usw1`. The one remaining `pdg-project-406720` is inside a header comment. All six referenced assets were separately confirmed `all_users_can_read`. The first publish silently did not take — the bundle still served the old script, because the Apps dialog republishes from the saved script path, not the editor buffer, so pasting into an unsaved tab re-publishes the script the app is already bound to. Checking the app returned HTTP 200 would have missed this entirely: fetch the bundle and grep it.
9	Master drained — nothing durable local-only, `/mnt/nvme_scratch` included	PASS 2026-08-28, after a second sweep that looked for unmirrored rather than for large. Verified present: 2022 tile lists at `inference/2022q3_south/` (3.61 GB + 3.83 GB) and the t65 build at `products/t65_build/` — 5,374 objects, exactly the local file count. The first audit's size-ordered walk had missed four small-but-unreproducible trees, all now drained: Heidi's acquisition state (§3b), the `file:///outputs/mlflow` tracking store (128 MB, 27 runs — real, not stale: MLflow 2.x cannot use `gs://` as a tracking URI), `multiscale_poc_eval` (1.9 GB of cached `probs_.npz` the ledger cites as the family-M evidence, of which only the logs were mirrored), and the two VM-creation scripts, which existed only on the VM they created* and are now in `computing/vm_provisioning/`. Home directories were never in the audit either; checked, and they hold only git checkouts. The 189 GB MAE corpus remains the one recorded, deliberate loss.
10	Billing — no `rts-*` resource left in PDG; `abruptthawmapping` line items as expected	SUBSTANTIALLY PASS 2026-09-01 — PDG holds *no `rts-` instance, disk, address, Cloud Run service or image*. `rts-review-vm-ip` released, our three images deleted with `lake_drainage_test` (not ours) untouched, `rts-mapping-v2` and `rts-mapping-v2-usc1` deleted. Only `rts-mapping-v2-usw1` remains, emptying under an age-0 lifecycle rule. `pdg-planet-data` audited object-by-object and cleared for deletion (§5c). Remaining PDG resources — `download-vm`, two `gke-water-cluster-` nodes, `pdg-` buckets — were all confirmed not ours*.
11	Docs true — the README.md registry matches `gcloud compute instances list`	PASS 2026-08-28 — both projects enumerated. `abruptthawmapping`: `rts-ops`, `rts-review`. PDG: `a100-8x-train`, `rts-review-vm`, plus `download-vm` and two `gke-water-cluster-` nodes that the registry already flags not ours — do not touch* (verified: no labels, no metadata, bare Ubuntu, predates our work in that zone). Registry corrected for the compressed deadline: both PDG rows now retire 08-31, not 09-06, and `rts-ops` is recorded as fully provisioned.

5b. Project-wide audit — everything of ours in PDG, 2026-09-01

Earlier sweeps checked the resource types the migration happened to touch. This one enumerates the whole project and works backwards, which is the only way to find things nobody remembered creating:

gcloud asset search-all-resources --scope=projects/pdg-project-406720     --query="name:(rts OR mapping OR slump OR thaw OR infer OR review OR planet)"

Five hits, and two were orphans no earlier sweep had looked for:

Item	Verdict
`rts-review-allow-http` (firewall)	ours, orphaned — its VM was deleted 08-29 and the rule outlived it. Deleted 09-01
`rts-review-app@pdg-project-406720` (service account)	ours, superseded by the same-named SA in the new project. Deleted 09-01
`rts-mapping-v2-usw1` (bucket)	purging under the age-0 lifecycle rule
`pdg-planet-data` (bucket)	cleared for deletion 2026-09-01 — every one of its 5,000,891 objects verified present in the new bucket (§5c)
`planet-orders@pdg-project-406720` (service account)	left alone — ownership ambiguous. Our deliveries provably use `planet-orders@abruptthawmapping` (see below), and this one dies with the project. Deleting another team's identity on a guess buys nothing

Swept and empty or none of ours: snapshots, custom images, BigQuery datasets, Pub/Sub topics, Secret Manager (API never enabled), Vertex AI notebooks, Cloud Run, instance templates, instance groups, disks, addresses. The 30 firewall rules include exactly one of ours; the instance templates/groups are all gke-water-cluster-*.

Earth Engine is not in the asset inventory and had to be walked separately — it matters most, because EE assets die with the project rather than with a bucket. All 9 assets under projects/pdg-project-406720/assets have a counterpart in projects/abruptthawmapping/assets: RTS_Sentinel_ROI, south_density_10km, south_likelihood_95m, south_mask, south_rts, south_rts_candidates, south_rts_centroids, south_rts_high_confidence, south_rts_t65. Zero missing.

Do not use search-all-resources to confirm a deletion. It still listed the firewall and the service account minutes after both were deleted and the deletions were confirmed — the index refreshes on a lag. Use it to find things; confirm with the live per-service API.

Which planet-orders@ is actually delivering. Both projects have one, which is a trap worth naming: if Heidi's key belonged to the PDG copy, her run would break the moment PDG closes. It does not. Only planet-orders@abruptthawmapping holds storage.objectUser on gs://rts-arctic-usw1, and deliveries are landing there (+37,900 quads, §5 row 6) — so the key in use is the new-project one, whose user-managed key dates from 2023-02-08 and long predates PDG.

5c. Clearing `pdg-planet-data` for deletion — 2026-09-01

PDG's admin is deleting everything, not retaining the bucket. That inverts the question the runbook had answered: not "who deletes it" but "is our copy complete enough that its deletion loses nothing?" — which needs evidence, not the earlier courtesy reasoning.

Structure first: pdg-planet-data holds only global_quarterly/, no loose root objects, and years 2019/ 2022/ 2025/ — all three present in the new bucket. So the whole bucket is our acquisition; none of PDG's own data is mixed in.

Then every object, walked as 5,430 parallel chunks split at <year>/q3/<col>:


source (PDG)	5,000,891 objects / 39,459,568,819,002 bytes
destination (`rts-arctic-usw1`)	5,001,875 objects — more, because Heidi's run was delivering during the audit
missing from destination	0
differing	0

The source count is identical to the 2026-08-28 frozen measurement, which independently confirms nothing new has landed in PDG since the final sync. Cleared for deletion.

Also found and rescued: gs://pdg-project-406720_cloudbuild/source/ held 7 Cloud Build source tarballs (1.29 GB) — ours, from the May and August image builds. Their outputs are already migrated and verified, and their inputs are in git, so they were arguably redundant; copied to gs://rts-arctic-us/build_context_archive/source/ anyway and verified (7 = 7, byte-identical), because 1.3 GB is cheaper than relying on a reproducibility argument. pdg-storage-default was checked here on its top-level prefix names only, and that was not enough — see §5d.

planet-orders@pdg-project-406720 may be deleted with everything else: our deliveries provably use the abruptthawmapping copy (§5b), and nothing of ours authenticates with the PDG one.

5d. `pdg-storage-default` — the inventory was wrong a fifth time, and it was the expensive one

§5c cleared this bucket by reading its top-level prefix names (LostLakes/, UAFmodeling/, viz_workflow/, …), all of which are plainly PDG's. Two of those prefixes had ours inside them.

The big one: 511 GB of Sentinel-2, one level down inside `sentinel2/`

sentinel2/Woodwell_sent2_74-84N_cloudmask_summer_2024/ — 398 GeoTIFFs, 511 GB, COLDLINE, written 2025-11-26/28. Covering 74–82°N, which data/s2_rgb_data.md §2 names as this project's circumpolar_north_domain. It was a previous S2 download attempt made outside this repository (confirmed by Yili) — which is exactly why nothing found it:

it is not in the repo — no file, config or doc mentions it;
it is not in the migration's 7-leg pair list, which was built from the repo;
it does not match our cell_id convention (Lat74_Lon-102.tif, not W1020_N0740);
it is summer 2024, while our north job is S2_RGB/2025_north/ — already migrated, and genuinely a different dataset, so its presence in the new bucket "explained" the north domain and made this look redundant.

Every one of those signals argued against it being ours, and all of them were about the repo, not about the data. Rescued to gs://rts-arctic-usw1/rescued_pdg/ — same region, so the copy is server-side. Its 1.15 GB single-tile prototype (sentinel2/…T07WFR_SR|SCL.tif.tif, 2025-10-01, the surface-reflectance + scene-classification pair a cloudmask is built from) went with it.

The rule this breaks. Every other check in this migration keyed off the repo: the pair list, the path map, the audit queries. That is sound for work the repo knows about and blind to work it does not — and a solo research project accumulates a lot of the latter. A bucket is not cleared by "nothing in the repo points at it."

The smaller one: three legacy RTS archives in `working/`

Object	Size	Verdict
`RTS_PlanetScope_4BandRGB_1024_Banks_clean_v2.zip`	4.91 GB	ours — a labelled training set (`train_img/`, `README.md`, 1,637 members)
`rts_vit_sem_seg_1024inputs.zip`	1.04 GB	ours — RTS ViT semantic-segmentation run (`model_final.pth`, `metrics.json`)
`rts_vit_sem_seg_1024inputs_Banks.zip`	1.04 GB	ours — the same, Banks Island
`vit_sem_seg_1024input_13400itr.zip` / `.pth`	2.15 GB	not ours — unpacks to `vit_sem_seg/``infrastructure/`
`infrastructure.zip`	1.90 GB	not ours — 1,510 `TALO_mask_*.TIF` infrastructure masks
`iwp_mask_rcnn_vitdet_1024input.zip`	1.23 GB	not ours — ice-wedge polygons

2024, from the detectron2/ViT work that preceded RTSmappingDL v2. Copied to gs://rts-arctic-us/legacy_pdg_working/ and CRC32C-verified (dE1hLA==, Lu6sxQ==, Wdx2YQ==), 6.99 GB.

Ownership was decided by reading the archives, not their names. Four of the seven have names that suggest nothing either way; a zip's central directory is at the end of the file, so a Range: request for the last 200 KB lists every member without moving the other 11 GB. That is what separated rts_vit_sem_seg_* (ours) from vit_sem_seg_1024input_13400itr (infrastructure) — two names one underscore apart, holding different projects' models. The same trick cleared workflows_optimization/rts_ray_pipeline/ (5.35 GB, and rts in the name): its checkpoints are RTS_v6_tcvis / s2-tcvis-final-large trained on pd-dgx-a100, i.e. AWI's DARTS model in PDG's Ray harness, not ours.

Closing it properly: rank by bytes, not by name

The structural walk and the name sweep both have the same weakness — they depend on someone having named something recognisably. Woodwell_sent2_… was caught only because a human happened to put the institution in the folder name. Had it been test2/, nothing would have flagged it.

So the bucket was finally closed on the one property that cannot be hidden: size. A full walk of all 1,562,015 objects, aggregated by depth-2 prefix — pdg-storage-default holds 3.60 TB, and 556 prefixes are ≥1 GB. Strike out everything unmistakably PDG's (MAPLE, Landsat, clowder, the lake/water/viz families, landsattrend) and exactly ten remain — every one of them already adjudicated:

Prefix	GB	Verdict
`sentinel2/Woodwell_sent2_74-84N_cloudmask_summer_2024`	511.0	ours — rescued
`workflows_optimization/rts_ray_pipeline`	5.3	AWI's DARTS checkpoints
`working/RTS_PlanetScope_4BandRGB_1024_Banks_clean_v2.zip`	4.9	ours — rescued
`working/infrastructure.zip`	1.9	PDG — TALO masks
`working/iwp_mask_rcnn_vitdet_1024input.zip`	1.2	PDG — ice-wedge polygons
`sentinel2/…T07WFR_SR.tif.tif`	1.1	ours — rescued
`working/vit_sem_seg_1024input_13400itr.pth`	1.1	PDG — infrastructure model
`working/rts_vit_sem_seg_1024inputs_Banks.zip`	1.0	ours — rescued
`working/rts_vit_sem_seg_1024inputs.zip`	1.0	ours — rescued
`working/vit_sem_seg_1024input_13400itr.zip`	1.0	PDG — infrastructure model

Zero unaccounted. Our 511 GB was the second-largest object group in the entire bucket and five inventories walked past it. Sorting by bytes would have found it on day one.

The check to run first, next time. Rank every prefix by total size and adjudicate the top of the list. It is one pass, it needs no knowledge of naming conventions, no repo, and no memory of what was created — and it degrades gracefully: what it cannot see is small, and what is small is cheap to lose. Name and structure sweeps are the follow-up, not the opener.

The sweep re-run with the list it should have had — zero new findings

Prefix	GB
`sentinel2/Woodwell_sent2_74-84N_…`	511.00	ours — rescued (now matches on `sentinel`)
`working/RTS_PlanetScope_4BandRGB_1024_Banks_clean_v2.zip`	4.91	ours — rescued
`sentinel2/…T07WFR_SR.tif.tif` + `…_SCL.tif.tif`	1.15	ours — rescued
`working/rts_vit_sem_seg_1024inputs{,_Banks}.zip`	2.08	ours — rescued
`workflows_optimization/rts_ray_pipeline`	5.35	AWI's DARTS
`viz_workflow/output`, `viz_workflow/lakes`	2.62	PDG's — matched `Quad` on `WGS1984Quad`, the OGC tile-matrix set behind the Permafrost Discovery Gateway. Ours is `WebMercatorQuad`/EPSG:3857 and lives in our own bucket
`workflows_optimization/lake_drainage`, `pdg_clowder_data/Lake_Drainage_Pipeline`	~0	PDG's — matched `Arctic` on `Pan-Arctic …_freq.png`

Two independent methods — rank by size, and grep every full path with a vocabulary that includes the institution, not just the project — now agree exactly, and both agree with the targeted work. That is the closure: pdg-storage-default holds nothing of ours.

What actually works: open every prefix, don't read its name

Five inventories, five misses, all the same shape — a prefix listed and never opened. The fix is cheap and should have been the first thing done. A delimiter walk costs one API call per prefix, so three levels of a 1.56 M-object bucket takes seconds rather than the two hours a full walk needs:

it = client.list_blobs(bucket, prefix=prefix, delimiter="/",
                       fields="items(name,size),prefixes,nextPageToken")
loose, subs = [b for page in it.pages for b in page], sorted(it.prefixes)   # then recurse into subs

That walk is what surfaced both working/ and sentinel2/…74-84N…. A name-regex sweep of all 1,562,015 object names (rts|slump|thaw|banks|arctic|planetscope|abrupt) was run alongside it as a cross-check and returned six prefixes: the three working/ archives, rts_ray_pipeline (AWI's), and two PDG Pan-Arctic …_freq.png lake plots. It did not return the 511 GB.

Why it missed, corrected. The first reading of this was that the sweep could not see the folder — that only the directory name gave it away. That is wrong, and worth correcting because it points at the wrong lesson. blob.name is the full path, so sentinel2/Woodwell_sent2_74-84N_…/Lat74_Lon-102.tif was in scope all along. The sweep missed it because the pattern was too narrow — no woodwell, no sent2, no sentinel. The keyword list had been built from this repo's vocabulary, and the whole point of that folder is that it was made outside this repo. A keyword sweep can only find what you already think of; that is its real limit, not an inability to see directories.

6. Teardown (irreversible — only after §5)

Gate cleared 2026-08-28 — §5 row 1 PASS, every leg compared object-by-object with missing = 0. Runnable form: teardown.ps1. Progress:

Step	State
Soft-delete cleared on all three buckets	done 2026-08-28 — all carried the 7-day default (604,800 s), which keeps deleted objects billable for a week
Cloud Run service `rts-review` (superseded 08-04, never torn down)	done 2026-08-28
`rts-review-vm` stopped	done 2026-08-28 — after verifying the new app was ahead of it
`a100-8x-train` stopped	done 2026-08-28 — `--discard-local-ssd=false`, so the t65 build survives on disk as well as in GCS
Delete VMs, release `rts-review-vm-ip`, delete our three images	done 2026-08-29
Delete `rts-mapping-v2`, `rts-mapping-v2-usc1`	done (usc1 2026-09-01, after the `staging/` rescue and a whole-bucket re-verify)
Empty and delete `rts-mapping-v2-usw1`	DONE 2026-09-01 — the age-0 rule took it from 20.9 TB to 56 GB in its first pass (`total_bytes`, 08-29 → 08-31); a live count at 17:45 UTC showed 2,337,904 objects left of the original ~42 M, and by 18:54 UTC the rule had finished and the empty bucket was deleted. `describe` now returns 404. No versioning and no retention policy, so the rule reached everything — it needed nobody, which was the point of choosing it over a client-side `rm` loop
Rescue from `pdg-storage-default`	done 2026-09-01 — 511 GB of north-domain S2 (398/398, 0 missing, 0 differing) + its 1.15 GB prototype pair + 6.99 GB of legacy RTS archives, all CRC32C-verified (§5d)
Final sweep of every other PDG surface	done 2026-09-01 — see below
`pdg-planet-data` released to PDG for deletion	done 2026-09-01 — audited first (§5c); nothing of ours depends on it

Ordering trap worth restating: release the static IP after its VM is deleted, or the release is refused as in-use.

Final sweep, 2026-09-01 — what is left in PDG and who owns it. Confirmed against each live service API, not the asset inventory (which lags deletions):

Surface	State
Instances	`download-vm`, `gke-water-cluster-` ×2 — none ours*
Disks	the same three, plus one GKE PVC — none ours
Addresses	one NAT auto-IP — none ours; `rts-review-vm-ip` released
Cloud Run	empty
Artifact Registry	`lake_drainage_test` only — our three images gone
Firewall	no `rts-*` rule; `rts-review-allow-http` deleted
Service accounts	`rts-review-app@` deleted; `planet-orders@pdg-…` deliberately left (§5b)
Project IAM	no binding for any of our identities
Cloud Functions, Dataproc, Cloud SQL	APIs never enabled — stronger than an empty list
Earth Engine	9 assets, all 9 present in `abruptthawmapping` (re-listed live 09-01)
Buckets	All three of ours are deleted. What remains is PDG's own, and all three were closed out by evidence rather than by assumption: `pdg-planet-data` (only `global_quarterly/{2019,2022,2025}`, every object verified present in the new bucket), `pdg-storage-default` (§5d — closed twice over, by size census and by widened full-path sweep, 0 unaccounted), `_cloudbuild` (7 tarballs, archived)

Residue spot-check. 601 objects sampled evenly across the 689,870 the purge had not yet reached were each looked up in rts-arctic-usw1: 0 missing, 0 differing. This is not a second gate — §5 row 1 already compared all ~42 M objects with missing = 0, and the residue is a strict subset of that. It re-tests the one assumption behind the word "subset": that nothing has been written to the PDG bucket since. Nothing can be — the master is deleted, the EE queue cancelled, Cloud Run gone, and no service account retains write access.

The soft-delete tail closed itself. rts-mapping-v2-usw1 was carrying 394 GB of soft-deleted-object, flat since 08-30: disabling soft-delete stops new soft deletes but lets objects already in that state serve out their original 7-day window, which would have billed until about 2026-09-04. Deleting the bucket purged it immediately, so that tail is gone too.

teardown.ps1 stalled part-way through step 4 (2026-08-29) — gcloud storage rm --recursive died somewhere inside usw1's 42 M objects, leaving usc1 completely untouched. That stall is what bought the time to find the usc1/staging/ gap in §4d, so it was a fortunate failure, but it is a bad way to delete tens of millions of objects: it is a client-side loop making one DELETE call per object over hours, and anything that interrupts the client loses the rest.

Use a lifecycle rule for the large bucket instead. GCS applies it server-side and asynchronously, so it completes whether or not we still hold access — which matters when the whole point is that access is about to be revoked:

'{"rule":[{"action":{"type":"Delete"},"condition":{"age":0}}]}' > purge.json
gcloud storage buckets update gs://rts-mapping-v2-usw1 --lifecycle-file=purge.json

Soft-delete must already be off (§6 step 0), or the purge just moves 20 TB into a week of soft-delete billing. Verified 0 on both buckets before applying. The rule empties the bucket; the empty bucket itself costs nothing and can be removed by anyone who still has access.

Do not read storage/v2/total_bytes as a live gauge. Checking whether Heidi's restart was delivering, that metric showed rts-arctic-usw1 flat to the gigabyte across three days while 37,882 orders were placed — which reads exactly like deliveries failing. It is a coarse, roughly daily snapshot, and it was simply stale. list_delivered() counted +37,900 quads over the same window. The metric is trustworthy for the historical question §4d used it for (what a now-deleted bucket used to hold) and misleading for "is data arriving right now" — count objects for that.

The original sequence

Stop cron and tmux on the master; final delta-sync; re-list each source prefix once more for anything that landed late.
Cut reviewers to the new URL, then retire rts-review-vm.
Delete a100-8x-train (only after its disks are drained and signed off) and rts-review-vm; release rts-review-vm-ip; delete the Cloud Run service rts-review.
Delete our three images from pdg-artifact-registry; leave lake_drainage_test.
Disable soft-delete first — 7-day retention means deleted objects keep billing — then delete rts-mapping-v2, rts-mapping-v2-usw1, rts-mapping-v2-usc1.
pdg-planet-data: theirs to delete, and verified safe to do so. Originally written as "hand back, do not delete" on the assumption PDG would keep the bucket. They are deleting everything instead, which makes the question "is our copy complete?" rather than "who deletes it?" — answered in §5c: 0 missing, 0 differing across all 5,000,891 objects. Still not ours to delete, but nothing is lost when they do.
Confirm no rts-* VM, disk, address, Cloud Run service or bucket remains in PDG.

Navigation Menu

Uh oh!

PDG wind-down — migration to `abruptthawmapping` (Sept 2026)

The deadline is now Monday 2026-08-31, not 09-07

1. What was there

2. Decisions

Two findings worth keeping

3. Path history — old → new

3b. Day-1 audit — what on the master is genuinely local-only

4. Runbook

Copy method

4a. Operational hazard found the hard way: the checkout is live infrastructure

4b. Provisioning — run these from `ARCHITECTURE`

1. APIs — DONE 2026-08-27

2. The three buckets

3. Lifecycle — imagery ages into Coldline

4. Artifact Registry

5. `rts-ops` — the migration workhorse

5a. Egress — a `--no-address` VM has none, and this step was missing

5b. On-box provisioning — state 2026-08-27

5c. `gcloud` CLI and ADC are different credentials — the copy needs both

6. Verify before copying anything

4c. Earth Engine assets — migrated 2026-08-28

4d. The inventory was wrong a fourth time — `usc1/staging/`, caught during teardown

`usw1` could no longer be enumerated — so it was confirmed a different way

5. Verification — the gate before deletion

5b. Project-wide audit — everything of ours in PDG, 2026-09-01

5c. Clearing `pdg-planet-data` for deletion — 2026-09-01

5d. `pdg-storage-default` — the inventory was wrong a fifth time, and it was the expensive one

The big one: 511 GB of Sentinel-2, one level down inside `sentinel2/`

The smaller one: three legacy RTS archives in `working/`

Closing it properly: rank by bytes, not by name

The sweep re-run with the list it should have had — zero new findings

What actually works: open every prefix, don't read its name

6. Teardown (irreversible — only after §5)

The original sequence

Uh oh!

FilesExpand file tree

pdg_migration.md

Latest commit

History

pdg_migration.md

File metadata and controls

PDG wind-down — migration to abruptthawmapping (Sept 2026)

The deadline is now Monday 2026-08-31, not 09-07

1. What was there

2. Decisions

Two findings worth keeping

3. Path history — old → new

3b. Day-1 audit — what on the master is genuinely local-only

4. Runbook

Copy method

4a. Operational hazard found the hard way: the checkout is live infrastructure

4b. Provisioning — run these from ARCHITECTURE

1. APIs — DONE 2026-08-27

2. The three buckets

3. Lifecycle — imagery ages into Coldline

4. Artifact Registry

5. rts-ops — the migration workhorse

5a. Egress — a --no-address VM has none, and this step was missing

5b. On-box provisioning — state 2026-08-27

5c. gcloud CLI and ADC are different credentials — the copy needs both

6. Verify before copying anything

4c. Earth Engine assets — migrated 2026-08-28

4d. The inventory was wrong a fourth time — usc1/staging/, caught during teardown

usw1 could no longer be enumerated — so it was confirmed a different way

5. Verification — the gate before deletion

5b. Project-wide audit — everything of ours in PDG, 2026-09-01

5c. Clearing pdg-planet-data for deletion — 2026-09-01

5d. pdg-storage-default — the inventory was wrong a fifth time, and it was the expensive one

The big one: 511 GB of Sentinel-2, one level down inside sentinel2/

The smaller one: three legacy RTS archives in working/

Closing it properly: rank by bytes, not by name

The sweep re-run with the list it should have had — zero new findings

What actually works: open every prefix, don't read its name

6. Teardown (irreversible — only after §5)

The original sequence

PDG wind-down — migration to `abruptthawmapping` (Sept 2026)

4b. Provisioning — run these from `ARCHITECTURE`

5. `rts-ops` — the migration workhorse

5a. Egress — a `--no-address` VM has none, and this step was missing

5c. `gcloud` CLI and ADC are different credentials — the copy needs both

4d. The inventory was wrong a fourth time — `usc1/staging/`, caught during teardown

`usw1` could no longer be enumerated — so it was confirmed a different way

5c. Clearing `pdg-planet-data` for deletion — 2026-09-01

5d. `pdg-storage-default` — the inventory was wrong a fifth time, and it was the expensive one

The big one: 511 GB of Sentinel-2, one level down inside `sentinel2/`

The smaller one: three legacy RTS archives in `working/`