From c65e4858019ee09ca54c052d554f9565479dbf34 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 28 Aug 2026 18:02:21 +0000 Subject: [PATCH 01/10] docs: record the CARTO basemap API-key outage across the suite MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CARTO began burning an "API KEY REQUIRED" watermark into unauthenticated raster tiles from basemaps.cartocdn.com, server-side. The request still returns HTTP 200 with a valid PNG, so nothing errors, logs, or falls back — the map just renders defaced. All nine companion apps are hit on first load, because every one of them puts CartoDB.Positron on its landing site-picker map. Driver-Cascade itself ships no Leaflet map and is unaffected. Adds docs/SUITE-BASEMAP-INCIDENT-2026-08.md with the verified diagnosis, the reproduction, the non-causes eliminated with evidence (package versions, referer/CORS, CSP, app strings), the per-app call-site table, the decided fix (Esri.WorldGrayCanvas plus a CSS-inverted tile pane for dark), the rejected alternatives, and a resume checklist. Records the lesson and a dated handoff entry. Docs only: docs/ and .claude/ are outside DEPLOY_APP_FILES, so no artifact, manifest, or deploy surface moves. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01738HeGPEU9NEoczL8RXnu1 --- .claude/agents/LESSONS.md | 14 ++ docs/BUILD-TEST-HANDOFF.md | 53 +++++- docs/SUITE-BASEMAP-INCIDENT-2026-08.md | 226 +++++++++++++++++++++++++ 3 files changed, 292 insertions(+), 1 deletion(-) create mode 100644 docs/SUITE-BASEMAP-INCIDENT-2026-08.md diff --git a/.claude/agents/LESSONS.md b/.claude/agents/LESSONS.md index 2c99478..11ff467 100644 --- a/.claude/agents/LESSONS.md +++ b/.claude/agents/LESSONS.md @@ -123,3 +123,17 @@ artifacts. If that fails, the environment cannot produce publishable artifacts, full stop — do not ship them and do not re-register hashes from them. Text artifacts (the codebook CSV) still match, so a diff where only the compressed RDS move is the tell-tale signature of an encoding, not content, difference. +- [2026-08-28] cass · confirmed · A third-party basemap can break EVERY app in the suite at once with no + error, no log line, and no deploy: on 2026-08-28 CARTO began burning an "API KEY REQUIRED" watermark into + unauthenticated `basemaps.cartocdn.com` raster tiles server-side. The request still returns HTTP 200 and a + valid PNG, so nothing fails loudly — the map just renders defaced. All nine companion apps were hit on + FIRST LOAD because every one of them puts `CartoDB.Positron` on its landing/site-picker map; Driver-Cascade + was untouched only because it ships no Leaflet map. DECISIVE TEST when a hosted map "looks wrong": fetch one + raw tile with curl and LOOK AT THE IMAGE (`curl -s -o t.png "https://a.basemaps.cartocdn.com/light_all/6/13/24.png"`). + Do not start from the R package — the two `leaflet.providers` versions in the suite (2.0.0 and 3.0.0) ship + byte-identical CartoDB/Esri url templates with no `{apikey}` placeholder, and the apps pinning each were + equally affected, which by itself rules the package out. Referer/Origin/CORS were also ruled out: the same + watermarked bytes come back with browser headers and a live Connect Cloud referer. GENERAL LESSON: a free + tile provider is an unversioned, uncontracted runtime dependency of the deploy surface that no manifest pin + covers — the fix (`Esri.WorldGrayCanvas`) has a shelf life too, because Esri has already announced the same + sunset for the legacy `server.arcgisonline.com` endpoints. Full record: `docs/SUITE-BASEMAP-INCIDENT-2026-08.md`. diff --git a/docs/BUILD-TEST-HANDOFF.md b/docs/BUILD-TEST-HANDOFF.md index d2934e5..c860edf 100644 --- a/docs/BUILD-TEST-HANDOFF.md +++ b/docs/BUILD-TEST-HANDOFF.md @@ -1,6 +1,6 @@ # Build, test, and handoff record -Last updated: 2026-08-06 +Last updated: 2026-08-28 This is the durable operating record for the NEON Driver Cascade repository. Read the whole document before doing work. Keep it factual and current so a new session @@ -4635,3 +4635,54 @@ Rules: new upstream sites are excluded by decision rather than by omission. (b) preserves the current published family; (a) expands it and needs a coverage review. Do not simply append a row to silence the guard. + +### 2026-08-28 - [Claude] suite basemap outage: CARTO "API KEY REQUIRED" watermark + +- **Reported:** the maps in the NEON explorer apps display "need api key for + layers/basemap" across the basemap, in a lot — as it turns out, all — of the suite. +- **Diagnosed:** CARTO now burns an `API KEY REQUIRED / carto.com/basemaps/apikey` + watermark into unauthenticated raster tiles from `basemaps.cartocdn.com`, + server-side. The tile request still returns HTTP 200 with a valid PNG, so no app + errors, logs, or falls back — the map simply renders defaced. Verified by fetching + `light_all` (`CartoDB.Positron`) and `dark_all` (`CartoDB.DarkMatter`) tiles and + inspecting the images. Both variants are affected. +- **Scope:** all **nine** companion apps, **on first load** — every one of them puts + `CartoDB.Positron` on its landing/site-picker map, which needs no interaction. + Apps whose main explore map defaults to an `Esri.*` layer look correct until the + user switches the Basemap select to "Light". **Driver-Cascade is unaffected:** it + ships no Leaflet map (no `leaflet`, `addTiles`, or `addProviderTiles` in + `global.R`, `ui.R`, `server.R`, or `R/`), and this change touched only `docs/` and + `.claude/`, both outside the `DEPLOY_APP_FILES` allowlist, so no artifact, + manifest, or deploy surface moved. +- **Ruled out with evidence, so no one re-walks it:** not an R package regression + (`leaflet.providers` 2.0.0 and 3.0.0 ship byte-identical CartoDB/Esri url + templates with no `{apikey}` placeholder, and the apps pinning each version were + equally affected); not a leaflet-providers API-key code path (the bundled JS only + throws `No such provider`); not referer/Origin/CORS (identical watermarked bytes + returned with browser headers and a live Connect Cloud referer); not CSP, not a + manifest pin, not something a redeploy clears; and no app's own source contains an + "api key" string. +- **Decided (owner, 2026-08-28):** `CartoDB.Positron` → `Esri.WorldGrayCanvas` + (keyless, fetched and visually confirmed clean); `CartoDB.DarkMatter` → CSS-invert + the Leaflet **tile pane only**, synthesising a dark canvas from the same grey + basemap so markers, labels and the attribution control are untouched; existing + `Esri.*` choices unchanged. A CARTO API key was rejected (public in any + client-side tile URL, nine deployments to manage, and CARTO is retiring raster + basemaps regardless). +- **Caveat carried on the fix:** Esri has already announced the same sunset for the + legacy `server.arcgisonline.com` basemap endpoints this moves onto. This is a fix + with a shelf life, recorded as a standing risk rather than presented as settled. +- **Not done here:** the nine companion repos are **not yet patched**. Their per-file + call sites, the CSS-invert open questions, the max-zoom check, and a rebuild recipe + for the local sibling checkouts are all in + `docs/SUITE-BASEMAP-INCIDENT-2026-08.md` §6. +- **Environment limit found:** headless-browser verification of the deployed apps is + impossible in this container — Chromium is installed and Playwright configured, but + every navigation fails `net::ERR_CONNECTION_RESET`, including `https://example.com`, + routed through the agent proxy or not. `curl` works. The evidence here is therefore + tile-level: decisive for the cause, but no one has yet *seen* a fixed app. +- **Next action:** settle the CSS-invert selector and the `Canvas/World_Light_Gray_Base` + max zoom, then apply the patch to the nine companion repos — one branch and draft PR + each, respecting each repo's default branch (Driver-Cascade `master`; Small Mammal and + Vegetation `main`) and its CI manifest gate, which is byte-exact in the siblings where + Driver-Cascade's is semantic. diff --git a/docs/SUITE-BASEMAP-INCIDENT-2026-08.md b/docs/SUITE-BASEMAP-INCIDENT-2026-08.md new file mode 100644 index 0000000..a7f0a71 --- /dev/null +++ b/docs/SUITE-BASEMAP-INCIDENT-2026-08.md @@ -0,0 +1,226 @@ +# Suite incident: CARTO stamps "API KEY REQUIRED" on every basemap tile + +**Status: DIAGNOSED — root cause verified, fix decided, not yet applied to the nine apps.** +**Opened 2026-08-28 · Driver-Cascade branch `claude/neon-maps-api-key-eocg8r` · tagged `[Claude]`.** + +This is the suite-wide record for the basemap outage reported on 2026-08-28: the maps in the +NEON explorer apps display a large diagonal watermark reading **"API KEY REQUIRED — +carto.com/basemaps/apikey"** across the basemap. Driver-Cascade is the suite hub, so the +advisory lives here even though **Driver-Cascade itself is NOT affected** (it ships no Leaflet +map — verified: no `leaflet`, `addTiles`, or `addProviderTiles` anywhere in `global.R`, `ui.R`, +`server.R`, or `R/`). + +--- + +## 1. What is actually happening + +CARTO now watermarks **unauthenticated** raster tiles served from +`https://{s}.basemaps.cartocdn.com/…`. The watermark is **burned into the tile image +server-side**. The request still returns `HTTP 200` with a valid PNG, so nothing in the app +errors, logs, or falls back — the map simply renders defaced. + +### Reproduction (verified live, 2026-08-28) + +```sh +curl -s -o carto_light.png "https://a.basemaps.cartocdn.com/light_all/6/13/24.png" # CartoDB.Positron +curl -s -o carto_dark.png "https://a.basemaps.cartocdn.com/dark_all/6/13/24.png" # CartoDB.DarkMatter +``` + +Both return `HTTP 200`, `image/png`, 256×256 (6465 B and 5884 B respectively). Opening either +image shows a real Colorado basemap overprinted with a grey diagonal +`API KEY REQUIRED / carto.com/basemaps/apikey`. Both the light and dark variants are affected. + +### What it is NOT — ruled out with evidence + +Each of these was checked and eliminated, so no future session needs to re-walk them: + +- **Not an R package regression.** `leaflet.providers` **2.0.0** and **3.0.0** were both + downloaded from CRAN and their bundled `leaflet-providers.js` compared directly. The + `CartoDB` and `Esri` URL templates are **byte-identical between the two versions**, and + neither carries an `{apikey}` placeholder. Eight apps pin `leaflet.providers` 3.0.0 and the + Water-Chemistry app pins 2.0.0 — **both are equally affected**, which is itself proof the + package version is not the variable. +- **Not a leaflet-providers API-key check.** The bundled JS only ever throws + `No such provider (…)` / `No such variant of …`. It has no API-key error path. The + `` placeholders in that file belong to Thunderforest, Jawg, + Mapbox, MapTiler, TomTom, HERE and OpenWeatherMap — **providers this suite does not use**. +- **Not a referer / origin / CORS block.** The tile was re-fetched with a browser + `User-Agent`, with `Referer` set to a live `*.share.connect.posit.cloud` app URL, and with + full `Origin` + `Sec-Fetch-*` headers. **All three return the identical watermarked bytes** + (`md5` prefix `00f0cd56cfe1` in every case). +- **Not a Posit Connect Cloud problem**, not a CSP problem, not a manifest-pin problem, and + not something a redeploy can clear. Re-deploying the identical manifest re-fetches the same + watermarked tiles. +- **Not an app-side string.** `grep` across all nine repos finds **no** occurrence of + "api key" / "apikey" / "access token" in any app's own R, JS, CSS or HTML. + +### Esri is currently clean + +Tiles from `https://server.arcgisonline.com/ArcGIS/rest/services/{variant}/MapServer/tile/{z}/{y}/{x}` +were fetched and visually inspected for `World_Topo_Map`, `World_Imagery`, `World_Street_Map`, +`NatGeo_World_Map` and `Canvas/World_Light_Gray_Base` — **all keyless, all unwatermarked**. +That is why the apps whose *main* map defaults to an Esri layer look fine until the user +switches the Basemap select to "Light". + +> **Caveat that belongs on the number:** Esri has publicly announced that these legacy ArcGIS +> Online basemap tile services are deprecated and being sunset in favour of the newer ArcGIS +> basemap layer service (which is keyed). Moving to Esri is a **fix with a shelf life**, not a +> permanent answer. See §5. + +--- + +## 2. Blast radius — all nine companion apps, on first load + +Every companion app puts a `CartoDB.Positron` layer on its **landing / site-picker map**, which +renders with **zero user interaction**. So the watermark is the first thing a visitor sees in +all nine apps, regardless of what the main explore map defaults to. + +| App | Repo | CARTO call sites (file:line) | Watermark visible without interaction | +|---|---|---|---| +| Small Mammal | `neon-small-mammal-tracker-app` | `server.R:1189` (pickerMap), `server.R:2515` (DarkMatter fallback), `ui.R:622-623` (choices) | **Yes** — picker map | +| Plant Diversity | `neon-plant-diversity` | `R/map_picker.R:58`, `ui.R:458` (choices) | **Yes** — picker map | +| Vegetation Structure | `neon-vegetation-structure-explorer` | `R/map_picker.R:88`, `server.R:1375`, `server.R:1398`, `ui.R:259-260` (`selected =`) | **Yes** — picker *and* main map default | +| Breeding Birds | `neon-breeding-birds` | `server.R:711`, `server.R:723`, `ui.R:298` (choices) | **Yes** — `nationalPicker` | +| Plant Phenology | `neon-plant-phenology-explorer` | `server.R:145`, `server.R:177`, `server.R:779`, `ui.R:215` (choices) | **Yes** — `nationalMap` *and* main map default | +| Mosquito Pulse | `neon-mosquito-pulse` | `server.R:578`, `server.R:588`, `ui.R:229` (choices) | **Yes** — `nationalPicker` | +| Ground Beetle | `neon-ground-beetle-tracker` | `R/map_picker.R:57`, `server.R:1633` (theme-aware Positron/DarkMatter) | **Yes** — picker *and* main map, both themes | +| My Little Inverts | `neon-my-little-inverts` | `server.R:940`, `server.R:963` (`providers$CartoDB.Positron`) | **Yes** — `nationalPicker` | +| Water Chemistry | `neon-waterchemistry-analyte-viewer-app` | `app.R:2110` | **Yes** — the only map | +| **Driver-Cascade** | `NEON-Driver-Cascade` | **none** | **No — unaffected** | + +Line numbers are from each repo's default-branch HEAD as cloned on 2026-08-28. **Re-verify them +before patching** — they are a navigation aid, not a contract. + +> ⚠️ **Branch defaults are SPLIT across the suite.** Driver-Cascade is `master`; Small Mammal +> and Vegetation are `main`. Never assume — check each repo before branching or pushing. + +--- + +## 3. The decided fix + +Owner decision, 2026-08-28: + +1. **`CartoDB.Positron` → `Esri.WorldGrayCanvas`** (variant `Canvas/World_Light_Gray_Base`). + Keyless, verified unwatermarked, and a pale minimal canvas that fills the same visual role + as Positron. Confirmed to render over Alaska as well as the lower 48. +2. **`CartoDB.DarkMatter` → CSS-invert the grey canvas.** There is **no keyless dark canvas** + in `leaflet.providers` 3.0.0. Rather than take a key or drop the dark theme, synthesise the + dark basemap by filtering **only the Leaflet tile pane**, leaving markers, labels, popups and + the attribution control untouched (they live in separate panes). The exact selector and the + Shiny-side wiring must be verified before it ships — see the open item in §6. +3. **Leave existing `Esri.*` choices alone.** `Esri.WorldTopoMap` ("Terrain") and + `Esri.WorldImagery` ("Satellite") are unaffected and stay as they are. +4. **Update the visible labels** in each `selectInput("view", "Basemap", …)` so "Light" still + maps to a sensible provider string. + +### Rejected alternatives, and why + +- **Take a CARTO API key** (free tier, 5M tiles/month). Rejected: the key is unavoidably public + in a client-side tile URL, it needs managing across nine separate Connect Cloud deployments, + and CARTO is retiring raster basemaps regardless — so it buys a cosmetic reprieve and leaves + the same cliff in place. +- **`OpenStreetMap.Mapnik`.** Rejected: the OSMF tile usage policy discourages exactly this + kind of hosted-app embedding, and the style is far busier than a canvas behind circle markers. + (Noted: a request from this container returned `HTTP 200` with an `x-blocked: Access denied` + header — OSMF blocks datacentre IPs.) +- **`USGS.USTopo` / `USGS.USImagery`.** Keyless and live, and thematically apt for a US network. + Rejected as the *canvas* replacement because `USGSTopo` is a full-colour topographic sheet + with highway shields and dense labels — unusable behind the suite's circle markers. Worth + keeping in mind as a future "Topo" option. +- **Drop the dark basemap.** Rejected: the ground-beetle app deliberately re-themes its map + with the rest of the UI, and losing that is a real regression. + +--- + +## 4. Applying it — what a patch must respect + +- **Docs-only changes in Driver-Cascade are outside the deploy surface.** `scripts/manifest_files.R` + allowlists only `global.R`, `ui.R`, `server.R`, `R/cascade_helpers.R`, `R/site_metadata.R`, + `www/{cascade.css,cascade.js,styles.css}` and the four `data/` artifacts. `docs/` and + `.claude/` are not in it, so this document does not touch the manifest gate. +- **In the companion repos the patch DOES touch `ui.R` / `server.R` / `R/*.R`**, which are on + their deploy surfaces. A source-only edit changes no package, so `manifest.json` should not + need regeneration — **but each repo's CI manifest gate must be checked**, because the sibling + gates are byte-exact where Driver-Cascade's is semantic. +- **A push to the watched branch is the deploy.** Each companion repo is watched by Posit + Connect Cloud; merging is what ships. Nine merges = nine deploys. +- Match each repo's local idiom: pipe style (`%>%` vs `|>`), namespacing (`leaflet::` vs bare), + and its own `CLAUDE.md` / `AGENTS.md` house rules. + +--- + +## 5. Standing risk to record + +Both the old and the new basemap are third-party freebies with no contract behind them. CARTO +withdrew its free raster tier with no warning that reached this project; Esri has already +announced the same intent for the legacy endpoints this fix moves onto. **Treat the basemap as +a dependency with an owner and a review date, not as scenery.** Options that would durably +de-risk it — and are explicitly NOT part of this fix: + +- Self-host a minimal vector or raster canvas for the ~81 NEON site locations (the suite only + ever needs a national overview plus site-level zoom, not a world basemap). +- Draw a simple state/coastline outline from a committed GeoJSON — no tile server at all. The + `prototypes/site-explorer` map already proves this suite can render real geography with no + tiles and no key. +- Add a suite CI check that fetches one tile from each configured provider and fails on a + non-image, an unexpected byte-size shift, or a known watermark signature. + +--- + +## 6. Resume checklist — pick this up cold + +Work completed on `claude/neon-maps-api-key-eocg8r` in Driver-Cascade: + +- [x] Root cause identified and verified by direct tile fetch + image inspection. +- [x] Non-causes eliminated with evidence (package versions, referer, CORS, CSP, app strings). +- [x] Replacement `Esri.WorldGrayCanvas` fetched and visually confirmed clean. +- [x] All nine companion apps enumerated with per-file call sites. +- [x] Owner decision recorded: Esri grey canvas + CSS-inverted tile pane for dark. +- [x] This advisory + `LESSONS.md` + `BUILD-TEST-HANDOFF.md` entries committed. + +Open, in order: + +- [ ] **Verify the CSS-invert approach concretely** before it ships: confirm the Leaflet 1.x + pane structure (`.leaflet-tile-pane` vs `.leaflet-overlay-pane` / `.leaflet-marker-pane`), + settle the exact filter (`invert(1) hue-rotate(180deg)` plus brightness/contrast trim), + confirm the attribution control is not inverted, and decide how it is toggled from each + app's existing theme state (a class on the map container vs `htmlwidgets::onRender`). +- [ ] **Confirm `Canvas/World_Light_Gray_Base` max zoom** against + `https://server.arcgisonline.com/ArcGIS/rest/services/Canvas/World_Light_Gray_Base/MapServer?f=json` + and check it is deep enough for the plot-level maps (Positron went to zoom 20). +- [ ] **Check marker contrast** in each app against the paler grey canvas — several palettes + were tuned against Positron's near-white. +- [ ] **Apply the patch to the nine companion repos**, one branch + draft PR each, respecting + each repo's default branch (`master` vs `main`) and CI manifest gate. +- [ ] **Re-check the deployed apps** after each merge; Connect Cloud redeploys on push. +- [ ] Consider the suite CI tile-canary from §5. + +### Local working state + +The nine companion repos were cloned shallow + blobless + sparse (code present, `data/` and +`assets/` deliberately not checked out) under `/home/user/tgilbert14/`. That checkout is +**ephemeral** — this container is reclaimed after the session. To rebuild it: + +```sh +for r in neon-small-mammal-tracker-app neon-plant-diversity \ + neon-vegetation-structure-explorer neon-breeding-birds \ + neon-plant-phenology-explorer neon-mosquito-pulse \ + neon-ground-beetle-tracker neon-my-little-inverts \ + neon-waterchemistry-analyte-viewer-app; do + git clone --depth 1 --filter=blob:none --no-checkout \ + "https://github.com/tgilbert14/$r" "/home/user/tgilbert14/$r" + git -C "/home/user/tgilbert14/$r" sparse-checkout set --no-cone \ + '/*' '!/data/**' '!/data-sample/**' '!/assets/**' '!/docs/assets/**' + git -C "/home/user/tgilbert14/$r" checkout HEAD +done +``` + +### Environment note for the next session + +**A headless-browser check of the deployed apps was not possible here.** Chromium is installed +and Playwright is configured, but every navigation fails with `net::ERR_CONNECTION_RESET` — +including `https://example.com` — so the browser has no egress in this container even routed +through the agent proxy. `curl` works fine. A session that *can* drive a browser should +confirm the watermark on a live app and confirm it is gone after the fix; until then the +evidence here is tile-level, which is decisive for the cause but does not substitute for +seeing a fixed app. From 19d3a35ae8e28085b84792f92d0c0ff508242a87 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 28 Aug 2026 19:30:42 +0000 Subject: [PATCH 02/10] =?UTF-8?q?docs:=20correct=20the=20basemap=20fix=20?= =?UTF-8?q?=E2=80=94=20the=20Esri=20canvas=20is=20blank=20at=20plot=20scal?= =?UTF-8?q?e?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A nine-app audit and a four-lens adversarial review refuted the fix recorded in the previous commit. The diagnosis is unchanged; the remedy moved. Measured: Canvas/World_Light_Gray_Base has had no content update since 2021. At z16 it returns a single-colour blank tile for 17 of NEON's 46 terrestrial sites; SCBI z15 and z16 are byte-identical; above z16 every request is the same "Map data not yet available" placeholder. Verified by hand for SCBI (z13 = 224 distinct colours, z16 = 1). Every other candidate tested is 0/46 blank. So the swap is right for the national pickers at z2-5 and wrong for every plot-scale map, and the proposed maxNativeZoom guard upscales an already-blank tile rather than fixing it. The CSS-invert dark workstream is cancelled, superseded rather than deferred: Canvas/World_Dark_Gray_Base is a real, keyless, labelled dark canvas, verified live. The earlier "there is no keyless dark canvas" claim was false — it asked what leaflet-providers exposes, never what the provider serves. Three further defects: the swap makes marker contrast ~8-10% worse, not better; leaflet.providers emits a stale Esri attribution that omits the OpenStreetMap credit the service's own copyrightText requires, trading a compliant credit line for a non-compliant one; and one app uses the object form providers$CartoDB.Positron, which a quoted-string grep misses. Replaces the plan with a per-map-role one, records the 22 call sites across 9 repos, maps the gate asymmetry (two repos have no ci.yml), and raises two owner questions that block rollout. Docs only: docs/ and .claude/ are outside DEPLOY_APP_FILES. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01738HeGPEU9NEoczL8RXnu1 --- .claude/agents/LESSONS.md | 19 ++ docs/BUILD-TEST-HANDOFF.md | 50 +++ docs/SUITE-BASEMAP-INCIDENT-2026-08.md | 409 +++++++++++++++++++------ 3 files changed, 379 insertions(+), 99 deletions(-) diff --git a/.claude/agents/LESSONS.md b/.claude/agents/LESSONS.md index 11ff467..34bb886 100644 --- a/.claude/agents/LESSONS.md +++ b/.claude/agents/LESSONS.md @@ -137,3 +137,22 @@ tile provider is an unversioned, uncontracted runtime dependency of the deploy surface that no manifest pin covers — the fix (`Esri.WorldGrayCanvas`) has a shelf life too, because Esri has already announced the same sunset for the legacy `server.arcgisonline.com` endpoints. Full record: `docs/SUITE-BASEMAP-INCIDENT-2026-08.md`. +- [2026-08-28] cass · confirmed · "Unwatermarked" is not "usable", and HTTP 200 proves neither. The obvious fix + for the CARTO watermark — swap `CartoDB.Positron` for the keyless `Esri.WorldGrayCanvas` — was REFUTED by + measurement after it had already been written down as decided. `Canvas/World_Light_Gray_Base` has had no + content update since 2021: at z16 it returns a tile with exactly ONE distinct colour (RGB 239,239,239) for + 17 of NEON's 46 terrestrial sites, and SCBI z15/z16 are byte-identical, so the `maxNativeZoom = 16` mitigation + upscales an already-blank tile and HIDES the defect from an eyeball check. DECISIVE TEST for any basemap + change: decode the tile and count distinct RGB values at the zooms the app actually uses (the picker's z3-4 + AND the plot map's z13-16) — a single-colour tile is blank, and Esri's "Map data not yet available" + placeholder is md5 `f27d9de7f80c13501f470595e327aa6d`. Three further traps found the same way: the swap makes + marker contrast ~8-10% WORSE (three sub-audits asserted the opposite — #efefef is darker than Positron's + #fafaf8, and every marker palette is darker still); `leaflet.providers` hardcodes a STALE Esri attribution + ("DeLorme, NAVTEQ") that omits the OpenStreetMap credit the service's own live `copyrightText` requires, so + the swap trades a compliant credit line for a non-compliant one; and a `grep '"CartoDB'` misses + `leaflet::providers$CartoDB.Positron` (the object form, used in My Little Inverts) — sweep with + `-E 'CartoDB[."$]|cartocdn'`. GENERAL LESSON: ask what the PROVIDER serves, not what the wrapper exposes. The + "there is no keyless dark canvas" claim in the first draft of this incident was false — `Canvas/World_Dark_Gray_Base` + is real, keyless and labelled; it is simply absent from leaflet-providers. That single mis-framed question had + already produced a signed-off CSS-invert workstream that was pure waste. Full record + per-role plan: + `docs/SUITE-BASEMAP-INCIDENT-2026-08.md`. diff --git a/docs/BUILD-TEST-HANDOFF.md b/docs/BUILD-TEST-HANDOFF.md index c860edf..3481d1b 100644 --- a/docs/BUILD-TEST-HANDOFF.md +++ b/docs/BUILD-TEST-HANDOFF.md @@ -4686,3 +4686,53 @@ Rules: each, respecting each repo's default branch (Driver-Cascade `master`; Small Mammal and Vegetation `main`) and its CI manifest gate, which is byte-exact in the siblings where Driver-Cascade's is semantic. + +### 2026-08-28 (later) - [Claude] the basemap fix was refuted by its own audit + +- **What changed:** the fix recorded in the entry above — blanket + `CartoDB.Positron` → `Esri.WorldGrayCanvas` plus a CSS-invert dark canvas — was + put through a nine-app audit and a four-lens adversarial review. **All four + lenses refuted it.** The plan in `docs/SUITE-BASEMAP-INCIDENT-2026-08.md` is now + a per-map-ROLE plan (§4), and two owner decisions (§5) block rollout. The + diagnosis in the earlier entry is unchanged and still correct; only the remedy moved. +- **Why, measured:** `Canvas/World_Light_Gray_Base` has had no content update since + 2021. At z16 it returns a single-colour blank tile (`RGB(239,239,239)`) for **17 of + 46** NEON terrestrial sites; SCBI z15 and z16 are byte-identical; above z16 every + request is the same 2,521-byte "Map data not yet available" placeholder. Verified + by hand for SCBI: z13 = 224 distinct colours (hairline roads, no labels), z16 = 1. + `Esri.WorldTopoMap`, `Esri.WorldImagery`, `USGS.USTopo`, `USGS.USImageryTopo`, + `OpenStreetMap.Mapnik` and `CartoDB.Positron` are all 0/46 blank. So the swap is + sound for the national pickers (z2-5) and wrong for every plot-scale map. +- **The proposed `maxNativeZoom = 16` mitigation does not mitigate** — it upscales an + already-blank tile and hides the defect from a visual check. +- **The CSS-invert workstream is cancelled, superseded not deferred.** + `Canvas/World_Dark_Gray_Base` is a real, keyless, labelled dark canvas — verified + live (z4: 5,456 B, 283 distinct colours, dominant `RGB(63,63,65)`). It is simply + absent from `leaflet.providers`, so it needs a raw `addTiles()`. The earlier + "there is no keyless dark canvas" line was FALSE; the question asked what the + wrapper exposed, never what the provider serves. +- **Three further defects the audit surfaced:** the swap makes marker contrast ~8-10% + WORSE (Esri `#efefef` is darker than Positron `#fafaf8`, and every marker palette is + darker still); `leaflet.providers` emits a stale Esri attribution ("DeLorme, NAVTEQ") + that omits the OpenStreetMap credit the service's own live `copyrightText` requires, + so the swap would trade a compliant credit line for a non-compliant one; and + `neon-my-little-inverts` uses the object form `leaflet::providers$CartoDB.Positron`, + which a quoted-string grep misses entirely. Sweep with `-E 'CartoDB[."$]|cartocdn'`. +- **Also found:** ground-beetle's dark branch is read inside `renderLeaflet`, so dark + mode cannot be deferred to a later wave; small-mammal `server.R:2491`/`:2514` suppress + attribution entirely and are non-compliant today with CARTO and equally with Esri; + and `neon-my-little-inverts` and `neon-waterchemistry-analyte-viewer-app` have **no + `ci.yml` at all** — the two repos with the two non-standard idioms are the two with + no pre-merge gate. No repo in the suite asserts any provider string, which is why + this shipped unnoticed. +- **Nothing at risk:** docs-only again; `docs/` and `.claude/` stay outside + `DEPLOY_APP_FILES`, no artifact or manifest moved, and no companion repo was touched. +- **Next action (owner decision, blocking):** two questions in + `docs/SUITE-BASEMAP-INCIDENT-2026-08.md` §5 that could dissolve most of the work — + (1) is there a University of Arizona institutional ArcGIS Online licence, which + removes the "keyless or bust" constraint entirely; (2) should the picker be tileless + at all, given `prototypes/site-explorer` already renders real geography with no tiles + and the suite once shipped a tileless scattergeo picker. The free CARTO key is + deliberately re-opened as a third option: it restores Positron AND DarkMatter with + zero re-tuning and no blank tiles. Do not start the nine-repo rollout before these + are answered. diff --git a/docs/SUITE-BASEMAP-INCIDENT-2026-08.md b/docs/SUITE-BASEMAP-INCIDENT-2026-08.md index a7f0a71..7276258 100644 --- a/docs/SUITE-BASEMAP-INCIDENT-2026-08.md +++ b/docs/SUITE-BASEMAP-INCIDENT-2026-08.md @@ -1,6 +1,8 @@ # Suite incident: CARTO stamps "API KEY REQUIRED" on every basemap tile -**Status: DIAGNOSED — root cause verified, fix decided, not yet applied to the nine apps.** +**Status: DIAGNOSED — root cause verified and measured. The FIRST proposed fix was refuted by evidence (§3) +and replaced by a per-role plan (§4). Two owner decisions (§5) now block rollout. Nothing applied to the +nine apps yet.** **Opened 2026-08-28 · Driver-Cascade branch `claude/neon-maps-api-key-eocg8r` · tagged `[Claude]`.** This is the suite-wide record for the basemap outage reported on 2026-08-28: the maps in the @@ -96,110 +98,321 @@ before patching** — they are a navigation aid, not a contract. --- -## 3. The decided fix - -Owner decision, 2026-08-28: - -1. **`CartoDB.Positron` → `Esri.WorldGrayCanvas`** (variant `Canvas/World_Light_Gray_Base`). - Keyless, verified unwatermarked, and a pale minimal canvas that fills the same visual role - as Positron. Confirmed to render over Alaska as well as the lower 48. -2. **`CartoDB.DarkMatter` → CSS-invert the grey canvas.** There is **no keyless dark canvas** - in `leaflet.providers` 3.0.0. Rather than take a key or drop the dark theme, synthesise the - dark basemap by filtering **only the Leaflet tile pane**, leaving markers, labels, popups and - the attribution control untouched (they live in separate panes). The exact selector and the - Shiny-side wiring must be verified before it ships — see the open item in §6. -3. **Leave existing `Esri.*` choices alone.** `Esri.WorldTopoMap` ("Terrain") and - `Esri.WorldImagery` ("Satellite") are unaffected and stay as they are. -4. **Update the visible labels** in each `selectInput("view", "Basemap", …)` so "Light" still - maps to a sensible provider string. - -### Rejected alternatives, and why - -- **Take a CARTO API key** (free tier, 5M tiles/month). Rejected: the key is unavoidably public - in a client-side tile URL, it needs managing across nine separate Connect Cloud deployments, - and CARTO is retiring raster basemaps regardless — so it buys a cosmetic reprieve and leaves - the same cliff in place. -- **`OpenStreetMap.Mapnik`.** Rejected: the OSMF tile usage policy discourages exactly this - kind of hosted-app embedding, and the style is far busier than a canvas behind circle markers. - (Noted: a request from this container returned `HTTP 200` with an `x-blocked: Access denied` - header — OSMF blocks datacentre IPs.) -- **`USGS.USTopo` / `USGS.USImagery`.** Keyless and live, and thematically apt for a US network. - Rejected as the *canvas* replacement because `USGSTopo` is a full-colour topographic sheet - with highway shields and dense labels — unusable behind the suite's circle markers. Worth - keeping in mind as a future "Topo" option. -- **Drop the dark basemap.** Rejected: the ground-beetle app deliberately re-themes its map - with the rest of the UI, and losing that is a real regression. +## 3. The first proposed fix — and why the evidence overturned it + +The initial plan was: swap every `CartoDB.Positron` for `Esri.WorldGrayCanvas`, and synthesise a dark +canvas with a CSS filter. **A measured audit refuted that plan on four independent grounds.** Both of the +decisive findings were then re-verified by hand, and both hold. The record is kept here rather than +quietly rewritten, because the *reason* the first answer was wrong is the useful part. + +### 3.1 `Esri.WorldGrayCanvas` goes BLANK at plot scale — measured + +`Esri.WorldGrayCanvas` is the variant `Canvas/World_Light_Gray_Base`. Esri's legacy raster canvas has had +**no content update since 2021**, and at NEON's rural sites it simply runs out of data: + +| Tile (SCBI, 38.893 N 78.140 W) | Distinct RGB values in the 256×256 tile | +|---|---| +| `World_Light_Gray_Base` z13 | **224** — hairline roads, no labels, near-white | +| `World_Light_Gray_Base` z16 | **1** — a single flat `RGB(239,239,239)`. Literally blank | +| `World_Topo_Map` z13 | **3,912** — roads, contours, named streets | +| `World_Topo_Map` z16 | **717** — full detail | + +The audit ran the same measurement across the coordinates of **all 46 NEON terrestrial sites**: at z16, +**17 of 46 (37%) return a single-colour blank tile** — including SCBI, TREE, UNDE, ORNL, CPER, CLBJ, SRER, +WOOD, DCFS, UKFS, OAES, NIWO, KONZ, BONA, DEJU, HEAL, TOOL. At z15, 9 of 46. Above z16 every request +returns an identical 2,521-byte "Map data not yet available" placeholder. Every other candidate tested — +`Esri.WorldTopoMap`, `Esri.WorldImagery`, `USGS.USTopo`, `USGS.USImageryTopo`, `OpenStreetMap.Mapnik`, and +`CartoDB.Positron` itself — was **0/46 blank**. + +**Consequence:** on the plot-scale maps, whose entire purpose is 20 m vegetation plots and 10 m trap grids, +the "fix" would trade a defaced-but-legible basemap for markers floating on a featureless grey field. In +Vegetation Structure that blank field lands under the app's "held is not zero" empty-state message — a +science-communication regression, not a cosmetic one. + +**The proposed `maxNativeZoom = 16` mitigation does not mitigate.** SCBI z15 and z16 are byte-identical; +the option upscales an already-blank tile and *hides* the defect from a visual check. It is a zoom-control +fix only — it stops Leaflet hard-stopping the zoom slider at 16, nothing more. + +### 3.2 There IS a keyless dark canvas — the CSS-invert plan is unnecessary + +The earlier claim in this document that "there is no keyless dark canvas" was **wrong**, and the error was +in the question, not the search: the check asked whether *leaflet-providers* exposes a dark canvas, never +whether *the provider* serves one. It does. + +``` +https://server.arcgisonline.com/ArcGIS/rest/services/Canvas/World_Dark_Gray_Base/MapServer/tile/{z}/{y}/{x} +https://server.arcgisonline.com/ArcGIS/rest/services/Canvas/World_Dark_Gray_Reference/MapServer/tile/{z}/{y}/{x} +``` + +Verified live: z4 returns a 5,456-byte JPEG, **283 distinct colours, dominant `RGB(63,63,65)`** — a real, +dark, unwatermarked canvas carrying country and ocean labels. It is keyless and needs no referer. It is +absent from `leaflet.providers`, so it needs a raw `addTiles()` rather than `addProviderTiles()`. + +**This supersedes the CSS-invert decision.** A real dark basemap beats a synthesised one, costs the same +raw `addTiles()` call, and avoids filtering artefacts entirely. The CSS-invert route is recorded as +**REJECTED — superseded**, not as an open task. + +### 3.3 The swap makes marker contrast WORSE, not better + +Measured grounds: Positron land `#fafaf8`, Esri Base land `#efefef`. Every marker palette in the suite is +*darker* than both, so a darker ground **reduces** contrast. Computed ratios, Positron → Esri: + +| Marker | Positron | Esri Base | +|---|---|---| +| birds tundra `#7fc7ec` | 1.78 | 1.62 | +| birds NA `#9aa6b2` | 2.37 | 2.16 | +| phenology NA `#c4c0b2` | 1.74 | 1.58 | +| veg `grey_na` `#cfd6dd` | 1.40 | 1.28 | +| plant-diversity comp-low `#E7E0CC` | 1.26 | 1.15 | + +Uniformly ~8–10% worse. Three of the per-app audits asserted contrast would *improve*; they had the sign +backwards. Worse, the white 1–1.5 px marker ring every picker uses scores **1.10:1** against `#efefef` — +with the base's linework gone, markers float with neither ground contrast nor surrounding structure. + +### 3.4 The swap is an attribution REGRESSION + +`leaflet.providers` 3.0.0 hardcodes the Esri canvas attribution as `Tiles © Esri — Esri, DeLorme, NAVTEQ`. +The service's own live metadata disagrees: + +```sh +curl -s "https://server.arcgisonline.com/ArcGIS/rest/services/Canvas/World_Light_Gray_Base/MapServer?f=pjson" +# copyrightText: "Esri, HERE, Garmin, © OpenStreetMap contributors, and the GIS user community" +``` + +So the bundled string credits two entities that are no longer licensors (DeLorme was absorbed by Garmin, +NAVTEQ by HERE) and **omits the required OpenStreetMap credit** — separately an ODbL produced-work +attribution failure. The `CartoDB` entry in the same file emits `© OpenStreetMap contributors © CARTO`, +which is *correct today*. The naive swap therefore trades a compliant credit line for a non-compliant one. +In a repo whose house rule is "the caveat goes ON the number", shipping an affirmative misstatement of +provenance to fix a cosmetic watermark is the wrong trade. + +The same stale-string defect already affects the Esri options this plan leaves alone (`World_Topo_Map`'s +live text also names HERE, Garmin and OpenStreetMap; `World_Imagery`'s names Vantor and Earthstar). + +**Separately non-compliant today:** `neon-small-mammal-tracker-app/server.R:2491` and `:2514` set +`leafletOptions(attributionControl = FALSE)`, suppressing tile attribution entirely. That violates CARTO's +terms now and would violate Esri's after the swap. + +### 3.5 Dark mode cannot be deferred + +`neon-ground-beetle-tracker/server.R:1633` reads `tiles <- if (is_dark()) "CartoDB.DarkMatter" else +"CartoDB.Positron"` **inside** `renderLeaflet`, so flipping the theme toggle actively re-renders the map +with DarkMatter. Patching only the light branch leaves every dark-mode visitor a full-bleed 540 px +watermark. This must land in the same wave, not "separately". + +--- + +## 4. The corrected plan — split by map ROLE + +The single biggest correction: **there is no one right basemap here, because these apps have two kinds of +map with opposite requirements.** + +### 4a. National site-picker maps (z2–5) → `Esri.WorldGrayCanvas`, keyless + +At picker zoom the grey canvas is fine — 0/46 blank, and it carries country and ocean labels (verified by +eye at z4). It is a "pick a dot" navigation control, not a detail map. **Caveat to carry:** it shows +country outlines and a state-boundary hairline but *not* state or city names, where Positron did. If that +reads too bare, pair it with the keyless label layer +`Canvas/World_Light_Gray_Reference` (verified live, transparent PNG, 5,346 B at z4) via a raw `addTiles()` +— at the cost of doubling this suite's exposure to the sunsetting Esri service. + +Apply to: small-mammal `server.R:1189`; plant-diversity `R/map_picker.R:58`; veg-structure +`R/map_picker.R:88`; breeding-birds `server.R:711` and `:723`; mosquito `server.R:578` and `:588`; +phenology `server.R:145` and `:177`; little-inverts `server.R:940` and `:963`; water-chemistry `app.R:2110`; +ground-beetle `R/map_picker.R:57`. + +### 4b. Per-site / plot-scale maps → NOT the grey canvas + +Do **not** put `Esri.WorldGrayCanvas` on any per-site map or in any Basemap dropdown. Either drop the +"Light" choice or point it at a provider with content at z13–16 (`Esri.WorldTopoMap`, `USGS.USTopo` and +`OpenStreetMap.Mapnik` all measured 0/46 blank). + +Two apps currently **default** their plot map to Light and must change: `neon-vegetation-structure-explorer/ui.R:259` +(`selected = "CartoDB.Positron"`) and `neon-plant-phenology-explorer/ui.R:215` (Light is first, therefore +selected). Small Mammal, Birds and Plant Diversity already default to Satellite/Terrain. + +### 4c. Dark → `Canvas/World_Dark_Gray_Base` via raw `addTiles()` + +Not a CSS filter. See §3.2. + +### 4d. Attribution must be set explicitly + +Because the bundled provider strings are stale and drop the OSM credit (§3.4), any call that adopts an +Esri layer should pass an explicit `attribution` matching the service's live `copyrightText`, and +`attributionControl = FALSE` must come out of small-mammal `server.R:2491`/`:2514`. + +### 4e. A string sweep is not sufficient + +`neon-my-little-inverts/server.R:940` and `:963` use the **object form** +`leaflet::addProviderTiles(leaflet::providers$CartoDB.Positron)` — no quoted provider name. A +`grep '"CartoDB'` leaves that app fully watermarked. Use: + +```sh +grep -rn --include='*.R' -E 'CartoDB[."$]|cartocdn' . +``` + +The complete CARTO surface is **22 lines across 9 repos**. `R/map_picker.R` is a *forked shared module* +present in three repos (ground-beetle:57, plant-diversity:58, veg-structure:88) and must move in lockstep. --- -## 4. Applying it — what a patch must respect - -- **Docs-only changes in Driver-Cascade are outside the deploy surface.** `scripts/manifest_files.R` - allowlists only `global.R`, `ui.R`, `server.R`, `R/cascade_helpers.R`, `R/site_metadata.R`, - `www/{cascade.css,cascade.js,styles.css}` and the four `data/` artifacts. `docs/` and - `.claude/` are not in it, so this document does not touch the manifest gate. -- **In the companion repos the patch DOES touch `ui.R` / `server.R` / `R/*.R`**, which are on - their deploy surfaces. A source-only edit changes no package, so `manifest.json` should not - need regeneration — **but each repo's CI manifest gate must be checked**, because the sibling - gates are byte-exact where Driver-Cascade's is semantic. -- **A push to the watched branch is the deploy.** Each companion repo is watched by Posit - Connect Cloud; merging is what ships. Nine merges = nine deploys. -- Match each repo's local idiom: pipe style (`%>%` vs `|>`), namespacing (`leaflet::` vs bare), - and its own `CLAUDE.md` / `AGENTS.md` house rules. +## 5. Two questions that should have been asked first + +Both could dissolve most of the work above. Neither has been put to the owner: + +1. **Is there a University of Arizona institutional ArcGIS Online licence?** (`tsgilbert@arizona.edu` — + near-universal at US R1s.) That yields a keyed, supported, non-sunsetting Esri basemap at no cost and + removes the "keyless or bust" constraint this entire plan is built on. +2. **Why a tile basemap for the picker at all?** It needs a national outline and ~46 dots. `prototypes/site-explorer` + already proves this suite can render real geography with no tiles and no key; `docs/_phase23_plan.md` + P3-3 specifies a plotly `scattergeo` for exactly this map; and the Water Chemistry repo's + `assets/wc_sitemap.png` shows **the suite already shipped a tileless scattergeo picker once.** It is the + only option with no vendor, no key, no watermark and no sunset. + +A third option, deliberately re-opened: **the free CARTO key** (no account, emailed on request, ~5M +tiles/month fair use) restores the exact current design — Positron *and* DarkMatter, retina, maxZoom 20, +zero blank tiles, no palette or CSS re-tuning. Its real costs are that the key is visible client-side +(inherent to browser tile fetching, and it is free and rotatable) and that CARTO requires visible +attribution — which small-mammal suppresses today regardless. It was filed as a last resort; on the +measured evidence it deserved to be a leading option. --- -## 5. Standing risk to record +## 6. Standing risk + +Esri's published retirement schedule: **October 2026** legacy globe services; **March 2028** phase 1 of +legacy basemap retirement (World Imagery/Clarity); **December 2029** phase 2 (World Topographic Map). The +Light Gray Canvas raster basemap is *already labelled deprecated*, and Esri states the legacy raster +basemaps have been in mature support 4+ years with **no content updates since 2021** — which is the direct +cause of the blanking in §3.1. + +So: no imminent cliff, but this adopts a **frozen, decaying** service. And several apps move from 2-of-3 to +3-of-3 Esri plus the always-visible picker, so every map surface in those apps then retires on one date. +That concentration is the genuine durability defect. **Disposition: `HOLD` on any blanket Esri swap; +`ADOPT` scoped to the national pickers only**, with a scheduled re-check. + +Durable options, none part of this fix: self-host a minimal canvas for ~46 points; draw states/coastlines +from a committed GeoJSON; or go tileless per §5.2. + +--- + +## 7. Gate asymmetry across the nine repos — checked + +Not uniform, and it matters for rollout order: + +- **`neon-my-little-inverts` and `neon-waterchemistry-analyte-viewer-app` have NO `ci.yml`** — only + `post-deploy.yml` and `refresh-data.yml`. The two repos with the two non-standard idioms (object-form + provider access; native `|>` pipe and a `leaflet.providers` 2.0.0 pin) are the two with **zero pre-merge + parse or manifest gate**. Inverts still ships a 55-file `manifest.json` containing `ui.R` and `server.R` + that nothing regenerates or verifies on a PR. +- **`neon-vegetation-structure-explorer` has no `post-deploy.yml`** — no smoke verification after deploy. +- small-mammal and veg-structure carry a separate `regenerate-manifest.yml` path. +- **No repo asserts any provider string anywhere.** Nothing today would catch a revert, or a new map added + with the old default. That is why this shipped unnoticed. + +Verified safe on both pins: `Esri.WorldGrayCanvas` is present in `leaflet.providers` 3.0.0 +(`R/providers_data.R:94`, variant `Canvas/World_Light_Gray_Base`, `maxZoom = 16L`) **and** 2.0.0 +(`:38`, `:244`), so the water-chemistry app's older pin is safe. Note `leaflet` 2.2.3 +(`R/plugin-providers.R:44-50`) runs `check = TRUE` and **hard-errors** on an unknown provider name — a typo +is a loud runtime failure at first render, not a silent blank. + +--- -Both the old and the new basemap are third-party freebies with no contract behind them. CARTO -withdrew its free raster tier with no warning that reached this project; Esri has already -announced the same intent for the legacy endpoints this fix moves onto. **Treat the basemap as -a dependency with an owner and a review date, not as scenery.** Options that would durably -de-risk it — and are explicitly NOT part of this fix: +## 8. What must NOT be repeated -- Self-host a minimal vector or raster canvas for the ~81 NEON site locations (the suite only - ever needs a national overview plus site-level zoom, not a world basemap). -- Draw a simple state/coastline outline from a committed GeoJSON — no tile server at all. The - `prototypes/site-explorer` map already proves this suite can render real geography with no - tiles and no key. -- Add a suite CI check that fetches one tile from each configured provider and fails on a - non-image, an unexpected byte-size shift, or a known watermark signature. +- Do not answer "what is the canvas max zoom?" from `MapServer?f=json`. It declares **LODs to level 23**, + while `leaflet.providers` hardcodes `maxZoom = 16L` and real tiles above ~16 are the 2,521-byte + placeholder. The prescribed method returns the wrong answer. +- Do not treat `HTTP 200` as evidence a tile is good. That assumption is exactly what let the CARTO + watermark ship unnoticed, and it equally hides a blank Esri tile. **Decode the image and count distinct + colours**; a single-colour tile is blank, and the "Map data not yet available" placeholder has md5 + `f27d9de7f80c13501f470595e327aa6d`. --- -## 6. Resume checklist — pick this up cold - -Work completed on `claude/neon-maps-api-key-eocg8r` in Driver-Cascade: - -- [x] Root cause identified and verified by direct tile fetch + image inspection. -- [x] Non-causes eliminated with evidence (package versions, referer, CORS, CSP, app strings). -- [x] Replacement `Esri.WorldGrayCanvas` fetched and visually confirmed clean. -- [x] All nine companion apps enumerated with per-file call sites. -- [x] Owner decision recorded: Esri grey canvas + CSS-inverted tile pane for dark. -- [x] This advisory + `LESSONS.md` + `BUILD-TEST-HANDOFF.md` entries committed. - -Open, in order: - -- [ ] **Verify the CSS-invert approach concretely** before it ships: confirm the Leaflet 1.x - pane structure (`.leaflet-tile-pane` vs `.leaflet-overlay-pane` / `.leaflet-marker-pane`), - settle the exact filter (`invert(1) hue-rotate(180deg)` plus brightness/contrast trim), - confirm the attribution control is not inverted, and decide how it is toggled from each - app's existing theme state (a class on the map container vs `htmlwidgets::onRender`). -- [ ] **Confirm `Canvas/World_Light_Gray_Base` max zoom** against - `https://server.arcgisonline.com/ArcGIS/rest/services/Canvas/World_Light_Gray_Base/MapServer?f=json` - and check it is deep enough for the plot-level maps (Positron went to zoom 20). -- [ ] **Check marker contrast** in each app against the paler grey canvas — several palettes - were tuned against Positron's near-white. -- [ ] **Apply the patch to the nine companion repos**, one branch + draft PR each, respecting - each repo's default branch (`master` vs `main`) and CI manifest gate. -- [ ] **Re-check the deployed apps** after each merge; Connect Cloud redeploys on push. -- [ ] Consider the suite CI tile-canary from §5. +## 9. Resume checklist — pick this up cold + +Done: + +- [x] Root cause verified by direct tile fetch + image inspection; non-causes eliminated with evidence. +- [x] All nine apps audited; 22 CARTO call sites inventoried with verified file:line. +- [x] First plan (blanket `Esri.WorldGrayCanvas` + CSS-invert dark) **refuted on four grounds**, measured. +- [x] `Canvas/World_Dark_Gray_Base` confirmed keyless and real — CSS-invert superseded. +- [x] Blank-at-plot-scale quantified (37% of NEON sites at z16); attribution regression identified. +- [x] Gate asymmetry across the nine repos mapped. + +Open, in order — **first two are owner decisions and block the rest**: + +- [ ] **Answer §5.1** — is there a UA institutional ArcGIS licence? +- [ ] **Answer §5.2 / the CARTO key** — tileless picker, free CARTO key, or keyless Esri split by role? +- [ ] Apply §4 per-role patch to the nine repos, one branch + draft PR each, respecting each repo's default + branch (Driver-Cascade `master`; Small Mammal and Vegetation `main`) and its gate config from §7. + Land ground-beetle's dark branch in the same wave (§3.5). +- [ ] Fix attribution (§3.4) — explicit strings; remove `attributionControl = FALSE`. +- [ ] Re-tune marker strokes and the palest fills against the new ground (§3.3); note `www/styles.css` is + manifest-tracked in several repos, so bundle it into the same commit. +- [ ] Add the regression guard: a CI grep asserting zero matches for `CartoDB[."$]|cartocdn`, plus a + scheduled tile canary that decodes tiles and fails on a single-colour or placeholder result. +- [ ] Record the basemap contract in `docs/neonize-playbook.md` §2g — it currently names **no provider at + all**, so nothing stops the next app reintroducing CARTO. +- [ ] Visually verify each deployed app after merge. `HTTP 200` is not verification (§8). + +### Per-app call-site inventory + +Line numbers verified against each repo's default-branch HEAD on 2026-08-28. Re-verify before patching. + +**Small Mammal** — `neon-small-mammal-tracker-app` +- `server.R:1189` — CartoDB.Positron · ALWAYS — CONFIRMED, and the confirmation is stronger than the original audit stated. This +- `server.R:2492` — dynamic · The Plot map tab (output$map, server.R:2511; leafletOutput("map") at ui.R:631, inside the +- `server.R:2515` — CartoDB.DarkMatter · Conditional empty state only — CONFIRMED not on the default path, but the ORIGINAL AUDIT S +- `ui.R:619` — Basemap choices vector + +**Plant Diversity** — `neon-plant-diversity` +- `R/map_picker.R:58` — CartoDB.Positron · ALWAYS — the national site-picker map inside div(id="splash") (ui.R:106), placed by mapPic +- `ui.R:458` — CartoDB.Positron · ONLY IF THE USER PICKS IT — third option in the "Basemap" selectInput on the Map tab (ui.R +- `server.R:1495` — dynamic · The per-site plot map on the Map tab, shown after a site loads. The tile layer is always v +- `ui.R:456` — Basemap choices vector + +**Vegetation Structure** — `neon-vegetation-structure-explorer` +- `R/map_picker.R:88` — CartoDB.Positron · ALWAYS ON LOAD, zero interaction — CONFIRMED. This is the national site-picker map on the +- `server.R:1398` — CartoDB.Positron · CORRECTED — NOT on the cold-load path. This is the default basemap of the main Map tab, bu +- `server.R:1375` — CartoDB.Positron · Conditional, and — corrected — also gated behind a site load plus a Map-tab click. This is +- `ui.R:259` — Basemap choices vector + +**Breeding Birds** — `neon-breeding-birds` +- `server.R:723` — CartoDB.Positron · ALWAYS ON LOAD, VERIFIED. output$nationalPicker (server.R:709) renders into leafletOutput( +- `server.R:711` — CartoDB.Positron · CORRECTED — NOT "always on load". The original audit's visibility string opens with "alway +- `server.R:645` — dynamic · Opt-in and post-load, VERIFIED TWICE OVER. (a) output$map (server.R:629) begins `obs <- rv +- `ui.R:298` — Basemap choices vector + +**Plant Phenology** — `neon-plant-phenology-explorer` +- `server.R:177` — CartoDB.Positron · ALWAYS ON LOAD — highest-impact site in the app. VERIFIED: this is output$nationalMap (ren +- `server.R:145` — CartoDB.Positron · Empty-state fallback of that same always-on landing map. VERIFIED: server.R:144 filters si +- `server.R:779` — CartoDB.Positron · DEFAULT BASEMAP of the main explore map (output$map, renderLeaflet opens at server.R:743; +- `ui.R:215` — Basemap choices vector + +**Mosquito Pulse** — `neon-mosquito-pulse` +- `server.R:588` — CartoDB.Positron · ALWAYS on load — VERIFIED. This is the national site-picker map rendered into leafletOutpu +- `server.R:578` — CartoDB.Positron · Fallback branch of the same always-on picker map — VERIFIED at server.R:574-582. Reached o +- `server.R:537` — Dynamic · Main per-site trap-grid map, rendered at ui.R:230 inside the Map nav_panel (ui.R:222). Rea +- `ui.R:229` — Basemap choices vector + +**Ground Beetle** — `neon-ground-beetle-tracker` +- `R/map_picker.R:57` — CartoDB.Positron · always on load — this is the landing/site-picker map, the app's front door. VERIFIED: ui.R +- `server.R:1630` — CartoDB.Positron · the DEFAULT basemap of the Biogeography tab map (output$map opens at server.R:1628; ui.R:4 + +**My Little Inverts** — `neon-my-little-inverts` +- `server.R:963` — CartoDB.Positron · VERIFIED, with one correction to the original audit. This is the real landing/site-picker +- `server.R:940` — CartoDB.Positron · VERIFIED. Degraded-state fallback inside the SAME output$nationalPicker renderLeaflet (ser + +**Water Chemistry** — `neon-waterchemistry-analyte-viewer-app` +- `app.R:2110` — CartoDB.Positron · ALWAYS — CONFIRMED by re-reading the UI tree, not just the section comment. This is the on +- `app.R:2014` — CartoDB.Positron · NEVER user-visible — source comment only. Byte-verified verbatim at line 2014 (2-space ind ### Local working state -The nine companion repos were cloned shallow + blobless + sparse (code present, `data/` and -`assets/` deliberately not checked out) under `/home/user/tgilbert14/`. That checkout is -**ephemeral** — this container is reclaimed after the session. To rebuild it: +The nine companion repos were cloned shallow + blobless + sparse (code present, `data/` and `assets/` not +checked out) under `/home/user/tgilbert14/`. That checkout is **ephemeral**. To rebuild it: ```sh for r in neon-small-mammal-tracker-app neon-plant-diversity \ @@ -215,12 +428,10 @@ for r in neon-small-mammal-tracker-app neon-plant-diversity \ done ``` -### Environment note for the next session +### Environment note -**A headless-browser check of the deployed apps was not possible here.** Chromium is installed -and Playwright is configured, but every navigation fails with `net::ERR_CONNECTION_RESET` — -including `https://example.com` — so the browser has no egress in this container even routed -through the agent proxy. `curl` works fine. A session that *can* drive a browser should -confirm the watermark on a live app and confirm it is gone after the fix; until then the -evidence here is tile-level, which is decisive for the cause but does not substitute for -seeing a fixed app. +**A headless-browser check of the deployed apps was not possible here.** Chromium is installed and +Playwright configured, but every navigation fails `net::ERR_CONNECTION_RESET` — including +`https://example.com` — so the browser has no egress in this container, proxied or not. `curl` works. All +evidence in this document is tile-level and source-level: decisive for the cause and for the measured +blanking, but **nobody has yet seen a fixed app**. From c25ca8a9431f27becb359c107495d15ca9be6967 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 28 Aug 2026 21:53:01 +0000 Subject: [PATCH 03/10] docs: adopt the free CARTO key as the basemap fix MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Owner decision: authenticate the existing basemaps rather than replace them. Keeping CartoDB.Positron and CartoDB.DarkMatter is the only option that changes nothing visually — same tiles, same maxZoom 20, same retina, zero blank tiles, no palette or CSS re-tuning, no contrast or attribution regression, and ground-beetle's dark theme keeps working. Every defect in the refutation exists because the alternatives moved the basemap. Records the key mechanics: it is a ?key= query parameter, and addProviderTiles() cannot carry it because the pinned CartoDB template has an {r} slot but no {apikey} placeholder, so it needs a raw addTiles() with attribution by hand. Adds an add_suite_basemap() helper, keyed with a keyless Esri-canvas fallback so a missing or revoked key degrades to a clean canvas instead of a defaced one. States plainly that the key is not a secret: it rides in the tile URL and every request is client-side, and CARTO's terms ban server-side proxying, so it cannot be hidden. Sys.getenv keeps it out of nine public git histories and makes rotation a Connect Cloud setting — nothing more. Adds the rollout section: every companion repo has a byte-exact manifest gate and all app sources are on the deploy surface, so the manifest must be regenerated in the same commit, never by hand. Two repos have no CI at all and ship a manifest nothing verifies. Small Mammal forbids automation pushing to main, so this opens PRs rather than merging them. Retires the Esri.WorldGrayCanvas swap and the CSS-invert dark canvas, and keeps the per-role split only as the fallback path. Docs only: docs/ and .claude/ are outside DEPLOY_APP_FILES. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01738HeGPEU9NEoczL8RXnu1 --- docs/BUILD-TEST-HANDOFF.md | 51 +++++ docs/SUITE-BASEMAP-INCIDENT-2026-08.md | 245 +++++++++++++++++-------- 2 files changed, 215 insertions(+), 81 deletions(-) diff --git a/docs/BUILD-TEST-HANDOFF.md b/docs/BUILD-TEST-HANDOFF.md index 3481d1b..e72ffc1 100644 --- a/docs/BUILD-TEST-HANDOFF.md +++ b/docs/BUILD-TEST-HANDOFF.md @@ -4736,3 +4736,54 @@ Rules: deliberately re-opened as a third option: it restores Positron AND DarkMatter with zero re-tuning and no blank tiles. Do not start the nine-repo rollout before these are answered. + +### 2026-08-28 (decision) - [Claude] basemap: take the free CARTO key + +- **Owner decision:** take the free CARTO basemap key and keep `CartoDB.Positron` + and `CartoDB.DarkMatter` exactly as they are. Disposition `ADOPT`. This is the + only option that changes nothing visually — same tiles, same `maxZoom = 20`, + same retina, zero blank tiles, no marker-palette or CSS re-tuning, no contrast + regression, no attribution regression, and ground-beetle's dark theme keeps + working untouched. Every defect recorded in the previous entry exists *because* + the alternatives moved the basemap; this one does not. +- **Retired by this decision:** the blanket `Esri.WorldGrayCanvas` swap (REJECTED + — blank at plot scale) and the CSS-invert dark canvas (REJECTED, superseded). + The per-map-role split survives only as the keyless fallback inside the new + helper, so a missing or revoked key degrades to a clean canvas rather than a + defaced one. The two blocking questions from the previous entry (UA institutional + ArcGIS licence; tileless picker) are no longer needed for the fix and move to the + register as durability options. +- **BLOCKED on one owner action no agent can perform:** request the key at + (email + domain + one-line description; + emailed straight back, no account, no approval queue; fair use 5M tile + requests/calendar month). Everything downstream is written and waiting. +- **Key mechanics settled and recorded in §4 of the incident doc:** it is a `?key=` + QUERY PARAMETER; `addProviderTiles()` cannot carry it (the pinned CartoDB template + has an `{r}` slot but no `{apikey}` placeholder), so it needs a raw `addTiles()` + with attribution supplied by hand. An `add_suite_basemap()` helper is written, + keyed with a keyless Esri-canvas fallback. +- **The key is NOT a secret, and the plan says so explicitly.** It rides in the tile + URL and every request is issued client-side, so it lands in page source regardless; + CARTO's terms §9.c bans server-side proxying, so it cannot be hidden. + `Sys.getenv("CARTO_BASEMAP_KEY")` buys exactly two things — it keeps the key out of + nine public git histories and makes rotation a Connect Cloud setting rather than + nine releases. Connect Cloud does support this (content settings → Variables, + encrypted at rest, read at runtime; NOT part of `manifest.json`). Cost: nine apps + × one manual variable, and a republish is likely needed — untested. +- **Cannot be verified without the real key:** an INVALID key returns the + byte-identical watermarked tile as no key at all (same ETag), so there is no + negative test. Smoke-test the URL form once with the real key before any PR opens. +- **Rollout gate discovered and recorded (§5):** every companion repo has a + byte-exact `manifest.json` gate and `ui.R`/`server.R`/`R/*.R`/`global.R` are all on + the deploy surface, so a source edit REQUIRES the manifest regenerated in the same + commit. Never hand-edit it — Small Mammal and Vegetation carry a blessed + `regenerate-manifest.yml` whose own header says it exists to end the loop that + "made the ChatGPT/Codex cover rework fail merges over and over". The two repos with + NO `ci.yml` (little-inverts, water-chemistry) still ship a manifest and have nothing + to catch a stale one — do those last. Small Mammal `DEPLOY.md:9-13` forbids + automation pushing to `main`: open PRs, do not merge them. +- **Nothing at risk:** docs-only again; `docs/` and `.claude/` stay outside + `DEPLOY_APP_FILES`; no companion repo touched. +- **Next action:** owner requests the key; then smoke-test the URL form, set the + Connect variable, and canary ONE repo end to end — patch, manifest, merge, deploy, + and actually look at the live map — before touching the other eight. diff --git a/docs/SUITE-BASEMAP-INCIDENT-2026-08.md b/docs/SUITE-BASEMAP-INCIDENT-2026-08.md index 7276258..5bd9c60 100644 --- a/docs/SUITE-BASEMAP-INCIDENT-2026-08.md +++ b/docs/SUITE-BASEMAP-INCIDENT-2026-08.md @@ -1,7 +1,8 @@ # Suite incident: CARTO stamps "API KEY REQUIRED" on every basemap tile -**Status: DIAGNOSED — root cause verified and measured. The FIRST proposed fix was refuted by evidence (§3) -and replaced by a per-role plan (§4). Two owner decisions (§5) now block rollout. Nothing applied to the +**Status: DIAGNOSED and DECIDED. The first proposed fix was refuted by evidence (§3). The owner's decision +is to take the free CARTO key and keep the existing basemaps unchanged (§4). BLOCKED on one owner action — +requesting the key at — which no agent can do. Nothing applied to the nine apps yet.** **Opened 2026-08-28 · Driver-Cascade branch `claude/neon-maps-api-key-eocg8r` · tagged `[Claude]`.** @@ -202,79 +203,149 @@ watermark. This must land in the same wave, not "separately". --- -## 4. The corrected plan — split by map ROLE - -The single biggest correction: **there is no one right basemap here, because these apps have two kinds of -map with opposite requirements.** - -### 4a. National site-picker maps (z2–5) → `Esri.WorldGrayCanvas`, keyless - -At picker zoom the grey canvas is fine — 0/46 blank, and it carries country and ocean labels (verified by -eye at z4). It is a "pick a dot" navigation control, not a detail map. **Caveat to carry:** it shows -country outlines and a state-boundary hairline but *not* state or city names, where Positron did. If that -reads too bare, pair it with the keyless label layer -`Canvas/World_Light_Gray_Reference` (verified live, transparent PNG, 5,346 B at z4) via a raw `addTiles()` -— at the cost of doubling this suite's exposure to the sunsetting Esri service. - -Apply to: small-mammal `server.R:1189`; plant-diversity `R/map_picker.R:58`; veg-structure -`R/map_picker.R:88`; breeding-birds `server.R:711` and `:723`; mosquito `server.R:578` and `:588`; -phenology `server.R:145` and `:177`; little-inverts `server.R:940` and `:963`; water-chemistry `app.R:2110`; -ground-beetle `R/map_picker.R:57`. - -### 4b. Per-site / plot-scale maps → NOT the grey canvas - -Do **not** put `Esri.WorldGrayCanvas` on any per-site map or in any Basemap dropdown. Either drop the -"Light" choice or point it at a provider with content at z13–16 (`Esri.WorldTopoMap`, `USGS.USTopo` and -`OpenStreetMap.Mapnik` all measured 0/46 blank). - -Two apps currently **default** their plot map to Light and must change: `neon-vegetation-structure-explorer/ui.R:259` -(`selected = "CartoDB.Positron"`) and `neon-plant-phenology-explorer/ui.R:215` (Light is first, therefore -selected). Small Mammal, Birds and Plant Diversity already default to Satellite/Terrain. - -### 4c. Dark → `Canvas/World_Dark_Gray_Base` via raw `addTiles()` - -Not a CSS filter. See §3.2. +## 4. DECIDED: take the free CARTO key (owner, 2026-08-28) + +**Disposition: `ADOPT`.** Keep `CartoDB.Positron` and `CartoDB.DarkMatter` exactly as they are and +authenticate them. This is the only option that changes nothing visually: same tiles, same `maxZoom = 20`, +same retina, zero blank tiles, no marker-palette or CSS re-tuning, no contrast regression, no attribution +regression, and the ground-beetle dark theme keeps working unchanged. Every defect in §3 exists *because* +the other options moved the basemap; this one does not move it. + +### 4.1 The one step only the owner can do + +**Request the key at .** Verbatim from that page: *"Tell us your email, +the domain you will use the basemaps on, and roughly what you are building. We email the key straight back — +there is no approval queue and you do not need a CARTO account."* A JS-less fallback is emailing CARTO +support. Fair use is **5 million tile requests per calendar month**, counted across raster and vector — for +nine low-traffic academic apps that is orders of magnitude of headroom. + +Domain to give: the Connect Cloud share domain (`*.share.connect.posit.cloud`). **Not confirmed** whether +that domain is actually enforced as a referer lock or merely recorded — CARTO documents no allowlist +feature and there is no console, since there is no account. Treat it as informational. + +### 4.2 The key is NOT a secret — this is load-bearing + +CARTO's key rides in the tile URL, and every tile request is issued **client-side by the browser**. The key +therefore lands in page source and in the network tab on first map paint, whatever you do. CARTO's terms +**§9.c prohibits server-side proxying or caching of tiles**, so routing them through the Shiny server to +hide the key is not allowed either. + +So: **treat it as a public, rate-limited identifier, not a credential.** Storing it in +`Sys.getenv("CARTO_BASEMAP_KEY")` buys exactly two things — it keeps the key out of nine public git +histories, and it makes rotation a Connect Cloud setting change rather than nine releases. That is worth +doing. It does not make the key private, and no part of this plan should be written as though it does. + +Posit Connect Cloud **does** support this: content settings have a *Variables* section ("Add, update, or +remove your secret environment variables"), values are encrypted at rest, and `Sys.getenv()` reads them at +runtime. They are **not** part of `manifest.json` — the manifest describes files and packages only. Cost to +be honest about: **nine apps × one manual variable each**, and Posit's docs do not say whether an edit takes +effect immediately or needs a republish — **assume a republish until tested**. + +### 4.3 The code — one helper, keyed with a keyless fallback + +`addProviderTiles()` **cannot** carry the key: the pinned `leaflet.providers` CartoDB template has an `{r}` +retina slot but no `{apikey}` placeholder. It must be a raw `addTiles()` with attribution supplied by hand. + +Add to each app's `global.R` (or the top of `app.R` for water-chemistry): + +```r +# --- basemap ----------------------------------------------------------------- +# CARTO began watermarking UNAUTHENTICATED basemaps.cartocdn.com raster tiles on +# 2026-08-26 (see docs/SUITE-BASEMAP-INCIDENT-2026-08.md in NEON-Driver-Cascade). +# This key is a PUBLIC, rate-limited identifier, not a credential: it rides in the +# tile URL and is visible in the browser. Sys.getenv keeps it out of git and makes +# rotation a Connect Cloud setting rather than a release. +CARTO_KEY <- Sys.getenv("CARTO_BASEMAP_KEY", "") + +# CARTO's terms require BOTH credits visible; do not suppress the attribution control. +CARTO_ATTR <- paste( + '© OpenStreetMap contributors', + '© CARTO') +ESRI_CANVAS_ATTR <- 'Tiles © Esri — Esri, HERE, Garmin, © OpenStreetMap contributors' + +# variant: "light_all" (= CartoDB.Positron) or "dark_all" (= CartoDB.DarkMatter). +# With a key: the exact tiles the suite has always used. Without one: a keyless Esri +# canvas — never a watermarked tile. The fallback is deliberately NOT used at plot +# scale; see §3.1 (Esri's canvas is blank at z16 for 37% of NEON sites). +add_suite_basemap <- function(map, variant = "light_all", ...) { + if (nzchar(CARTO_KEY)) { + leaflet::addTiles(map, + urlTemplate = sprintf( + "https://{s}.basemaps.cartocdn.com/%s/{z}/{x}/{y}{r}.png?key=%s", variant, CARTO_KEY), + attribution = CARTO_ATTR, + options = leaflet::tileOptions(subdomains = "abcd", maxZoom = 20, + detectRetina = TRUE, ...)) + } else { + leaflet::addTiles(map, + urlTemplate = sprintf( + "https://server.arcgisonline.com/ArcGIS/rest/services/Canvas/World_%s_Gray_Base/MapServer/tile/{z}/{y}/{x}", + if (identical(variant, "dark_all")) "Dark" else "Light"), + attribution = ESRI_CANVAS_ATTR, + options = leaflet::tileOptions(maxNativeZoom = 16, maxZoom = 19, ...)) + } +} +``` -### 4d. Attribution must be set explicitly +Then every call site becomes `add_suite_basemap(map, "light_all")` / `"dark_all"`, and the `input$view` +dropdowns keep their `Esri.*` options routed through `addProviderTiles()` as today, with `"CartoDB.Positron"` +/ `"CartoDB.DarkMatter"` dispatched to the helper. -Because the bundled provider strings are stale and drop the OSM credit (§3.4), any call that adopts an -Esri layer should pass an explicit `attribution` matching the service's live `copyrightText`, and -`attributionControl = FALSE` must come out of small-mammal `server.R:2491`/`:2514`. +> **Smoke-test the URL form ONCE before rolling to nine repos.** The `?key=` query-parameter shape is taken +> from CARTO's own docs. It could not be verified here: an *invalid* key returns the **byte-identical +> watermarked tile** as no key at all (same ETag), so there is no negative test — only a real key proves it. +> Fetch one tile with the real key and confirm the watermark is gone before any PR is opened. -### 4e. A string sweep is not sufficient +### 4.4 Attribution must be restored at the same time -`neon-my-little-inverts/server.R:940` and `:963` use the **object form** -`leaflet::addProviderTiles(leaflet::providers$CartoDB.Positron)` — no quoted provider name. A -`grep '"CartoDB'` leaves that app fully watermarked. Use: +`neon-small-mammal-tracker-app/server.R:2491` and `:2514` set `leafletOptions(attributionControl = FALSE)`. +CARTO's terms require visible CARTO **and** OpenStreetMap credit, so those two lines are **non-compliant +today** and must come out as part of this change — not as a follow-up. -```sh -grep -rn --include='*.R' -E 'CartoDB[."$]|cartocdn' . -``` +### 4.5 What this decision retires -The complete CARTO surface is **22 lines across 9 repos**. `R/map_picker.R` is a *forked shared module* -present in three repos (ground-beetle:57, plant-diversity:58, veg-structure:88) and must move in lockstep. +- The blanket `Esri.WorldGrayCanvas` swap — **REJECTED** (blank at plot scale, §3.1). +- The CSS-invert dark canvas — **REJECTED, superseded** (§3.2; and moot now that Positron/DarkMatter stay). +- The per-map-role split — **retained only as the keyless fallback path** in `add_suite_basemap()`, so a + missing or revoked key degrades to a clean canvas instead of a defaced one. +- The two questions in the previous revision (UA institutional ArcGIS licence; tileless picker) — **not + needed for this fix.** Both stay on the register as durability options, since CARTO is retiring raster + basemaps eventually (§6). --- -## 5. Two questions that should have been asked first - -Both could dissolve most of the work above. Neither has been put to the owner: - -1. **Is there a University of Arizona institutional ArcGIS Online licence?** (`tsgilbert@arizona.edu` — - near-universal at US R1s.) That yields a keyed, supported, non-sunsetting Esri basemap at no cost and - removes the "keyless or bust" constraint this entire plan is built on. -2. **Why a tile basemap for the picker at all?** It needs a national outline and ~46 dots. `prototypes/site-explorer` - already proves this suite can render real geography with no tiles and no key; `docs/_phase23_plan.md` - P3-3 specifies a plotly `scattergeo` for exactly this map; and the Water Chemistry repo's - `assets/wc_sitemap.png` shows **the suite already shipped a tileless scattergeo picker once.** It is the - only option with no vendor, no key, no watermark and no sunset. - -A third option, deliberately re-opened: **the free CARTO key** (no account, emailed on request, ~5M -tiles/month fair use) restores the exact current design — Positron *and* DarkMatter, retina, maxZoom 20, -zero blank tiles, no palette or CSS re-tuning. Its real costs are that the key is visible client-side -(inherent to browser tile fetching, and it is free and rotatable) and that CARTO requires visible -attribution — which small-mammal suppresses today regardless. It was filed as a last resort; on the -measured evidence it deserved to be a leading option. +## 5. Rollout — what a patch must clear in each repo + +**The manifest is the real gate, and it bites at runtime as well as in CI.** Every companion repo ships a +`manifest.json` with per-file checksums, and `ui.R` / `server.R` / `R/*.R` / `global.R` are all on the deploy +surface. So a source edit **requires the manifest to be regenerated in the same commit.** + +- **Never hand-edit `manifest.json`.** Small Mammal and Vegetation Structure carry a blessed + `.github/workflows/regenerate-manifest.yml` — `workflow_dispatch`-only, refuses to run on `main`, + regenerates twice and requires byte-identical output. Its own header says it exists to end the loop that + "made the ChatGPT/Codex cover rework fail merges over and over". **Use it.** +- Byte-exact manifest gates, per repo: small-mammal `ci.yml:156`; breeding-birds `:185`; ground-beetle + `:129`; mosquito `:111`; phenology `:165`; plant-diversity `:165-177` (regenerates twice, requires + identical sha256, then `verify_bundle.R`); veg-structure `:179-195` (regenerates twice, `verify_manifest.R`, + then a `git status --porcelain` equivalent). +- **`neon-my-little-inverts` and `neon-waterchemistry-analyte-viewer-app` have NO `ci.yml`** — both still + ship a `manifest.json`, so their patch needs a **manually regenerated** manifest with nothing to catch a + stale one. Highest-risk two repos; do them last, with the most care. +- **Driver-Cascade enforces manifest checksums at RUNTIME** (`global.R:106-112`, before any repo code is + sourced; `DEPLOY.md:74-76`). It has no map so it is unaffected — but if that integrity pattern is ever + promoted to the siblings, a stale-manifest basemap patch stops being a red check and becomes a + **production outage**. +- **No test anywhere in the suite asserts a provider string** — a grep across every `scripts/`, + `test_*`, `check_*`, `verify_*` and smoke file in all nine repos returns zero hits. CI sources the app but + never fetches a tile, so **CI cannot detect a bad provider or a defaced tile.** That is exactly why this + shipped unnoticed. +- **Branch targets differ and so do the deploy rules.** Driver-Cascade: Connect watches `master` and a push + to `master` *is* the deploy. Small Mammal: Connect watches `main` and `DEPLOY.md:9-13` states **"no + automation may push there directly — a human-reviewed merge is the explicit production decision."** + So: open a PR per repo; **do not merge them.** + +Suggested order, safest first: ground-beetle → mosquito → phenology → breeding-birds → plant-diversity → +veg-structure → small-mammal → little-inverts → water-chemistry. Canary the first one end to end (patch, +manifest, merge, deploy, look at the live map) before touching the rest. --- @@ -288,8 +359,10 @@ cause of the blanking in §3.1. So: no imminent cliff, but this adopts a **frozen, decaying** service. And several apps move from 2-of-3 to 3-of-3 Esri plus the always-visible picker, so every map surface in those apps then retires on one date. -That concentration is the genuine durability defect. **Disposition: `HOLD` on any blanket Esri swap; -`ADOPT` scoped to the national pickers only**, with a scheduled re-check. +**Disposition: `ADOPT` the keyed CARTO basemaps (§4); `HOLD` on any blanket Esri swap** — Esri is now only +the keyless fallback in `add_suite_basemap()`, which is why that fallback is capped at `maxNativeZoom = 16` +and is not used to justify plot-scale detail. CARTO states raster basemaps are being retired in favour of +vector, so the key buys time, not permanence: schedule a re-check. Durable options, none part of this fix: self-host a minimal canvas for ~46 points; draw states/coastlines from a committed GeoJSON; or go tileless per §5.2. @@ -341,20 +414,30 @@ Done: - [x] Blank-at-plot-scale quantified (37% of NEON sites at z16); attribution regression identified. - [x] Gate asymmetry across the nine repos mapped. -Open, in order — **first two are owner decisions and block the rest**: - -- [ ] **Answer §5.1** — is there a UA institutional ArcGIS licence? -- [ ] **Answer §5.2 / the CARTO key** — tileless picker, free CARTO key, or keyless Esri split by role? -- [ ] Apply §4 per-role patch to the nine repos, one branch + draft PR each, respecting each repo's default - branch (Driver-Cascade `master`; Small Mammal and Vegetation `main`) and its gate config from §7. - Land ground-beetle's dark branch in the same wave (§3.5). -- [ ] Fix attribution (§3.4) — explicit strings; remove `attributionControl = FALSE`. -- [ ] Re-tune marker strokes and the palest fills against the new ground (§3.3); note `www/styles.css` is - manifest-tracked in several repos, so bundle it into the same commit. -- [ ] Add the regression guard: a CI grep asserting zero matches for `CartoDB[."$]|cartocdn`, plus a - scheduled tile canary that decodes tiles and fails on a single-colour or placeholder result. +- [x] Owner decision recorded: **take the free CARTO key**, keep Positron/DarkMatter unchanged (§4). +- [x] Key mechanics settled: `?key=` query param, raw `addTiles()`, helper + keyless fallback written (§4.3). +- [x] Connect Cloud confirmed to support runtime env vars via a Variables UI; key confirmed NOT secret (§4.2). +- [x] Per-repo manifest gates, branch targets and deploy rules mapped (§5). + +Open, in order — **the first is an owner action and blocks everything after it**: + +- [ ] **Request the key at ** — email + domain + one-line description, + emailed straight back, no account, no queue. *No agent can do this step.* +- [ ] **Smoke-test the URL form once** with the real key (§4.3) — an invalid key is byte-identical to no key, + so only a real one proves the format. +- [ ] Set `CARTO_BASEMAP_KEY` in Connect Cloud content settings → Variables, for each of the nine apps. + Assume a republish is needed for it to take effect. +- [ ] Add the `add_suite_basemap()` helper to each app and route its call sites through it (§4.3), removing + `attributionControl = FALSE` from small-mammal `server.R:2491`/`:2514` in the same change (§4.4). +- [ ] **Regenerate `manifest.json` in the same commit, never by hand** — use `regenerate-manifest.yml` where + it exists; hand-regenerate for the two repos with no CI (§5). +- [ ] One canary repo end to end — patch, manifest, merge, deploy, *look at the live map* — before the rest. +- [ ] Open a PR per repo; **do not merge them** (Small Mammal `DEPLOY.md:9-13`: a human-reviewed merge is the + production decision). +- [ ] Add the regression guard: a CI grep asserting no unkeyed `cartocdn` URL, plus a scheduled tile canary + that decodes a tile and fails on a watermark, a single-colour result, or the Esri placeholder. - [ ] Record the basemap contract in `docs/neonize-playbook.md` §2g — it currently names **no provider at - all**, so nothing stops the next app reintroducing CARTO. + all**, so nothing stops the next app reintroducing an unkeyed CARTO layer. - [ ] Visually verify each deployed app after merge. `HTTP 200` is not verification (§8). ### Per-app call-site inventory From b19eb1c6a0360a6c1012f790d4335d1f7f7dae72 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 28 Aug 2026 22:24:20 +0000 Subject: [PATCH 04/10] docs: record that platform API tokens do not unlock basemap tiles Tested a Workspace API Access Token scoped to the Maps API against the raster basemap CDN at confirmed origin cache misses under five auth forms; every response was still watermarked. The token is valid, but the maps scope covers CARTO's Maps API, not the public basemap CDN, and CARTO's key page confirms platform credentials only cover in-platform use. External embedding needs the form-issued basemap key. Also records that the CDN ignores the query string in its cache key, so a key smoke test must use an origin-MISS tile. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01738HeGPEU9NEoczL8RXnu1 --- docs/SUITE-BASEMAP-INCIDENT-2026-08.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/docs/SUITE-BASEMAP-INCIDENT-2026-08.md b/docs/SUITE-BASEMAP-INCIDENT-2026-08.md index 5bd9c60..e49a68f 100644 --- a/docs/SUITE-BASEMAP-INCIDENT-2026-08.md +++ b/docs/SUITE-BASEMAP-INCIDENT-2026-08.md @@ -223,6 +223,18 @@ Domain to give: the Connect Cloud share domain (`*.share.connect.posit.cloud`). that domain is actually enforced as a referer lock or merely recorded — CARTO documents no allowlist feature and there is no console, since there is no account. Treat it as informational. +**Verified 2026-08-28 — a CARTO *platform* API Access Token does NOT work as the basemap key.** The owner +created a Workspace token scoped to the Maps API and it was tested against the raster CDN at confirmed +origin cache misses (`x-cache: MISS`) under five auth forms — `?key=`, `?access_token=`, `?api_key=`, +`?apikey=`, and an `Authorization: Bearer` header. Every response was still the watermarked tile. The +token itself is valid (the platform API recognizes it and echoes `allowed_apis: ["maps"]`); the "maps" +scope is CARTO's Maps API for data layers, not the public basemap CDN. CARTO's key page confirms the +split: platform credentials cover Builder/Workflows and in-platform use — external embedding needs the +form-issued basemap key, a separate credential. **Do not re-test platform tokens; use the form.** +Also learned: the CDN ignores the query string in its cache key (a keyed request can get `x-cache: HIT` +on the unkeyed cached tile), so any smoke test of the real key must be done on an origin-MISS tile — +deep zoom over an obscure spot — or the cached watermarked tile will masquerade as a key failure. + ### 4.2 The key is NOT a secret — this is load-bearing CARTO's key rides in the tile URL, and every tile request is issued **client-side by the browser**. The key From 3674787828f31747e6a3715324c3bd2b7e3fe20e Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 28 Aug 2026 22:30:00 +0000 Subject: [PATCH 05/10] docs: record that the key form is captcha-gated; email fallback exists HubSpot's API-submission path for the basemap key form returns FORM_HAS_RECAPTCHA_ENABLED, so the form must be human-submitted in a browser. The page's own fallback is support-basemaps@carto.com, which issues a key by hand. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01738HeGPEU9NEoczL8RXnu1 --- docs/SUITE-BASEMAP-INCIDENT-2026-08.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/SUITE-BASEMAP-INCIDENT-2026-08.md b/docs/SUITE-BASEMAP-INCIDENT-2026-08.md index e49a68f..2c9f027 100644 --- a/docs/SUITE-BASEMAP-INCIDENT-2026-08.md +++ b/docs/SUITE-BASEMAP-INCIDENT-2026-08.md @@ -223,6 +223,12 @@ Domain to give: the Connect Cloud share domain (`*.share.connect.posit.cloud`). that domain is actually enforced as a referer lock or merely recorded — CARTO documents no allowlist feature and there is no console, since there is no account. Treat it as informational. +**Also verified 2026-08-28 — the key-request form cannot be submitted programmatically.** It is a HubSpot +form (portal `474999`, form `4545f275-9bc1-408f-ba2f-963a53a14803`) and HubSpot's API-submission path +returns `FORM_HAS_RECAPTCHA_ENABLED` — captcha is enforced server-side, so a human must submit it in a +browser. The page's own fallback: email **support-basemaps@carto.com** and "we will issue a key by hand." +Do not re-attempt an API submission. + **Verified 2026-08-28 — a CARTO *platform* API Access Token does NOT work as the basemap key.** The owner created a Workspace token scoped to the Maps API and it was tested against the raster CDN at confirmed origin cache misses (`x-cache: MISS`) under five auth forms — `?key=`, `?access_token=`, `?api_key=`, From b5ddf82ec29282df77779c2d69b7ae285f112a36 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 28 Aug 2026 23:08:38 +0000 Subject: [PATCH 06/10] docs: key issued and verified; ground-beetle canary PR open The owner obtained the basemap key via the form; it verifies clean at CDN origin misses for both variants, and previously-cached watermarked zooms return clean with it, so the CDN caches keyed responses separately. Records the canary flow: code first, manifest from the pinned CI validator's artifact in a follow-up commit, first run intentionally red at the byte gate. The key itself lives in Connect Cloud variables and the owner's email, never in any repo. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01738HeGPEU9NEoczL8RXnu1 --- docs/SUITE-BASEMAP-INCIDENT-2026-08.md | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/docs/SUITE-BASEMAP-INCIDENT-2026-08.md b/docs/SUITE-BASEMAP-INCIDENT-2026-08.md index 2c9f027..5f7b33d 100644 --- a/docs/SUITE-BASEMAP-INCIDENT-2026-08.md +++ b/docs/SUITE-BASEMAP-INCIDENT-2026-08.md @@ -437,12 +437,20 @@ Done: - [x] Connect Cloud confirmed to support runtime env vars via a Variables UI; key confirmed NOT secret (§4.2). - [x] Per-repo manifest gates, branch targets and deploy rules mapped (§5). -Open, in order — **the first is an owner action and blocks everything after it**: - -- [ ] **Request the key at ** — email + domain + one-line description, - emailed straight back, no account, no queue. *No agent can do this step.* -- [ ] **Smoke-test the URL form once** with the real key (§4.3) — an invalid key is byte-identical to no key, - so only a real one proves the format. +Open, in order: + +- [x] **Key requested and issued** (2026-08-28, owner via the form; the platform `api_token` route was + tested first and does NOT work — see §4.1). Key starts `cb1_…`; it lives in the owner's email and in + Connect Cloud variables, **never in any repo**. +- [x] **Key verified against live tiles**: clean at CDN origin misses for `light_all` AND `dark_all`, and — + the deployment worry — the previously-cached watermarked picker zooms return CLEAN with the key (the + CDN caches keyed responses separately). Unkeyed control stays watermarked. `?key=` is the confirmed form. +- [x] **Canary PR open**: Ground-Beetle-Tracker#22 (branch `claude/carto-basemap-key`) — adds + `add_suite_basemap()` to `global.R`, routes both call sites (`R/map_picker.R:57`, + `server.R:1633-1634`), dark toggle preserved via `dark_all`. Manifest deliberately untouched in the + first commit: the pinned CI validator regenerates it and the validated candidate artifact lands in a + follow-up commit (AGENTS.md forbids hand edits; the repo has no regenerate-manifest.yml dispatch). + Expect the canary's FIRST run red at the byte-match gate — that is the designed flow. - [ ] Set `CARTO_BASEMAP_KEY` in Connect Cloud content settings → Variables, for each of the nine apps. Assume a republish is needed for it to take effect. - [ ] Add the `add_suite_basemap()` helper to each app and route its call sites through it (§4.3), removing From 443bf2135dab7d5d715e3871131c6d67c38d2373 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 29 Aug 2026 02:49:01 +0000 Subject: [PATCH 07/10] docs: record the completed nine-repo basemap rollout MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Ground Beetle canary merged, deployed green, and the owner confirmed the live map, which closed the last unproven link — the Connect variable really does reach the running app. All eight remaining repos are now patched with PRs open. Records the rollout table, the split default branches checked per repo, and the two no-CI repos handled differently from each other and from the rest: Inverts source-only because its AGENTS.md forbids regenerating the manifest and its release identity hashes the source, Water Chemistry with its one MD5 updated because a stale checksum there would silently drop the fix. Also notes why no ui.R dropdown needed changing: the helper accepts a provider name or a CARTO variant, so choice vectors and defaults stay exactly as they were. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01738HeGPEU9NEoczL8RXnu1 --- docs/SUITE-BASEMAP-INCIDENT-2026-08.md | 43 ++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/docs/SUITE-BASEMAP-INCIDENT-2026-08.md b/docs/SUITE-BASEMAP-INCIDENT-2026-08.md index 5f7b33d..7624e80 100644 --- a/docs/SUITE-BASEMAP-INCIDENT-2026-08.md +++ b/docs/SUITE-BASEMAP-INCIDENT-2026-08.md @@ -464,8 +464,51 @@ Open, in order: that decodes a tile and fails on a watermark, a single-colour result, or the Esri placeholder. - [ ] Record the basemap contract in `docs/neonize-playbook.md` §2g — it currently names **no provider at all**, so nothing stops the next app reintroducing an unkeyed CARTO layer. +- [x] **Canary MERGED and confirmed live** — Ground Beetle #22 merged as `e136aeb`; `main` CI, the + production-verification workflow and Pages all green, no issue opened, and the owner confirmed the + live map shows Positron/DarkMatter with no watermark. That closed the last unproven link: the + Connect `CARTO_BASEMAP_KEY` variable really does reach the running app. +- [x] **All eight remaining repos patched, pushed, PRs open** (2026-08-29) — see the rollout table below. +- [ ] Merge the eight PRs (owner decision per repo) and set `CARTO_BASEMAP_KEY` in each app's Connect + Cloud Variables **before** merging, or the deploy shows the keyless Esri fallback. - [ ] Visually verify each deployed app after merge. `HTTP 200` is not verification (§8). + +### Rollout status (2026-08-29) + +The helper landed in every app as `add_suite_basemap()`. It accepts **either** a leaflet provider name or a +CARTO variant, which is why **no `ui.R` Basemap dropdown needed changing anywhere** — the choice vectors and +their defaults are untouched, CARTO entries route to keyed tiles, and every `Esri.*` entry passes straight +through to `addProviderTiles()` exactly as before. + +| App | PR | Base | Call sites | Notes | +|---|---|---|---|---| +| Ground Beetle | #22 **MERGED** | `main` | `R/map_picker.R:57`, `server.R:1633` | canary; live map confirmed | +| Mosquito Pulse | #12 | `master` | `server.R:537`, `:578`, `:588` | — | +| Breeding Birds | #6 | `master` | `server.R:645`, `:711`, `:723` | manifest gate also covers `docs/release.json` | +| Plant Phenology | #12 | `master` | `server.R:145`, `:177`, `:779` | "Light" is the **default** basemap | +| Plant Diversity | #18 | `master` | `R/map_picker.R:58`, `server.R:1495` | `www/runtime-receipt.txt` regenerated with the repo's own script | +| Vegetation Structure | #16 | `main` | `R/map_picker.R:88`, `server.R:1375`, `:1398` | `ui.R` `selected =` made it the **default** | +| Small Mammal | #93 | `main` | `server.R:1189`, `:2492`, `:2515` | **also removes `attributionControl = FALSE`** (§4.4) | +| My Little Inverts | #10 | `main` | `server.R:940`, `:963` | **object form**; source-only, needs validator regen | +| Water Chemistry | #19 | `main` | `app.R:2110` | no CI; manifest MD5 updated in-PR | + +**Default branches really are split** — `master` for Mosquito, Birds, Phenology, Plant Diversity; +`main` for Ground Beetle, Vegetation, Small Mammal, Inverts, Water Chem. Checked per repo, never assumed. + +**The seven repos with CI follow the canary's two-step flow:** first run goes red at the byte-match gate by +design, its validated manifest artifact is committed as a second commit, second run goes green. + +**The two repos with no `ci.yml` were handled differently, and differently from each other:** +- *My Little Inverts* — `AGENTS.md` forbids hand-editing or casually regenerating `manifest.json`, and its + `release/production-identity.json` hashes `global.R`/`ui.R`/`server.R` into `runtime_payload_sha256`. + Nothing was fabricated: the PR is **source-only and explicitly not mergeable** until both are regenerated + in the clean validator. +- *Water Chemistry* — no such prohibition, and `write_manifest.R` warns a stale checksum can make Connect + serve yesterday's bytes, which would silently drop the fix. Its manifest is plain per-file MD5s with the + package block restored verbatim from a reviewed lock, so the single `app.R` checksum was updated in-PR + after verifying all six committed checksums reproduce exactly. + ### Per-app call-site inventory Line numbers verified against each repo's default-branch HEAD on 2026-08-28. Re-verify before patching. From 21ef586aaecc10e73ac4ed6deda8d5da4d3faebf Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 29 Aug 2026 03:25:59 +0000 Subject: [PATCH 08/10] docs: record the rollout outcome across the nine apps MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Six of nine are green or merged. Every manifest shuttle showed the same thing: manifest.json was the only file that differed, with all data, indexes and receipts already byte-identical to the branch. Three are blocked on the owner for two distinct reasons. Mosquito is a CI shape gap — its artifact upload is conditional on an earlier failure and sits before the byte gate, so the validated manifest is discarded exactly when it is needed; five siblings upload unconditionally. Breeding Birds and Inverts bind the app source into generated authority, so regenerating needs R in the pinned validator; both PRs carry the validator's own commands verbatim. Also corrects a mistake worth keeping: no ci.yml is not no CI. Inverts and Water Chemistry both validate from refresh-data.yml, and Water Chemistry's connect_cold_start actually cold-boots the deploy bundle. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01738HeGPEU9NEoczL8RXnu1 --- docs/SUITE-BASEMAP-INCIDENT-2026-08.md | 38 ++++++++++++++++++++------ 1 file changed, 29 insertions(+), 9 deletions(-) diff --git a/docs/SUITE-BASEMAP-INCIDENT-2026-08.md b/docs/SUITE-BASEMAP-INCIDENT-2026-08.md index 7624e80..f4bc9a3 100644 --- a/docs/SUITE-BASEMAP-INCIDENT-2026-08.md +++ b/docs/SUITE-BASEMAP-INCIDENT-2026-08.md @@ -483,19 +483,39 @@ through to `addProviderTiles()` exactly as before. | App | PR | Base | Call sites | Notes | |---|---|---|---|---| -| Ground Beetle | #22 **MERGED** | `main` | `R/map_picker.R:57`, `server.R:1633` | canary; live map confirmed | -| Mosquito Pulse | #12 | `master` | `server.R:537`, `:578`, `:588` | — | -| Breeding Birds | #6 | `master` | `server.R:645`, `:711`, `:723` | manifest gate also covers `docs/release.json` | -| Plant Phenology | #12 | `master` | `server.R:145`, `:177`, `:779` | "Light" is the **default** basemap | -| Plant Diversity | #18 | `master` | `R/map_picker.R:58`, `server.R:1495` | `www/runtime-receipt.txt` regenerated with the repo's own script | -| Vegetation Structure | #16 | `main` | `R/map_picker.R:88`, `server.R:1375`, `:1398` | `ui.R` `selected =` made it the **default** | -| Small Mammal | #93 | `main` | `server.R:1189`, `:2492`, `:2515` | **also removes `attributionControl = FALSE`** (§4.4) | -| My Little Inverts | #10 | `main` | `server.R:940`, `:963` | **object form**; source-only, needs validator regen | -| Water Chemistry | #19 | `main` | `app.R:2110` | no CI; manifest MD5 updated in-PR | +| Ground Beetle | #22 | `main` | `R/map_picker.R:57`, `server.R:1633` | **MERGED + DEPLOYED**; live map confirmed by owner | +| Plant Diversity | #18 | `master` | `R/map_picker.R:58`, `server.R:1495` | **GREEN, ready**; `www/runtime-receipt.txt` regenerated with the repo's own node script — matched the validator byte-for-byte | +| Small Mammal | #93 | `main` | `server.R:1189`, `:2492`, `:2515` | **GREEN, ready**; also removes `attributionControl = FALSE` (§4.4) | +| Vegetation Structure | #16 | `main` | `R/map_picker.R:88`, `server.R:1375`, `:1398` | **GREEN, ready**; `ui.R` `selected =` made it the default | +| Water Chemistry | #19 | `main` | `app.R:2110` | **GREEN, ready**; passed a real `connect_cold_start` — proves the hand-set manifest MD5 and the `app.R` patch both boot | +| Plant Phenology | #12 | `master` | `server.R:145`, `:177`, `:779` | **GREEN, ready**; "Light" is the default basemap | +| Mosquito Pulse | #12 | `master` | `server.R:537`, `:578`, `:588` | **BLOCKED on owner** — everything passed except the manifest byte gate, and this repo's artifact upload is conditional on an *earlier* failure, so no validated manifest is exported. Needs `Rscript scripts/write_manifest.R` | +| Breeding Birds | #6 | `master` | `server.R:645`, `:711`, `:723` | **BLOCKED on owner** — `write_release_stamp.R` binds `global.R`/`ui.R`/`server.R`; failed at the stamp check before packages installed. Needs manifest **then** stamp | +| My Little Inverts | #10 | `main` | `server.R:940`, `:963` | **BLOCKED on owner** — object form; validator lives in `refresh-data.yml` (not `ci.yml`), failed at "Reject a stale committed identity". Producer artifact byte-compared: 36/36 identical | **Default branches really are split** — `master` for Mosquito, Birds, Phenology, Plant Diversity; `main` for Ground Beetle, Vegetation, Small Mammal, Inverts, Water Chem. Checked per repo, never assumed. +**Six of nine are green or merged.** The two-step flow (first run red at the byte gate by design → commit its +validated manifest artifact → second run green) worked for Ground Beetle, Plant Diversity, Small Mammal, +Vegetation Structure and Plant Phenology. In every shuttle the ONLY file that differed was `manifest.json` — +every data file, search index and receipt in the validator's artifact was already byte-identical to the +branch, which is independent evidence the patch moved nothing it shouldn't. + +**Three are blocked on the owner, for two distinct reasons — neither faked:** +- *Mosquito Pulse* — a CI-shape gap, not a code problem: its `Upload unvalidated manifest` step is + `if: failure() && …` and sits BEFORE the byte gate, so when the gate is the only failure nothing has failed + yet, the upload is skipped, and the validated manifest dies with the runner. Five sibling repos upload + theirs unconditionally. Worth aligning separately; deliberately not changed in a basemap PR. +- *Breeding Birds* and *My Little Inverts* — their generated authority **binds the app source** + (`write_release_stamp.R`; `runtime_payload_sha256` in `production-identity.json`), so editing `global.R` + or `server.R` invalidates it by construction. Regenerating needs R in the pinned validator. Each PR carries + the validator's own command sequence, lifted verbatim from its workflow. + +**Correction worth keeping:** "no `ci.yml`" is NOT the same as "no CI". My Little Inverts and Water Chemistry +both run validators from `refresh-data.yml`. Water Chemistry's even includes a `connect_cold_start` job that +cold-boots the deploy bundle — the strongest single check in the suite, and it passed. + **The seven repos with CI follow the canary's two-step flow:** first run goes red at the byte-match gate by design, its validated manifest artifact is committed as a second commit, second run goes green. From 27f0dee9adb0793fec7ad5a33c3f2664fb9d5f79 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 31 Aug 2026 16:03:45 +0000 Subject: [PATCH 09/10] docs: record the Birds master corruption, the Water Chem blank map, and the last two shuttles Adds section 10 to the incident record and three lessons. Two problems the rollout had not anticipated: - Breeding Birds' master carries an unresolved merge -- nine conflict-marker lines inside manifest.json's files map -- so the file is invalid JSON on the branch Connect Cloud watches, and CI failed on it. Cause and blast radius documented; the repair rides in PR #6. - Water Chemistry's map is blank for a reason that is not in the code. The live app provably runs the merged bytes and all three tile endpoints answer 200, which leaves the key's value: a missing key is loud (CARTO's watermark), a whitespace-padded one is silent. That asymmetry is the lesson worth keeping. Also records how Birds' authority was regenerated without an R runtime -- by reproducing the known-good stamp byte-for-byte first -- and how the Mosquito and Inverts shuttles were validated. Flags the outstanding work plainly: the hardening is in one repo, the other eight still carry the unhardened helper. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01738HeGPEU9NEoczL8RXnu1 --- .claude/agents/LESSONS.md | 29 ++++++ docs/BUILD-TEST-HANDOFF.md | 48 +++++++++ docs/SUITE-BASEMAP-INCIDENT-2026-08.md | 137 +++++++++++++++++++++++++ 3 files changed, 214 insertions(+) diff --git a/.claude/agents/LESSONS.md b/.claude/agents/LESSONS.md index 34bb886..66c3044 100644 --- a/.claude/agents/LESSONS.md +++ b/.claude/agents/LESSONS.md @@ -156,3 +156,32 @@ is real, keyless and labelled; it is simply absent from leaflet-providers. That single mis-framed question had already produced a signed-off CSS-invert workstream that was pure waste. Full record + per-role plan: `docs/SUITE-BASEMAP-INCIDENT-2026-08.md`. +- [2026-08-31] cass · confirmed · A generated artifact regenerated on the WRONG BASE is worse than one not + regenerated at all: it looks like progress and it lands on the deploy branch. Breeding Birds `master` head + `08eb093 "update"` shipped NINE conflict-marker lines inside `manifest.json`'s `files` map, making the file + invalid JSON and failing every gate that parses it (`verify_manifest.R`, `write_release_stamp.R`, Connect's + own bundle read) — on the branch Posit Connect Cloud watches. Root cause: `8128680` ran the regeneration on + `efda16e`, a months-old line with neither the release work nor the change it claimed to regenerate for, then + was merged without resolving the collision. CHECK `git merge-base --is-ancestor HEAD` BEFORE running + any regeneration script, and treat conflict markers in a generated file as a production incident, not a merge + nit. Blast radius was bounded only by luck: `git diff --name-status bb18be3 origin/master` returned exactly + one line, so the corrupt side could be discarded wholesale. +- [2026-08-31] cass · confirmed · You CAN regenerate a deterministic authority artifact without its runtime — + but only by reproducing a KNOWN-GOOD one byte-for-byte first. Birds' schema-v3 release stamp (a two-phase, + self-referential manifest→stamp→manifest contract) was reimplemented outside R and validated by reproducing + the committed `bb18be3` stamp exactly: same 125 payload files, same receipt digests, same `payload_sha256`, + same `release_id`. Only then was it applied to the new tree. That is not hand-editing — the implementation is + checked against ground truth before it is trusted, and a mismatch anywhere aborts. Same technique validated + the manifest MD5 model (121/121 checksums reproduced). Corollary: a contract digest that EXCLUDES the files + map cannot move on a source-only edit, so carrying it over verbatim is provable, not assumed. +- [2026-08-31] cass · confirmed · A free-tier API key is a runtime input, so validate it like one. The CARTO + basemap key rides in the tile URL via `sprintf()`; a value pasted into Connect Cloud's Variables field with a + TRAILING NEWLINE interpolates straight into the URL and every tile request fails — the map goes BLANK behind + an otherwise-working Leaflet frame. Note the asymmetry that makes this nasty: a MISSING key is loud (CARTO's + "API KEY REQUIRED" watermark, the incident that started all this), a MALFORMED key is silent, and neither + errors or logs. Tile URLs travel over the Shiny websocket, so it is invisible from outside the container. + Fix is three characters of defence — `trimws()` plus `grepl("^[A-Za-z0-9_-]+$", key)` — so a padded paste + still authenticates and a mangled one degrades to a working fallback instead of nothing, plus a `message()` + in the fallback branch so the next occurrence is one log line rather than a blank rectangle. GENERAL LESSON: + when a human pastes a value into a settings box, the code owns the whitespace. Full record: + `docs/SUITE-BASEMAP-INCIDENT-2026-08.md` §10.3. diff --git a/docs/BUILD-TEST-HANDOFF.md b/docs/BUILD-TEST-HANDOFF.md index e72ffc1..b88c462 100644 --- a/docs/BUILD-TEST-HANDOFF.md +++ b/docs/BUILD-TEST-HANDOFF.md @@ -4787,3 +4787,51 @@ Rules: - **Next action:** owner requests the key; then smoke-test the URL form, set the Connect variable, and canary ONE repo end to end — patch, manifest, merge, deploy, and actually look at the live map — before touching the other eight. + +## 2026-08-31 [Claude] — clear the last three PRs; two new problems found + +**Done.** The three blocked basemap PRs are unblocked; all nine repos now have a +mergeable PR. Two problems surfaced that the rollout had not anticipated — one of +them a production incident unrelated to basemaps. Full record: +`docs/SUITE-BASEMAP-INCIDENT-2026-08.md` §10. + +- **Breeding Birds `master` was broken, and it is the branch Connect watches.** Head + `08eb093 "update"` carries an unresolved merge: nine conflict-marker lines committed + inside `manifest.json`'s `files` map, so the file is not valid JSON and every gate + that parses it fails. CI run #8 on that commit failed. Cause: `8128680` regenerated + the manifest on `efda16e`, a months-old base with neither PR #5's release work nor + the basemap change. Bounded by luck — `git diff --name-status bb18be3 origin/master` + is exactly one line — so the corrupt side was discarded wholesale. The repair rides + in PR #6: one merge fixes `master` and ships the basemap change together. +- **Birds' authority regenerated without R, and proved before use.** The schema-v3 + stamp is deterministic, so it was reimplemented and validated by reproducing the + committed `bb18be3` stamp byte-for-byte (125 payload files, both receipt digests, + `payload_sha256`, and the derived `release_id` — all four match) before being applied + to the new tree. Manifest MD5 model validated the same way, 121/121. Exactly three + checksums moved; the six non-file contract fields are unchanged, so + `manifest_contract_sha256` holds by construction. +- **Mosquito and Inverts shuttled.** Mosquito's CI now exports its validated manifest + unconditionally (the gap was CI shape, not code); its artifact's 112 checksums all + match the tree, everything else differing is 73 package `Built` timestamps — a fresh + case for promoting this repo's semantic `compare_manifests.R` to the byte-exact + siblings. Inverts' dispatched validator succeeded in all four jobs; its three + authority files were shuttled after byte-comparing the whole branch. +- **Water Chemistry's map is blank, and the cause is NOT the code.** The live app is + provably running the merged bytes (`ddl-runtime-receipt` matches the six file MD5s + exactly), the helper is byte-identical to Ground Beetle's, and all three tile + endpoints answer 200. That leaves the key's *value*: a trailing newline in Connect + Cloud's Variables field interpolates into the tile URL and blanks the basemap + silently — a *missing* key is loud (the watermark), a *malformed* one is not. + PR #20 adds `trimws()` + a shape check + a `message()` in the fallback branch. + +**⚠️ Not verified end-to-end.** The Water Chem diagnosis is airtight except for the +actual stored value of `CARTO_BASEMAP_KEY`, which only the Connect settings page shows. +Tile URLs travel over the Shiny websocket, so they are not observable from a container. + +**Nothing at risk here:** docs-only in this repo; `docs/` and `.claude/` stay outside +`DEPLOY_APP_FILES`. + +**Next action:** land the §10.3 hardening in the remaining EIGHT repos in one pass — +they all carry the unhardened helper and are each one padded paste away from the same +silent blank map. Each needs its own manifest regeneration, so run it as a rollout +using the shuttle flow already documented here, not as a drive-by patch. diff --git a/docs/SUITE-BASEMAP-INCIDENT-2026-08.md b/docs/SUITE-BASEMAP-INCIDENT-2026-08.md index f4bc9a3..df42bc4 100644 --- a/docs/SUITE-BASEMAP-INCIDENT-2026-08.md +++ b/docs/SUITE-BASEMAP-INCIDENT-2026-08.md @@ -607,3 +607,140 @@ Playwright configured, but every navigation fails `net::ERR_CONNECTION_RESET` `https://example.com` — so the browser has no egress in this container, proxied or not. `curl` works. All evidence in this document is tile-level and source-level: decisive for the cause and for the measured blanking, but **nobody has yet seen a fixed app**. + +--- + +## 10. Session 2026-08-31 — the last three PRs, and two new problems + +Six of nine were already merged or green. This session cleared the three blocked ones and turned up two +things the rollout had not anticipated: a **corrupted `manifest.json` on the Breeding Birds deploy branch**, +and a **second, quieter basemap failure mode** that the helper as shipped does not survive. + +### 10.1 Breeding Birds — `master` was carrying an unresolved merge + +Not a basemap problem. Found while checking why the owner's regeneration had not reached the PR. + +`master` head `08eb093 "update"` has **nine conflict-marker lines and six superseded checksums committed +into `manifest.json`**, in three blocks inside the `files` map. The file is **not valid JSON**, so every gate +that parses it fails: `verify_manifest.R`, `write_release_stamp.R`, and Connect's own bundle read. CI run #8 +on that commit failed, and `master` is the branch Connect Cloud watches. + +How it happened: `8128680 "build: regenerate the manifest and release stamp for the basemap change"` has +parent `efda16e` — a months-old line carrying **neither** PR #5's release work **nor** the basemap change. It +regenerated four checksums against the wrong tree. Merging that into `bb18be3` collided on `manifest.json` +and the markers were committed unresolved. + +**Nothing else was lost.** `git diff --name-status bb18be3 origin/master` returns exactly one line — +`M manifest.json`. Master's tree is otherwise byte-identical to the last known-good commit, so discarding +the corrupt side restores it completely. The repair rides in PR #6 rather than a separate PR: one merge both +fixes `master` and ships the basemap change. The resolved tree is identical to the PR branch before the merge. + +**Lesson for the log:** a generated artifact regenerated on the wrong base is worse than one not regenerated +at all — it looks like progress and it lands on the deploy branch. Check `git merge-base --is-ancestor` before +running any regeneration script. + +### 10.2 Regenerating Birds' authority without R — and proving it first + +No R runtime here, and Birds' authority is a two-phase, self-referential contract (prestamp manifest → +release stamp → final manifest). It is also **fully deterministic**, so it was reimplemented rather than +hand-computed — and, decisively, **validated by reproducing the known-good `bb18be3` stamp byte-for-byte +before being used**: + +| field | reproduced | committed at `bb18be3` | +|---|---|---| +| payload files | 125 | 125 | +| `source_receipt_sha256` | `55f30d25…` | ✅ same | +| `environment_receipt_sha256` | `03f4bc77…` | ✅ same | +| `payload_sha256` | `2ce22cd3…` | ✅ same | +| `release_id` | `sha256:043aa41d…` | ✅ same | + +All four match, including the `release_id` derived from the carried-over `manifest_contract_sha256`. The +manifest MD5 model was validated the same way: all 121 committed checksums reproduce from the tree. + +Only then was it applied to the PR tree. What moved: `payload_sha256 → d62ebd6d…`, `release_id → +sha256:9492231a…`, and exactly three manifest checksums (`global.R`, `server.R`, `data/release_stamp.json`). +The six non-file contract fields are byte-identical to `bb18be3`, so `manifest_contract_sha256` is unchanged +**by construction** — the stamp's contract digest excludes the `files` map, which is why a source-only edit +cannot move it. + +**This is the pattern to reuse**: reproduce a known-good generated artifact exactly, then and only then apply +the same computation to the new tree. It is not hand-editing, because the implementation is checked against +ground truth before it is trusted. + +### 10.3 Water Chemistry — a blank map, and the failure mode the helper missed + +Reported symptom: the site-picker map draws its Leaflet frame and attribution control but **no basemap tiles**. + +Ruled out by measurement, not inference: + +| Hypothesis | Test | Result | +|---|---|---| +| Stale deploy | `ddl-runtime-receipt` meta in the served HTML vs the joined MD5 of the six `WATER_RUNTIME_FILES` | **Exact match** — the live app runs the merged code | +| Bad call site | Helper + call site vs Ground Beetle's (which works) | Byte-identical | +| Dead tile servers | `curl` all three endpoints | All `200`: CARTO `light_all` keyed (`image/png`), both Esri canvases (`image/jpeg`) | + +That leaves the **key's value in Connect Cloud**, and one value explains it exactly: + +> A **missing** key is loud — CARTO serves the "API KEY REQUIRED" watermark, which is this whole incident. +> A key carrying a **trailing newline or a stray space** is silent. `sprintf()` interpolates it into the tile +> URL, every tile request is malformed, and the basemap goes blank behind an otherwise-working map. + +That is exactly what a paste out of the CARTO signup mail leaves in Connect Cloud's Variables field, and it is +**not observable from outside the container** — tile URLs travel over the Shiny websocket. + +**The fix (Water Chemistry PR #20):** stop trusting the value. + +```r +key <- trimws(Sys.getenv("CARTO_BASEMAP_KEY", "")) +if (grepl("^[A-Za-z0-9_-]+$", key)) { +``` + +Trim it, so a padded paste still authenticates. Validate its shape, so a mangled value takes the Esri +fallback — a real basemap instead of nothing. A correct key is unaffected by both. The fallback branch now +also `message()`s why it fired, so the next occurrence is one line in the Connect log instead of a blank +rectangle. + +**⚠️ Not verified end-to-end.** The reasoning is airtight on everything *except* the actual stored value of +`CARTO_BASEMAP_KEY` for that content item, which only the Connect Cloud settings page shows. If the map is +still blank after PR #20 deploys, the next datum to get is the tile host in the browser Network tab +(`cartocdn` keyed / `arcgisonline` / neither). + +### 10.4 OUTSTANDING — the hardening is in ONE repo, not nine + +**The other eight apps carry the unhardened helper.** They work today, so this session deliberately did not +churn eight byte-exact manifests to fix one app. But every one of them is one whitespace-padded paste away +from the same silent blank map, and the owner set the variable by hand in nine places. + +**Next session: land §10.3's three-line change in the remaining eight repos in one pass.** Each needs its own +manifest regeneration, so treat it as a rollout, not a patch — the same shuttle flow this incident already +documents. The `message()` line makes it self-diagnosing thereafter. + +### 10.5 Final rollout state + +| App | PR | Base | State at end of session | +|---|---|---|---| +| Ground Beetle | #22 | `main` | **MERGED + DEPLOYED**, live map confirmed by owner | +| Water Chemistry | #19 | `main` | **MERGED + DEPLOYED**; receipt-verified live. Blank map → **PR #20** (§10.3) | +| Plant Diversity | #18 | `master` | Green, ready | +| Small Mammal | #93 | `main` | Green, ready | +| Vegetation Structure | #16 | `main` | Green, ready | +| Plant Phenology | #12 | `master` | Green, ready | +| Mosquito Pulse | #12 | `master` | **UNBLOCKED** — CI now exports the validated manifest; artifact shuttled (§10.6) | +| My Little Inverts | #10 | `main` | **UNBLOCKED** — dispatched validator succeeded, all 3 authority files shuttled (§10.6) | +| Breeding Birds | #6 | `master` | **UNBLOCKED** — authority regenerated (§10.2) **and** repairs `master` (§10.1) | + +### 10.6 How the last two shuttles were done + +*Mosquito* — the CI-shape gap is fixed: an unconditional `upload-artifact` step now sits before the byte +gate, matching Ground Beetle's verbatim (same pinned action SHA). The run then exported +`mosquito-manifest-43c8892d…`, and its 112 file checksums all match the tree, including the two the basemap +change moves. Everything else differing from the committed manifest is a package `Built` timestamp — **73 of +them** — recording when the validator compiled each source package. That is precisely the non-determinism +this repo's byte-exact gate flaps on, and the reason the bytes must be *taken* from the validator rather than +reconstructed. **This is the case for promoting `compare_manifests.R` to the byte-exact siblings.** + +*Inverts* — the dispatched `refresh-data.yml` run (`skip_download`) succeeded in all four jobs against the PR +head, and its publish job wrote the validated tree to `automation/invert-data-refresh`. Before shuttling, the +whole branch was byte-compared: it differed from the PR branch in **exactly** `manifest.json`, +`release/production-identity.json` and `docs/release.json`. After the shuttle the PR tree is byte-identical to +the validated branch. `runtime_payload_sha256` moved because it hashes `global.R`/`ui.R`/`server.R`. From e8f10a3fd2e70996a032054c8d0801cd6e8d61bd Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 31 Aug 2026 16:25:18 +0000 Subject: [PATCH 10/10] docs: confirm the Built drift is upstream, and record the finished rollout Adds section 10.7 and brings the rollout table up to date. Birds' rerun failed at the second stamp verify, and the failure named a real design flaw rather than a mistake: schema-v3 hashes the whole packages block into manifest_contract_sha256, so package metadata the change never touched invalidates the release identity. The open question was whether adopting the validator's manifest converges or loops. It converges, and that is now measured rather than assumed: Birds and Mosquito, different repos and different runners, produced byte-identical Built values for all 83 packages they share, in the same 2026-08-12 window, and Mosquito's rerun after shuttling went green. Those are Posit Package Manager's binary build times for the pinned snapshot, not per-run compiles. Worth keeping for connor/neonize: in Mosquito this drift only trips a byte-diff gate, but in Birds it corrupts the release identity of an unchanged app -- the strongest case yet for promoting compare_manifests.R. Also records how to spot a Connect variable that was missed: that app comes up on the grey Esri canvas instead of Positron. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01738HeGPEU9NEoczL8RXnu1 --- docs/SUITE-BASEMAP-INCIDENT-2026-08.md | 60 ++++++++++++++++++++++---- 1 file changed, 52 insertions(+), 8 deletions(-) diff --git a/docs/SUITE-BASEMAP-INCIDENT-2026-08.md b/docs/SUITE-BASEMAP-INCIDENT-2026-08.md index df42bc4..067f33f 100644 --- a/docs/SUITE-BASEMAP-INCIDENT-2026-08.md +++ b/docs/SUITE-BASEMAP-INCIDENT-2026-08.md @@ -720,14 +720,20 @@ documents. The `message()` line makes it self-diagnosing thereafter. | App | PR | Base | State at end of session | |---|---|---|---| | Ground Beetle | #22 | `main` | **MERGED + DEPLOYED**, live map confirmed by owner | -| Water Chemistry | #19 | `main` | **MERGED + DEPLOYED**; receipt-verified live. Blank map → **PR #20** (§10.3) | -| Plant Diversity | #18 | `master` | Green, ready | -| Small Mammal | #93 | `main` | Green, ready | -| Vegetation Structure | #16 | `main` | Green, ready | -| Plant Phenology | #12 | `master` | Green, ready | -| Mosquito Pulse | #12 | `master` | **UNBLOCKED** — CI now exports the validated manifest; artifact shuttled (§10.6) | -| My Little Inverts | #10 | `main` | **UNBLOCKED** — dispatched validator succeeded, all 3 authority files shuttled (§10.6) | -| Breeding Birds | #6 | `master` | **UNBLOCKED** — authority regenerated (§10.2) **and** repairs `master` (§10.1) | +| Water Chemistry | #19 | `main` | **MERGED + DEPLOYED**; receipt-verified live. Blank map → #20, **also merged** (§10.3) | +| Plant Diversity | #18 | `master` | **MERGED** (owner, 15:28Z) | +| Small Mammal | #93 | `main` | **MERGED** (owner, 15:28Z); also restores the suppressed attribution control | +| Vegetation Structure | #16 | `main` | **MERGED** (owner, 15:28Z) | +| Plant Phenology | #12 | `master` | **MERGED** (owner, 15:28Z) | +| Mosquito Pulse | #12 | `master` | **MERGED** — CI now exports the validated manifest; artifact shuttled (§10.6) | +| My Little Inverts | #10 | `main` | **MERGED** — dispatched validator succeeded, all 3 authority files shuttled (§10.6) | +| Breeding Birds | #6 | `master` | Authority regenerated (§10.2, §10.7) **and** repairs `master` (§10.1) | + +**Eight of nine merged.** Breeding Birds is the last, and merging it also repairs `master`'s invalid manifest. + +**Spotting a missed Connect variable:** an app whose `CARTO_BASEMAP_KEY` was not set comes up on Esri's grey +canvas rather than Positron — correct and clean, but visibly greyer than Ground Beetle. That contrast is the +fastest cross-suite check that all nine variables really are set. ### 10.6 How the last two shuttles were done @@ -739,6 +745,44 @@ them** — recording when the validator compiled each source package. That is pr this repo's byte-exact gate flaps on, and the reason the bytes must be *taken* from the validator rather than reconstructed. **This is the case for promoting `compare_manifests.R` to the byte-exact siblings.** +### 10.7 CONFIRMED — the `Built` drift is upstream, and adopting the validator's manifest converges + +Birds' first rerun failed at the **second** stamp verify, and the failure named the real design flaw: + +``` +Error: Candidate release stamp does not match its payload and manifest contract. +``` + +Not a defect in the regeneration. That run's own manifest agreed with the committed one on **all 121 file +checksums**, and the *first* stamp verify passed on the committed pair. The 146 differing lines were **73 +package `Built` timestamps and nothing else** — and those feed `manifest_contract_sha256`, because the +schema-v3 contract hashes the entire `packages` block. + +**So package metadata that the change never touched invalidates the release identity.** That is the flaw. + +The open question was whether adopting the validator's manifest converges or loops forever. It converges, +and this is now *measured*, not assumed: + +| Evidence | Result | +|---|---| +| Birds run 33411176784 (16:09Z) vs Mosquito run 33409103282 (15:47Z) — different repos, different runners | All **83 shared packages** carry byte-identical `Built` values | +| The window | `2026-08-12 19:35:57 – 19:50:19 UTC`, identical in both | +| Mosquito's rerun after shuttling | **Green** — CI regenerated the manifest and it byte-matched | + +Those are **Posit Package Manager's own binary build times** for the pinned 2026-07-15 jammy snapshot, rebuilt +server-side on 2026-08-12 — not per-run compiles. Stable across runs and across repos. + +Birds was therefore fixed by taking the validator's manifest verbatim and **rebinding the stamp to it** +(`manifest_contract_sha256 80ac0012… → cf90c33b…`, `release_id 9492231a… → 243036a9…`), with +`payload_sha256` unchanged at `d62ebd6d…` because no source byte moved. The contract-digest implementation +was validated the same way as everything else here: it reproduces the known-good `80ac0012…` exactly from +the committed manifest before being used on the new one. + +**This is the strongest argument yet for promoting `compare_manifests.R`.** In Mosquito the drift only trips +a byte-diff gate. In Birds it corrupts the **release identity of an unchanged app** — an upstream binary +rebuild is enough to invalidate a release. A semantic comparison (package identity / version / checksum) +would be immune to both. Candidate follow-up for `connor` + `neonize`. + *Inverts* — the dispatched `refresh-data.yml` run (`skip_download`) succeeded in all four jobs against the PR head, and its publish job wrote the validated tree to `automation/invert-data-refresh`. Before shuttling, the whole branch was byte-compared: it differed from the PR branch in **exactly** `manifest.json`,