docs: diagnose the CARTO basemap outage, adopt the CARTO key, and record the nine-app rollout

[tgilbert14]

What happened

The maps across the NEON explorer suite started showing "API KEY REQUIRED — carto.com/basemaps/apikey" stamped across the basemap.

Root cause: CARTO now burns that watermark into unauthenticated raster tiles from basemaps.cartocdn.com, server-side. The tile request still returns HTTP 200 with a valid PNG, so nothing errors, logs, or falls back — the map simply renders defaced.

curl -s -o carto_light.png "https://a.basemaps.cartocdn.com/light_all/6/13/24.png"   # CartoDB.Positron
curl -s -o carto_dark.png  "https://a.basemaps.cartocdn.com/dark_all/6/13/24.png"    # CartoDB.DarkMatter

Blast radius: all nine companion apps, on first load — every one puts CartoDB.Positron on its landing/site-picker map, which needs no interaction. Driver-Cascade itself ships no Leaflet map and is unaffected.

Decision: take the free CARTO key — ADOPT

Authenticate the existing basemaps rather than replace them. Keeping CartoDB.Positron and CartoDB.DarkMatter is the only option that changes nothing visually: same tiles, same maxZoom = 20, same retina, zero blank tiles, no palette or CSS re-tuning, no contrast or attribution regression, and ground-beetle's dark theme keeps working untouched.

One step no agent can do: request the key at https://carto.com/basemaps/apikey — email + domain + one-line description, emailed straight back, no account, no approval queue, 5M tiles/month fair use.

Why not swap the provider — the first plan was refuted

The initial plan (blanket CartoDB.Positron → Esri.WorldGrayCanvas plus a CSS-invert dark canvas) went through a nine-app audit and a four-lens adversarial review. All four lenses refuted it. Two findings were re-verified by hand:

1. The replacement is measurably BLANK at plot scale. Canvas/World_Light_Gray_Base has had no content update since 2021.

Tile (SCBI)	Distinct RGB values
Light Gray Base z13	224 — hairline roads, no labels
Light Gray Base z16	1 — a single flat RGB(239,239,239)
World_Topo_Map z13	3,912
World_Topo_Map z16	717

Across all 46 NEON terrestrial sites, 17 (37%) return a single-colour blank tile at z16. Every other candidate — Esri Topo/Imagery, USGS, OSM, CARTO — is 0/46. The maxNativeZoom = 16 guard upscales an already-blank tile.

2. There IS a keyless dark canvas. Canvas/World_Dark_Gray_Base is real, keyless and labelled — so the CSS-invert workstream was unnecessary. The earlier "no keyless dark canvas exists" claim was false: it asked what leaflet-providers exposes, never what the provider serves.

Plus: the swap makes marker contrast ~8–10% worse; leaflet.providers emits a stale Esri attribution omitting the OpenStreetMap credit the service's own copyrightText requires; and one app uses leaflet::providers$CartoDB.Positron in object form, which a string grep misses.

Every one of those defects exists because the alternatives moved the basemap. The key doesn't.

The code

addProviderTiles() cannot carry the key — the pinned CartoDB template has an {r} retina slot but no {apikey} placeholder — so it needs a raw addTiles() with attribution by hand. §4.3 adds an add_suite_basemap() helper: keyed CARTO when the key is present, keyless Esri canvas when it isn't, so a missing or revoked key degrades to a clean canvas rather than a defaced one.

The key is not a secret. It rides in the tile URL and every request is client-side, so it lands in page source regardless; CARTO's terms §9.c bans server-side proxying, so it cannot be hidden. Sys.getenv("CARTO_BASEMAP_KEY") buys exactly two things — it keeps the key out of nine public git histories, and makes rotation a Connect Cloud setting instead of nine releases.

No negative test exists: an invalid key returns the byte-identical watermarked tile as no key at all (same ETag). Only the real key proves the URL form — smoke-test once before any rollout PR.

Rollout constraints found (§5)

• The manifest is the real gate. ui.R/server.R/R/*.R/global.R are all on each repo's deploy surface with per-file checksums, so the manifest must be regenerated in the same commit. Never hand-edit it — Small Mammal and Vegetation carry a blessed regenerate-manifest.yml for exactly this.
• neon-my-little-inverts and neon-waterchemistry-analyte-viewer-app have no ci.yml but still ship a manifest — nothing catches a stale one. Do them last.
• No test anywhere in the suite asserts a provider string, and CI never fetches a tile. That is why this shipped unnoticed.
• Small Mammal DEPLOY.md:9-13 forbids automation pushing to main — open PRs, do not merge them.

Changes

• docs/SUITE-BASEMAP-INCIDENT-2026-08.md — diagnosis, non-causes eliminated with evidence, the refutation and its measurements, the decision, the helper code, per-repo rollout gates, all 22 call sites, and a resume checklist.
• .claude/agents/LESSONS.md — two lessons: a free tile provider is an uncontracted runtime dependency no manifest pin covers; and "unwatermarked" is not "usable" — decode the tile and count colours, because HTTP 200 proves neither.
• docs/BUILD-TEST-HANDOFF.md — three dated [Claude] entries.

Deploy safety

Docs only. docs/ and .claude/ are outside the DEPLOY_APP_FILES allowlist, so no artifact, manifest, or deploy surface moves.

Known limitation

No headless-browser check was possible in this container (Chromium has no egress; every navigation fails ERR_CONNECTION_RESET, including example.com). All evidence is tile-level and source-level — decisive for the cause and the blanking, but nobody has yet seen a fixed app.

🤖 Generated with Claude Code

https://claude.ai/code/session_01738HeGPEU9NEoczL8RXnu1