Website scaffolding: portable site config, unified template, coherent URLs

[saulbert]

Phase 3 (website) kickoff: the scaffolding + glue layer that the rebrand sits on. Closes #83, closes #49.

What this does

Portability (the rebrand becomes a config change):

• Site identity comes from env vars — SITE_NAME, CONTACT_EMAIL, SITE_ATTRIBUTION — with defaults identical to today, so a redeploy of xd.saul.pw from master is brand-identical, and the staging site gets the new name purely via Netlify env settings.
• One shared template: scripts/wwwify.py now wraps xdfile/html.py instead of duplicating it. Prose pages get the navbar (it was commented out); HTML5 doctype + UTF-8 replace XHTML 1.0/ISO-8859-1 (output was already pure-ASCII entities, so this is declaration-only).

URL scheme (full inventory + strategy analysis behind these choices is summarized below):

• Front page is now prose, built from www/index.md (never built before — the live front page was the raw pubyear grid).
• The grid map moves to /pub/, which doubles as the publications index; new per-publication pages at /pub/<pubid>/ mean the map's links stop 404ing. /pub/ reads progressively: /pub/ → /pub/nyt → /pub/nyt1994 → /pub/nyt1994-01-01.
• Clue pages move /pub/clue/ → /clue/, mirroring /word/ (clues aren't publication-scoped; /pub/clue broke the progressive scheme).
• Prose pages build in directory form (/about/index.html) — immune to /about vs /about/ ambiguity.
• Navbar fixed: it pointed at /words and /clues, which have never matched what the generators write. Also fixed: the "Most Popular" dropdown had no parent anchor, so Words/Clues were unreachable by hover.

Nothing breaks (hard requirement): every URL that works today — and the S3-era inbound ones that 404 today — keeps working via real page or _redirects 301: /words//clues, /pub/clue/*, /xdiffs/*, /logs/*, /pubs/*, old zips, /doc/* (→ GitHub spec until a /format/ page exists), plus the two Google-Sheets redirects absorbed from gxd/redirects.tsv.

Deploy unblockers: bots.ts syntax error fixed (missing paren — blocked deploys); robots.txt no longer disallows all of /pub (only the ephemeral /pub/deep/); proper Netlify 404.html replaces the S3-era error.html.

Verified

• 291 tests pass (8 new in xdfile/tests/test_html.py); pyright clean on all touched files.
• make website-static + 37-pubyear-svg.py (full nyt run) + 34-mkwww-clues.py against the local gxd corpus: per-pub pages exist for every link the map emits, clue pages land in /clue/, prose front page survives the full website target.
• SITE_NAME='The Cross Reference' make website-static renames the site end-to-end with zero code change.

Open questions surfaced by the URL audit (not blocking, future tasks)

1. Incremental builds destroy URLs daily: /clue/<bc> beyond top-200 and /pub/deep/<xdid> pages only exist for last-day puzzles, and each atomic Netlify deploy deletes yesterday's. Generation-policy decision needed.
2. /word/ is capped at top-100 answers — expand?
3. Pre-1990 pubyear pages were deliberately stripped (13a8c90) but the pre-Shortzian trove is the crown jewel — restore decades on the map?
4. xd-puzzles.zip is linked from /data/ but its build line is commented out (Makefile:55) — restore or unlink.
5. Whether Netlify actually registers the edge function from the publish dir — will verify on the staging deploy.

[this message written by Claude Fable 5 and approved by @saulpw]