feat(seo): IndexNow submissions for changed pages

.github/workflows/indexnow-submit.yml

@@ -0,0 +1,105 @@
+name: "SEO: IndexNow submit"
+run-name: "IndexNow: ${{ github.event_name == 'workflow_dispatch' && inputs.scope || 'changed' }} URLs"
+
+# Tells the IndexNow engines (Bing, Yandex, Seznam, Naver, Yep) which pages
+# changed, instead of waiting for their next crawl. Google does not take part;
+# it keeps reading the sitemap.
+#
+# On every push to main that touches plots/, the diff is mapped to page URLs:
+#   plots/<spec>/specification.*                  → https://anyplot.ai/<spec>
+#   plots/<spec>/metadata/<lang>/<lib>.yaml        → https://anyplot.ai/<spec>/<lang>/<lib>
+#   plots/<spec>/implementations/<lang>/<lib>.<ext> → same page
+# A deleted implementation is submitted too — IndexNow is "this URL changed",
+# which covers removals.
+#
+# The key is public by design: it only proves the submitter controls the host,
+# and the engines verify it by fetching https://anyplot.ai/<key>.txt. The same
+# key appears in three places — app/public/<key>.txt (the file itself), the
+# exact-match `location` in app/nginx.conf (so a crawler UA is not proxied to
+# /seo-proxy and a 404), and INDEXNOW_KEY below. Keep all three in sync when
+# rotating it.
+
+on:
+  push:
+    branches: [main]
+    paths:
+      - 'plots/**'
+  workflow_dispatch:
+    inputs:
+      scope:
+        description: "'changed' submits the URLs touched by the latest commit on main; 'sitemap' submits every URL in the live sitemap (initial load, or after a long outage)"
+        required: false
+        default: changed
+        type: choice
+        options: [changed, sitemap]
+
+permissions:
+  contents: read
+
+concurrency:
+  group: indexnow-submit
+  cancel-in-progress: false
+
+env:
+  HOST: anyplot.ai
+  INDEXNOW_KEY: anyplot-indexnow-ab738f04ea92446a
+  SCOPE: ${{ github.event_name == 'workflow_dispatch' && inputs.scope || 'changed' }}
+
+jobs:
+  submit:
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
+        with:
+          # HEAD~1 is the previous main for a squash merge, which is how every
+          # pipeline PR lands; a rare multi-commit push submits its last commit.
+          fetch-depth: 2
+
+      - name: Collect URLs
+        id: urls
+        run: |
+          set -euo pipefail
+          if [ "$SCOPE" = "sitemap" ]; then
+            curl -sfS "https://${HOST}/sitemap.xml" \
+              | grep -oE '<loc>[^<]+</loc>' | sed -E 's#</?loc>##g' | sort -u > urls.txt
+          else
+            git diff --name-only HEAD~1 HEAD -- plots/ | awk -F/ '
+              $1 != "plots" || NF < 3 { next }
+              $3 ~ /^specification\./ { print "https://'"$HOST"'/" $2; next }
+              ($3 == "metadata" || $3 == "implementations") && NF == 5 {
+                lib = $5; sub(/\.[^.]+$/, "", lib)
+                print "https://'"$HOST"'/" $2
+                print "https://'"$HOST"'/" $2 "/" $4 "/" lib
+              }' | sort -u > urls.txt
+          fi
+          n=$(wc -l < urls.txt)
+          echo "count=$n" >> "$GITHUB_OUTPUT"
+          echo "::notice::${n} URL(s) to submit (scope: ${SCOPE})"
+          head -20 urls.txt
+
+      - name: Submit to IndexNow
+        if: steps.urls.outputs.count != '0'
+        run: |
+          set -euo pipefail
+          # 10,000 URLs per request is the protocol limit; the full sitemap is
+          # under that today (4.6k) but the split keeps this future-proof.
+          split -l 10000 -d urls.txt batch_
+          for f in batch_*; do
+            body=$(jq -n --arg host "$HOST" --arg key "$INDEXNOW_KEY" \
+              --arg loc "https://${HOST}/${INDEXNOW_KEY}.txt" \
+              --rawfile list "$f" \
+              '{host: $host, key: $key, keyLocation: $loc,
+                urlList: ($list | split("\n") | map(select(length > 0)))}')
+            code=$(curl -sS -o response.txt -w '%{http_code}' \
+              -X POST "https://api.indexnow.org/indexnow" \
+              -H "Content-Type: application/json; charset=utf-8" \
+              --data "$body")
+            n=$(grep -c . "$f")
+            case "$code" in
+              200|202) echo "::notice::IndexNow accepted ${n} URL(s) (HTTP ${code})" ;;
+              # 4xx is a protocol or key problem on our side; make it visible.
+              *) echo "::error::IndexNow returned HTTP ${code} for ${n} URL(s): $(head -c 300 response.txt)"; exit 1 ;;
+            esac
+          done

CHANGELOG.md

@@ -26,6 +26,18 @@ aggregate instead: an italic *Catalog* line at the end of the version section an
 
 ## [Unreleased]
 
+### Added
+
+- **IndexNow: changed pages are pushed to Bing, Yandex, Seznam, Naver and Yep instead of
+  waiting for a crawl** — Bing Webmaster Tools' first recommendation for the site. A public
+  key file (`app/public/<key>.txt`, served by an explicit nginx `location` so crawler UAs
+  are not proxied away from it) proves control of the host, and the new
+  `.github/workflows/indexnow-submit.yml` maps every push to `main` that touches `plots/`
+  onto the affected `/{spec}` and `/{spec}/{language}/{library}` URLs and POSTs them to
+  `api.indexnow.org` (10,000 per request; a deleted implementation is submitted too).
+  `workflow_dispatch` with `scope=sitemap` submits the whole live sitemap for the initial
+  load. Google does not take part and keeps reading the sitemap. The protocol is free.
+
 ### Fixed
 
 - **Infrastructure failures no longer spend a pair's generation budget** — the

app/nginx.conf

@@ -258,6 +258,14 @@ server {
         try_files $uri =404;
     }
 
+    # IndexNow key file (docs/reference/seo.md "IndexNow"). Bing verifies a
+    # submission by fetching it with a crawler UA, which the map above would
+    # otherwise send to /seo-proxy/<key>.txt and a 404. Same key as
+    # .github/workflows/indexnow-submit.yml; rotate both together.
+    location = /anyplot-indexnow-ab738f04ea92446a.txt {
+        try_files $uri =404;
+    }
+
     # llms.txt has no registered .well-known name, but agents guess this path
     # (docs/reference/seo.md "Discoverability") — answer with the file's
     # canonical location instead of the SPA shell, which soft-404'd it with

app/public/anyplot-indexnow-ab738f04ea92446a.txt

@@ -0,0 +1 @@
+anyplot-indexnow-ab738f04ea92446a

docs/reference/seo.md

@@ -499,6 +499,23 @@ onto the seo-proxy path, and `.github/workflows/bot-serving-check.yml` guards it
 daily against the Cloud Run origin (origin, not edge — so it reports on the
 nginx map no matter what the zone policy is, and will never catch edge drift).
 
+## IndexNow
+
+[IndexNow](https://www.indexnow.org/) is the free, open push protocol shared by
+Bing, Yandex, Seznam, Naver and Yep: instead of waiting for a crawl, the site
+posts the URLs that changed. Google does not take part and keeps reading the
+sitemap. Three pieces, kept in sync when the key is rotated:
+
+| Piece | Where | Purpose |
+|-------|-------|---------|
+| Key file | `app/public/<key>.txt` (served by nginx to every client, bots included — an explicit `location =` like `robots.txt`) | Proves the submitter controls `anyplot.ai`; the engines fetch it on every submission. The key is public by design. |
+| Submission workflow | `.github/workflows/indexnow-submit.yml` | On every push to `main` that touches `plots/`, maps the diff to page URLs (`/{spec}` and `/{spec}/{language}/{library}`) and POSTs them to `https://api.indexnow.org/indexnow`, 10,000 per request. A deleted implementation is submitted too — the protocol means "this URL changed". |
+| Manual full load | `gh workflow run indexnow-submit.yml -f scope=sitemap` | Submits every URL of the live sitemap; used once at rollout and after a long outage of the workflow. |
+
+The engines answer `200` or `202` for an accepted batch; `4xx` means a key or
+payload problem and fails the run so it is visible. Bing Webmaster Tools shows
+the received submissions under *IndexNow*.
+
 ## Discoverability for assistants
 
 Two session protocols of external assistants against the sister project

docs/workflows/overview.md

@@ -172,6 +172,7 @@ Located in `.github/workflows/`:
 | `report-validate.yml` | Validates user-submitted issue reports |
 | `sync-postgres.yml` | Syncs `plots/` filesystem state to PostgreSQL on push to main |
 | `sync-labels.yml` | Auto-syncs spec/impl labels after manual PR merges |
+| `indexnow-submit.yml` | Pushes changed page URLs to IndexNow (Bing, Yandex, Seznam, Naver, Yep) on every push to main that touches `plots/`; `workflow_dispatch` with `scope=sitemap` submits the whole sitemap |
 | `codeql.yml` | CodeQL scanning (actions, JavaScript/TypeScript, Python) on pushes to main, PRs and a weekly cron; `plots/**` is excluded from triggers and analysis, so pipeline PRs never start a scan |
 | `ci-lint.yml` | Ruff lint check on PRs |
 | `ci-tests.yml` | Unit + integration tests on PRs |