feat(seo): collect per-URL index state and per-family search performance from the Search Console API (weekly)

[koala73]

Parent: #8602 · Related: #6036, which runs the manual measurement cycle. This issue automates its Google half.

Outcome

A command that answers two questions for each URL family every week, from the Search Console API rather than UI transcription:

1. Which of our URLs are indexed, and which did Google decline and why?
2. Which families earn impressions and clicks per indexed page?

Until this exists, every decision about growing or pruning a page family is a guess. The 2026-09-24 coverage export has counts only. We know 1,486 URLs are "Crawled, currently not indexed" but not which ones.

Human prerequisites (owner only; an agent cannot do these)

1. Create a Google Cloud service account with the Search Console API enabled. Add its email as a Restricted user on the worldmonitor.app Domain property.
2. Store its JSON key where loadEnvFile() finds it, under one variable such as GSC_SERVICE_ACCOUNT_JSON (base64). Add the same value as a GitHub Actions secret if the weekly job should run in CI.
3. Comment the property ID's format here (sc-domain:worldmonitor.app or a URL prefix) without secrets.

The agent can finish everything below against recorded fixtures before step 1 happens. Only the live-run criterion waits on the owner.

Scope (agent)

1. Collector. Add a collector under scripts/, for example scripts/seo-gsc-collect.mjs, that:
   • calls searchanalytics.query with dimensions page (and query for page-level top queries), for the 28-day and 90-day windows, paginating past 25k rows;
   • calls urlInspection.index.inspect on a stratified sample of up to 2,000 URLs a day (the API quota). Draw the sample from the three live sitemaps, with every family represented. Record coverageState, indexingState, googleCanonical against userCanonical, lastCrawlTime and pageFetchState;
   • loads credentials only through loadEnvFile() and never prints secrets;
   • keeps missing values as null with a reason, never zero. This matches the contract in scripts/seo-ai-visibility-collector.mjs.
2. Family mapping. Reuse PAGE_FAMILIES in scripts/seo-ai-visibility-scorecard.mjs:55-76. It does not yet cover sources, compare, research, use_cases, docs_zh, markdown_twins (*.md, llms*.txt) or story_shares. Extend the domain and do not fork it. Every sitemap URL must map to exactly one family, and a test enforces this.
3. Report. Write a dated snapshot next to the existing baselines in docs/research/seo-ai-visibility/ (for example gsc/YYYY-MM-DD.json), following their conventions. Also write a markdown summary. Per family it shows URLs declared, sampled, indexed %, top non-indexed reasons, impressions, clicks, CTR, impressions per indexed URL, and the 5 best and 5 worst URLs.
   • Feed the per-family numbers into the existing scorecard instead of starting a parallel report.
4. Schedule. Add a weekly GitHub Actions workflow that runs the collector and opens or updates a PR with the new snapshot. Gate it on the secret, so it skips cleanly when the secret is absent.
5. Tests. Use fixture API responses: pagination, a quota-exhausted 429, a URL with googleCanonical ≠ userCanonical, and an unmapped URL (which must fail).

Out of scope

• Bing, AI-platform observations and referral analytics (ops(growth): execute the first credentialed SEO and AI visibility measurement cycle #6036).
• Acting on the findings. The tracker issue owns decisions, such as removing markdown twins from the sitemap.

Acceptance criteria

• node scripts/seo-gsc-collect.mjs --fixtures tests/fixtures/gsc/ produces a snapshot and summary deterministically. The tests pass under npm run test:data.
• Every URL in the three live sitemaps maps to exactly one family, and a test asserts it.
• No credential, property secret or user-level data appears in committed output. A test greps the snapshot for private_key, client_email and similar.
• After the owner completes the prerequisites, one live run is attached to this issue. It must show which families make up the "Crawled, currently not indexed" population. This answers the tracker's open question.

Guardrails

• Use loadEnvFile(). Never add an env parser or read credentials from $HOME or an absolute path.
• Any new file containing an external URL (googleapis.com) needs node scripts/source-attribution.mjs --write and then node scripts/docs-stats.mjs --check.
• Never merge. Report the PR as ready.

[koala73]

Reinforced by the drilldown, and one requirement to add

The #8602 drilldown is in: #8602 (comment)

Short version: this issue just became the highest-value child, and the coverage report cannot substitute for it. The 1,486 "Crawled – currently not indexed" population turned out to contain no corpus content pages at all — 503/1,000 sampled rows are /docs/_next/* render assets already serving noindex, 245 are non-www hosts that already 3xx, 122 are .json data files, and the actionable residue is ~178 machine-readable files (now #8608). The corpus is indexed. It ranks for nothing. Only searchanalytics.query can tell us why, which is precisely what this collector adds.

Add this requirement to the scope

Bucket by response content-type and x-robots-tag before reporting any coverage count, and report indexability for HTML pages separately.

A report that counts coverage rows without that split reproduces the exact error #8602 made. Two reasons it is load-bearing, not tidiness:

1. 61% of the bucket is .js/.css. A headline "crawled-not-indexed" number dominated by subresources moves on Mintlify redeploys — hash-versioned chunk names churn on every docs build — rather than on anything we control. The number would look alarming and be uninformative.
2. noindex is the discriminator, and only a live fetch reveals it. urlInspection alone does not tell you that /docs/pricing.md serves noindex while /pricing.md does not; that asymmetry is what produced fix(seo): ~200 machine-readable URLs (resilience.json, *.openapi.yaml, .well-known/*) answer 200 with no X-Robots-Tag — the bulk of the actionable crawled-not-indexed population #8608, and it is invisible in the coverage export. The collector should record the x-robots-tag it observes alongside coverageState, and treat "crawled-not-indexed and no noindex and content-type: text/html" as the only cell that warrants attention.

Suggested minimum buckets, orthogonal to PAGE_FAMILIES: html, data (.json/.yaml), markdown-twin (.md/llms*.txt), subresource (.js/.css/images), feed (.xml). Scope item 2 already says every sitemap URL must map to exactly one family and a test must enforce it — the same test should assert exactly one kind.

Two fixture cases worth adding

The drilldown hit both, and both would silently corrupt a report:

• A row whose live status contradicts its recorded coverageState. 428 of the 666 reported 404s already return 3xx today, and 298 of those were fixed by fix(seo): scope the Mintlify docs proxy to www + redirect prefix-less /api-reference/* (#5345) #5347 well before the export. Any trend line built on coverageState alone reports a defect that no longer exists. The collector should carry a live-probe status next to Google's state and flag disagreement rather than resolving it silently.
• A non-www host. happy, commodity, tech, finance, energy, api and the apex are all in the Domain property and appear across every bucket (521 of 1,000 rows in the redirect export). Scope item 2 lists the families to add but not the hosts. Every row needs a host dimension or the per-family denominators are wrong.

Reference numbers for the first live run

When step 1 of the prerequisites lands, the run should reproduce these from the API. They are the 2026-09-24 hand-parse, so a live run that disagrees means the collector is wrong, not the baseline:

Export	Reported	Sampled	Already noindex or 3xx today	Live 200, no noindex
Crawled, currently not indexed	1,486	1,000	~825	178
Not found (404)	666	666 (complete)	428	— (237 still 404)
Page with redirect	1,685	1,000	all expected	0

Note for scope item 3: the crawled and redirect exports are Google's example-URL cap of 1,000, with unspecified ordering. They are not random samples. The report must not extrapolate a sampled share to the reported total without saying so — the 404 export is the only one of the three that is complete.

[koala73]

Agent half is built and open as #8619 — the service account is now the only blocker

Everything in "Scope (agent)" items 1-5 is implemented and tested against recorded fixtures. Acceptance criteria 1, 2 and 3 are met; criterion 4 (one live run) is the only one outstanding, and it needs the prerequisites below.

# works today, no credential, deterministic
node scripts/seo-gsc-collect.mjs --fixtures tests/fixtures/gsc/

# works the moment the service account exists
node scripts/seo-gsc-collect.mjs --live

What you need to do — about 15 minutes

1. Create a Google Cloud service account with the Search Console API enabled.
2. In Search Console, add its email as a Restricted user on the worldmonitor.app Domain property.
3. Base64 the JSON key into GSC_SERVICE_ACCOUNT_JSON in .env.local, and set GSC_PROPERTY. Both are documented in .env.example on the PR.
4. Add the same secret to GitHub Actions if the weekly job should run in CI. The workflow is gated on it and skips cleanly when absent, so merging before you do this changes nothing.
5. Reply here with the property ID format only (sc-domain:worldmonitor.app or a URL prefix) — no secret.

One finding from building it that is worth knowing now

The live path fetches /blog/sitemap-index.xml, which is a <sitemapindex> rather than a urlset, and the inventory builder was counting an index by its children instead of as pages. The first live run would have reported the entire blog family as undeclared while it was earning impressions — a false diagnosis of exactly the kind this issue exists to prevent. Fixed in 0cbfddcce4 with a stubbed-fetch test. It could not have surfaced from fixtures alone, which is a reminder that the live run is a real acceptance step and not a formality.

What the first live run should be checked against

The 2026-09-24 hand-parse, which is not hardcoded anywhere — it sits in the fixture manifest as reportedTotals on a 15-URL recording, so the run reproduces or contradicts it independently:

Export	Reported	Sampled	Already noindex or 3xx	Live 200, no noindex
Crawled, currently not indexed	1,486	1,000	~825	178
Not found (404)	666	666 (complete)	428	— (237 still 404)

And the question that unblocks #8602 and #8605: impressions and clicks per indexed URL, per family, with crises against countries and chokepoints as controls, plus whether any non-brand query reaches an impression at all today.

Two judgement calls that would benefit from your view

/country-instability-index/ is mapped to country_pages, and /accuracy/ to research. Both are documented in scripts/lib/seo-url-taxonomy.mjs; neither is forced. Say so and either moves.

[koala73]

Prerequisites 1–3 are done and verified live on 2026-09-25.

• The property is sc-domain:worldmonitor.app. The service account has siteRestrictedUser, and Restricted is enough for all three calls below.
• GSC_SERVICE_ACCOUNT_JSON (base64 service-account JSON) is set in .env.local and as a GitHub Actions secret. The GitHub copy can't be read back, so the first workflow run is what proves it matches.
• Token exchange with the webmasters.readonly scope returns 200. sites.list returns 200. searchAnalytics.query (dimension page, 2026-08-26..2026-09-22) returns 200. urlInspection.index.inspect returns 200: /crises/sudan-conflict/ is PASS, "Submitted and indexed".

Implementation note: build the JWT with jose (SignJWT + importPKCS8, already in node_modules). A hand-rolled crypto.createSign JWT got a generic 400 invalid_request from the token endpoint with the same key.

Early signal for the report: in that 28-day window, / alone has 264,028 impressions and 91,845 clicks. /dashboard is next at 75,847 impressions, and the top blog post has 10,542. Search traffic is heavily brand and homepage.