fix(seo): ~200 machine-readable URLs (resilience.json, *.openapi.yaml, .well-known/*) answer 200 with no X-Robots-Tag — the bulk of the actionable crawled-not-indexed population

[koala73]

Parent: #8602 · Evidence: #8602 (comment)

Outcome

Every non-HTML URL we publish for machines — corpus dataset downloads, OpenAPI documents, .well-known descriptors — answers 200 and carries X-Robots-Tag: noindex, follow. It stays fetchable for agents and stops occupying Google's index queue.

Why

The #8602 drilldown parsed the 2026-09-24 "Crawled – currently not indexed" export (1,000 of 1,486 sampled) and re-probed every row live. After removing rows that already carry noindex or already redirect, 178 of 1,000 answer 200 today with no noindex, and they are almost entirely machine-readable files:

103  /countries/<slug>/resilience.json
 20  other .json  (countries/<slug>/cii.json, countries/resilience-ranking.json,
                   chokepoints/<slug>/reference.json, chokepoints/status.json,
                   crises/<slug>/tracker.json, sources/search-index.json,
                   docs/snapshots/*.json, openapi.json,
                   .well-known/{agent-skills/index.json,mcp/server-card.json,webhook-sample.json})
  8  /docs/api/*.openapi.yaml
 18  markdown + txt  (.well-known/agent-skills/*/SKILL.md, llms.txt, api/llms.txt,
                      pricing.md, ai-search.md, security.txt)

Zero /countries/<slug>/ or /crises/<slug>/ HTML pages appear in the sample. This is the largest actionable population the coverage export produced, and it is in no existing child issue.

Two things make it a real defect rather than cosmetics:

1. Mintlify already solved this for its own twins and we did not for ours. https://www.worldmonitor.app/docs/pricing.md serves x-robots-tag: noindex; https://www.worldmonitor.app/pricing.md serves no such header. 25/25 probed Mintlify /docs/*.md twins carry it, 0 of ours do.
2. The pattern is already established in our own config. vercel.json sets X-Robots-Tag: noindex, follow on /stocks, /stocks/(.*), /stocks.md and /blog/rss.xml. This issue applies the same header to the rest of the same class.

Link: <…>; rel="canonical" is already present on these files (#4999) and does not help — a self-canonical tells Google which URL is authoritative, not that the document should be left out of the index.

Why not robots.txt

robots.txt is the wrong tool here and #7660's reasoning applies directly. /countries/<slug>/resilience.json is discovered two legitimate ways, both of which a Disallow would break:

href="/countries/iran/resilience.json"                                          # visible download link
"contentUrl":"https://www.worldmonitor.app/countries/iran/resilience.json"      # schema.org DataDownload

A disallowed URL is never fetched, so the DataDownload claim in our structured data would point at something Google is forbidden to verify. noindex keeps the file fetchable and the claim honest.

Scope

1. Data files — unambiguous, do these. Add X-Robots-Tag: noindex, follow in vercel.json headers for:

   • /countries/(.*)\.json, /chokepoints/(.*)\.json, /crises/(.*)\.json, /sources/(.*)\.json, /research/(.*)\.json
   • /docs/api/(.*)\.openapi\.yaml, /openapi.json, /openapi.yaml, /plugin.json
   • /docs/snapshots/(.*)\.json
   • /.well-known/(.*)\.json

   Match the existing entries' shape and placement. Do not touch robots.txt, robots.*.txt, sitemap*.xml, schemamap.xml or /blog/rss.xml — the sitemaps must stay plain, and rss already has the header.

2. Agent-facing markdown and llms surfaces — owner decision required, do not guess. llms.txt, llms-full.txt, api/llms.txt, the 12 sitemap-declared .md twins, agents.md, developers.md, and .well-known/agent-skills/*/SKILL.md are the GEO/AI-citation surface. Adding noindex is correct for Google, but it is not established that OAI-SearchBot, PerplexityBot or Claude-SearchBot treat X-Robots-Tag: noindex as "do not index" rather than "do not cite". Until that is known, changing these risks trading a cosmetic Search Console count for AI-search citability, which is the opposite of the tradeoff we want.

   Deliver this half as a written recommendation on this issue, not a code change: name each surface, state what is known about each crawler's handling, and propose the call. The safe interim step that needs no crawler assumption is dropping the 12 non-HTML entries from sitemap-main.xml (scripts/build-sitemap.mjs:95-106) — a Google sitemap is a request to index, and these are not pages we want ranked. llms.txt discovery does not depend on the XML sitemap; robots.txt and the Link headers carry it.

3. Gate. Extend tests/deploy-config.test.mjs (which already guards the X-Robots-Tag and Content-Signal contracts) so that every vercel.json headers source matching a data-file extension asserts noindex, and so sitemap-main.xml's generator cannot re-add a non-HTML <loc> without the test failing. Grep that file for the existing robots-tag helpers before adding a new one.

Out of scope

• /docs/_next/static/chunks/* — 503 of the 1,000 sampled rows, all already serving noindex. Mintlify's assets, nothing to do.
• Mintlify's /docs/*.md twins — already noindex.
• The 428 stale 404 records and 245 stale non-www rows in the same export — already fixed upstream, they age out.
• Anything about ranking. This issue removes noise from the index queue; it will not move impressions. See feat(seo): collect per-URL index state and per-family search performance from the Search Console API (weekly) #8606.

Acceptance criteria

• After deploy, curl -sI -A "<Googlebot UA>" https://www.worldmonitor.app/countries/iran/resilience.json shows x-robots-tag: noindex, follow and still 200. Same for cii.json, a chokepoints/*/reference.json, a crises/*/tracker.json and a /docs/api/*.openapi.yaml. Paste the output in the PR.
• The visible download link and the schema.org contentUrl on /countries/iran/ are unchanged, and the file still downloads. Structured data still validates.
• tests/deploy-config.test.mjs passes, and fails when the noindex entry for /countries/(.*)\.json is removed. Prove it by deleting that entry locally and pasting the red output.
• npm run typecheck and the touched npm run test:data suites pass.
• Scope item 2 is answered in a comment on this issue with a named recommendation per surface, whether or not any code ships for it.

Guardrails

• vercel.json is also mirrored by the nginx confs for self-hosting. Check whether deploy/ carries an equivalent robots-tag block before assuming vercel.json alone is the contract, and tests/deploy-config.test.mjs is the suite that catches the drift.
• Editing vercel.json near the CSP entries can change an inline-script hash. Run tests/deploy-config.test.mjs locally before pushing.
• Do not add a Disallow line. fix(seo): unbounded map-state URL space is burning crawl budget — 1,271 redirects and rising #7660 settled that robots.txt is a crawl control, not a canonicalization or de-indexing tool, and the DataDownload structured data depends on these URLs staying fetchable.

[koala73]

Scope item 2 answered: per-surface recommendation for the AI-citation surface

This is the acceptance criterion that asks for a named recommendation per surface rather than a code change. The premise that made item 2 a deferral is unverified and stays unverified: I checked, and neither this repo nor any vendor documentation I can point at states whether OAI-SearchBot, PerplexityBot or Claude-SearchBot treat X-Robots-Tag: noindex as "do not index" or as "do not cite". Google's handling is not in doubt; theirs is. Everything below is graded by whether it needs that unknown resolved.

A. Drop the 12 non-HTML entries from sitemap-main.xml — do it, no assumption required

scripts/build-sitemap.mjs:95-106 declares /pricing.md, /support.md, /ai-search.md, /developers.md, /mcp-server.md, /openapi.md, /sdks.md, /world-monitor.md, /api-versioning.md, /llms.txt, /llms-full.txt, /api/llms.txt.

An XML sitemap is a request to index. We are asking Google to index twelve files we do not want ranked, then treating its refusal as a defect. This needs no theory about AI crawlers at all: removing a <loc> does not affect fetchability, and none of the agent-discovery paths depend on the XML sitemap — robots.txt carries the Sitemap: directive, the files carry self-canonical Link headers (#4999), and llms.txt is discovered at its well-known path. Recommend: remove all 12. Pair it with the gate the issue asks for, so the generator cannot re-add a non-HTML <loc>.

B. llms.txt, llms-full.txt, api/llms.txt — leave alone

This is the AI-citation manifest and the single surface where the unknown actually bites. Cost of being wrong is asymmetric: noindexing them buys the removal of 7 rows from a coverage report, and risks the discovery surface the whole GEO effort is built on. Recommend: no noindex. Do A and accept the rows. Revisit only if #8606 shows them causing measurable harm, which counting rows in a report cannot demonstrate.

C. The 12 markdown twins (pricing.md, agents.md, developers.md, …) — noindex, and there is real evidence for it

These are markdown duplicates of HTML pages we do want indexed, which makes them a duplicate-content cost rather than a citation asset — an AI answering from /pricing.md and one answering from /pricing cite the same claims.

The evidence is not speculation: Mintlify already serves x-robots-tag: noindex on its own /docs/*.md twins (25/25 probed), and Mintlify's entire product thesis is docs discoverability for AI. A vendor with a direct commercial stake in AI citability shipped noindex on exactly this file class. That is the closest thing to a settled answer available, and it points the opposite way from the cautious default. Recommend: noindex these in a follow-up PR, separately from A so the two can be reverted independently.

D. .well-known/* — make it consistent, either way

The shipped PR noindexes .well-known/*.json (agent-card.json, ai-catalog.json, mcp/server-card.json, webhook-sample.json, agent-skills/index.json) while leaving .well-known/agent-skills/*/SKILL.md alone. That is incoherent: index.json is the index of those same SKILL.md files, so one is noindexed and the things it points at are not.

The case for noindexing all of it: these are consumed by runtime agent clients that fetch a known URL from a manifest and ignore robots directives entirely — they are not search-crawler surfaces, so noindex costs nothing real. The case against: SKILL.md is prose an assistant could quote, which puts it in bucket B.

Recommend: keep the .json manifests noindexed as shipped, and leave SKILL.md un-noindexed with B. That is the same data-vs-prose line the rest of the rule set draws, and it makes the split intentional rather than accidental. Worth a reviewer's explicit nod on the PR either way.

E. robots.txt, robots.*.txt, sitemap*.xml, schemamap.xml — never

A noindexed sitemap is simply dropped. Untouched, and the shipped gate asserts null on each so it stays that way.

How to know if this was right

None of the above is measurable from the coverage export, which is what got this tracker's original diagnosis wrong. The check belongs in #8606: after A ships, the 12 URLs should leave "Crawled – currently not indexed" without any family losing impressions or AI-referral sessions. If a family does lose them, B was the right call and C should be reverted.

Remaining work on this issue

• Item 1 — data-file headers (PR open)
• Item 3a — the noindex gate (PR open)
• Item 2 — A (sitemap removal) and C (markdown twins), as a follow-up PR
• Item 3b — the gate preventing sitemap-main.xml from re-adding a non-HTML <loc>; it belongs with A, since there is nothing to assert until the entries are gone

The open PR deliberately does not close this issue.

[koala73]

Correction to the scope-item-2 recommendation above: section D was wrong

My earlier comment recommended keeping .well-known/*.json noindexed while leaving SKILL.md alone, and justified it as a "data vs prose" line. That reasoning does not survive opening the file.

/.well-known/agent-skills/index.json is 17 kB, and its instructions field is this:

"instructions": "World Monitor is a live, machine-readable global-intelligence API and MCP server.
 Reach for it when a task needs fresh, correlated, structured data about the state of the world —
 JSON to reason over, not prose to re-parse.\n\nWhen to use these skills:\n- Use `fetch-news-digest`
 when the user asks what ..."

That is prose written to persuade an agent to call us. I sorted the surface by file extension when the property that matters is whether the content is agent-facing persuasion. On that test index.json is more sensitive than the SKILL.md files it lists, so section D had the split backwards. agent-card.json, ai-catalog.json and mcp/server-card.json are the same class.

Revised section D: leave all of .well-known/ alone. Reverted in 48adc8e40f on #8614, with tests/deploy-config.test.mjs now asserting those paths stay indexable so it cannot be re-added silently.

This also removes the need to resolve the crawler-semantics unknown for this decision. The benefit of noindexing .well-known was about one row in a coverage export; the cost was unquantified risk to agent discovery. The asymmetry decides it on its own, which is the same argument section B already made for llms.txt.

The recommendation table, corrected

Surface	Call	Why
The 12 non-HTML entries in sitemap-main.xml	Remove them	A sitemap is a request to index. Needs no assumption about any crawler.
llms.txt, llms-full.txt, api/llms.txt	Leave	The citation manifest. Asymmetric downside.
The 12 .md twins (pricing.md, agents.md, …)	Noindex, follow-up PR	Duplicates of HTML we want ranked, and Mintlify ships noindex on its own .md twins — the one piece of real vendor evidence available.
All of .well-known/, JSON included	Leave (corrected)	Agent-discovery surface, and index.json carries the pitch prose.
robots.txt, sitemap*.xml, schemamap.xml	Never	A noindexed sitemap is simply dropped.

What ships in #8614 is now only the genuinely inert set: corpus dataset downloads (the 103 resilience.json files and siblings), /docs/snapshots/*.json, /docs/api/*.openapi.yaml, and the three root service descriptors.