Tighten robots.txt to cut duplicate bot crawl

[ChristopherChudzicki]

What are the relevant tickets?

N/A

Description (What does it do?)

Rewrites the robots.txt rules (frontends/main/src/app/robots.ts) to stop crawlers from fetching URL variants that duplicate content they can already reach through canonical URLs, and blocks AI-training crawlers.

Every ?resource= drawer URL canonicalizes to /search?resource=<id>&resource_title=<slug>, and the resources sitemap enumerates every resource at exactly that URL — so crawler fetches of drawer overlays on /c/, /news, home, and faceted search are pure duplicate load. A 7-day Fastly log analysis found this crawl is the dominant source of origin-reaching traffic: bot resource-URL crawl is roughly half of all origin-reaching requests (Applebot alone fans out over ~2.5M /c/ + /news drawer URLs/week), and because each per-resource URL is fetched about once, the CDN can't absorb it. These are full page renders, so this is most of our real origin compute.

The new rules: allow only the canonical drawer form (/search?resource=) and the bare /search landing page; disallow resource-carrying URLs site-wide, internal search result pages (/search?q=/facets, per Google's own guidance), and _rsc router-prefetch payloads (never part of rendered or indexed content); extend the app-only disallows (/enrollmentcode/, /organization/, /website_content/drafts); exempt link-preview fetchers (Facebook/Twitter/Slack/etc.) via their own group so og: cards keep working on shared drawer URLs; and block AI-training crawlers (GPTBot, CCBot, meta-externalagent, Bytespider, ClaudeBot, Amazonbot) plus the Google-Extended/Applebot-Extended training opt-out tokens, which cost no search visibility.

Resource indexing is preserved: search engines keep the sitemap-enumerated canonical drawer URLs, and the drawer variants they lose all declare canonicals pointing at the form that stays crawlable.

How can this be tested?

Unit test pins both MITOL_NOINDEX branches and the exact rule groups: yarn test frontends/main/src/app/robots.test.ts.

For manual validation, run the app with MITOL_NOINDEX=false, fetch /robots.txt, and check sample URLs against the rules — /search?resource=123 and /search allowed; /search?q=python, /c/topic/physics/?resource=5, and /?resource=5 disallowed. Google Search Console's robots.txt tester (or any RFC 9309 checker) confirms the longest-match precedence: Allow: /search?resource= (17 chars) beats Disallow: /*?resource= (12) and Disallow: /search (7).

Additional Context

The Allow: /search?resource= rule is a literal prefix match, so it depends on resource staying the first query param in canonical drawer URLs — documented on resourceDrawerSearch in common/urls.ts, whose output was already pinned by urls.test.ts. Rule order within the group is Allow-first as a courtesy to naive first-match parsers; compliant crawlers use longest-match, where order is irrelevant.

This only affects robots-compliant crawlers. The disguised scraper traffic we've seen (rotating fake-browser UAs from a Tencent-cloud netblock) never fetches robots.txt and needs a Fastly rate-limit/ASN block, tracked separately. meta-externalads intentionally stays under the default group: on our logs it only fetches the UAI+B2C pages we advertise on Meta, so it keeps access to everything except drawer-overlay and _rsc URLs.

[github-actions]

OpenAPI Changes

No changes detected

View full changelog

Unexpected changes? Ensure your branch is up-to-date with main (consider rebasing).

[Copilot]

Pull request overview

This PR updates the Next.js robots.ts metadata route to reduce duplicate, origin-reaching crawler traffic by restricting crawlable URL variants (especially ?resource= drawer overlays) while preserving indexing via canonical /search?resource= URLs and the sitemap.

Changes:

• Replace the single default robots.txt rule set with multiple user-agent groups (default crawl rules, link-preview bots exemption, AI-training crawlers blocked).
• Add an explicit invariant note tying canonical drawer URL query-param ordering to the robots.txt allow rule.
• Add a unit test that pins both MITOL_NOINDEX branches and the exact generated rule groups.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
frontends/main/src/common/urls.ts	Documents the invariant that resource must remain the first query param in canonical drawer URLs.
frontends/main/src/app/robots.ts	Rewrites robots rules into multiple groups to block duplicate drawer overlays, internal search result pages, and _rsc payloads while allowing canonical drawer URLs.
frontends/main/src/app/robots.test.ts	Adds unit tests to pin the emitted robots metadata for indexing and no-indexing modes.

[ChristopherChudzicki]

Changed below to meta-externalads per https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.1

Crawlers set their own name, which is called a product token, to find relevant groups. The product token MUST contain only uppercase and lowercase letters ("a-z" and "A-Z"), underscores ("_"), and hyphens ("-").