Prevent next.learn.mit.edu from being indexed via robots.txt

[blarghmatey]

Description/Context

next.learn.mit.edu is the raw Kubernetes-hosted NextJS backend that Fastly proxies for learn.mit.edu. It should never be directly indexed by search engines — all public traffic and crawling should go through learn.mit.edu (the Fastly-fronted canonical URL).

Currently the app's robots.ts route handler uses the MITOL_NOINDEX environment variable to decide what to serve:

• MITOL_NOINDEX=true → Disallow: / (used in CI and QA environments)
• MITOL_NOINDEX=false → Allow: / with specific disallows (used in production)

Because production sets MITOL_NOINDEX=false, the robots.txt served at next.learn.mit.edu/robots.txt currently says Allow: / — making the domain crawlable. Google is actively indexing pages at next.learn.mit.edu as a result.

The ol-infrastructure repo is addressing this at the HTTP header level by injecting X-Robots-Tag: noindex, nofollow at the Traefik Gateway for next.learn.mit.edu (with Fastly stripping that header before forwarding responses to learn.mit.edu users). This issue covers the complementary robots.txt fix in the app itself.

Plan/Design

Modify frontends/main/src/app/robots.ts to check whether the incoming request's host matches the canonical public origin (NEXT_PUBLIC_ORIGIN, e.g. learn.mit.edu). If the host does not match the canonical origin, serve a Disallow: / robots.txt regardless of the MITOL_NOINDEX value.

Reference:

• NEXT_PUBLIC_ORIGIN is already available in the environment (https://learn.mit.edu in production)
• The Next.js robots.ts route handler has access to request headers via next/headers
• Google's documentation on X-Robots-Tag: https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag

[ChristopherChudzicki]

@blarghmatey Should we wait until we are sure search engines have de-indexed next.learn.mit.edu before we give search engines a hint not to crawl it?