Block search engines from indexing OVS - #1576
Merged
Merged
Conversation
Adds a noindex/nofollow meta tag to base.html (inherited by every page) and fixes robots.txt, which had an empty Disallow value that was actually permitting all crawling. Fixes mitodl/hq#12798
There was a problem hiding this comment.
Pull request overview
Adds site-wide search-engine indexing controls through HTML metadata and crawler directives.
Changes:
- Adds
noindex, nofollowto the application base template. - Blocks crawling through
robots.txt. - Adds regression tests for both directives.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
ui/templates/base.html |
Adds robots metadata to application pages. |
static/robots.txt |
Disallows crawler access site-wide. |
ui/views_test.py |
Tests metadata rendering. |
odl_video/robots_test.py |
Tests crawler directives. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- Add RobotsTagMiddleware to set X-Robots-Tag: noindex, nofollow on every response, covering pages that don't extend base.html (e.g. Django admin), per review feedback. - Revert static/robots.txt for now: blocking all crawling immediately would prevent search engines from ever fetching pages to observe the noindex directive, leaving already-indexed pages stuck. Ship noindex first; add Disallow: / in a follow-up once existing pages have had time to drop out of the index. - Tighten robots.txt test to an exact content match (dropped for now since robots.txt is unchanged in this PR).
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.
Suppressed comments (1)
odl_video/settings.py:104
- Place this middleware first in the stack. With the current ordering,
SecurityMiddlewarecan return the default HTTP-to-HTTPS redirect before calling downstream middleware (SECURE_SSL_REDIRECTdefaults toTrueatodl_video/settings.py:48), so that response never receives the promisedX-Robots-Tagheader. Making this middleware outermost ensures it also decorates responses short-circuited by other Django middleware.
"odl_video.middleware.RobotsTagMiddleware",
umar8hassan
approved these changes
Aug 18, 2026
umar8hassan
left a comment
umar8hassan
left a comment
Contributor
There was a problem hiding this comment.
Looks Good 👍
Just make sure to leave a note on the issue that we have deliberately removed the Disallow from robots.txt for now. Maybe open a follow up sub-issue for that.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
<meta name="robots" content="noindex, nofollow">toui/templates/base.html, inherited by every page on the site (including error pages)RobotsTagMiddlewareto set theX-Robots-Tag: noindex, nofollowresponse header on every response, covering pages that don't extendbase.html(Django admin, DRF browsable API, etc.)Fixes mitodl/hq#12798
Phased rollout (see review discussion)
static/robots.txtis intentionally not changed in this PR. Blocking crawling outright (Disallow: /) immediately would prevent Google from ever re-fetching pages to observe the newnoindexdirective, which would leave already-indexed pages (like the one in the issue's screenshot) stuck in search results indefinitely instead of dropping out.Plan:
noindexonly, so search engines can still crawl, see the directive, and drop existing pages from their index.Disallow: /torobots.txtonce existing pages have had time to be recrawled and removed (or sooner, via a manual Search Console removal request for the specific already-indexed URL).Test plan
uv run pytest ui/views_test.py odl_video/— no new failures (pre-existing local-env failures unrelated to this change, due to missing local Postgres)uv run ruff format --check/uv run ruff check— cleandocker compose up: rendered<head>on a live page contains thenoindexmeta tag;/admin/login/response has theX-Robots-Tagheader