Serve sitemap via static robots.txt; drop dead Django robots view (#1252) - #1308
Merged
Conversation
) Follow-up to the sitemap work. On the servers Apache serves the static top-level ./robots.txt from the project checkout and never forwards /robots.txt to Django, so the robots_txt view added earlier was dead code in every server environment (it only ran under local runserver, which diverges from prod). Confirmed via the test deploy: /sitemap.xml is served by Django (WSGIServer) while /robots.txt comes from Apache as a static file, and the served content matches this tracked ./robots.txt. - robots.txt: advertise the sitemap (Sitemap: line); keep allow-all crawling. - Remove website/views/robots.py, its re-export, and the /robots.txt route; add a comment in urls.py pointing future edits at the static file. - Drop the now-irrelevant robots view tests; sitemap tests unchanged. The dynamic sitemap (django.contrib.sitemaps) is unchanged and already verified on the test server. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jun 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #1307 (sitemap work). Refs #1252.
Why
The test deploy revealed how
/robots.txtis actually served:/sitemap.xml→ Django (Server: WSGIServer) — our dynamic sitemap works./robots.txt→ Apache static file (Server: Apache, withETag/Last-Modified/Accept-Ranges) served from the project checkout on the Docker host — Apache never forwards/robots.txtto Django.The served content matches the repo's tracked top-level
./robots.txt(whose git history — "changed the robots.txt to reflect the behavior we want for the production site" — confirms it's the production robots file). So the Djangorobots_txtview added in #1307 was dead code in every server environment; it only ran under localrunserver, which diverges from prod. The test-server "don't index" concern it tried to solve is already handled at the server level (Apache stampsX-Robots-Tag: noindex, nofollowon the whole test host).Changes
robots.txt— add theSitemap:directive (points crawlers like Bing/Google at the dynamic sitemap); keep allow-all crawling, matching the existing production behavior.website/views/robots.py, its re-export inwebsite/views/__init__.py, and the/robots.txtURL route. Added a comment inmakeabilitylab/urls.pypointing future edits at the static./robots.txt.The dynamic sitemap itself (
django.contrib.sitemaps) is unchanged and already verified live on the test server.Single source of truth
robots.txtis now unambiguously a static file served by Apache. To change crawler rules or the sitemap URL, edit the top-level./robots.txt.Testing
Ran 165 tests ... OK(3 fewer than before = the removed robots view tests)./robots.txtpicks up the newSitemap:line (confirming the repo file is the served file), then crawl-validate the sitemap before cutting a prod tag.Note (no UI change)
Non-visual change; no screenshots.
🤖 Generated with Claude Code