Add dynamic sitemap.xml and robots.txt (#1252)

[jonfroehlich]

Closes #1252.

What this does

Adds a dynamic XML sitemap and an environment-aware robots.txt, both served by Django (no web-server access required).

The sitemap is generated from our querysets via django.contrib.sitemaps, so it's always current — no static file to regenerate when content changes. See the issue discussion for why this beats the crawler/plugin/manual-file approaches.

Changes

• settings.py — add 'django.contrib.sitemaps' to INSTALLED_APPS. Deliberately not installing django.contrib.sites: without it the framework falls back to RequestSite, deriving the domain from the request host, so the sitemap emits the correct domain across local/test/prod with no per-env config and no extra DB migration.
• website/sitemaps.py — four sitemaps:
  • StaticViewSitemap — /, /people/, /publications/, /projects/, /awards/, /news/
  • ProjectSitemap — visible projects only (is_visible=True), lastmod from updated
  • PersonSitemap — people with ≥1 Position, excluding the placeholder url_name, lastmod from bio_datetime_modified
  • NewsSitemap — news items, lastmod from date
  • Publications have no per-publication detail page, so they're covered by the static /publications/ entry.
• website/views/robots.py — robots.txt: PROD allows crawling and advertises the sitemap; every other environment (notably the TEST server, DJANGO_ENV=TEST) returns Disallow: / so the test site isn't indexed and can't compete with prod in search results.
• makeabilitylab/urls.py — route sitemap.xml and robots.txt above the website.urls catch-all so it can't shadow them.
• website/tests/test_sitemap.py — 9 regression tests (XML content-type, static pages present, visible-vs-private project, person with/without position, news item, robots PROD vs non-PROD).

Sample output (local dev server)

GET /sitemap.xml (excerpt):

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" ...>
<url><loc>http://localhost:8571/</loc><changefreq>weekly</changefreq><priority>0.8</priority></url>
<url><loc>http://localhost:8571/publications/</loc>...</url>
<url><loc>http://localhost:8571/project/sidewalk/</loc><lastmod>2024-09-19</lastmod>...</url>
<url><loc>http://localhost:8571/member/jonfroehlich/</loc><lastmod>2023-12-30</lastmod>...</url>
<url><loc>http://localhost:8571/news/project-sidewalk-launch/</loc><lastmod>2016-08-31</lastmod>...</url>
</urlset>

GET /robots.txt (dev/test — non-PROD):

User-agent: *
Disallow: /

GET /robots.txt (PROD):

User-agent: *
Allow: /

Sitemap: https://makeabilitylab.cs.washington.edu/sitemap.xml

Testing

• New website/tests/test_sitemap.py — 9 tests, all pass.
• Full suite green: Ran 168 tests ... OK.
• Verified live via curl against the local dev server.

Note (no UI change)

This is a non-visual change, so no before/after screenshots — sample output above stands in. After this merges and deploys to prod, the sitemap still needs to be submitted once in Google Search Console (a manual, account-side step).

🤖 Generated with Claude Code