From 2a6112034e5b1551e970ca5adae5ddc5effca283 Mon Sep 17 00:00:00 2001 From: Jon Froehlich Date: Mon, 15 Jun 2026 15:33:43 -0700 Subject: [PATCH] docs: document Apache static-vs-Django routing (robots.txt & sitemap.xml) Records a non-obvious server behavior discovered while shipping the sitemap (#1252): Apache serves real files directly and only proxies to Django for paths with no matching file. So /robots.txt is the static repo-root file (edit it to change crawler rules), /sitemap.xml is dynamic (Django), Django sees requests as http (sitemap pins protocol=https), and the test host is noindex'd by an Apache X-Robots-Tag header. Added under Server Configuration in DEPLOYMENT.md + TOC entry. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/DEPLOYMENT.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index b065333e..d79ae34b 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -16,6 +16,7 @@ This document covers the Makeability Lab website's production infrastructure, de - [Server Configuration](#server-configuration) - [Configuration File](#configuration-file) - [Environment Variables](#environment-variables) + - [Static Files vs. Dynamic Requests (Apache routing)](#static-files-vs-dynamic-requests-apache-routing) - [Debugging \& Logging](#debugging--logging) - [Log Files](#log-files) - [Accessing Logs via Web](#accessing-logs-via-web) @@ -127,6 +128,15 @@ Django reads database credentials and secret keys from `config.ini`. This file: Production-specific settings are configured in `settings.py` using values from `config.ini`. Local development uses different defaults specified in `docker-compose-local-dev.yml`. +### Static Files vs. Dynamic Requests (Apache routing) + +On both servers, Apache sits in front of the Django container. It serves any URL that maps to a **real file** directly, and only **proxies to Django** (over plain HTTP) for paths that have no matching file. This has a few non-obvious consequences: + +- **`/robots.txt` is a static file** — it is the top-level [`robots.txt`](../robots.txt) committed in the repo root, served by Apache from the project checkout. To change crawler rules or the advertised sitemap, edit that file and deploy. A Django view/route for `/robots.txt` would be dead code on the servers (it only runs under local `runserver`, which diverges from production). +- **`/sitemap.xml` is dynamic** — no such file exists, so Apache proxies it to Django's `django.contrib.sitemaps` (see `website/sitemaps.py`), which builds the XML from the database on each request. +- **Django sees requests as HTTP, not HTTPS.** Apache terminates TLS and proxies to Django over plain HTTP, so `request.scheme` is `http`. Any code that builds absolute URLs from the request (e.g. the sitemap) must force `https` explicitly — the sitemaps do this via `protocol = "https"`. +- **The test server is never indexed.** Apache stamps `X-Robots-Tag: noindex, nofollow` on every response from the test host, so staging stays out of search engines regardless of its `robots.txt`. + ## Debugging & Logging ### Log Files