Merge pull request #1444 from makeabilitylab/1443-nightly-db-backup

Nightly pg_dump into the postgres volume, with backup health reporting (#1443)

Author: jonfroehlich

CLAUDE.md

@@ -132,6 +132,8 @@ the existing viewset/serializer pattern and keep `v1` fields additive-only
 - `TIME_ZONE = 'America/Los_Angeles'`. `ML_WEBSITE_VERSION` in settings is shown in the admin header and used in release tagging.
 - **Logging (#1283):** `debug.log` lives at `LOG_DIR/debug.log`, where `LOG_DIR` is `$ML_LOG_DIR` or `<BASE_DIR>/media` (`/code/media` in the container). Keep it inside `MEDIA_ROOT` — that's the tree bind-mounted to the shared CSE filesystem, so it's what makes the log readable over SSH at all. `ML_LOG_DIR` is unset everywhere today; it exists for non-`/code` hosts. `MEDIA_ROOT` is web-served, so never log anything sensitive. If the dir isn't writable the file handler degrades to a `NullHandler` rather than crashing `django.setup()`, and since there's no console on the servers that state surfaces via `/version.json` (`log_to_file`) and a superuser-only callout on the admin dashboard. Rotation uses `concurrent-log-handler` (#1439) because Gunicorn's 3 workers share one file — the stdlib `RotatingFileHandler` races on rollover across processes. Its lock file goes in a per-uid temp dir (`/tmp/makelab-log-locks-<uid>`), never the web-served media root and never shared across users. If the package isn't importable (the bind-mounted checkout can be ahead of the image's site-packages) or no lock dir is usable, the handler degrades to the stdlib `RotatingFileHandler` instead of crashing `django.setup()`; `/version.json` reports which one is live as `log_rotation`. `django.db.backends` is pinned to INFO so per-query SQL doesn't dominate the log (or the lock).
 
+- **Database backups (#1443):** a `db-backup` sidecar service (in *both* compose files) runs `scripts/pg_backup.sh` hourly; the script is a single pass that writes one dated `pg_dump | gzip` per UTC day to `pg_backups/` **inside the postgres data volume**, prunes past `BACKUP_RETENTION_DAYS` (14, never the newest dump), and writes `status.json` to a small volume the website container mounts read-only. Dumps go inside the data volume on purpose — that's the volume CSE IT snapshots, so every snapshot carries a consistent restore point. Two things are load-bearing: the sidecar's `entrypoint` **must** stay overridden (the postgres image's own entrypoint would start a second server on that `PGDATA`), and the scheduling loop lives in the compose file rather than in the script (`docker compose up -d` only recreates containers whose *config* changed, so a loop inside the bind-mounted script would run stale code forever after a deploy). Django only ever *reads* the status: `website/utils/backup_status.py` → `/version.json` (`backup_ok`), a superuser callout on the admin dashboard shown only when stale/failed, and a panel on Data Health. Dumps contain `Person.email` — never move one under a web-served path. Restore procedure, the `initdb`-refuses-a-non-empty-directory gotcha, and the two Docker-based restore harnesses (`scripts/test_backup_restore*.sh`): `docs/BACKUPS.md`.
+
 ### Container startup side effects (`docker-entrypoint.sh`)
 
 Every container start runs, in order: `collectstatic` → `makemigrations` → `migrate` → `makemigrations website` → `migrate website` → `delete_unused_files` → `thumbnail_cleanup` → `generate_slugs_for_old_news_items` → `auto_close_project_roles` → `remove_year_from_forum_name` → `fix_sortedm2m_columns` → `seed_sidewalk_participants` → `warm_api_thumbnails` → `runserver 0.0.0.0:8000`. The repeated `makemigrations website` step is intentional (fixes first-run issues). If you add a one-shot data migration command under `website/management/commands/`, decide whether it belongs in this startup sequence.

docker-compose-local-dev.yml

@@ -72,6 +72,66 @@ services:
       timeout: 5s
       retries: 5
 
+  # ===========================================================================
+  # DATABASE BACKUP SERVICE (#1443)
+  # ===========================================================================
+  # Writes a dated, gzipped pg_dump into the postgres data volume once a day.
+  #
+  # On the servers this exists because volume-level snapshots of a *live*
+  # postgres data directory are only probably restorable; the dump is the
+  # guaranteed-consistent restore point. It runs locally too so that the same
+  # code path is exercised in development rather than only in production.
+  #
+  # Two things here are load-bearing and should not be "simplified":
+  #   1. `entrypoint` is overridden. Left alone, the postgres image's own
+  #      entrypoint would try to start a second database server on this PGDATA.
+  #   2. The scheduling loop lives here, not inside pg_backup.sh, because
+  #      `docker compose up -d` only recreates containers whose config changed.
+  #      A loop inside the bind-mounted script would keep running stale code
+  #      after a deploy.
+  #
+  # To force a backup immediately instead of waiting for the next pass:
+  #   docker compose -f docker-compose-local-dev.yml exec db-backup \
+  #     bash /backup-scripts/pg_backup.sh
+  #
+  # To verify and restore, see docs/BACKUPS.md.
+  db-backup:
+    # Same image as `db` so the pg_dump binary version matches the server.
+    image: 'postgres:16'
+    restart: always
+
+    environment:
+      # libpq connection settings; must match the `db` service above.
+      - PGHOST=db
+      - PGUSER=admin
+      - PGPASSWORD=password
+      - PGDATABASE=makeability
+      # Delete dumps older than this, but never the most recent one.
+      - BACKUP_RETENTION_DAYS=14
+      # How long to wait between passes, and after a failed pass.
+      - BACKUP_POLL_SECONDS=3600
+      - BACKUP_RETRY_SECONDS=300
+
+    volumes:
+      # The database volume, so dumps land inside the thing that gets snapshotted.
+      - postgres-data:/var/lib/postgresql/data
+      # Small shared volume for the status file Django reads.
+      - backup-status:/var/backup-status
+      # The backup script, mounted as a directory so edits survive git checkouts.
+      - ./scripts:/backup-scripts:ro
+
+    # `sleep ... & wait $!` plus the leading `trap`, instead of a plain
+    # foreground `sleep`, so `docker compose stop`/`down` return promptly:
+    # bash only acts on a trapped signal between foreground commands, so a
+    # SIGTERM arriving during a plain `sleep 3600` would sit unhandled for up
+    # to an hour and this container would always eat the full stop grace
+    # period before Docker escalates to SIGKILL.
+    entrypoint: ["/bin/bash", "-c", "trap 'exit 0' TERM INT; while true; do if bash /backup-scripts/pg_backup.sh; then sleep \"$${BACKUP_POLL_SECONDS}\" & wait $!; else sleep \"$${BACKUP_RETRY_SECONDS}\" & wait $!; fi; done"]
+
+    depends_on:
+      db:
+        condition: service_healthy
+
   # ===========================================================================
   # WEBSITE SERVICE (Django Application)
   # ===========================================================================
@@ -94,6 +154,9 @@ services:
       # This enables "live reloading"—when you edit local files, the changes
       # are immediately visible inside the container without rebuilding.
       - .:/code
+      # Read-only view of the backup status file so Django can report backup
+      # health on the admin dashboard and /version.json (#1443).
+      - backup-status:/var/backup-status:ro
 
     healthcheck:
       # Check if Django is responding
@@ -158,4 +221,6 @@ services:
 # Run 'docker volume ls' to see all volumes.
 # Run 'docker volume rm postgres-data' to delete (WARNING: destroys all data).
 volumes:
-  postgres-data:
+  postgres-data:
+  # Disposable: holds only the backup status.json, regenerated every pass.
+  backup-status:

docker-compose.yml

@@ -10,6 +10,45 @@ services:
       - POSTGRES_PASSWORD=password
     volumes:
       - db-data:/var/lib/postgresql/data
+  # Consistent daily pg_dump into the postgres data volume (#1443). CSE IT's
+  # ZFS snapshots capture the raw volume, which is only *probably* restorable
+  # for a live database; this writes a guaranteed-consistent restore point
+  # inside that same volume so every snapshot carries one. See docs/BACKUPS.md.
+  #
+  # Uses the same image as `db` so pg_dump's version always matches the server.
+  # `entrypoint` MUST stay overridden: the postgres image's own entrypoint would
+  # otherwise try to bring up a second server on this PGDATA.
+  #
+  # The scheduling loop lives here rather than inside pg_backup.sh on purpose —
+  # `docker compose up -d` only recreates a container whose config changed, so a
+  # loop inside the bind-mounted script would keep running the old code forever
+  # after a deploy. Re-invoking the script each pass makes logic changes deploy.
+  #
+  # The `sleep ... & wait $!` (rather than a plain foreground `sleep`) and the
+  # leading `trap` are so `docker compose stop`/`down` return promptly: bash
+  # only acts on a trapped signal between foreground commands, so with a plain
+  # `sleep 3600` a SIGTERM sent while idle would sit unhandled for up to an
+  # hour and this container would always eat the full stop grace period before
+  # Docker escalates to SIGKILL. Backgrounding the sleep and `wait`-ing on it
+  # makes the trap run immediately instead.
+  db-backup:
+    image: "${POSTGRES_IMAGE:-postgres}"
+    restart: always
+    environment:
+      - PGHOST=db
+      - PGUSER=admin
+      - PGPASSWORD=password
+      - PGDATABASE=makeability
+      - BACKUP_RETENTION_DAYS=${BACKUP_RETENTION_DAYS:-14}
+      - BACKUP_POLL_SECONDS=${BACKUP_POLL_SECONDS:-3600}
+      - BACKUP_RETRY_SECONDS=${BACKUP_RETRY_SECONDS:-300}
+    volumes:
+      - db-data:/var/lib/postgresql/data
+      - backup-status:/var/backup-status
+      - ./scripts:/backup-scripts:ro
+    entrypoint: ["/bin/bash", "-c", "trap 'exit 0' TERM INT; while true; do if bash /backup-scripts/pg_backup.sh; then sleep \"$${BACKUP_POLL_SECONDS}\" & wait $!; else sleep \"$${BACKUP_RETRY_SECONDS}\" & wait $!; fi; done"]
+    depends_on:
+      - db
   website:
     environment:
       - DJANGO_ENV=${DJANGO_ENV:-TEST}
@@ -23,6 +62,7 @@ services:
       - .:/code
       - ${MEDIA_PATH:-./media}:/code/media
       - ${CONFIG_PATH:-./config-dev.ini}:/code/config.ini # for loading vars into ConfigParser in Django
+      - backup-status:/var/backup-status:ro # read-only: Django reports backup health (#1443)
     depends_on:
       - db
     command: ["./docker-entrypoint.sh", "db", "python", "manage.py"]
@@ -32,3 +72,8 @@ volumes:
   db-data:
     external: true
     name: "makeabilitylabcswashingtonedu_${POSTGRES_VOLUME:-postgres-data}"
+  # Holds only the backup status.json that Django reads to report backup health.
+  # Intentionally NOT external and NOT something CSE IT needs to back up: it is
+  # disposable state, regenerated on the next backup pass. The dumps themselves
+  # live in db-data above, which is the volume that gets snapshotted.
+  backup-status:

docs/BACKUPS.md

@@ -0,0 +1,196 @@
+# Database backups and restore
+
+How the Makeability Lab website's data is backed up, how to check that it's
+actually happening, and how to restore it. See issue
+[#1443](https://github.com/makeabilitylab/makeabilitylabwebsite/issues/1443).
+
+**Read the [Restoring](#restoring) section before you need it.** The one step
+people get wrong under pressure is documented there: Postgres will not
+initialize into a non-empty data directory, and the dumps live *inside* that
+directory.
+
+## What is backed up, and by whom
+
+| Data | Where it lives | How it's protected |
+| --- | --- | --- |
+| Uploaded media (PDFs, images) | `/cse/web/research/makelab/www[-test]/` on the shared CSE filesystem | CSE IT's standard snapshot schedule — hourly, weekly, monthly, plus off-site to UW's lolo service. Retained 1 year. Plain files, so a snapshot is always consistent. |
+| Code and schema | git | GitHub |
+| **Database contents** | the `db` container's named volume (`db-data` → `/var/lib/postgresql/data`) | Two tiers, below |
+
+The database is the only piece that needs special handling, because a
+filesystem-level snapshot of a **live** Postgres data directory is not
+guaranteed to be transaction-consistent. Restoring one behaves like recovering
+from a hard power cut — usually fine, occasionally not.
+
+So there are two tiers:
+
+| Tier | Cadence | Guarantee |
+| --- | --- | --- |
+| CSE IT's ZFS snapshot of the raw volume | hourly | *Probably* restorable. Postgres crash recovery is designed for exactly this case. |
+| Our `pg_dump`, written **into** that volume | daily | Guaranteed consistent restore point. |
+
+Because the dump lives inside the volume that gets snapshotted, every snapshot
+automatically carries a known-good dump. A snapshot from six months ago contains
+that day's dump, which is why in-volume retention only needs to be 14 days.
+
+> **The dump cadence is the guaranteed worst-case RPO, and more snapshots don't
+> improve it.** Every hourly snapshot taken between two dumps contains the *same*
+> dump. Hourly snapshots give more copies of one restore point, not more restore
+> points.
+
+## How it works
+
+A `db-backup` sidecar service in `docker-compose.yml` runs
+[`scripts/pg_backup.sh`](../scripts/pg_backup.sh) once an hour. The script is a
+single pass: if today's dump doesn't exist yet it makes one, prunes anything past
+retention, and writes a status file.
+
+- **Dumps:** `/var/lib/postgresql/data/pg_backups/makeability-YYYY-MM-DD.sql.gz`
+  (UTC date, mode 0600).
+- **Retention:** 14 days, but the newest dump is *never* pruned regardless of
+  age — otherwise a backup that had been failing for longer than the retention
+  window would end with pruning deleting the last good dump too.
+- **Status:** `status.json` on a small shared volume that the website container
+  mounts read-only.
+
+Two things in the compose config are load-bearing and shouldn't be "simplified":
+
+1. `entrypoint` is overridden. Left alone, the postgres image's own entrypoint
+   would try to start a second database server on that `PGDATA`.
+2. The scheduling loop lives in `docker-compose.yml`, not inside
+   `pg_backup.sh`. `docker compose up -d` only recreates a container whose
+   *config* changed, so a loop inside the bind-mounted script would keep running
+   stale code after a deploy — and there's no shell access to restart it by hand.
+
+## Checking that backups are actually running
+
+Because the dumps sit in a Docker volume on a host nobody has a shell on — and
+inside a `PGDATA` the website container can't even traverse — the status file is
+the only way to observe this. Three places surface it:
+
+- **`/version.json`** — `backup_ok`, `last_backup_at`, `backup_age_hours`,
+  `backup_count`. No auth needed; use this for any external check.
+- **Admin dashboard** — a superuser-only warning callout, shown *only* when
+  backups are stale or failing.
+- **Admin → Data Health** — a panel with last success, age, size, how many
+  dumps are retained, and the last error.
+
+"Stale" means the newest dump is more than 36 hours old (`BACKUP_STALE_AFTER_HOURS`).
+That's 1.5× the daily cadence, so one missed run doesn't cry wolf but a second
+consecutive one does.
+
+## Restoring
+
+### The gotcha, first
+
+The dumps live at `pg_backups/` **inside** the Postgres data directory. `initdb`
+refuses to initialize into a non-empty directory, so you cannot wipe the database
+and leave the backups sitting there. **Copy the dump out of the volume first.**
+This is pinned by a test in `scripts/test_backup_restore.sh` so the warning can't
+silently go stale.
+
+### Restoring locally (development, or verifying a dump)
+
+```bash
+# 1. Get the dump out of the volume and onto your machine.
+docker compose -f docker-compose-local-dev.yml cp \
+  db-backup:/var/lib/postgresql/data/pg_backups/makeability-2026-08-07.sql.gz .
+
+# 2. Stop the stack and destroy the database volume.
+docker compose -f docker-compose-local-dev.yml down
+docker volume rm makeabilitylabwebsite_postgres-data
+
+# 3. Bring just the database back up on a fresh, empty volume.
+docker compose -f docker-compose-local-dev.yml up -d db
+
+# 4. Restore.
+gunzip -c makeability-2026-08-07.sql.gz | \
+  docker compose -f docker-compose-local-dev.yml exec -T db \
+    psql -v ON_ERROR_STOP=1 -U admin -d makeability
+
+# 5. Start the site and confirm Django agrees the database is complete.
+docker compose -f docker-compose-local-dev.yml up -d
+docker compose -f docker-compose-local-dev.yml exec website python manage.py migrate --check
+```
+
+Step 5 is the real test. `migrate --check` exits non-zero if Django thinks
+migrations are pending, which is how you'd catch a restore that brought back
+tables but not the `django_migrations` table.
+
+### Restoring production or test
+
+**This requires someone with Docker access on the host** — `grabthar` for
+production, `docker-test2` for test. The maintainer does not have that (see the
+server access model in `CLAUDE.md`), so a production restore means opening a
+ticket with UW CSE IT. Send them this section.
+
+The steps are the same as above, against `docker-compose.yml` and the external
+volume `makeabilitylabcswashingtonedu_postgres16-data`. Before destroying
+anything:
+
+1. **Copy the chosen dump somewhere off the volume first.** If the volume itself
+   is the problem, ask CSE IT to recover the dump from a ZFS snapshot or from
+   lolo instead — the dump inside a snapshot is exactly what this whole scheme
+   exists to provide.
+2. **Take a copy of the current broken volume before overwriting it.** A
+   corrupt database still contains data; a hasty restore over the top of it
+   destroys any chance of salvaging rows the dump predates.
+3. Restore, then confirm via `/version.json` and by loading the site.
+
+There's no ad-hoc "back up right now" button, deliberately — see the follow-up
+note in #1443. If you need a fresh dump before something risky and you can't
+reach the host, the practical options are to wait for the next pass or ask CSE
+IT to run one.
+
+## Testing the backups
+
+Two harnesses, both self-contained and namespaced so they never touch your real
+stack, database, or volumes. **An untested backup is not a backup** — run these
+after any change to `pg_backup.sh` or the compose wiring.
+
+```bash
+# Mechanics: dump → destroy → restore, on a synthetic schema built to break a
+# naive dump (unicode, embedded quotes and newlines, NULLs, binary columns,
+# a 540 KB row, views, foreign keys, sequences). ~1 minute.
+bash scripts/test_backup_restore.sh
+
+# The real thing: builds the actual Makeability Lab schema with `migrate`, seeds
+# through the ORM (sortedm2m through-tables, rich text), backs up, destroys the
+# volume, restores, and asserts Django accepts the result — `migrate --check`
+# and `manage.py check` both pass. Needs a built website image. ~3 minutes.
+bash scripts/test_backup_restore_django.sh
+```
+
+The Django-side unit tests (status file parsing, staleness, failure reporting)
+run in the normal suite:
+
+```bash
+python manage.py test website.tests.test_backup_status --settings=makeabilitylab.settings_test
+```
+
+## Security
+
+The dumps contain personal data — `Person.email`, which is deliberately withheld
+from the public API. They are written mode 0600 into a Docker volume.
+
+**Never move a dump under `media/`, `static/`, or any other web-served path.**
+Everything under those is publicly downloadable. The status file is safe to
+surface in the admin because it carries no row data.
+
+## Configuration
+
+Set on the `db-backup` service in `docker-compose.yml`:
+
+| Variable | Default | Meaning |
+| --- | --- | --- |
+| `BACKUP_RETENTION_DAYS` | 14 | Delete dumps older than this (never the newest). |
+| `BACKUP_MIN_KEEP` | 1 | Dumps always kept regardless of age. |
+| `BACKUP_POLL_SECONDS` | 3600 | Time between passes. |
+| `BACKUP_RETRY_SECONDS` | 300 | Time between passes after a failure. |
+
+Django-side, in `settings.py`:
+
+| Setting | Default | Meaning |
+| --- | --- | --- |
+| `BACKUP_STATUS_FILE` | `/var/backup-status/status.json` | Where to read status from (`ML_BACKUP_STATUS_FILE`). |
+| `BACKUP_STALE_AFTER_HOURS` | 36 | Age at which the dashboard warns (`ML_BACKUP_STALE_AFTER_HOURS`). |