diff --git a/OPERATIONS.md b/OPERATIONS.md index fbc08406..49aa2c7d 100644 --- a/OPERATIONS.md +++ b/OPERATIONS.md @@ -14,15 +14,22 @@ The backup command creates a private, timestamped directory containing: - `manifest.txt`: format, timestamp, revision, and an explicit no-secrets claim; - `checksums.sha256`: integrity hashes for every artifact. -The destination must be an absolute path and cannot be `/` or the operator's home -directory. The command removes a partial destination if creation fails. It does not -encrypt artifacts, upload them off-site, or choose retention; use an approved encrypted -backup target with access controls before production. +The approved backup root and upload root must already exist as absolute canonical +directories. The command rejects filesystem root, the operator home directory, the +current working directory, any root whose literal path resolves through a symbolic-link +component, symbolic links anywhere inside the upload tree, hard-linked upload files, +and non-regular upload entries. This prevents a successful backup from referring to +files outside the intended upload tree or producing an archive that its own restore +workflow cannot safely consume. A partial timestamped destination is removed if backup +creation fails. + +The command does not encrypt artifacts, upload them off-site, or choose retention; +use an approved encrypted backup target with least-privilege access before production. ```bash DATABASE_URL='postgresql://…' \ -SYNAPSE_UPLOAD_DIR='/absolute/path/to/uploads' \ -SYNAPSE_BACKUP_DIR='/absolute/path/to/private/backups' \ +SYNAPSE_UPLOAD_DIR='/absolute/canonical/path/to/uploads' \ +SYNAPSE_BACKUP_DIR='/absolute/canonical/path/to/private/backups' \ npm run backup ``` @@ -103,15 +110,63 @@ Never point the database assertion at a different deployment from the HTTP targe ## Restore drill -The drill is deliberately destructive to its target database. Use a newly created, -empty disposable database and an empty disposable uploads directory. It refuses to -run without the exact confirmation value, verifies checksums and archive paths, -restores both stores, confirms Prisma migration state, and reconciles upload metadata. +Use a newly created, dedicated disposable database and an empty disposable uploads +directory. Do not target `postgres`, `template0`, or `template1`. The drill refuses to +run without the exact confirmation value. It canonicalizes the restore upload target +and rejects linked path components, then verifies backup checksums and fully +materializes archive name/type listings before checking traversal and member types. +Only ordinary regular-file and directory archive members are accepted; symbolic links, +hard links, FIFOs, devices, and other special members fail closed before extraction. +The complete listing is consumed from a file, so validation cannot be short-circuited +by a producer receiving SIGPIPE. + +Before import, the drill first proves that its `postgres` maintenance connection can +execute the exact idempotent `ALTER DATABASE … ALLOW_CONNECTIONS true` operation needed +for failure cleanup. If the maintenance database is unavailable, CONNECT is denied, +the target is missing, or the role lacks that permission, the drill aborts before +freezing the disposable target. It then creates one protected PostgreSQL session. That +session disables new connections to the disposable target, terminates other target +sessions, and refuses to continue if any competing connection remains. The same +protected session then runs the database preflight and applies SQL generated from the +backup, eliminating the race between an emptiness check and restore. A cleanup path +reconnects through the already-validated maintenance connection to re-enable target +connections if the protected session fails after closing them. + +The database preflight requires exactly one standard `public` schema owned by +`pg_database_owner`. It rejects unexpected `public` schema ACLs; outside the owner, +only ordinary non-grantable `USAGE` for `PUBLIC` is accepted. It also queries user +schemas, namespace-owned relations/functions/types/operators/text-search objects, +extensions, event triggers, publications/subscriptions, foreign-data objects, +transforms, large objects, custom access methods/casts, non-default languages, +pre-existing default ACL rows, and prepared transactions for the target database. +A separate system-schema check rejects namespace-owned objects with user-assigned OIDs +at or above PostgreSQL's `FirstNormalObjectId` boundary (`16384`) inside the +system-named schemas `pg_catalog` and `information_schema`; ordinary low-OID built-in +and initdb objects remain accepted. Any detected user-defined object, unsafe `public` +schema metadata, default privilege, prepared transaction, system-schema contamination, +or competing session fails the empty-target gate. + +Because the target is independently verified and isolated, restore does not use +`pg_restore --clean` and does not intentionally drop existing objects. The archive is +rendered to SQL with owner/privilege restoration disabled, then applied with psql +`ON_ERROR_STOP` inside the protected target session. Connections are re-enabled only +after that protected restore completes. + +After the upload archive is extracted, restore-only upload reconciliation verifies +that every restored attachment row has a matching regular file in the restored upload +directory before removing orphan files or rebuilding quota accounting. A database dump +that references a file missing from the archive therefore fails the drill instead of +being reported healthy. Normal production startup does not enable this completeness +check; it retains the standard `./uploads` reconciliation behaviour. + +The drill then confirms Prisma migration state and reconciles the exact restored upload +directory through the restore-only explicit argument. Normal production startup still +reconciles only its standard `./uploads` directory. ```bash SYNAPSE_BACKUP_PATH='/absolute/path/to/one/synapse-backup' \ RESTORE_DATABASE_URL='postgresql://…/synapse_restore_drill' \ -RESTORE_UPLOAD_DIR='/absolute/path/to/empty/restore-uploads' \ +RESTORE_UPLOAD_DIR='/absolute/canonical/path/to/empty/restore-uploads' \ RESTORE_CONFIRM='EMPTY_DISPOSABLE_DATABASE' \ npm run restore:drill ``` diff --git a/scripts/backup.sh b/scripts/backup.sh index 81fc4d70..9c1ee749 100755 --- a/scripts/backup.sh +++ b/scripts/backup.sh @@ -4,25 +4,78 @@ set -euo pipefail : "${DATABASE_URL:?DATABASE_URL is required}" : "${SYNAPSE_BACKUP_DIR:?SYNAPSE_BACKUP_DIR must be an absolute backup directory}" -case "$SYNAPSE_BACKUP_DIR" in - /*) ;; - *) echo "SYNAPSE_BACKUP_DIR must be absolute" >&2; exit 2 ;; -esac +canonical_existing_directory() { + TARGET_PATH="$1" node <<'NODE' +const fs = require('node:fs'); +const os = require('node:os'); +const path = require('node:path'); -if [[ "$SYNAPSE_BACKUP_DIR" == "/" || "$SYNAPSE_BACKUP_DIR" == "$HOME" ]]; then - echo "Refusing unsafe backup root" >&2 - exit 2 -fi +const input = process.env.TARGET_PATH; +if (!path.isAbsolute(input)) { + console.error(`Directory path must be absolute: ${input}`); + process.exit(2); +} +const resolved = path.resolve(input); +let stat; +try { + stat = fs.lstatSync(resolved); +} catch { + console.error(`Directory does not exist: ${resolved}`); + process.exit(2); +} +if (!stat.isDirectory()) { + console.error(`Path is not a directory: ${resolved}`); + process.exit(2); +} +const canonical = fs.realpathSync(resolved); +if (canonical !== resolved) { + console.error(`Directory path must not contain symbolic-link components: ${resolved}`); + process.exit(2); +} +if (resolved === path.parse(resolved).root || resolved === os.homedir() || resolved === process.cwd()) { + console.error(`Refusing unsafe directory root: ${resolved}`); + process.exit(2); +} +process.stdout.write(resolved); +NODE +} -upload_dir="${SYNAPSE_UPLOAD_DIR:-$PWD/uploads}" -if [[ ! -d "$upload_dir" ]]; then - echo "Upload directory does not exist: $upload_dir" >&2 - exit 2 -fi +backup_root="$(canonical_existing_directory "$SYNAPSE_BACKUP_DIR")" +upload_dir="$(canonical_existing_directory "${SYNAPSE_UPLOAD_DIR:-$PWD/uploads}")" + +UPLOAD_ROOT="$upload_dir" node <<'NODE' +const fs = require('node:fs'); +const path = require('node:path'); + +const root = process.env.UPLOAD_ROOT; +function walk(directory) { + for (const entry of fs.readdirSync(directory, { withFileTypes: true })) { + const fullPath = path.join(directory, entry.name); + const stat = fs.lstatSync(fullPath); + if (stat.isSymbolicLink()) { + console.error(`Upload tree contains a symbolic link: ${fullPath}`); + process.exit(2); + } + if (stat.isDirectory()) { + walk(fullPath); + continue; + } + if (!stat.isFile()) { + console.error(`Upload tree contains a non-regular file: ${fullPath}`); + process.exit(2); + } + if (stat.nlink !== 1) { + console.error(`Upload tree contains a hard-linked file: ${fullPath}`); + process.exit(2); + } + } +} +walk(root); +NODE umask 077 timestamp="$(date -u +%Y%m%dT%H%M%SZ)" -destination="$SYNAPSE_BACKUP_DIR/synapse-$timestamp" +destination="$backup_root/synapse-$timestamp" if [[ -e "$destination" ]]; then echo "Backup destination already exists: $destination" >&2 exit 2 diff --git a/scripts/reconcile-uploads.ts b/scripts/reconcile-uploads.ts index cbba276c..8daf2f87 100644 --- a/scripts/reconcile-uploads.ts +++ b/scripts/reconcile-uploads.ts @@ -1,11 +1,36 @@ import { mkdir, readdir, rm } from "node:fs/promises"; +import { homedir } from "node:os"; import path from "node:path"; import { prisma } from "../src/lib/prisma"; -const uploadDir = path.join(process.cwd(), "uploads"); -const trashDir = path.join(uploadDir, ".trash"); +const RESTORE_UPLOAD_OVERRIDE_CONFIRM = "restore-drill-disposable-upload-target"; + +function resolveUploadDir() { + const configured = process.argv[2]; + if (!configured) return path.join(process.cwd(), "uploads"); + + if (process.env.SYNAPSE_UPLOAD_DIR_CONFIRM !== RESTORE_UPLOAD_OVERRIDE_CONFIRM) { + throw new Error( + `explicit upload target requires SYNAPSE_UPLOAD_DIR_CONFIRM=${RESTORE_UPLOAD_OVERRIDE_CONFIRM}` + ); + } + if (!path.isAbsolute(configured)) { + throw new Error("explicit upload target must be an absolute path"); + } + + const resolved = path.resolve(configured); + const root = path.parse(resolved).root; + if (resolved === root || resolved === homedir() || resolved === process.cwd()) { + throw new Error("Refusing unsafe explicit upload target"); + } + + return resolved; +} async function main() { + const explicitRestoreTarget = Boolean(process.argv[2]); + const uploadDir = resolveUploadDir(); + const trashDir = path.join(uploadDir, ".trash"); await mkdir(uploadDir, { recursive: true }); // Query the authoritative database before removing anything. If the database @@ -19,8 +44,28 @@ async function main() { .filter((filename): filename is string => Boolean(filename)) ); + const uploadEntries = await readdir(uploadDir, { withFileTypes: true }); + const regularFiles = new Set( + uploadEntries.filter((entry) => entry.isFile()).map((entry) => entry.name) + ); + + // The explicit target is available only to the disposable restore drill. In + // that mode, reconciliation must prove completeness as well as remove stale + // files: every attachment row restored from PostgreSQL must have its matching + // regular file before any orphan cleanup can make the result look healthy. + if (explicitRestoreTarget) { + const missingReferenced = [...referenced].filter( + (filename) => !regularFiles.has(filename) + ); + if (missingReferenced.length > 0) { + throw new Error( + `restore upload target is missing ${missingReferenced.length} referenced attachment file(s)` + ); + } + } + let removedOrphans = 0; - for (const entry of await readdir(uploadDir, { withFileTypes: true })) { + for (const entry of uploadEntries) { if (!entry.isFile() || referenced.has(entry.name)) continue; await rm(path.join(uploadDir, entry.name), { force: true }); removedOrphans++; diff --git a/scripts/restore-drill.sh b/scripts/restore-drill.sh index 8a936e26..41bf5085 100755 --- a/scripts/restore-drill.sh +++ b/scripts/restore-drill.sh @@ -12,10 +12,45 @@ fi case "$SYNAPSE_BACKUP_PATH" in /*) ;; *) echo "SYNAPSE_BACKUP_PATH must be absolute" >&2; exit 2 ;; esac case "$RESTORE_UPLOAD_DIR" in /*) ;; *) echo "RESTORE_UPLOAD_DIR must be absolute" >&2; exit 2 ;; esac -if [[ "$RESTORE_UPLOAD_DIR" == "/" || "$RESTORE_UPLOAD_DIR" == "$HOME" ]]; then - echo "Refusing unsafe restore upload target" >&2 - exit 2 -fi + +RESTORE_UPLOAD_DIR="$(TARGET_PATH="$RESTORE_UPLOAD_DIR" node <<'NODE' +const fs = require('node:fs'); +const os = require('node:os'); +const path = require('node:path'); + +const resolved = path.resolve(process.env.TARGET_PATH); +if (resolved === path.parse(resolved).root || resolved === os.homedir() || resolved === process.cwd()) { + console.error(`Refusing unsafe restore upload target: ${resolved}`); + process.exit(2); +} + +let existing = resolved; +while (!fs.existsSync(existing)) { + const parent = path.dirname(existing); + if (parent === existing) { + console.error(`Could not locate an existing restore-target ancestor: ${resolved}`); + process.exit(2); + } + existing = parent; +} +if (fs.realpathSync(existing) !== existing) { + console.error(`Restore upload target path must not contain symbolic-link components: ${resolved}`); + process.exit(2); +} +if (fs.existsSync(resolved)) { + const stat = fs.lstatSync(resolved); + if (!stat.isDirectory()) { + console.error(`Restore upload target is not a directory: ${resolved}`); + process.exit(2); + } + if (fs.realpathSync(resolved) !== resolved) { + console.error(`Restore upload target must not resolve through a symbolic link: ${resolved}`); + process.exit(2); + } +} +process.stdout.write(resolved); +NODE +)" for file in database.dump uploads.tar.gz manifest.txt checksums.sha256; do if [[ ! -f "$SYNAPSE_BACKUP_PATH/$file" ]]; then @@ -29,28 +64,294 @@ done shasum -a 256 -c checksums.sha256 ) +archive_names="$(mktemp)" +archive_listing="$(mktemp)" +restore_sql="$(mktemp)" +connections_closed=0 +postgres_admin_url="" +restore_database_name="" + +reenable_target_connections() { + if [[ "$connections_closed" != "1" || -z "$postgres_admin_url" || -z "$restore_database_name" ]]; then + return 0 + fi + + psql "$postgres_admin_url" --no-psqlrc --set=ON_ERROR_STOP=1 --set=target_db="$restore_database_name" >/dev/null <<'SQL' +SELECT format('ALTER DATABASE %I ALLOW_CONNECTIONS true', :'target_db') \gexec +SQL + connections_closed=0 +} + +cleanup_restore_drill() { + status=$? + set +e + reenable_target_connections + rm -f -- "$archive_names" "$archive_listing" "$restore_sql" + exit "$status" +} +trap cleanup_restore_drill EXIT + +tar -tzf "$SYNAPSE_BACKUP_PATH/uploads.tar.gz" > "$archive_names" while IFS= read -r entry; do case "$entry" in /*|../*|*/../*|*/..) echo "Unsafe upload archive entry: $entry" >&2; exit 2 ;; esac -done < <(tar -tzf "$SYNAPSE_BACKUP_PATH/uploads.tar.gz") +done < "$archive_names" + +tar -tvzf "$SYNAPSE_BACKUP_PATH/uploads.tar.gz" > "$archive_listing" +if grep -Evq '^[-d]' "$archive_listing"; then + echo "Upload archive contains non-regular or non-directory entries; refusing unsafe restore" >&2 + exit 2 +fi if [[ -d "$RESTORE_UPLOAD_DIR" && -n "$(find "$RESTORE_UPLOAD_DIR" -mindepth 1 -maxdepth 1 -print -quit)" ]]; then echo "Restore upload target must be empty" >&2 exit 2 fi -postgres_restore_url="$(RESTORE_DATABASE_URL="$RESTORE_DATABASE_URL" node -e ' - const url = new URL(process.env.RESTORE_DATABASE_URL); - url.searchParams.delete("schema"); - process.stdout.write(url.toString()); -')" -pg_restore --exit-on-error --clean --if-exists --no-owner --no-privileges \ - --dbname="$postgres_restore_url" "$SYNAPSE_BACKUP_PATH/database.dump" +mapfile -t postgres_connection < <(RESTORE_DATABASE_URL="$RESTORE_DATABASE_URL" node <<'NODE' +const target = new URL(process.env.RESTORE_DATABASE_URL); +target.searchParams.delete('schema'); +const databaseName = decodeURIComponent(target.pathname.replace(/^\//, '')); +if (!databaseName || ['postgres', 'template0', 'template1'].includes(databaseName)) { + console.error('RESTORE_DATABASE_URL must name a dedicated disposable database'); + process.exit(2); +} +const admin = new URL(target); +admin.pathname = '/postgres'; +process.stdout.write(`${target.toString()}\n${admin.toString()}\n${databaseName}\n`); +NODE +) + +postgres_restore_url="${postgres_connection[0]:-}" +postgres_admin_url="${postgres_connection[1]:-}" +restore_database_name="${postgres_connection[2]:-}" +if [[ -z "$postgres_restore_url" || -z "$postgres_admin_url" || -z "$restore_database_name" ]]; then + echo "Could not resolve disposable restore database connection details" >&2 + exit 2 +fi + +# Prove the exact maintenance connection and permission path needed by failure +# cleanup before making the disposable target inaccessible. This is idempotent while +# the target still accepts connections and fails closed if `postgres` is unavailable, +# CONNECT is denied, the target is missing, or ALTER DATABASE is not permitted. +psql "$postgres_admin_url" --no-psqlrc --set=ON_ERROR_STOP=1 --set=target_db="$restore_database_name" >/dev/null <<'SQL' +SELECT format('ALTER DATABASE %I ALLOW_CONNECTIONS true', :'target_db') \gexec +SQL + +# Generate SQL before closing target connections. Execution happens later through +# one protected psql session so the emptiness check and restore cannot be raced. +pg_restore --no-owner --no-privileges \ + --file="$restore_sql" "$SYNAPSE_BACKUP_PATH/database.dump" + +connections_closed=1 +{ + cat <<'SQL' +\set ON_ERROR_STOP on + +-- Freeze the disposable database before validating it. The current psql session +-- remains connected; new connections are refused until the restore completes. +SELECT format('ALTER DATABASE %I ALLOW_CONNECTIONS false', current_database()) \gexec +SELECT pg_terminate_backend(pid) +FROM pg_stat_activity +WHERE datname = current_database() + AND pid <> pg_backend_pid(); + +DO $synapse_connections$ +BEGIN + IF EXISTS ( + SELECT 1 + FROM pg_stat_activity + WHERE datname = current_database() + AND pid <> pg_backend_pid() + ) THEN + RAISE EXCEPTION 'Restore database still has another active connection'; + END IF; +END +$synapse_connections$; + +DO $synapse_preflight$ +DECLARE + existing_objects bigint; + system_schema_objects bigint; + public_schema_owner text; + unsafe_public_acl bigint; +BEGIN + IF (SELECT count(*) FROM pg_namespace WHERE nspname = 'public') <> 1 THEN + RAISE EXCEPTION 'Restore database must contain exactly one standard public schema'; + END IF; + + SELECT pg_get_userbyid(nspowner) + INTO public_schema_owner + FROM pg_namespace + WHERE nspname = 'public'; + IF public_schema_owner <> 'pg_database_owner' THEN + RAISE EXCEPTION 'Restore database public schema has unexpected owner: %', public_schema_owner; + END IF; + + SELECT count(*) + INTO unsafe_public_acl + FROM pg_namespace n + CROSS JOIN LATERAL aclexplode(COALESCE(n.nspacl, acldefault('n', n.nspowner))) acl + WHERE n.nspname = 'public' + AND acl.grantee <> n.nspowner + AND NOT ( + acl.grantee = 0 + AND acl.privilege_type = 'USAGE' + AND NOT acl.is_grantable + ); + IF unsafe_public_acl <> 0 THEN + RAISE EXCEPTION 'Restore database public schema has unexpected privileges'; + END IF; + + -- PostgreSQL reserves OIDs below FirstNormalObjectId (16384) for built-in/initdb + -- objects. Any namespace-owned object at or above that boundary inside a system + -- schema was created later and makes this supposed disposable target non-pristine. + SELECT count(*) + INTO system_schema_objects + FROM ( + SELECT 1 FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace + WHERE n.nspname IN ('pg_catalog', 'information_schema') AND c.oid >= 16384 + UNION ALL + SELECT 1 FROM pg_proc p JOIN pg_namespace n ON n.oid = p.pronamespace + WHERE n.nspname IN ('pg_catalog', 'information_schema') AND p.oid >= 16384 + UNION ALL + SELECT 1 FROM pg_type t JOIN pg_namespace n ON n.oid = t.typnamespace + WHERE n.nspname IN ('pg_catalog', 'information_schema') AND t.oid >= 16384 + UNION ALL + SELECT 1 FROM pg_collation c JOIN pg_namespace n ON n.oid = c.collnamespace + WHERE n.nspname IN ('pg_catalog', 'information_schema') AND c.oid >= 16384 + UNION ALL + SELECT 1 FROM pg_conversion c JOIN pg_namespace n ON n.oid = c.connamespace + WHERE n.nspname IN ('pg_catalog', 'information_schema') AND c.oid >= 16384 + UNION ALL + SELECT 1 FROM pg_operator o JOIN pg_namespace n ON n.oid = o.oprnamespace + WHERE n.nspname IN ('pg_catalog', 'information_schema') AND o.oid >= 16384 + UNION ALL + SELECT 1 FROM pg_opclass o JOIN pg_namespace n ON n.oid = o.opcnamespace + WHERE n.nspname IN ('pg_catalog', 'information_schema') AND o.oid >= 16384 + UNION ALL + SELECT 1 FROM pg_opfamily o JOIN pg_namespace n ON n.oid = o.opfnamespace + WHERE n.nspname IN ('pg_catalog', 'information_schema') AND o.oid >= 16384 + UNION ALL + SELECT 1 FROM pg_ts_config c JOIN pg_namespace n ON n.oid = c.cfgnamespace + WHERE n.nspname IN ('pg_catalog', 'information_schema') AND c.oid >= 16384 + UNION ALL + SELECT 1 FROM pg_ts_dict d JOIN pg_namespace n ON n.oid = d.dictnamespace + WHERE n.nspname IN ('pg_catalog', 'information_schema') AND d.oid >= 16384 + UNION ALL + SELECT 1 FROM pg_ts_parser p JOIN pg_namespace n ON n.oid = p.prsnamespace + WHERE n.nspname IN ('pg_catalog', 'information_schema') AND p.oid >= 16384 + UNION ALL + SELECT 1 FROM pg_ts_template t JOIN pg_namespace n ON n.oid = t.tmplnamespace + WHERE n.nspname IN ('pg_catalog', 'information_schema') AND t.oid >= 16384 + UNION ALL + SELECT 1 FROM pg_statistic_ext s JOIN pg_namespace n ON n.oid = s.stxnamespace + WHERE n.nspname IN ('pg_catalog', 'information_schema') AND s.oid >= 16384 + ) AS user_created_system_schema_objects; + + IF system_schema_objects <> 0 THEN + RAISE EXCEPTION 'Restore database system schemas contain % user-created object(s)', system_schema_objects; + END IF; + + SELECT count(*) + INTO existing_objects + FROM ( + SELECT 1 FROM pg_namespace + WHERE nspname NOT IN ('pg_catalog', 'information_schema', 'public') + AND nspname !~ '^pg_(toast|temp)' + UNION ALL + SELECT 1 FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace + WHERE n.nspname = 'public' + UNION ALL + SELECT 1 FROM pg_proc p JOIN pg_namespace n ON n.oid = p.pronamespace + WHERE n.nspname = 'public' + UNION ALL + SELECT 1 FROM pg_type t JOIN pg_namespace n ON n.oid = t.typnamespace + WHERE n.nspname = 'public' + UNION ALL + SELECT 1 FROM pg_collation c JOIN pg_namespace n ON n.oid = c.collnamespace + WHERE n.nspname = 'public' + UNION ALL + SELECT 1 FROM pg_conversion c JOIN pg_namespace n ON n.oid = c.connamespace + WHERE n.nspname = 'public' + UNION ALL + SELECT 1 FROM pg_operator o JOIN pg_namespace n ON n.oid = o.oprnamespace + WHERE n.nspname = 'public' + UNION ALL + SELECT 1 FROM pg_opclass o JOIN pg_namespace n ON n.oid = o.opcnamespace + WHERE n.nspname = 'public' + UNION ALL + SELECT 1 FROM pg_opfamily o JOIN pg_namespace n ON n.oid = o.opfnamespace + WHERE n.nspname = 'public' + UNION ALL + SELECT 1 FROM pg_ts_config c JOIN pg_namespace n ON n.oid = c.cfgnamespace + WHERE n.nspname = 'public' + UNION ALL + SELECT 1 FROM pg_ts_dict d JOIN pg_namespace n ON n.oid = d.dictnamespace + WHERE n.nspname = 'public' + UNION ALL + SELECT 1 FROM pg_ts_parser p JOIN pg_namespace n ON n.oid = p.prsnamespace + WHERE n.nspname = 'public' + UNION ALL + SELECT 1 FROM pg_ts_template t JOIN pg_namespace n ON n.oid = t.tmplnamespace + WHERE n.nspname = 'public' + UNION ALL + SELECT 1 FROM pg_statistic_ext s JOIN pg_namespace n ON n.oid = s.stxnamespace + WHERE n.nspname = 'public' + UNION ALL + SELECT 1 FROM pg_extension WHERE extname <> 'plpgsql' + UNION ALL + SELECT 1 FROM pg_event_trigger + UNION ALL + SELECT 1 FROM pg_publication + UNION ALL + SELECT 1 FROM pg_subscription + WHERE subdbid = (SELECT oid FROM pg_database WHERE datname = current_database()) + UNION ALL + SELECT 1 FROM pg_foreign_data_wrapper + UNION ALL + SELECT 1 FROM pg_foreign_server + UNION ALL + SELECT 1 FROM pg_user_mapping + UNION ALL + SELECT 1 FROM pg_transform + UNION ALL + SELECT 1 FROM pg_largeobject_metadata + UNION ALL + SELECT 1 FROM pg_am WHERE oid >= 16384 + UNION ALL + SELECT 1 FROM pg_cast WHERE oid >= 16384 + UNION ALL + SELECT 1 FROM pg_default_acl + UNION ALL + SELECT 1 FROM pg_prepared_xacts WHERE database = current_database() + UNION ALL + SELECT 1 FROM pg_language + WHERE lanname NOT IN ('internal', 'c', 'sql', 'plpgsql') + ) AS user_defined_objects; + + IF existing_objects <> 0 THEN + RAISE EXCEPTION 'Restore database must be empty; found % user-defined object(s)', existing_objects; + END IF; +END +$synapse_preflight$; +SQL + cat "$restore_sql" + cat <<'SQL' + +-- Re-open the disposable target only after the protected restore finishes. +SELECT format('ALTER DATABASE %I ALLOW_CONNECTIONS true', current_database()) \gexec +SQL +} | psql "$postgres_restore_url" --no-psqlrc --set=ON_ERROR_STOP=1 >/dev/null +connections_closed=0 + mkdir -p "$RESTORE_UPLOAD_DIR" tar -xzf "$SYNAPSE_BACKUP_PATH/uploads.tar.gz" -C "$RESTORE_UPLOAD_DIR" DATABASE_URL="$RESTORE_DATABASE_URL" npx prisma migrate status >/dev/null -DATABASE_URL="$RESTORE_DATABASE_URL" npm run uploads:reconcile >/dev/null +SYNAPSE_UPLOAD_DIR_CONFIRM='restore-drill-disposable-upload-target' \ +DATABASE_URL="$RESTORE_DATABASE_URL" \ +npm run uploads:reconcile -- "$RESTORE_UPLOAD_DIR" >/dev/null echo "Restore drill passed: database migrations and upload reconciliation are healthy" diff --git a/tests/backup-restore.test.ts b/tests/backup-restore.test.ts index df4e7c7c..5cf828e7 100644 --- a/tests/backup-restore.test.ts +++ b/tests/backup-restore.test.ts @@ -2,9 +2,17 @@ import assert from "node:assert/strict"; import { readFile } from "node:fs/promises"; import test from "node:test"; -test("backup captures database, uploads, manifest, and checksums without secrets", async () => { +test("backup captures only a canonical regular upload tree with integrity metadata", async () => { const script = await readFile("scripts/backup.sh", "utf8"); + assert.match(script, /canonical_existing_directory/); + assert.match(script, /path\.isAbsolute\(input\)/); + assert.match(script, /Directory path must be absolute/); + assert.match(script, /fs\.realpathSync\(resolved\)/); + assert.match(script, /Directory path must not contain symbolic-link components/); + assert.match(script, /stat\.isSymbolicLink\(\)/); + assert.match(script, /stat\.nlink !== 1/); + assert.match(script, /hard-linked file/); assert.match(script, /pg_dump --format=custom --no-owner --no-privileges/); assert.match(script, /url\.searchParams\.delete\("schema"\)/); assert.match(script, /tar -czf .*uploads\.tar\.gz/); @@ -13,15 +21,129 @@ test("backup captures database, uploads, manifest, and checksums without secrets assert.match(script, /trap cleanup_failed_backup ERR INT TERM/); }); -test("restore drill is explicit, integrity-checked, and validates application state", async () => { +test("restore drill fails closed on archive, target path, database state, concurrent access, cleanup access, and missing uploads", async () => { const script = await readFile("scripts/restore-drill.sh", "utf8"); + const reconcile = await readFile("scripts/reconcile-uploads.ts", "utf8"); assert.match(script, /RESTORE_CONFIRM:-.*EMPTY_DISPOSABLE_DATABASE/); assert.match(script, /shasum -a 256 -c checksums\.sha256/); - assert.match(script, /pg_restore --exit-on-error --clean --if-exists/); - assert.match(script, /url\.searchParams\.delete\("schema"\)/); + assert.match(script, /fs\.realpathSync\(existing\) !== existing/); + assert.match(script, /symbolic-link components/); + assert.match(script, /archive_listing="\$\(mktemp\)"/); + assert.match(script, /tar -tvzf .* > "\$archive_listing"/); + assert.match(script, /grep -Evq '\^\[-d\]' "\$archive_listing"/); + assert.doesNotMatch(script, /tar -tvzf[^\n]*\|[^\n]*grep -q/); + assert.match(script, /non-regular or non-directory entries/); + + assert.match(script, /dedicated disposable database/); + assert.match(script, /ALLOW_CONNECTIONS false/); + assert.match(script, /ALLOW_CONNECTIONS true/); + assert.match(script, /pg_terminate_backend\(pid\)/); + assert.match(script, /Restore database still has another active connection/); + assert.match(script, /connections_closed=1/); + assert.match(script, /reenable_target_connections/); + assert.match(script, /postgres_admin_url/); + const cleanupProbe = script.indexOf("Prove the exact maintenance connection and permission path"); + const freezeMarker = script.indexOf("connections_closed=1"); + assert.ok(cleanupProbe >= 0 && freezeMarker > cleanupProbe); + assert.match( + script.slice(cleanupProbe, freezeMarker), + /psql "\$postgres_admin_url"[\s\S]*ALTER DATABASE %I ALLOW_CONNECTIONS true/ + ); + + assert.match(script, /public_schema_owner <> 'pg_database_owner'/); + assert.match(script, /aclexplode\(COALESCE\(n\.nspacl, acldefault\('n', n\.nspowner\)\)\)/); + assert.match(script, /public schema has unexpected privileges/); + + const systemSchemaStart = script.indexOf("INTO system_schema_objects"); + const existingObjectsStart = script.indexOf("INTO existing_objects"); + assert.ok(systemSchemaStart >= 0 && existingObjectsStart > systemSchemaStart); + const systemSchemaBlock = script.slice(systemSchemaStart, existingObjectsStart); + assert.match(systemSchemaBlock, /n\.nspname IN \('pg_catalog', 'information_schema'\)/); + assert.match(systemSchemaBlock, /oid >= 16384/); + for (const catalog of [ + "pg_class", + "pg_proc", + "pg_type", + "pg_collation", + "pg_conversion", + "pg_operator", + "pg_opclass", + "pg_opfamily", + "pg_ts_config", + "pg_ts_dict", + "pg_ts_parser", + "pg_ts_template", + "pg_statistic_ext", + ]) { + assert.match(systemSchemaBlock, new RegExp(catalog)); + } + assert.match(script, /system schemas contain % user-created object/); + + for (const catalog of [ + "pg_namespace", + "pg_class", + "pg_proc", + "pg_type", + "pg_collation", + "pg_conversion", + "pg_operator", + "pg_opclass", + "pg_opfamily", + "pg_ts_config", + "pg_ts_dict", + "pg_ts_parser", + "pg_ts_template", + "pg_statistic_ext", + "pg_extension", + "pg_event_trigger", + "pg_publication", + "pg_subscription", + "pg_foreign_data_wrapper", + "pg_foreign_server", + "pg_user_mapping", + "pg_transform", + "pg_largeobject_metadata", + "pg_am", + "pg_cast", + "pg_default_acl", + "pg_prepared_xacts", + "pg_language", + ]) { + assert.match(script, new RegExp(catalog)); + } + assert.match( + script, + /pg_subscription[\s\S]*subdbid = \(SELECT oid FROM pg_database WHERE datname = current_database\(\)\)/ + ); + assert.match(script, /pg_am WHERE oid >= 16384/); + assert.match(script, /SELECT 1 FROM pg_default_acl/); + assert.match(script, /pg_prepared_xacts WHERE database = current_database\(\)/); + assert.match(script, /Restore database must be empty/); + + assert.match(script, /pg_restore --no-owner --no-privileges/); + assert.match(script, /--file="\$restore_sql"/); + assert.doesNotMatch(script, /pg_restore[^\n]*--clean/); + assert.match(script, /\\set ON_ERROR_STOP on/); + assert.match(script, /searchParams\.delete\(['"]schema['"]\)/); assert.match(script, /prisma migrate status/); - assert.match(script, /npm run uploads:reconcile/); + assert.match( + script, + /SYNAPSE_UPLOAD_DIR_CONFIRM='restore-drill-disposable-upload-target'[\s\S]*npm run uploads:reconcile -- "\$RESTORE_UPLOAD_DIR"/ + ); + + assert.doesNotMatch(reconcile, /process\.env\.SYNAPSE_UPLOAD_DIR/); + assert.match(reconcile, /process\.argv\[2\]/); + assert.match(reconcile, /SYNAPSE_UPLOAD_DIR_CONFIRM/); + assert.match(reconcile, /restore-drill-disposable-upload-target/); + assert.match(reconcile, /path\.isAbsolute\(configured\)/); + assert.match(reconcile, /resolved === root/); + assert.match(reconcile, /resolved === homedir\(\)/); + assert.match(reconcile, /resolved === process\.cwd\(\)/); + assert.match(reconcile, /path\.join\(process\.cwd\(\), "uploads"\)/); + assert.match(reconcile, /explicitRestoreTarget = Boolean\(process\.argv\[2\]\)/); + assert.match(reconcile, /missingReferenced/); + assert.match(reconcile, /restore upload target is missing .* referenced attachment file/); }); test("runbook keeps production recovery gated on owner and operator evidence", async () => { @@ -29,5 +151,9 @@ test("runbook keeps production recovery gated on owner and operator evidence", a assert.match(runbook, /encrypted off-site destination/); assert.match(runbook, /target recovery point and recovery time objectives/); + assert.match(runbook, /prepared transactions/); + assert.match(runbook, /system-named schemas/); + assert.match(runbook, /maintenance connection/); + assert.match(runbook, /every restored attachment row/); assert.match(runbook, /backup and recovery remain a release blocker/); });