A failed logical backup still uploads a 20-byte empty object under a normal filename

[jacobecox]

Summary

A failed logical backup still uploads an object. pg_dumpall writes nothing, gzip emits a valid empty stream on EOF, and the upload completes before the shell reacts — leaving a ~20-byte "backup" in the bucket under a normal, timestamped filename.

The job itself correctly exits non-zero. The problem is the artifact it leaves behind.

Evidence

Measured on postgres-backup:17.1.0 against a real cluster. Two objects in the same prefix:

2026-08-24 20:10:38     20 Bytes  postgres-2026-08-24T20-10-29Z.sql.gz   <- failed run
2026-08-24 20:15:17    1.1 KiB    postgres-2026-08-24T20-15-10Z.sql.gz   <- good run

Decompressed:

object	lines	content
20-byte	0	empty
1.1 KiB	163	valid pg_dumpall, includes CREATE ROLE

Job log for the failed run:

Starting Postgres backup job
Running pg_dumpall against <proxy>:5432
pg_dumpall: error: connection to server ... failed: server closed the connection unexpectedly

...immediately followed by the AWS CLI acquiring credentials and uploading.

Cause

/usr/local/bin/backup.sh:

pg_dumpall --host="${PG_HOST}" --port="${PG_PORT}" --username="${PG_USER}" \
  | gzip \
  | aws s3 cp - "s3://${BACKUP_BUCKET}/${BACKUP_PREFIX}/${FILENAME}"

set -euo pipefail is present and works — the script exits non-zero. But in a pipeline the stages run concurrently: gzip sees EOF and emits a 20-byte empty gzip, aws s3 cp uploads it successfully, and only then does the shell observe the failed first stage. The same shape applies to the gcp (gsutil cp -) and minio branches.

Why it matters

Nothing distinguishes a failed backup from a real one except its size. A user — or a restore script — listing the prefix and taking the most recent object can restore an empty dump over a live database. The failure is silent at exactly the moment it matters.

This is one of three defects found in the same audit whose only symptom appears when you try to restore; the other two were in the templates and are fixed there.

Suggested fix

Dump to a temp file, verify it is non-empty (and ideally that gunzip -t passes), then upload:

tmp="$(mktemp)"; trap 'rm -f "$tmp"' EXIT
pg_dumpall --host="${PG_HOST}" --port="${PG_PORT}" --username="${PG_USER}" | gzip > "$tmp"
[ -s "$tmp" ] && gzip -t "$tmp" || { echo "dump failed or empty; not uploading" >&2; exit 1; }
aws s3 cp "$tmp" "s3://${BACKUP_BUCKET}/${BACKUP_PREFIX}/${FILENAME}"

Trade-off worth naming: this needs local disk for the dump, where streaming does not. If streaming must be preserved for large databases, the alternative is to upload to a .partial key and rename only after the pipeline succeeds, so a failed run never leaves an object at the real name.

Interim mitigation

postgres-highly-available and postgres-multi-location READMEs now warn that a zero-length object is a failed run, not a backup (controlplane-com/templates#479).