Add weekly restic restore-verification drill (#139) - #188
Merged
Conversation
Weekly restic-verify CronJob: structure check, rotating 1/5 read-data subset, per-tag freshness, and a real restore drill with sentinel assertions for the seven small tags. Failure signal is a single ResticVerifyStale dead-man rule via the #138 alerting stack. Assisted-by: AI
Four tasks: verify CronJob (sentinel paths verified against live snapshots), ResticVerifyStale rule, README runbook, and post-merge live verification including a deliberate red-path test. Assisted-by: AI
Four-phase drill: repo check with rotating 1/5 read-data subset (full pack coverage every 5 weeks), per-tag snapshot freshness (<48h), and a real restore of the seven small tags with sentinel assertions. No in-job alerting plumbing -- the ResticVerifyStale dead-man rule is the single failure signal. Assisted-by: AI
Dead-man rule on kube_cronjob_status_last_successful_time for restic-verify, mirroring ResticBackupStale: one expression catches failed, crashed, and never-scheduled drills. 8d = weekly + 1d grace. Assisted-by: AI
Assisted-by: AI
Assisted-by: AI
The Sunday 04:00 forget --prune holds an exclusive restic lock; a prune overrun past 05:00 would instantly burn both drill attempts. Wrap restic with --retry-lock 30m. Also correct the freshness comment's offset-slop claim (looser, not stricter, for positive offsets). Assisted-by: AI
Assisted-by: AI
CI runs pre-commit over all files; the spec/plan commits missed the mdformat pass locally. Assisted-by: AI
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
restic-verifyCronJob (Sun 05:00 ET, one hour after the Sunday prune):repo check with a rotating
--read-data-subset=n/5(100% pack coverage every 5 weeks, ~32 GiB egress/run ≈ $0.90/mo),per-tag snapshot freshness (< 48 h, all 8 tags),
and a real restore of the 7 small tags (~1.8 GiB) with
--verifyplus sentinel-file assertions.syncthing-data(176 GiB) is deliberately not restored — covered by the rotating data read and the freshness check.ResticVerifyStaledead-man alert (no successful drill in > 8 d),mirroring
ResticBackupStale; no in-job Kuma/SMTP plumbing by design.--retry-lock)so a Sunday prune overrun can't burn both job attempts.
docs/superpowers/.Closes Backup: periodic restic restore-verification drill #139.
Notes for review:
(the original prose details were preserved);
pipe-table rows are single physical lines — the documented exception to semantic line breaks.
(
restic ls latest --tag <tag>), including the Bitnamidata/PG_VERSIONnesting.k8s/backup/kustomization.yaml(
prune: falseuntouched; the backup/forget CronJobs stay manually managed).Before merge
restic-credentialsSecret by name only.emptyDirscratch).k8s/backup/README.md, "Restore-verification drill").Test plan
Pre-merge (done on this branch):
kubectl kustomize --load-restrictor LoadRestrictionsNone k8s/backupbuilds.promtool check rules: SUCCESS, 4 rules.-S info, SC3040 excluded for busyboxpipefail) clean on the embedded script.Post-merge (plan Task 4, on gandalf):
flux reconcilebackup + homelab-monitoring; CronJob appears.All phases passed, success timestamp advances.ResticVerifyStalehealth=ok/inactive; staleness query returns minutes, not days.🤖 Built with AI assistance.