Skip to content

fix(volume-backups): restart services after backup failure - #5081

Closed
bestmaa wants to merge 1 commit into
Dokploy:canaryfrom
bestmaa:codex/fix-volume-backup-restart
Closed

fix(volume-backups): restart services after backup failure#5081
bestmaa wants to merge 1 commit into
Dokploy:canaryfrom
bestmaa:codex/fix-volume-backup-restart

Conversation

@bestmaa

@bestmaa bestmaa commented Aug 14, 2026

Copy link
Copy Markdown

Problem

When Turn Off Container During Backup is enabled, the generated volume-backup script stops the application or Compose service before creating the archive. Because the script runs with set -e, a backup failure exits immediately and skips the restart command, leaving the service offline until someone restarts it manually.

This is especially disruptive for transient failures such as an unavailable image registry or a failed ubuntu image pull.

What changed

  • Run the backup step in a subshell and capture its exit status.
  • Always execute the service restart after the backup attempt.
  • Preserve and return the original backup failure status after the restart, so the deployment remains marked as failed and the upload is skipped.
  • Use the same restart-safe command flow for applications, Compose stacks, and regular Compose containers.
  • Add regression tests covering both failure and success command sequences.

Impact

Services that are intentionally stopped for a consistent volume backup are brought back online even when archive creation fails. Successful backup and upload behavior remains unchanged.

Validation

  • pnpm --filter=dokploy exec vitest --config __test__/vitest.config.ts run __test__/backups — 21 tests passed
  • pnpm --filter=@dokploy/server typecheck
  • pnpm --filter=dokploy typecheck
  • Biome check on the changed files

Fixes #4263

Greptile Summary

The PR changes volume-backup command generation so stopped application and Compose services are restarted after archive failures while retaining failure signaling.

  • Introduces a shared restart-safe shell-command generator.
  • Applies the flow to application, Compose stack, and regular Compose-container backups.
  • Adds regression coverage for successful backups and backup failures followed by successful restarts.

Confidence Score: 4/5

The restart sequencing needs correction before merging because a failed restart can override the original backup failure that this change promises to preserve.

The saved backup status is checked only after a fallible restart executes under errexit, so simultaneous backup and restart failures surface the wrong exit status and diagnostics.

Files Needing Attention: packages/server/src/utils/volume-backups/backup.ts

Reviews (1): Last reviewed commit: "fix(volume-backups): restart services af..." | Re-trigger Greptile

Greptile also left 1 inline comment on this PR.

Context used:

@bestmaa
bestmaa marked this pull request as ready for review August 14, 2026 11:41
@bestmaa
bestmaa requested a review from Siumauricio as a code owner August 14, 2026 11:41
@dosubot dosubot Bot added the size:M This PR changes 30-99 lines, ignoring generated files. label Aug 14, 2026
Comment on lines +31 to +35
set -e
${startCommand}
if [ "$DOKPLOY_VOLUME_BACKUP_STATUS" -ne 0 ]; then
exit "$DOKPLOY_VOLUME_BACKUP_STATUS"
fi

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Restart overrides backup status

When both the backup and restart fail, set -e exits at the restart command before the saved backup status is checked, causing deployment diagnostics and notifications to report the secondary restart failure instead of the original backup failure.

Suggested change
set -e
${startCommand}
if [ "$DOKPLOY_VOLUME_BACKUP_STATUS" -ne 0 ]; then
exit "$DOKPLOY_VOLUME_BACKUP_STATUS"
fi
${startCommand}
DOKPLOY_VOLUME_RESTART_STATUS=$?
set -e
if [ "$DOKPLOY_VOLUME_BACKUP_STATUS" -ne 0 ]; then
exit "$DOKPLOY_VOLUME_BACKUP_STATUS"
fi
if [ "$DOKPLOY_VOLUME_RESTART_STATUS" -ne 0 ]; then
exit "$DOKPLOY_VOLUME_RESTART_STATUS"
fi

Knowledge Base Used: Backups and Schedules

@bestmaa

bestmaa commented Aug 14, 2026

Copy link
Copy Markdown
Author

Replaced by #5082 using the contributor-owned branch name fix/volume-backup-restart. The code and verified commit are unchanged.

@bestmaa bestmaa closed this Aug 14, 2026
@bestmaa
bestmaa deleted the codex/fix-volume-backup-restart branch August 14, 2026 11:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:M This PR changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Container not restarted after volume backup fails mid-run when "Turn Off Container During Backup" is enabled

1 participant