fix(proxy): cancel restart policy before draining the old generation - #106
Merged
Conversation
Both port-holder generations run with --restart unless-stopped. drain makes the old proxy exit on its own - unlike docker stop, that does not mark the container deliberately stopped, so Docker's restart policy resurrects it within its sub-second backoff. The un-forced docker container rm then races that restart and loses essentially every time, leaving the host half-promoted: both generations bound via SO_REUSEPORT, promote_next_container never run, reboot exiting non-zero. docker update --restart=no BEFORE draining removes the race entirely - there is no window in which the exited container can be restarted. Closes #105
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #105
Problem
kamal proxy rebooton a host already on the port-holder architecture fails mid-handoff. Both generations run with--restart unless-stopped.drainmakes the old proxy process exit on its own — unlikedocker stop, that does not mark the container deliberately stopped, so Docker's restart policy resurrects it within its sub-second backoff. The un-forceddocker container rminhandoff_generationraces that restart and loses essentially every time. Result: bothkamal-proxy(old gen, resurrected) andkamal-proxy-nextserving via SO_REUSEPORT,promote_next_containernever run, reboot exiting non-zero with the host half-promoted.Fix
Cancel the restart policy BEFORE draining — the issue's preferred variant, because it leaves no timing window at all:
Kamal::Commands::Proxy#disable_restart— new command builder,docker update --restart=no kamal-proxy(argv-array style, matching siblings)Kamal::Cli::Proxy::Reboot#handoff_generation— runs it firstSibling-path audit:
handoff_generationis the only caller ofdrainon a proxy container.stop_and_replaceandmigrate_to_holderusedocker container stop(marks the container deliberately stopped — no resurrection) pluscontainer prune --force, so neither shares the race.Test plan
test/commands/proxy_test.rb— newdisable_restarttest asserting the exact argv:docker update --restart=no kamal-proxytest/cli/proxy_test.rb— the port-holder handoff test now assertsdocker update --restart=noappears in the output AND, via index comparison, that it runs strictly before the drain commandbundle exec rubocop --parallel— clean (218 files)git stash) to fail identically on the unmodified baseline; they pass in CIDeviations & judgment calls
handoff_generationdrains a proxy container.stop_and_replaceandmigrate_to_holderusedocker container stop(marks the container deliberately stopped, so the restart policy does not resurrect it) followed bycontainer prune --force— no shared race. No other fix needed.disable_restarttakes noname:keyword (unlikewait_for_exit/remove_stopped_container) — only one caller exists and it targets the default container; avoided an unused parameter.--restart=noas a single token (matching the issue text) rather than the two-token"--restart", "no"style used by thedocker runbuilders —docker updateaccepts both; the single token reads unambiguously in printed/logged output.handoff_generationvia the reboot path) using output index comparison, rather than adding a near-duplicate test.CommandsBuilderTest#test_hybrid_builder_with_local_registry,CliBuildTest#test_push_with_remote_builder_checks_both_the_builder_and_the_remote_context) verified pre-existing on baseline viagit stash— the two known Apple-Silicon host-arch failures (names differ from the older grep hint in.claude/rules/testing.md, but the failure mode — amd64/arm64 platform swap — matches exactly). Not regressions.handoff_generationhad none before this change, anddisable_restartfailing before drain leaves the host fully intact (old gen still serving, restart policy merely disabled; the next reboot re-runs the handoff). Out of scope per the issue.