fix(prune): scope container pruning to a role and destination - #68
Merged
Conversation
`Kamal::Commands::Prune#app_containers` selected removal candidates with `label=service=<service>` alone and kept the newest `retain` of them. Two consequences: - Roles share the window. All roles deploy at the same version, so with four roles and `retain: 5` only about the last one-and-a-quarter deploys survive, and a busy sibling role pushes another role's current container out of the window. - Destinations share it too, so pruning staging could remove production's containers on a shared host. That is a correctness problem, not just disk hygiene: a container kamal-proxy has put to sleep for scale-to-zero is `exited`, so it is a removal candidate. Once it is removed the proxy still holds its reference and every wake 404s, leaving the service returning 503 until someone redeploys. Prune now runs once per role on each host, filtering on `service`, `destination` and `role` - the same filter set `Kamal::Commands::App` already uses. Since `retain` is validated to be at least 1, a role's newest container is now never removed, and a slept container is always its role's newest because sleeping happens to the current release. Refs #62
This was referenced Jul 29, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Kamal::Commands::Prune#app_containerspicked removal candidates withlabel=service=<service>and nothing else, then kept the newestretainof them. Every role and every destination of a service shared one window.That is why a slept container can disappear. A container kamal-proxy has put to sleep is
exited, so it is already adocker rmcandidate; sharing the window with sibling roles means it does not even need five older releases of its own role to fall out of it. Once the container is gone the proxy still holds its reference, every wake 404s, and the service 503s until someone redeploys — the failure zoolutions/kamal-proxy#58 can only make visible, not prevent.Prune now runs once per role on each host, filtering on
service,destinationandrole— the same filter setKamal::Commands::App#container_filters(lib/kamal/commands/app.rb:120) has always used.retainis already validated to be at least 1, so:There is a second, quieter bug fixed here. All roles deploy at the same version, so with four roles and
retain: 5the old filter retained roughly the last 1.25 deploys, not 5. Per-role retention makesretain_containersmean what the docs say it means. Image retention is unaffected:tagged_imageskeys off images referenced by any container, and the retained versions are the same set.Changes
lib/kamal/commands/prune.rbapp_containers(retain:, role:)—role:required; addsdestinationandrolefilterslib/kamal/cli/prune.rbcontainersiteratesKAMAL.roles_on(host);descand--retainsay "per role"lib/kamal/configuration/docs/configuration.ymlretain_containersdocumented as per-roleCloses #62 for the prune half. The other half of the issue — passing a
service/role/destinationselector tokamal-proxy deployso the proxy resolves its target at wake time — is blocked on the proxy and is not in this PR (see below).Test plan
test/commands/prune_test.rb—app containersasserts the full filter set forretain: 5andretain: 3test/commands/prune_test.rb— a sibling role's filter never leaks into another role's prune commandtest/commands/prune_test.rb— destination-scoped prune emitslabel=destination=stagingtest/cli/prune_test.rb—containersandcontainers --retain 10end-to-end through the Thor commandtest/cli/prune_test.rb—deploy_with_roles.ymlproduces a separate prune per roletest/cli/proxy_test.rb—upgradeassertion updated (it runs a deploy, which prunes)bundle exec rubocop --parallel— 202 files, no offensesbuilder_test.rb:277,build_test.rb:51), both untouched by this changeNo integration run: this changes command construction only, and
test/integrationhas no prune assertions.Deviations & judgment calls
grep -ri idle lib/finds onlysshkit.pool_idle_timeout— and adding one now would be dead config, because kamal-proxy has no--idle-timeoutflag yet (feat(server): idle controller and container lifecycle seam for scale-to-zero (partial) kamal-proxy#58 landed theIdleControllerand lifecycle seam, not the CLI wiring). Emitting an unknown flag would breakkamal-proxy deploy. Per-role scoping gets the same protection with no new config and no proxy coordination, so this PR does not need to be release-paired with a proxy image — the ordering constraint in the issue does not apply to it.kamal-proxy deployis blocked on the same missing proxy support. It stays open on Prune removes containers kamal-proxy has put to sleep (scale-to-zero) #62.destinationfilter is included even though the issue only named roles. It is the identical failure mode (one destination's prune removing another's live container), the identical line of code, and it aligns prune withApp#container_filters. Behavior change worth knowing:kamal prune -d stagingno longer prunes production's stopped containers.role:is a required keyword, not optional, so no unscoped call path survives to be used by accident.kamal rollbackstarts an existing older container rather than creating one, so after a rollback the live container is not the newest by creation time. If that container then sleeps andretainnewer stopped containers of the same role exist, it is still a prune candidate. Closing that needs the proxy to resolve its target by label at wake time — option (2), blocked.deploy.ymlnow linger. Accepted: kamal does not clean up removed roles anywhere else either, and the alternative (a service-wide sweep that excludes each role's newest id) needs agrep -vover a shell substitution that silently matches everything when any role has no containers.tagged_imagesfrees the same images as before. Only the containers' thin r/w layers are extra.