Skip to content

fix(prune): scope container pruning to a role and destination - #68

Merged
mhenrixon merged 1 commit into
dashfrom
issue-62-prune-sleeping
Jul 29, 2026
Merged

fix(prune): scope container pruning to a role and destination#68
mhenrixon merged 1 commit into
dashfrom
issue-62-prune-sleeping

Conversation

@mhenrixon

Copy link
Copy Markdown
Collaborator

Summary

Kamal::Commands::Prune#app_containers picked removal candidates with label=service=<service> and nothing else, then kept the newest retain of them. Every role and every destination of a service shared one window.

That is why a slept container can disappear. A container kamal-proxy has put to sleep is exited, so it is already a docker rm candidate; sharing the window with sibling roles means it does not even need five older releases of its own role to fall out of it. Once the container is gone the proxy still holds its reference, every wake 404s, and the service 503s until someone redeploys — the failure zoolutions/kamal-proxy#58 can only make visible, not prevent.

Prune now runs once per role on each host, filtering on service, destination and role — the same filter set Kamal::Commands::App#container_filters (lib/kamal/commands/app.rb:120) has always used. retain is already validated to be at least 1, so:

A role's newest container is never removed — and a slept container is always its role's newest, because sleeping happens to the current release.

There is a second, quieter bug fixed here. All roles deploy at the same version, so with four roles and retain: 5 the old filter retained roughly the last 1.25 deploys, not 5. Per-role retention makes retain_containers mean what the docs say it means. Image retention is unaffected: tagged_images keys off images referenced by any container, and the retained versions are the same set.

Changes

File Change
lib/kamal/commands/prune.rb app_containers(retain:, role:)role: required; adds destination and role filters
lib/kamal/cli/prune.rb containers iterates KAMAL.roles_on(host); desc and --retain say "per role"
lib/kamal/configuration/docs/configuration.yml retain_containers documented as per-role

Closes #62 for the prune half. The other half of the issue — passing a service/role/destination selector to kamal-proxy deploy so the proxy resolves its target at wake time — is blocked on the proxy and is not in this PR (see below).

Test plan

  • test/commands/prune_test.rbapp containers asserts the full filter set for retain: 5 and retain: 3
  • test/commands/prune_test.rb — a sibling role's filter never leaks into another role's prune command
  • test/commands/prune_test.rb — destination-scoped prune emits label=destination=staging
  • test/cli/prune_test.rbcontainers and containers --retain 10 end-to-end through the Thor command
  • test/cli/prune_test.rbdeploy_with_roles.yml produces a separate prune per role
  • test/cli/proxy_test.rbupgrade assertion updated (it runs a deploy, which prunes)
  • bundle exec rubocop --parallel — 202 files, no offenses
  • Unit suite — 1168 runs, 0 errors; the only 2 failures are the known Apple-Silicon arch-dependent builder tests (builder_test.rb:277, build_test.rb:51), both untouched by this change

No integration run: this changes command construction only, and test/integration has no prune assertions.

Deviations & judgment calls

  • The issue's option (1) could not be built as written, so it was built differently. The issue proposed skipping proxy-managed containers "when an idle timeout is configured for that role". The gem has no idle-timeout key — grep -ri idle lib/ finds only sshkit.pool_idle_timeout — and adding one now would be dead config, because kamal-proxy has no --idle-timeout flag yet (feat(server): idle controller and container lifecycle seam for scale-to-zero (partial) kamal-proxy#58 landed the IdleController and lifecycle seam, not the CLI wiring). Emitting an unknown flag would break kamal-proxy deploy. Per-role scoping gets the same protection with no new config and no proxy coordination, so this PR does not need to be release-paired with a proxy image — the ordering constraint in the issue does not apply to it.
  • The issue's option (2) is not attempted. Passing a label selector to kamal-proxy deploy is blocked on the same missing proxy support. It stays open on Prune removes containers kamal-proxy has put to sleep (scale-to-zero) #62.
  • The destination filter is included even though the issue only named roles. It is the identical failure mode (one destination's prune removing another's live container), the identical line of code, and it aligns prune with App#container_filters. Behavior change worth knowing: kamal prune -d staging no longer prunes production's stopped containers.
  • role: is a required keyword, not optional, so no unscoped call path survives to be used by accident.
  • Known remaining hole, deliberately not fixed. kamal rollback starts an existing older container rather than creating one, so after a rollback the live container is not the newest by creation time. If that container then sleeps and retain newer stopped containers of the same role exist, it is still a prune candidate. Closing that needs the proxy to resolve its target by label at wake time — option (2), blocked.
  • Orphaned roles are no longer pruned. Prune only iterates configured roles, so containers of a role deleted from deploy.yml now linger. Accepted: kamal does not clean up removed roles anywhere else either, and the alternative (a service-wide sweep that excludes each role's newest id) needs a grep -v over a shell substitution that silently matches everything when any role has no containers.
  • Disk impact considered and found neutral. More containers are retained per host, but they reference the same set of image versions, so tagged_images frees the same images as before. Only the containers' thin r/w layers are extra.

`Kamal::Commands::Prune#app_containers` selected removal candidates with
`label=service=<service>` alone and kept the newest `retain` of them. Two
consequences:

- Roles share the window. All roles deploy at the same version, so with four
  roles and `retain: 5` only about the last one-and-a-quarter deploys survive,
  and a busy sibling role pushes another role's current container out of the
  window.
- Destinations share it too, so pruning staging could remove production's
  containers on a shared host.

That is a correctness problem, not just disk hygiene: a container kamal-proxy
has put to sleep for scale-to-zero is `exited`, so it is a removal candidate.
Once it is removed the proxy still holds its reference and every wake 404s,
leaving the service returning 503 until someone redeploys.

Prune now runs once per role on each host, filtering on `service`,
`destination` and `role` - the same filter set `Kamal::Commands::App`
already uses. Since `retain` is validated to be at least 1, a role's newest
container is now never removed, and a slept container is always its role's
newest because sleeping happens to the current release.

Refs #62
@mhenrixon mhenrixon self-assigned this Jul 29, 2026
@mhenrixon mhenrixon added the enhancement New feature or request label Jul 29, 2026
@mhenrixon
mhenrixon merged commit 3783e68 into dash Jul 29, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Prune removes containers kamal-proxy has put to sleep (scale-to-zero)

1 participant