Prune removes containers kamal-proxy has put to sleep (scale-to-zero)

[mhenrixon]

kamal deploy runs prune on every deploy (lib/kamal/cli/main.rb, kamal:cli:prune:all). Kamal::Commands::Prune#app_containers selects containers to remove with:

docker ps -q -a --filter status=exited --filter label=service=<service> | tail -n +6 | xargs docker rm

A container that kamal-proxy has put to sleep is exited. So once a service accumulates more than five stopped containers — old releases plus the sleeping one — the sleeping container becomes a docker rm candidate.

After it is removed the proxy still holds its reference, so every wake gets a 404 from the daemon. With zoolutions/kamal-proxy#58's backoff that degrades to one failed start attempt per 30s and a service that returns 503 until someone redeploys, rather than a retry storm — but it is still a hard outage, and nothing on the proxy side can prevent it. The proxy's deploy-time preflight only proves the reference resolved at deploy time.

Why this can't be fixed proxy-side

The proxy learns its container reference from the --target string, which under Kamal is the container short id. It has no way to stop the gem removing that container, and no way to distinguish "pruned" from "never existed" after the fact.

Options, roughly in order of preference

1. Exclude proxy-managed containers from the prune filter. Have the gem skip containers the proxy is responsible for — e.g. by a label the gem already sets when it deploys with an idle timeout configured.
2. Have the gem pass a stable label selector to the proxy instead of a per-release container id, so the proxy resolves the container at wake time rather than holding a reference that prune can invalidate. This also fixes the redeploy-while-sleeping case for free.
3. Raise the retention count when idle timeouts are configured. Cheapest, but only widens the window rather than closing it.

Option 2 is the most robust and is worth considering before the proxy side settles, because it changes what the proxy stores.

Release ordering

The proxy half is not merged yet (zoolutions/kamal-proxy#58 is a partial draft). This needs to land in the same release pair as the finished proxy feature — proxy image first, then the gem, per the usual MINIMUM_VERSION ordering.

Refs zoolutions/kamal-proxy#19, zoolutions/kamal-proxy#58

[mhenrixon]

Read the gem to ground this. Three concrete findings, one of which corrects how I framed the issue.

The labels already exist

Kamal::Configuration::Role#default_labels (lib/kamal/configuration/role.rb:256):

{ "service" => config.service, "role" => name, "destination" => config.destination }

Every app container already carries them. So the proxy could resolve its container at wake time with a native Docker filter — GET /containers/json?filters={"label":["service=X","role=web"]} — with no new labelling work in the gem. The only gem change is passing the selector to kamal-proxy deploy instead of (or alongside) the container id.

container_prefix (role.rb:171) is <service>-<role>-<destination>, stable across deploys, with only the version appended in container_name. So a name prefix would work too, but labels are the better primitive — Docker filters on them natively, and it is exactly what prune itself already selects on.

Correction: these are not alternatives

I listed "exclude from prune" and "label selector" as options 1 and 2, ranked. That was wrong — they fix different halves and both are needed.

• Label selector fixes redeploy while sleeping: the proxy stops holding a per-release id that a new deploy invalidates. It does not help when prune has removed the container, because then no selector finds anything either. The container is gone.
• Excluding proxy-managed containers from prune is the only thing that fixes the original report.

The prune filter, confirmed

Kamal::Commands::Prune#app_containers (lib/kamal/commands/prune.rb:16):

docker(:ps, "-q", "-a", *service_filter, *stopped_containers_filters) | tail -n +{retain+1} | docker rm

with stopped_containers_filters = status=created, status=exited, status=dead. A slept container is exited, so it is in scope, and tail -n +6 means it survives only while fewer than five other stopped containers exist for that service.

Worth noting the blast radius is narrower than I first said: prune is per-service and runs as part of that service's own deploy, which re-targets the proxy anyway. The genuinely dangerous paths are a standalone kamal prune all, and any deploy that prunes a sibling role of a service whose other role is asleep.

Suggested shape

1. Give prune a way to skip containers the proxy is managing — simplest is a filter exclusion when an idle timeout is configured for that role.
2. Separately, have the gem pass service/role/destination to kamal-proxy deploy so the proxy resolves the container at wake time rather than storing an id.

(2) is worth settling before the proxy-side persistence lands, since it changes what goes in the state file. (1) can follow.