feat(server): idle controller and container lifecycle seam for scale-to-zero (partial)

[mhenrixon]

Status: partial — foundation + two fixes. Not the whole feature.

Issue #19 is size:L and it is genuinely multi-day. This PR lands the state machine, the docs decision, and a CI fix found along the way. The wiring into Service/Router/CLI is not here, so nothing about proxy behavior changes yet. docs/plans/2026-07-29-scale-to-zero.md carries the rest.

Three independent commits, reviewable separately:

1. docs: dash is the main branch (644c741)

The docs told every workflow to root branches off main "to keep them upstream-PR-able". Per your call, that benefit is not one this fork wants, and the cost was concrete: GitHub resolves pull_request workflows from the head branch, and main's ci.yml triggers only on main — so a main-rooted PR into dash ran zero checks, including golangci-lint, which is not installed locally.

Updated: CLAUDE.md branch map, .claude/rules/git-workflow.md, /lfg, /plan, /github-review-pr. /plan also gained a rule to verify a "port basecamp#N" issue still points at an open PR — two have gone stale under us (#63→basecamp#225, basecamp#197→basecamp#228).

2. fix(test): the suite was making a real Let's Encrypt call (1e36b89)

TestRouter_GetCertificate_RegistryStillServesHostScopedServices — my own test, from #50 — deployed a TLS service with no certificate paths. createCertManager handed it an autocert.Manager; the test called GetCertificate, the registry declined as designed, and the fall-through reached autocert, which went to the real Let's Encrypt production directory and blocked for its full internal timeout.

	internal/server suite
before	321s
after	23.4s

That one test was 300 of the 320 seconds. It also violated .claude/rules/testing.md outright — "no real ACME/Let's Encrypt calls … never hit the real endpoint" — so CI was making an outbound request to a rate-limited production endpoint on every run. Found while timing the suite for #19, unrelated to it otherwise; merge this one regardless of what you decide about the rest.

3. feat(server): the idle controller (a063390)

IdleController + the ContainerLifecycle seam. Ports the design of basecamp#228 — which supersedes the closed basecamp#197 the issue names — with four deliberate departures, each a defect in basecamp#228:

#	basecamp#228	Here
1	A failed stop still marks the service asleep	Rolls back to active. Otherwise an unmounted socket or a pruned container is a permanent outage: every later request held for the wake timeout then 503, for containers running perfectly
2	A failed wake retries per request	Backs off, doubling to 30s. Otherwise one container start per inbound request, forever
3	Fresh timer per loop iteration	One deadline for the whole call. Otherwise a request arriving during a stop waits the full timeout for the stop and again for the start — twice the documented bound
4	Per-wake token	Generation counter, so any transition supersedes any in-flight lifecycle goroutine — a deploy landing mid-wake is not overwritten

Coalescing falls out of the mutex rather than extra machinery: startWakeLocked flips the state while still holding the lock, so 20 concurrent requests produce exactly one start.

Test plan

13 controller tests, -race at -count=3, no Docker and no network (fakeLifecycle counts atomic.Int64). Highlights:

• TestIdleController_CoalescesConcurrentWakes — 20 goroutines, starts == 1
• TestIdleController_StaysAwakeWhenContainersCannotBeStopped — the inverse of Add opt-in scale-to-zero for idle services basecamp/kamal-proxy#228's behavior
• TestIdleController_DoesNotSleepWithRequestsInFlight — the WebSocket/SSE guarantee at unit level; one unreleased BeginRequest means no stop, ever
• TestIdleController_BacksOffAfterAFailedWake — three requests, exactly one start attempt
• TestIdleController_ResetSupersedesAnInFlightWake — the generation token
• TestIdleState_NamesRoundTrip — stopping/waking fold to sleeping; unknown and empty → active
• make test, go vet ./..., gofmt -l internal/ cmd/ clean; go test -race ./internal/server ./internal/cmd — 1143 pass

Deviations & judgment calls

• The issue's premise was stale and the answer differed from last time. R5: Scale-to-zero idle services #19 says "port Add idle container support (scale-to-zero) basecamp/kamal-proxy#197 (open)". Add idle container support (scale-to-zero) basecamp/kamal-proxy#197 is closed, superseded by Add opt-in scale-to-zero for idle services basecamp/kamal-proxy#228. Unlike R4: On-demand TLS with ask endpoint #13 — where upstream had already merged and waiting was free — Add opt-in scale-to-zero for idle services basecamp/kamal-proxy#228 is only ready for review with no human maintainer review yet, so waiting has no known end. You chose to port.
• Scope is partial and I am saying so rather than padding it. The remaining work is the Docker client, the Service request gate, Router preflight + restore, LoadBalancer suspend/resume, persistence, and CLI flags — roughly 25 more tests. The plan is committed so none of the reasoning has to be rediscovered.
• Known hole, mitigated not fixed — this is the one to read. kamal deploy runs prune on every deploy, and Kamal::Commands::Prune#app_containers filters status=exited and pipes through tail -n +6 into docker rm. A sleeping container is exited. Past five stopped containers for a service, a sleeping one becomes a removal candidate; after that every wake 404s and the service 503s permanently. The deploy-time preflight cannot prevent it — it proves the reference resolves at deploy time. The backoff in this PR turns it from a silent retry storm into a visible failure. The real fix is gem-side in ../kamal (exclude proxy-managed containers from the prune filter, or pass a stable label selector) and needs its own issue.
• The security tradeoff is real and inherited deliberately. Mounting /var/run/docker.sock into the internet-facing proxy is root-equivalent on the host; an RCE in kamal-proxy becomes host compromise. Opt-in, and only mounted when the feature is enabled. Upstream's maintainer judged that acceptable in discussion Passenger-style idle shutdown and on-demand wake-up for hosting many low-traffic applications basecamp/kamal-proxy#222, and the ContainerLifecycle interface is the seam where a restricted host-side start/stop service replaces it without the controller changing. Recorded in ROADMAP.md so the next reader meets it with the anchor.
• Built in a worktree, not the main tree. The working tree had moved to feature/observability-batch with uncommitted work on R5: Observability batch (log format, OTel traceparent, metrics excludes) #20 — a parallel session. I did not touch it; .claude/worktrees/issue-19-scale-to-zero is where this was built.
• Unmeasured: Add opt-in scale-to-zero for idle services basecamp/kamal-proxy#228 reports a 3.15s cold wake for a real Rails app on a VPS. This implementation has not been measured, and that number must not be repeated as if it had been.
• Three pre-existing dash bugs the plan surfaced, none fixed here: WaitUntilHealthy reads waitForHealthyContext unlocked (load_balancer.go:153); BeginHealthChecks writes stateConsumer unsynchronized (target.go:285); Service.MarshalJSON reads s.active/s.rollout with no serviceLock. The third becomes far more reachable once sleep/wake persists state, so it must be fixed before the wiring lands.

Refs #19

[mhenrixon]

Follow-ups now tracked, and the third race is fixed in 67c1f69:

• Prune removes containers kamal-proxy has put to sleep (scale-to-zero) kamal#62 — prune removes sleeping containers. Gem-side; must ship in the same release pair as the finished proxy half. Includes the label-selector option, which is worth deciding before the proxy side settles because it changes what the proxy stores.
• R5: Reduce scale-to-zero cold-wake latency (tiered pause/stop) #59 — cold-wake latency. Notes why docker pause alone is the wrong answer (it reclaims no memory, which is the whole point of R5: Scale-to-zero idle services #19) and proposes a tiered pause-then-stop design instead.

67c1f69 fixes two of the three races from the plan. The third, waitForHealthyContext, is deliberately not changed: it is written once in NewLoadBalancer before publication and never again, so it is not a race today. It becomes one the moment ResumeFromSleep re-arms it, and the lock belongs in that commit where the second writer appears.