feat(server): Docker socket client behind the ContainerLifecycle seam

[mhenrixon]

Summary

The Docker backend for scale-to-zero (#19), behind the ContainerLifecycle seam that merged in #58. Three Engine API endpoints over a unix socket, no external dependency — the official SDK would add a large dependency tree to a proxy that has none.

Still inert. Nothing constructs or calls this yet; the idle controller only gets wired up when the request gate lands. Proxy behavior is unchanged.

Reaching /var/run/docker.sock is root-equivalent on the host, so the client is only built when the operator passes --docker-socket (that flag lands with the wiring).

Two defects found by attacking the first draft against a live daemon

I ran an adversarial pass over the first version — three lenses (wire contract, security, concurrency), 15 raw findings, each then independently re-verified by executing it. 13 were refuted. Two survived, both reproduced against a real Docker daemon, and both are fixed here.

1. /stop sent no t, so the daemon's kill deadline sat outside ours

Docker's SIGTERM→SIGKILL wait comes from the container's StopTimeout, which the proxy cannot see — kamal passes options: straight through to docker run, and compose has stop_grace_period.

Measured against a live daemon with a PID-1 that ignores SIGTERM:

-- default (daemon default 10s) --
StopContainer took 10.1s, err=<nil>
-- created with --stop-timeout 60 --
StopContainer took 30.0s, err=... context deadline exceeded
immediately after: running=true
           +30s:   exited  running=false   <-- daemon completed the stop we gave up on

The controller reads that error as a failed stop, rolls the service back to active and calls resume() — putting the targets back for a container Docker is about to kill. The service then serves 502s from a live-looking target until the next full idle period.

Fixed by deriving t from the caller's remaining budget, so the daemon's deadline lands inside ours with a margin. One correction to my own first diagnosis: http.Client.Timeout is not the binding deadline here — the controller's containerStopTimeout context is — so sending t is the load-bearing half, not touching the client timeout.

2. Redirects were followed, and Go downgrades a redirected POST to GET

Docker's router cleans the decoded path and answers 301 to the canonical form. Measured:

GET  /v1.54/containers/web-1%2F/json  -> 301 Location=/v1.54/containers/web-1/json
POST /v1.54/containers/web-1%2F/start -> 301 Location=/v1.54/containers/web-1/start

ContainerExists("web-1/") = <nil>                  <-- deploy preflight PASSES
StartContainer("web-1/")  = container not found    <-- wake FAILS

GET /json exists, GET /start does not (POST-only route), so the preflight accepts exactly the reference the wake path can never start. Against a redirecting socket proxy — the "hardened proxy" case this client is explicitly built for — a stop could return nil having stopped nothing. Fixed with CheckRedirect: http.ErrUseLastResponse; nothing in the Engine API legitimately redirects.

Other behavior worth knowing

• Version negotiation caches only a success. A briefly-unreachable daemon would otherwise pin the client to the fallback for the life of the process.
• /version gets a 1 MB read limit, errors get 4 KB. Different on purpose: a plugin-heavy host's Components array is far larger than any error body, and truncating that JSON turns a healthy daemon into a parse failure that silently drops to the fallback version.
• 304 is success on both start and stop — that is what makes a coalesced wake against an already-running container work, and what lets a proxy whose state file said "sleeping" for a container that never stopped heal itself.

Test plan

18 tests over a real unix listener, no Docker required, no network:

• NegotiatesAndUsesVersionedPaths — the wire contract, and that the version is negotiated once
• PathEscapesTheContainerReference — asserts on RequestURI, not URL.Path
• TreatsNotModifiedAsSuccess, ClassifiesMissingAndForbidden (404/403 → sentinels)
• FallsBackWhenVersionIsUnavailable — table over denied endpoint, non-JSON body, JSON without the field
• DoesNotCacheANegotiationFailure, ReadsALargeVersionPayload, TruncatesLongErrorBodies
• BoundsTheDaemonStopInsideTheCallersDeadline + StopWithoutADeadlineStillBoundsTheDaemon — finding 1
• DoesNotFollowRedirects — finding 2
• SurfacesAnUnreachableSocket, RespectsContextCancellation
• make test, go vet, gofmt clean; go test -race — 1174 pass

Deviations & judgment calls

• I caught one of my own tests being worthless. PathEscapesTheContainerReference originally asserted on r.URL.Path, which Go hands the handler already decoded — so it passed whether or not the code escaped anything. Switched to RequestURI and confirmed by mutation: strip url.PathEscape and it now fails with we/ird%20name instead of we%2Fird%20name. That class of test looks fine in review and tests nothing.
• dockerStopMargin is 5s and defaultDockerStopTimeout is 10s — both are judgment, not derivation. The margin has to cover a socket round trip and the daemon's own bookkeeping; 5s is generous. The default matches Docker's own, so containers that never configured a StopTimeout behave exactly as before. Easy to flip if either proves wrong in practice.
• A caller already out of budget gets t=1, not t=0. Docker reads 0 as "SIGKILL immediately", which denies the app any chance to shut down cleanly. One second is a poor deadline but a better default than none.
• Concurrent first-calls can each issue GET /version. N simultaneous wakes on a cold client means up to N negotiation requests. Judged acceptable rather than worth a singleflight: it happens once per process, the requests are cheap, and the alternative adds a second synchronisation path to a client whose only shared state is one string. Flagged rather than silently accepted — the verifier agreed it is a thundering herd, not a correctness bug.
• Not measured: no benchmark. This is not on a request hot path — it runs once per sleep and once per wake.

Refs #19