perf(health-check): retry quickly until a target first answers

[mhenrixon]

Summary

Cold wake: 1154 ms → 278 ms. Measured end to end against a real Docker daemon, same machine, same container, before and after.

#59 opened with one instruction — "Measure ours first. Before optimising, get a real number for this implementation on a real app." Now that #19 has landed, that was finally possible. The measurement changed the answer.

What the measurement found

A container whose app is ready almost instantly took 1154 ms to wake. That decomposes badly:

Component	Measured
docker start	~215 ms
app boot (python http.server)	~0 ms
our own overhead	~940 ms

The proxy log says exactly where it went:

15:52:44.733  Healthcheck failed ... connection refused
15:52:45.742  Target health updated ... healthy        ← 1009 ms later
15:52:45.743  Service awake

docker start returns when the container process is created, not when the application is accepting connections. So the immediate probe is routinely refused — and the next one came a full check interval later. The wake was quantised to the health-check interval: roughly a second of pure waiting, for nothing.

The fix

HealthCheck.run probes immediately, then retries from 50 ms doubling up to the configured interval until the target first answers, and settles to the configured interval afterwards.

• A running target is probed no harder than before — the fast cadence only applies before the first success.
• A container that never comes up backs off rather than spinning.
• Deploys get it too. Waiting for a new target to become healthy is the same wait, paying the same tax.

No new flag, no new state machine, no new config to explain.

Why not the tiered pause/stop this issue proposed

The issue's own design was docker pause for short idles and docker stop for long ones. The measurement says that is now the wrong first move:

• Of the remaining 278 ms, ~215 ms is docker start. I measured docker unpause at 75 ms median vs docker start at 215 ms — so the pause tier buys ~140 ms plus the app's own boot.
• For an instant-booting app that is marginal. For a Rails app (~3 s boot) it is still worth real seconds, because a paused process is already booted.
• But it costs a third controller state, a second threshold flag, and — critically — reclaims no memory, which is the entire point of R5: Scale-to-zero idle services #19.

So: the cheap, universal win is in, and the tier is now a data-informed choice rather than a guess. If Rails cold starts still annoy in practice, reopen it as its own issue — the ContainerLifecycle seam makes it additive.

Test plan

• RetriesQuicklyUntilTheFirstSuccess — a target ready 120 ms in is noticed without waiting out a 1 s interval. Fails on the old code at 881 ms, which is the production symptom in miniature.
• SettlesToTheConfiguredIntervalAfterSuccess — a healthy target is not probed at the wake cadence forever
• BackoffIsBoundedByTheConfiguredInterval — a container that never comes up is not hammered
• gofmt, go vet clean; go test -race — 1504 pass

Deviations & judgment calls

• I did not build what the issue asked for, and the measurement is why. R5: Reduce scale-to-zero cold-wake latency (tiered pause/stop) #59 proposed tiered pause/stop. Measuring first — which the issue itself demanded — showed the dominant cost was our own interval quantisation, not the container runtime. Fixing that is smaller, helps deploys too, and needs no new operator-facing concepts. The tier is deliberately deferred with the numbers above, not forgotten.
• I nearly reported my own test harness as a proxy bug. My first measurement showed requests returning 200 in 1 ms against a stopped container, with no wake in the log. That looked like a serious defect. It was my setup: I had deployed with --health-check-path / (python has no /up) and then measured requests to /, so the gate was correctly answering a health check for a sleeping service without waking it. Worth recording, because the false alarm was more convincing than the real bug.
• The backoff starts at 50 ms and doubles. Both numbers are judgment. 50 ms is small enough that a container listening almost immediately is noticed almost immediately; doubling to the configured interval means a slow boot is not probed hundreds of times. Easy to tune if it proves wrong.
• This changes deploy timing too, not just wakes. Faster is the intent, and it makes --health-check-interval mean "how often to check a healthy target" rather than also "how long a new one might sit ready but unnoticed" — arguably what operators assumed it meant already.
• Bundled a -race flake fix from feat(router): install the container runtime and complete scale-to-zero #66. TestService_CacheMissWakesASleepingService could race t.TempDir cleanup against a persist still in flight. Cleanup now takes saveLock. It is unrelated to the perf work, but it is a test I wrote, it is flaky on dash today, and it surfaced in this branch's -race run.
• The 278 ms is for an instant-booting container. A real Rails app will be dominated by its own boot; this removes the proxy's contribution, not the app's. No Rails measurement was taken — do not let 278 ms follow this into a release note as "the cold wake".

Closes #59