feat(san-cert): SAN batch resilience — eviction grace, shrink guard, renewal preflight, batch quarantine

[mhenrixon]

Summary

Closes the three gaps that let a dead tenant domain or a bad domain-source poll degrade or destroy SAN certificate coverage (the WM3 scenario: ~1100 tenant domains, many dead). No new flags; no RPC verbs; behavior-only.

• Eviction grace period (internal/server/domain_renewal.go) — a certificate whose domains were all evicted is never deleted before its own NotAfter: it stops renewing and keeps serving. Only a superseded cert (no domain maps through it anymore — a loaded replacement answers every handshake) is removed immediately. A bad poll + reconcile + recovery now costs zero new ACME orders. The per-domain expiry gauge is reported only by the cert a domain currently maps to, so lingering certs can't clobber their successor's metric.
• Shrink guard (internal/server/dynamic_domains.go) — a poll removing more than 30% of a service's applied domain set has its removals held (additions still apply) until 3 consecutive over-threshold polls confirm the shrink; a recovering poll clears the hold. Held domains keep serving existing certs but earn no new orders. Visible in kamal-proxy domains list (Removal held column) and domains stats; every held poll logs at Warn.
• Renewal member preflight (internal/server/domain_renewal.go) — every renewal probes its dynamic members before spending an order; unreachable ones follow the existing quarantine policy (defer while there's time, compact inside the 7-day window). Deploy-registered hosts are not probed (may be DNS-01-only by design); wildcards have nothing to answer on.
• Probe-based failure attribution (new internal/server/domain_failure.go) — identifyFailedDomains: parse lego's per-domain error lines; failing that, probe the members; failing that, hold the whole batch on the quarantine ladder. Fixes the renewer's quarantine-nobody hole (a doomed batch used to retry hourly, unbacked-off, until the whole cert expired) and gives the issuer culprit-aware survivor retries.
• Handshake-batch guard (new internal/server/san_cert_batch_guard.go) — provisionCertificate filters batch-mates through quarantine + preflight (they keep their pending slot), and attributes order failures: culprits quarantined, survivors restored. The triggering domain is never dropped. Wired via SetIssuanceGuard from NewDynamicDomainManager at boot.

Closes #96

Test plan

• make test — 1768 server-package tests green (21 new)
• go test -race ./internal/server/ — clean
• make lint — 0 issues (golangci-lint v2.11.3), gofmt -l empty, go vet clean
• make docker && docker run --rm kamal-proxy kamal-proxy -h — image smoke test
• Wipe scenario end-to-end at unit level: certs adopted → empty poll → reconcile → zero certs deleted, hold visible → recovery poll → zero new orders (TestCertRenewer_EvictedCertificateSurvivesSourceRecovery, TestDynamicDomainManager_ShrinkGuardHoldsEmptyPoll)
• Staging: point a service's --tls-domains-source at an endpoint, return an empty list once, confirm domains list shows held removals and no cert files disappear from the store

Deviations & judgment calls

• Discovery: since the plan was written, feat(wildcard-certs): per-zone DNS-01 provider selection #94 landed per-zone DNS-01 partitioning — certRenewer.renew now splits a renewal into one order per provider partition (renewPartition), and provisionCertificate narrows a handshake batch to one partition. The grace/probe logic wraps around that structure rather than the single-order flow the plan described.
• Judgment call: the eviction grace period distinguishes superseded from evicted. A cert none of whose domains map to it anymore (superseded — a replacement is loaded) is GC'd immediately as before; only a cert that still serves at least one mapped domain is kept until its own NotAfter. Blanket keep-everything would leave zombie certs clobbering the per-domain expiry gauges.
• Related judgment call: the per-domain expiry gauge is now reported only by the certificate a domain currently maps to — a lingering evicted cert must not overwrite its successor's metric (map iteration order made the winner random before).
• Deviation: the plan said renewal probes "all members"; it probes only dynamic members. A deploy-registered host may be DNS-01-only and unreachable over HTTP by design — probing it would wrongly quarantine a domain that renews fine. Same skip applies to probe-based culprit identification on the renewal path.
• Deviation: the plan's "wildcard members are never probed" renewal test is structurally unreachable (wildcards are never dynamic, so the dynamic-only rule already excludes them); the *. guard is kept as belt-and-braces and unit-tested in identifyFailedDomains instead.
• Judgment call: during a shrink hold, held domains keep their allowlist entry and any live certificate but earn no new ACME orders — issuance follows the polled list. A name the source stopped reporting shouldn't spend orders while its removal is in question.
• Judgment call: the shrink guard has no absolute floor — removing 1 of 2 domains (50%) is held like removing 600 of 1100. The cost of holding is ~2 extra polls (10 min at default interval); the cost of a false-negative wipe is the estate. Threshold and confirmation count are constants (30%, 3), not flags, per the interview decision.
• Judgment call: hold state is memory-only. A restart reloads the last applied (unshrunk) set, so the confirmation count restarts — conservative direction.
• Deviation from issuer semantics on the deploy path: an unattributable handshake-batch failure (generic ACME outage) quarantines nobody and restores everything to pending, exactly as the unguarded path did. The issuer/renewer quarantine the whole batch in that case; deploy-registered hosts are operator-curated and shouldn't ride a 24h ladder because Let's Encrypt had a bad day. Only culprits identified by error-parsing or probing are quarantined.
• Discovery/refactor: provisionCertificate used to collect the batch and mark it non-pending under one lock. The guard's probes do network I/O, so the provisioning slot is now claimed first (concurrent handshakes wait on it), then probes run unlocked, then the kept domains leave pending. No new race: the slot was always the mutual exclusion.
• Moved failedDomainsFromError from domain_issuer.go into the new domain_failure.go alongside identifyFailedDomains (shared by issuer + renewer + batch guard).

---

Summary by cubic

Improves SAN certificate resilience: prevents bad domain polls and failing tenants from tearing down coverage, and reduces wasted ACME orders with preflight checks and smarter failure handling. Addresses #96.

• New Features

  • Eviction grace period: fully-evicted certs keep serving until their own expiry; only superseded certs are removed immediately. Per-domain expiry is reported only by the cert a domain currently maps to.
  • Shrink guard: holds removals when a poll drops >30% of a service’s applied set until 3 consecutive polls confirm; additions still apply. Held domains keep serving but get no new orders. Visible in kamal-proxy domains list (Removal held) and domains stats. While holding, the source ETag is cleared to avoid 304 loops; the hold resets on redeploy.
  • Renewal preflight: probes dynamic members with bounded concurrency before renewal; unreachable ones are quarantined and deferred (when there’s time) or compacted out inside the 7‑day window. Skips deploy-registered hosts and wildcards.
  • Probe-based failure attribution: parse per-domain ACME errors; if absent, probe members (bounded concurrency); else quarantine the whole batch so retries back off. Shared by issuer and renewer.
  • Handshake-batch guard: filters quarantined/unreachable batch-mates out of provisioning (they keep their pending slot and don’t consume batch capacity). Probes every non-trigger mate, quarantines identified culprits on failure, and restores survivors; unattributable failures restore everyone. Claims the provisioning slot before probes, clears batch failure history on success, persists quarantine changes, and is wired via SetIssuanceGuard.

• Bug Fixes

  • Stops renewal batches from looping hourly with no backoff when ACME errors lack per-domain attribution.
  • Prevents a single bad poll from deleting certs or mass-evicting domains; recovery costs zero new ACME orders.
  • CLI/stats: domains list shows “Removal held”; domains stats counts held removals.

^{Written for commit 260ec16. Summary will update on new commits.}

[cubic-dev-ai]

All reported issues were addressed across 14 files

<sub>Tip: instead of fixing issues one by one fix them all with cubic

Re-trigger cubic</sub>