letsencrypt: first-deploy HTTP-01 race leaves the gateway cert failed for an hour

[dcmcand]

What happened

On a first deploy to Hetzner with certificate.type: letsencrypt, the gateway certificate failed to issue and stayed failed for an hour.

cert-manager created HTTP-01 challenges for all four hostnames on nebari-gateway-cert and asked Let's Encrypt to validate before Envoy had programmed the corresponding solver HTTPRoutes. Three hostnames validated; argocd. did not:

invalid: Error accepting authorization: acme: authorization error for argocd.<domain>:
400 urn:ietf:params:acme:error:connection: During secondary validation: <lb-ip>:
Fetching http://argocd.<domain>/.well-known/acme-challenge/<token>: Error getting validation data

A sibling challenge showed the same race in its pending state, naming the cause directly:

pending: Waiting for HTTP-01 challenge propagation: wrong status code '503', expected '200'

One invalid challenge invalidates the whole Order, so the four-hostname nebari-gateway-cert failed as a unit even though three of its hostnames had already validated. cert-manager then applied its post-failure backoff:

Issuing=False (Failed) The certificate request has failed to complete and will be retried:
  Failed to wait for order resource "..." to become ready: order is in "invalid" state
lastFailureTime: 2026-08-28T22:04:49Z

That is a 1 hour wait before the next attempt. Meanwhile the ArgoCD certificates application sits Degraded, and gateway-config / httproutes degrade with it, so a fresh cluster looks broken to anyone who deploys and checks.

The race is self-inflicted and transient: probing the solver path a couple of minutes later returned 200, and deleting the Certificate (letting ArgoCD's selfHeal recreate it) issued successfully on the first retry. The retry needed only the one challenge, since Let's Encrypt had cached the other three authorizations.

Why it matters

This is on the default first-deploy path for any provider using LetsEncrypt with a real domain. The failure is silent in the sense that nothing explains it - the operator sees certificates Degraded and an hour of no progress, with the actual cause buried in a Challenge resource's status.

It also consumes Let's Encrypt production rate limit. Failed validations are capped (5 per hostname per hour), so repeated deploys against the same domain while debugging can lock the domain out.

Suggested fix

Make cert issuance wait for its solver route to be programmed before validation is requested. Options, roughly in order of preference:

• Have the deploy gate Certificate creation on the Gateway reporting Programmed=True and its listeners having attached routes, rather than on the Gateway existing.
• Failing that, shorten the recovery: detect the invalid-order + lastFailureTime state during deploy and recreate the Certificate rather than leaving the operator to wait out cert-manager's hour.
• At minimum, surface the cause. When certificates is Degraded, print the failing Challenge's status.reason so the 503 is visible without hand-querying ACME resources.

Reproduction

1. Deploy to a real domain with certificate.type: letsencrypt (no acme.server override, i.e. production).
2. Watch kubectl get certificate,order,challenge -A during the first few minutes.

Timing-dependent: it reproduces when the ACME validation request beats Envoy's programming of the solver route, which is likelier on a cluster whose gateway is still coming up.

Environment

• nic built from main
• Hetzner provider, k3s v1.35.8+k3s1, 1 master + 3 workers
• Envoy Gateway v1.6.2 (also present at v1.9.1 - unrelated to the chart version)
• cert-manager v1.17.2, config.enableGatewayAPI: true
• LetsEncrypt production