Problem
Public ingress has been down since 2026-07-09. All workload hostnames resolve but refuse connections. Root cause: the DO CCM is stuck in a SyncLoadBalancerFailed loop (~1300 failures over 4.5 days):
```
SyncLoadBalancerFailed: failed to create load-balancer:
POST .../v2/load_balancers: 422 There is already a load balancer with that name
```
This is the Envoy Gateway CCM ordering bug: Envoy Gateway creates the LoadBalancer Service bare, CCM races to create a new LB before the adoption annotation lands, DO rejects subsequent attempts because an LB with that name already exists. PRs #20, #23, #24, #25 fought symptoms of this without fixing the root cause.
Solution
Migrate to DOKS-native Cilium Gateway API. Cilium creates the LoadBalancer Service with all annotations atomically -- no ordering race possible.
Key findings:
- DOKS 1.33+ manages the
cilium GatewayClass and Gateway API CRDs automatically -- confirmed live (kubectl get gatewayclass shows cilium already present)
- Gateway API
spec.infrastructure.annotations (GEP-1762, Standard status) passes LB annotations to the generated Service atomically at creation
- The DO Terraform module provisions a LB per cluster and outputs
lb_id -- Cilium adopts it via the annotation, keeping the Terraform-managed IP stable across redeploys
Changes
- Remove Envoy Gateway from the infrastructure ApplicationSet -- DOKS manages the CRDs and GatewayClass; the idle controller provides no value
- Add
charts/infrastructure/cilium-gateway/ -- thin chart with no upstream dependency; creates one Gateway per cluster with lbAnnotations passthrough for provider-specific LB adoption; LB ID flows in via clusters/<name>/overrides/cilium-gateway.yaml
- Switch all workload HTTPRoutes (
plausible, mealie, changedetection) to parentRef: public / cilium-gateway-system
- Enable
cloudflare.proxied: true in external-dns -- routes traffic through Cloudflare for DDoS protection and prepares the Cloudflare Worker failover path
Note on rate limiting (#26)
Envoy Gateway is being removed as the default GatewayClass. Cloudflare edge rate limiting covers the abuse protection need for now. Envoy Gateway can be added back as a scoped second GatewayClass when building the overengineered.dieu.dev load-test demo.
Manual cluster step required before this takes effect
```bash
STG=do-atl1-dieubernetes-stage-do-atl1
kubectl --context "$STG" patch cm cilium-config -n kube-system
--type=merge -p '{"data":{"enable-gateway-api":"true"}}'
kubectl --context "$STG" rollout restart ds/cilium -n kube-system
kubectl --context "$STG" rollout restart deploy/cilium-operator -n kube-system
```
Problem
Public ingress has been down since 2026-07-09. All workload hostnames resolve but refuse connections. Root cause: the DO CCM is stuck in a SyncLoadBalancerFailed loop (~1300 failures over 4.5 days):
```
SyncLoadBalancerFailed: failed to create load-balancer:
POST .../v2/load_balancers: 422 There is already a load balancer with that name
```
This is the Envoy Gateway CCM ordering bug: Envoy Gateway creates the LoadBalancer Service bare, CCM races to create a new LB before the adoption annotation lands, DO rejects subsequent attempts because an LB with that name already exists. PRs #20, #23, #24, #25 fought symptoms of this without fixing the root cause.
Solution
Migrate to DOKS-native Cilium Gateway API. Cilium creates the LoadBalancer Service with all annotations atomically -- no ordering race possible.
Key findings:
ciliumGatewayClass and Gateway API CRDs automatically -- confirmed live (kubectl get gatewayclassshowsciliumalready present)spec.infrastructure.annotations(GEP-1762, Standard status) passes LB annotations to the generated Service atomically at creationlb_id-- Cilium adopts it via the annotation, keeping the Terraform-managed IP stable across redeploysChanges
charts/infrastructure/cilium-gateway/-- thin chart with no upstream dependency; creates one Gateway per cluster withlbAnnotationspassthrough for provider-specific LB adoption; LB ID flows in viaclusters/<name>/overrides/cilium-gateway.yamlplausible,mealie,changedetection) toparentRef: public / cilium-gateway-systemcloudflare.proxied: truein external-dns -- routes traffic through Cloudflare for DDoS protection and prepares the Cloudflare Worker failover pathNote on rate limiting (#26)
Envoy Gateway is being removed as the default GatewayClass. Cloudflare edge rate limiting covers the abuse protection need for now. Envoy Gateway can be added back as a scoped second GatewayClass when building the overengineered.dieu.dev load-test demo.
Manual cluster step required before this takes effect
```bash
STG=do-atl1-dieubernetes-stage-do-atl1
kubectl --context "$STG" patch cm cilium-config -n kube-system
--type=merge -p '{"data":{"enable-gateway-api":"true"}}'
kubectl --context "$STG" rollout restart ds/cilium -n kube-system
kubectl --context "$STG" rollout restart deploy/cilium-operator -n kube-system
```