Skip to content

fix(ingress): migrate from Envoy Gateway to DOKS-native Cilium Gateway API #28

Description

@jonathandieu

Problem

Public ingress has been down since 2026-07-09. All workload hostnames resolve but refuse connections. Root cause: the DO CCM is stuck in a SyncLoadBalancerFailed loop (~1300 failures over 4.5 days):

```
SyncLoadBalancerFailed: failed to create load-balancer:
POST .../v2/load_balancers: 422 There is already a load balancer with that name
```

This is the Envoy Gateway CCM ordering bug: Envoy Gateway creates the LoadBalancer Service bare, CCM races to create a new LB before the adoption annotation lands, DO rejects subsequent attempts because an LB with that name already exists. PRs #20, #23, #24, #25 fought symptoms of this without fixing the root cause.

Solution

Migrate to DOKS-native Cilium Gateway API. Cilium creates the LoadBalancer Service with all annotations atomically -- no ordering race possible.

Key findings:

  • DOKS 1.33+ manages the cilium GatewayClass and Gateway API CRDs automatically -- confirmed live (kubectl get gatewayclass shows cilium already present)
  • Gateway API spec.infrastructure.annotations (GEP-1762, Standard status) passes LB annotations to the generated Service atomically at creation
  • The DO Terraform module provisions a LB per cluster and outputs lb_id -- Cilium adopts it via the annotation, keeping the Terraform-managed IP stable across redeploys

Changes

  • Remove Envoy Gateway from the infrastructure ApplicationSet -- DOKS manages the CRDs and GatewayClass; the idle controller provides no value
  • Add charts/infrastructure/cilium-gateway/ -- thin chart with no upstream dependency; creates one Gateway per cluster with lbAnnotations passthrough for provider-specific LB adoption; LB ID flows in via clusters/<name>/overrides/cilium-gateway.yaml
  • Switch all workload HTTPRoutes (plausible, mealie, changedetection) to parentRef: public / cilium-gateway-system
  • Enable cloudflare.proxied: true in external-dns -- routes traffic through Cloudflare for DDoS protection and prepares the Cloudflare Worker failover path

Note on rate limiting (#26)

Envoy Gateway is being removed as the default GatewayClass. Cloudflare edge rate limiting covers the abuse protection need for now. Envoy Gateway can be added back as a scoped second GatewayClass when building the overengineered.dieu.dev load-test demo.

Manual cluster step required before this takes effect

```bash
STG=do-atl1-dieubernetes-stage-do-atl1
kubectl --context "$STG" patch cm cilium-config -n kube-system
--type=merge -p '{"data":{"enable-gateway-api":"true"}}'
kubectl --context "$STG" rollout restart ds/cilium -n kube-system
kubectl --context "$STG" rollout restart deploy/cilium-operator -n kube-system
```

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions