Balancer: point balancerproject.org at Envoy LB by 2026-07-27

[themightychris]

For the Balancer team. Deadline: 2026-07-27 — after that the TLS cert expires and the site breaks.

balancerproject.org still points at the old ingress-nginx load balancer, which can no longer renew certificates (see #144). It must move to Envoy.

What to change

In whatever DNS provider hosts balancerproject.org (Google Cloud DNS, project unknown to us — we don't have access):

Record	Type	Value
balancerproject.org	A	45.79.246.168

www.balancerproject.org is a CNAME to the apex, so it follows automatically. Set TTL to 60 first if it isn't already, so rollback is fast.

What to expect

• Everything is staged on our side. We verified the ACME challenge already answers correctly through Envoy, so the cert issues automatically within ~1-2 minutes of the DNS change. No action needed from us.
• ~60–90 seconds of downtime at the moment of the flip: Envoy redirects HTTP→HTTPS, and its HTTPS listener doesn't come up until the cert lands.
• Rollback = set the A record back to 104.237.148.175. This only works until we decommission ingress-nginx, so go before then if you want a bail-out.

Do it whenever you like, but before Jul 27

Two hard limits:

1. 2026-07-27 — current cert expires.
2. ingress-nginx decommission (Migrate to Envoy Gateway: ingress-nginx → Gateway API + cert-manager gatewayHTTPRoute #144 phase 5.5) — blocked on this issue.

Please comment here when it's done.

[TineoC]

Done 💯

[TineoC]

It might take a few minutes, as TTL is 4 hours

[TineoC]

$ curl -v --max-time 10 http://balancerproject.org/ 2>&1 | head -30
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Host balancerproject.org:80 was resolved.
* IPv6: (none)
* IPv4: 45.79.246.168
*   Trying 45.79.246.168:80...
* Connected to balancerproject.org (45.79.246.168) port 80
* using HTTP/1.x
> GET / HTTP/1.1
> Host: balancerproject.org
> User-Agent: curl/8.12.1
> Accept: */*
> 
* Request completely sent off
< HTTP/1.1 404 Not Found
< date: Tue, 14 Jul 2026 16:27:40 GMT
< content-length: 0
< 

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connection #0 to host balancerproject.org left intact

Could you please confirm if the app is ready to use the new ip?

I don't think we have HTTPRoutes in this environment yet

[TineoC]

[TineoC]

DNS change complete — balancerproject.org → 45.79.246.168 (Envoy LB) ✅

Root cause found: HTTP returns 404 for the apex domain because the per-app Gateway/ListenerSet claims balancerproject.org for HTTPS only. With mergeGateways: true, the shared main-gateway's catch-all HTTP→HTTPS redirect stops applying to claimed hostnames. The www subdomain works because no Gateway explicitly claims it.

Fix

Two PRs opened:

1. balancer-main#517 — Adds balancer-http-redirect HTTPRoute attached to the http section of the ListenerSet, returning 301→HTTPS. Also includes kustomize hygiene fixes (disableNameSuffixHash, corrected ClusterIssuer annotation).

2. cfp-live-cluster#166 — Removes the conflicting standalone Gateway in _gateways/balancer.yaml (redundant with the ListenerSet approach).

Merge order

1. ✅ Merge balancer-main#517
2. ➡️ Trigger hologit rebuild (build-k8s-manifests workflow on main)
3. ➡️ Merge cfp-live-cluster#166

After deploy: HTTP should return 301→HTTPS, ACME challenge should complete, and the TLS cert should auto-issue within ~1-2 minutes.

[TineoC]

Status update

Done:

• ✅ balancer-main#517 — merged (added HTTP redirect route to ListenerSet)
• ✅ balancer-main#463 — merged (develop→main release, gets fix onto the branch hologit tracks)

Still needed (requires admin on this repo):

• ➡️ Trigger build-k8s-manifests workflow (push to main or manual dispatch) — regenerates the holobranch with the fixed ListenerSet + redirect
• ➡️ Merge cfp-live-cluster#166 — removes the conflicting standalone Gateway (safe only after hologit rebuild)

[TineoC]

@themightychris @dherbst — hologit rebuild on cfp-live-cluster is needed to pick up the balancer-main fix. Can someone trigger the build-k8s-manifests workflow (push to main or manual dispatch) and merge #166?

[themightychris]

merging #166 will do it

[TineoC]

Go ahead

[themightychris]

Done — balancerproject.org is on Envoy and serving a fresh cert.

The DNS change landed; I cleared a stale ACME authorization that had expired during the 57 days the hostname was stuck on nginx, and the cert issued immediately.

balancerproject.org   http=301 -> https   https=200
balancer-gw-tls       Ready, valid to 2026-10-12

The HTTP 404 reported in #166 is gone, and it resolved itself: it was cert-manager's ACME solver HTTPRoute (cm-acme-http-solver-*), which declares an exact hostname and therefore outranks http-redirect's catch-all, while only matching the challenge path. cert-manager deletes that route the moment the challenge succeeds. No Gateway change and no ListenerSet were needed — see #166 for the full trace.

Two things for the balancer team:

Your upstream Gateway API manifests aren't consumed by this cluster. Routing for balancerproject.org comes from _gateways/balancer.yaml here. gateway-listeners.yaml declares a ListenerSet, which Envoy Gateway v1.7.3 does not reconcile (XListenerSet CRD not found, skipping XListenerSet watch) — it would apply cleanly and be silently ignored. Revisit at EG v1.8. Adoption of v1.1.6 is tracked in #168.

www.balancerproject.org has never worked — the legacy cert covered only the apex, the Ingress served only the apex, and www fails TLS on nginx too. It CNAMEs to the apex and now reaches Envoy, but no listener matches its SNI. If you want www, it needs its own listener + cert.

[TineoC]

Are there any rules for www? In the gateway or you prefer to redirect www.balancerproject.org to balancerproject.org