GCP events edge: apply reports success ~10 minutes before the endpoint serves

[artemnikitin]

Summary

After a full GCP control-plane deploy, terraform apply reports success roughly
ten minutes before the events and status hostnames actually serve traffic.
During that window both hostnames reset the TLS connection
(SSL_ERROR_SYSCALL / Connection reset by peer), which is indistinguishable
from a broken deployment.

Two independent measurements, on 2026-07-25 and 2026-08-24, agree to within
about a minute. This is reproducible, not a one-off.

Measured timeline (2026-08-24, full fresh deploy)

Elapsed	Event
+0:00	Gateway/firework-events created
+1:09	control-plane pod Running and reconciling
+4:08	backend services and NEGs created, endpoints attached
+7:19	url map → target HTTPS proxy → global forwarding rule
+8:15	Gateway Programmed=True, Ready=True, address bound
+10:33	first byte served

The 2026-07-25 run: +7:26 to the forwarding rule, +9:56 to serving.

Throughout, nothing was wrong. Backends were HEALTHY, Certificate Manager
entries ACTIVE, DNS correct, and the control-plane pod had zero restarts and
was reconciling normally. The only missing piece was the GCLB front end, which
does not exist until the GKE Gateway controller finishes building it.

Where the time goes

Three roughly three-minute phases:

1. pod Ready → NEG created. GKE Gateway controller reconcile latency.
2. NEG → forwarding rule. The controller builds backend service, health
   check, url map, target proxy, and forwarding rule serially.
3. forwarding rule → first byte. Global load balancer propagation.

Phases 2 and 3 are Google-side. Nothing in this repository shortens them.

Note that the Gateway is applied before the control-plane pod starts, so there
is no remaining overlap to win by reordering the apply — the manifest is already
submitted as early as it can be.

Why this is worth fixing

Two separable problems:

• Speed. Ten minutes per full rebuild.
• Honesty. terraform apply exits 0 while the endpoint it just created
  refuses connections. terraform/control-plane/gcp/README.md already tells
  operators to wait for the Gateway before configuring the GitHub webhook, so
  the wait is known — it is just not enforced or surfaced.

Only the second is fully fixable here. The first is mostly avoidable rather
than shortenable.

Proposed direction

1. Make the slow path rare instead of faster (biggest practical win)

The ten minutes is the cost of building a GCLB front end from scratch. It is
paid only when the Gateway is created — that is, on a full teardown and rebuild.
It is not paid when only the control-plane image changes: the durable edge
stack already holds the static address, DNS records, DNS authorizations,
certificates, and certificate map, and README:130 notes that a normal apply
rolls the control-plane pod template in place.

So the iteration loop should be:

• keep the cluster and Gateway up between iterations;
• change controlplane_image / controlplane_deployment_revision and apply,
  which rolls the Deployment in seconds;
• destroy and rebuild only when something below the Deployment actually changed.

This is a documentation and workflow change, not a code change, and it removes
the ten minutes from the common case entirely. Worth stating explicitly in the
provider README next to the existing deploy instructions.

2. Stop reporting success before the endpoint serves

Add a readiness gate after the Gateway, matching the existing
terraform_data.validate_events_edge_wiring idiom rather than introducing a new
pattern:

resource "terraform_data" "events_gateway_ready" {
  depends_on = [
    kubectl_manifest.events_gateway,
    kubectl_manifest.events_route,
    kubectl_manifest.status_route,
  ]

  provisioner "local-exec" {
    command = <<-EOT
      kubectl wait --for=condition=Programmed \
        gateway/firework-events -n firework --timeout=15m
      # then poll until the front end actually answers
      for i in $(seq 1 60); do
        curl -sS -o /dev/null --max-time 10 \
          "https://${var.status_domain}/" && exit 0
        sleep 15
      done
      echo "gateway programmed but endpoint still not serving" >&2
      exit 1
    EOT
  }
}

Programmed=True alone is not sufficient — in the 2026-08-24 run it appeared at
+8:15 and the endpoint did not serve until +10:33. The condition and the probe
are both needed.

This adds wall-clock time to apply. It fixes correctness, not speed: the
operator waits either way, and this makes the waiting visible and bounded
instead of leaving them to discover a resetting endpoint.

3. Confirm the Gateway is not replaced on no-op applies — needs checking

kubectl_manifest.events_gateway builds its yaml_body with yamlencode over
local.effective_events_gateway_address_name and
local.effective_events_certificate_map_name, both resolved through try()
against the events-edge state file read by path. If that read ever resolves
differently between runs, yaml_body changes and the Gateway is replaced —
tearing down and rebuilding the entire GCLB on an apply that changed nothing.

The 2026-08-24 occurrence was a genuine full rebuild (the VPC, cluster, and node
instance templates were all created in the same window), so this was not the
cause there. But it has not been ruled out as a latent trigger. A terraform plan against an already-applied stack, with the events-edge state present and
then relocated, would settle it.

4. Regional instead of global load balancer — unverified, probably not worth it

A regional external Application Load Balancer
(gke-l7-regional-external-managed) is expected to program faster than the
global anycast one, because it does not propagate to the global front-end fleet.

This is unverified and likely not a good trade here. The durable edge stack
provisions a global address plus a Certificate Manager map consumed through
the networking.gke.io/certmap annotation, with DNS authorizations already
issued against that arrangement. Moving to a regional gateway requires different
address and certificate wiring, so it is not a gatewayClassName one-liner. It
also gives up global anycast. Recorded for completeness; measure before pursuing.

Acceptance criteria

• A full deploy either does not report success before the events and status
  hostnames serve, or prints an explicit, bounded wait with the current phase.
• The provider README documents the fast iteration path (image-only apply) and
  states plainly that a full rebuild costs roughly ten minutes at the edge.
• The expected phase sequence and its rough timings are written down where an
  operator debugging a resetting hostname will find them, including the point
  that healthy backends plus a missing forwarding rule means "still building",
  not "broken".
• It is confirmed whether a no-op apply can replace the Gateway, and if it can,
  the input is stabilised.

Diagnostic sequence for a resetting hostname

In order, stopping at the first thing missing:

1. reserved address RESERVED vs IN_USE;
2. a global forwarding rule for that address;
3. the target HTTPS proxy and its certificate map;
4. certificate map entries ACTIVE;
5. Gateway Programmed, not only Accepted and ResolvedRefs;
6. backend service health;
7. control-plane pod restarts and logs.

Backend services and NEGs existing is not evidence that the front end
exists. If the pod is healthy and reconciling while the forwarding rule is
absent, the deployment is fine and still building.