Skip to content

bug: moving a route's parentRefs to a Gateway of another GatewayClass leaves orphaned config in the old data plane #2833

Description

@johannes-engler-mw

Current Behavior

When an HTTPRoute's parentRefs are repointed from a Gateway managed by controller A to a
Gateway managed by controller B (a different GatewayClass.spec.controllerName), controller B
picks the route up correctly — but controller A never removes the configuration it had already
pushed
. Its data plane keeps serving the route indefinitely.

The chain, on master (be4aaef) and on the released 2.1.0:

  1. httproute_controller.go:75 — the reconciler is registered with
    For(&gatewayv1.HTTPRoute{}) and no controller-name predicate, so every controller
    instance watches every HTTPRoute. Repointing parentRefs bumps the generation, so
    GenerationChangedPredicate passes and controller A does reconcile the change.

  2. utils.go:357-359ParseRouteParentRefs skips parentRefs whose GatewayClass belongs to
    another controller:

    if string(gatewayClass.Spec.ControllerName) != config.ControllerConfig.ControllerName {
        continue
    }

    For controller A the route's only remaining parentRef is now controller B's Gateway, so the
    returned slice is empty.

  3. httproute_controller.go:172-174 — the empty slice returns early with no cleanup:

    if len(gateways) == 0 {
        return ctrl.Result{}, nil
    }

    Provider.Delete is only reachable from the IgnoreNotFound branch at :147, i.e. when the
    object is deleted from the cluster. There is no path for "this object used to be mine and no
    longer is".

  4. adc/client/client.go:290,303 — the periodic sync replays the in-memory store
    (ConfigManager.List()GetResources(name)); it never re-derives state from Kubernetes.
    The orphaned service therefore gets re-pushed to the data plane every sync period, forever.

Restarting the controller pod clears it, because startup rebuilds the store from the informer's
initial list and the route no longer produces an entry for controller A.

The same early return exists in all five Gateway API route reconcilers:

httproute_controller.go:172
grpcroute_controller.go:190
tcproute_controller.go:283
udproute_controller.go:283
tlsroute_controller.go:283

Why this is hard to notice

Every Kubernetes-level signal is correct and green while it happens:

  • the HTTPRoute's status.parents lists only controller B, with Accepted=True
  • controller B's Gateway shows the route in status.listeners[].attachedRoutes
  • controller A's Gateway shows the correct (reduced) attachedRoutes count
  • controller A logs no errors — its sync loop reports success every period
  • GitOps tooling reports the manifests as fully synced

The stale configuration is visible only in the old data plane's own control API
(:9090/v1/services). In our case the orphaned routes carried public hostnames, so this failed
open: hosts that were supposed to have been taken off the public gateway stayed publicly
reachable for over 21 hours, with the label metadata and modifiedIndex still showing values
from weeks before the change.

Prior art

PR #2543 ("fix: residual data issue when updating ingressClassName", merged 2025-09-08) fixed
exactly this failure mode for Ingress, and its description states the problem in general terms:

the controller simply ignores the Ingress resource. However, this behavior is problematic: the
corresponding configuration is not removed and remains in the data plane, which may result in
incorrect routing or stale configurations.

The resulting code in ingress_controller.go:159-166 calls Provider.Delete when the
IngressClass no longer matches. The Gateway API route reconcilers were never given the
equivalent.

Expected Behavior

When a route no longer references any Gateway managed by this controller, the controller removes
the configuration it previously pushed for that route, so the data plane stops serving it —
matching the Ingress/IngressClass behaviour from #2543.

Error Logs

None. That is a significant part of the problem: the controller logs no error at any log level,
and the ADC sync reports success on every tick while re-pushing the orphaned configuration.

Steps to Reproduce

  1. Install two APISIX ingress controller instances with distinct controller names, e.g.
    apisix.apache.org/apisix-ingress-controller and
    apisix.apache.org/apisix-ingress-controller-internal, each with its own GatewayClass and
    Gateway (gateway-a in namespace apisix, gateway-b in namespace apisix-internal).

  2. Create an HTTPRoute with parentRefs: [gateway-a] and a hostname, and confirm it serves:

    $ curl -H 'Host: demo.example.com' http://<gateway-a>/
    200
  3. Change only the parentRefs to gateway-b (keep the same name/namespace) and apply.

  4. Observe that controller B accepts the route and gateway-b serves it — and that gateway-a
    still serves it too:

    $ curl -H 'Host: demo.example.com' http://<gateway-a>/
    200        # expected: 404
  5. Confirm the orphan is in controller A's data plane, and that it is never removed:

    $ kubectl port-forward -n apisix pod/<apisix-pod> 9090:9090
    $ curl -s localhost:9090/v1/services | jq -r '.[].value | "\(.labels["k8s/name"]) \(.hosts)"'
    demo ["demo.example.com"]        # still present, indefinitely
  6. kubectl rollout restart deploy/apisix-ingress-controller -n apisix — the entry disappears,
    confirming the store is the only thing holding it.

The same reproduction applies to GRPCRoute, TCPRoute, UDPRoute and TLSRoute.

Environment

  • APISIX Ingress controller version: 2.1.0 (also reproduces on master, be4aaef)
  • ADC version: 0.26.0
  • Mode: standalone / ADC (config_provider: yaml)
  • Kubernetes cluster version: v1.34.9
  • Gateway API version: v1.3.0

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions