[WIP] fix: stop endpoint events from rebuilding Gateway proxies - #11341
Open
chandler-solo wants to merge 11 commits into
Open
[WIP] fix: stop endpoint events from rebuilding Gateway proxies#11341chandler-solo wants to merge 11 commits into
chandler-solo wants to merge 11 commits into
Conversation
# Description Prevent Pod and EndpointSlice events from triggering full Kubernetes Gateway proxy retranslation. These resources already flow through the KRT endpoint pipeline, which updates the affected xDS endpoint resources. The legacy controller-runtime watches additionally called the global proxy `Kick`, causing every Gateway proxy to be rebuilt as often as once per second in event-heavy clusters. ## Code changes - Remove the legacy Pod and EndpointSlice controller registrations. - Remove their reconcilers, whose only behavior was calling the global proxy `Kick`. - Add a regression test proving that Service events still request proxy recomputation while Pod and EndpointSlice events do not. - Add a changelog entry referencing solo-io/solo-projects#8013 with `resolvesIssue: false`. # Context Investigation of solo-io/solo-projects#8013 found sustained CPU usage and heap growth dominated by Gateway API route translation, including `buildProxy`, `translateGatewayHTTPRouteRule`, `setRouteAction`, and `RouteOptions.Clone`. Pod and EndpointSlice events currently enter two paths: 1. The KRT endpoint collections update endpoint resources in xDS. 2. The legacy controller-runtime reconcilers call `Kick`, causing all Gateway proxies and HTTPRoutes to be translated again. The second path is redundant for endpoint-only changes and creates substantial CPU and allocation churn in clusters with frequent workload or endpoint updates. This change addresses that unnecessary translation loop. It does not claim to fully resolve #8013; the independent cached Envoy snapshot retention addressed by #11309 has not yet landed. ## Interesting decisions The Pod and EndpointSlice controllers are removed instead of retained as no-op reconcilers. Their only behavior was calling `Kick`, and endpoint processing is already owned by the KRT collections. Service, Secret, route, policy, ReferenceGrant, and Namespace watches remain unchanged because those resources can affect proxy translation or validation. ## Testing steps ```bash CGO_ENABLED=0 go test -count=1 \ ./projects/gateway2/controller \ ./projects/gateway2/krtcollections \ ./projects/gateway2/proxy_syncer ``` The new envtest coverage uses a Service event as a positive control to verify that the `Kick` callback is active, then verifies that creating a Pod and EndpointSlice does not invoke it. ## Notes for reviewers Please verify that: - Pod and EndpointSlice events no longer reach the global proxy recompute trigger. - The KRT Pod and EndpointSlice collections remain unchanged and continue to update xDS endpoints. - Resources that can affect proxy translation continue to call `Kick`. - The changelog intentionally uses `resolvesIssue: false`. # Checklist: - [x] I have performed a self-review of my own code - [x] I have commented my code, particularly in hard-to-understand areas - [x] Documentation changes are not required because this does not change an API or user workflow - [x] I have added tests that prove my fix is effective ``` Signed-off-by: David L. Chandler <david.chandler@solo.io>
|
Issues linked to changelog: |
|
Visit the preview URL for this PR (updated for commit 9554a80): https://gloo-edge--pr11341-chandler-stop-endpoi-2gowlxo2.web.app (expires Tue, 11 Aug 2026 22:26:24 GMT) 🔥 via Firebase Hosting GitHub Action 🌎 Sign: 77c2b86e287749579b7ff9cadb81e099042ef677 |
Signed-off-by: David L. Chandler <david.chandler@solo.io>
Signed-off-by: David L. Chandler <david.chandler@solo.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Prevent Pod and EndpointSlice events from triggering full Kubernetes Gateway proxy retranslation.
These resources already flow through the KRT endpoint pipeline, which updates the affected xDS endpoint resources. The legacy controller-runtime watches additionally called the global proxy
Kick, causing every Gateway proxy to be rebuilt as often as once per second in event-heavy clusters.Code changes
Kick.resolvesIssue: false.Context
Investigation of solo-io/solo-projects#8013 found sustained CPU usage and heap growth dominated by Gateway API route translation, including
buildProxy,translateGatewayHTTPRouteRule,setRouteAction, andRouteOptions.Clone.Pod and EndpointSlice events currently enter two paths:
Kick, causing all Gateway proxies and HTTPRoutes to be translated again.The second path is redundant for endpoint-only changes and creates substantial CPU and allocation churn in clusters with frequent workload or endpoint updates.
This change addresses that unnecessary translation loop. It does not claim to fully resolve #8013; the independent cached Envoy snapshot retention addressed by #11309 has not yet landed.
Interesting decisions
The Pod and EndpointSlice controllers are removed instead of retained as no-op reconcilers. Their only behavior was calling
Kick, and endpoint processing is already owned by the KRT collections.Service, Secret, route, policy, ReferenceGrant, and Namespace watches remain unchanged because those resources can affect proxy translation or validation.
Testing steps
CGO_ENABLED=0 go test -count=1 \ ./projects/gateway2/controller \ ./projects/gateway2/krtcollections \ ./projects/gateway2/proxy_syncerThe new envtest coverage uses a Service event as a positive control to verify that the
Kickcallback is active, then verifies that creating a Pod and EndpointSlice does not invoke it.Notes for reviewers
Please verify that:
Kick.resolvesIssue: false.Checklist: