Skip to content

[WIP] fix: stop endpoint events from rebuilding Gateway proxies - #11341

Open
chandler-solo wants to merge 11 commits into
mainfrom
chandler/stop-endpoint-events-from-rebuilding-gw-proxies
Open

[WIP] fix: stop endpoint events from rebuilding Gateway proxies#11341
chandler-solo wants to merge 11 commits into
mainfrom
chandler/stop-endpoint-events-from-rebuilding-gw-proxies

Conversation

@chandler-solo

Copy link
Copy Markdown

Description

Prevent Pod and EndpointSlice events from triggering full Kubernetes Gateway proxy retranslation.

These resources already flow through the KRT endpoint pipeline, which updates the affected xDS endpoint resources. The legacy controller-runtime watches additionally called the global proxy Kick, causing every Gateway proxy to be rebuilt as often as once per second in event-heavy clusters.

Code changes

  • Remove the legacy Pod and EndpointSlice controller registrations.
  • Remove their reconcilers, whose only behavior was calling the global proxy Kick.
  • Add a regression test proving that Service events still request proxy recomputation while Pod and EndpointSlice events do not.
  • Add a changelog entry referencing solo-io/solo-projects#8013 with resolvesIssue: false.

Context

Investigation of solo-io/solo-projects#8013 found sustained CPU usage and heap growth dominated by Gateway API route translation, including buildProxy, translateGatewayHTTPRouteRule, setRouteAction, and RouteOptions.Clone.

Pod and EndpointSlice events currently enter two paths:

  1. The KRT endpoint collections update endpoint resources in xDS.
  2. The legacy controller-runtime reconcilers call Kick, causing all Gateway proxies and HTTPRoutes to be translated again.

The second path is redundant for endpoint-only changes and creates substantial CPU and allocation churn in clusters with frequent workload or endpoint updates.

This change addresses that unnecessary translation loop. It does not claim to fully resolve #8013; the independent cached Envoy snapshot retention addressed by #11309 has not yet landed.

Interesting decisions

The Pod and EndpointSlice controllers are removed instead of retained as no-op reconcilers. Their only behavior was calling Kick, and endpoint processing is already owned by the KRT collections.

Service, Secret, route, policy, ReferenceGrant, and Namespace watches remain unchanged because those resources can affect proxy translation or validation.

Testing steps

CGO_ENABLED=0 go test -count=1 \
  ./projects/gateway2/controller \
  ./projects/gateway2/krtcollections \
  ./projects/gateway2/proxy_syncer

The new envtest coverage uses a Service event as a positive control to verify that the Kick callback is active, then verifies that creating a Pod and EndpointSlice does not invoke it.

Notes for reviewers

Please verify that:

  • Pod and EndpointSlice events no longer reach the global proxy recompute trigger.
  • The KRT Pod and EndpointSlice collections remain unchanged and continue to update xDS endpoints.
  • Resources that can affect proxy translation continue to call Kick.
  • The changelog intentionally uses resolvesIssue: false.

Checklist:

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • Documentation changes are not required because this does not change an API or user workflow
  • I have added tests that prove my fix is effective

# Description

<!--
Please include a high level summary of the changes.

This bug fixes ... \ This new feature can be used to ...

_Fill out any of the following sections that are relevant and remove the others_
-->

## API changes

<!--
- Added x field to y resource
- ...
-->

## Code changes

<!--
- Fix error in `Foo()` function
- Add `Bar()` function
- ...
-->

## CI changes

<!--
- Adjusted schedule for x job
- ...
-->

## Docs changes

<!--
- Added guide about feature x to public docs
- Updated README to account for y behavior
- ...
-->

# Context

<!-- Users ran into this bug doing ... \ Users needed this feature to ...

See slack conversation [here](https://solo-io-corp.slack.com/archives/some/post)
-->

## Interesting decisions

<!-- We chose to do things this way because ... -->

## Testing steps

<!-- I manually verified behavior by ... -->

## Notes for reviewers

<!-- Be sure to verify intended behavior by ...

Please proofread comments on ...

This is a complex PR and may require a huddle to discuss ...
-->

# Checklist:

- [ ] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] I have added tests that prove my fix is effective or that my feature works

<!---
# Author reminders (delete before opening)
- Include a concise, user-facing changelog (for details, see https://github.com/solo-io/go-utils/tree/main/changelogutils) referencing the issue that is resolved
  - Include `resolvesIssue: false` unless the issue does not require a release to be resolved; only a subset of non-user-facing issues can be considered resolved without release
- Run codegen via `make -B install-go-tools generated-code`
- Follow guidelines laid out in the Gloo Edge [contribution guide](https://docs.solo.io/gloo-edge/latest/contributing/)
- If not ready for review, open a draft PR or apply the `work in progress` label
-->

# Description

Prevent Pod and EndpointSlice events from triggering full Kubernetes Gateway proxy retranslation.

These resources already flow through the KRT endpoint pipeline, which updates the affected xDS endpoint resources. The legacy controller-runtime watches additionally called the global proxy `Kick`, causing every Gateway proxy to be rebuilt as often as once per second in event-heavy clusters.

## Code changes

- Remove the legacy Pod and EndpointSlice controller registrations.
- Remove their reconcilers, whose only behavior was calling the global proxy `Kick`.
- Add a regression test proving that Service events still request proxy recomputation while Pod and EndpointSlice events do not.
- Add a changelog entry referencing solo-io/solo-projects#8013 with `resolvesIssue: false`.

# Context

Investigation of solo-io/solo-projects#8013 found sustained CPU usage and heap growth dominated by Gateway API route translation, including `buildProxy`, `translateGatewayHTTPRouteRule`, `setRouteAction`, and `RouteOptions.Clone`.

Pod and EndpointSlice events currently enter two paths:

1. The KRT endpoint collections update endpoint resources in xDS.
2. The legacy controller-runtime reconcilers call `Kick`, causing all Gateway proxies and HTTPRoutes to be translated again.

The second path is redundant for endpoint-only changes and creates substantial CPU and allocation churn in clusters with frequent workload or endpoint updates.

This change addresses that unnecessary translation loop. It does not claim to fully resolve #8013; the independent cached Envoy snapshot retention addressed by #11309 has not yet landed.

## Interesting decisions

The Pod and EndpointSlice controllers are removed instead of retained as no-op reconcilers. Their only behavior was calling `Kick`, and endpoint processing is already owned by the KRT collections.

Service, Secret, route, policy, ReferenceGrant, and Namespace watches remain unchanged because those resources can affect proxy translation or validation.

## Testing steps

```bash
CGO_ENABLED=0 go test -count=1 \
  ./projects/gateway2/controller \
  ./projects/gateway2/krtcollections \
  ./projects/gateway2/proxy_syncer
```

The new envtest coverage uses a Service event as a positive control to verify that the `Kick` callback is active, then verifies that creating a Pod and EndpointSlice does not invoke it.

## Notes for reviewers

Please verify that:

- Pod and EndpointSlice events no longer reach the global proxy recompute trigger.
- The KRT Pod and EndpointSlice collections remain unchanged and continue to update xDS endpoints.
- Resources that can affect proxy translation continue to call `Kick`.
- The changelog intentionally uses `resolvesIssue: false`.

# Checklist:

- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] Documentation changes are not required because this does not change an API or user workflow
- [x] I have added tests that prove my fix is effective
```

Signed-off-by: David L. Chandler <david.chandler@solo.io>
@solo-changelog-bot

Copy link
Copy Markdown

Issues linked to changelog:
https://github.com/solo-io/solo-projects/issues/8013

@github-actions

github-actions Bot commented Jul 29, 2026

Copy link
Copy Markdown

Visit the preview URL for this PR (updated for commit 9554a80):

https://gloo-edge--pr11341-chandler-stop-endpoi-2gowlxo2.web.app

(expires Tue, 11 Aug 2026 22:26:24 GMT)

🔥 via Firebase Hosting GitHub Action 🌎

Sign: 77c2b86e287749579b7ff9cadb81e099042ef677

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Migrated] Fix instructions for installing OIDC with Dex

1 participant