fix(gateway2): stop deep-copying RouteOptions per route rule (control-plane OOM)

[wkrause13]

Description

Fixes the dominant memory consumer behind the gloo control-plane OOMs: the Kubernetes Gateway API translation deep-copied the attached RouteOption's entire options tree — including multi-MB transformation templates — once per route rule, per listener, per translation cycle. With many routes referencing the same RouteOption, both allocation churn and retained heap scaled with routes × options-size instead of unique RouteOptions × options-size. User heap profiles show this exact chain (translateGatewayHTTPRouteRule → GetRouteOptionForRouteRule → ShallowMergeRouteOptions → RouteOptions.Clone → TransformationTemplate.Clone) holding 57–58% of a 14.9 GB live heap.

Design: one interned copy per unique RouteOption per translation pass (opt-in)

Feature flag: the optimization is gated behind the GG_ROUTE_OPTION_INTERNING env var on the control plane and is off by default. With it unset, translation behaves exactly as before this PR — each attached RouteOption is deep-copied per route rule (no UnsafeDisableDeepCopy, ShallowMergeRouteOptions seed). The two plugin correctness fixes below (commits 3–4) are not gated; they ship unconditionally because they are safe whether or not interning is enabled. Everything else in this section describes behavior when the flag is enabled.

When enabled, the RouteOption query looks objects up with client.UnsafeDisableDeepCopy (no per-rule copy out of the informer cache), deep-copies each unique RouteOption exactly once per query lifetime into an intern map, and feeds the merge from that interned copy. Every route referencing the same RouteOption shares the one interned copy's sub-messages, so memory scales with unique RouteOptions, not routes — and the informer cache is structurally unreachable from translation output: a translation plugin that mutates nested options in place can at worst contaminate one pass's output (self-healing on the next sync), never the cache that every other consumer reads.

The merged options remain a distinct top-level message per route, so plugins keep reassigning top-level fields safely. Intern entries are keyed by name and replaced when the cached object's resourceVersion moves, so a lookup can never be served stale options, and the map stays bounded even if a query were ever to outlive its pass. The query's per-pass lifetime is documented at the sites that decide it (K8sGatewayExtensions.CreatePluginRegistry, buildProxy).

An earlier revision of this PR shared the cache objects' sub-messages directly into the merged options (no interned copy). That kept the same asymptotics but made the informer cache reachable from — and corruptible by — translation output, turning "no plugin may ever mutate nested options" into a whole-program invariant. The audit below showed that invariant was already violated in-tree, so the design was revised to interning; it costs one clone per unique RouteOption per pass (~74 at production scale, vs ~2,000+ per-route clones before the fix).

Commits

1. fix(gateway2): share RouteOptions sub-messages instead of deep-cloning per route (cherry-picked from @puertomontt's Fix/routeoptions-shallow-copy-oom #11247, authorship preserved): adds ShallowCopyRouteOptions and uses it to seed the merged options.
2. fix(gateway2): stop deep-copying RouteOptions out of the cache per route rule: client.UnsafeDisableDeepCopy on the targetRef List and the extensionRef Get; extensionRef lookup becomes a direct typed Get (same-namespace local ref; NotFound semantics preserved); drops the now-impossible ErrTypesNotEqual case.
3. fix: stop writing resolved escapeCharacters back into the input template: pre-existing OSS bug surfaced by the audit — TranslateTransformation resolved the template → staged → Settings escapeCharacters inheritance by assigning the result into the input template. With shared options this write-back would have leaked resolved values across routes and masked later Settings changes (and under the earlier revision, persistently written them into the informer cache). It now resolves into a local passed down to translateTransformationTemplate.
4. fix(gateway2): clone HeaderManipulation before applying header modifier filters: second pre-existing mutator — applyRequestFilter/applyResponseFilter wrote into the route options' existing HeaderManipulation in place, reachable with shared sub-messages when parent filters are re-applied to delegated child routes.
5. fix(gateway2): intern one RouteOption copy per translation pass: the interning described above, plus the lifecycle documentation.
6. feat(gateway2): gate RouteOption interning behind GG_ROUTE_OPTION_INTERNING: puts the memory model (commits 1, 2, 5) behind the env flag, off by default; commits 3–4 stay unconditional. Follows the gloo env-flag convention — constant in projects/gloo/constants, a package-level var read once at init (the UseDetailedUnmarshalling precedent), captured per query in NewQuery so a translation pass can't flip mid-run. New query_sharing_test.go specs pin the default-off path (no UnsafeDisableDeepCopy, per-route deep copy); the interning specs now set the flag explicitly.

Memory measurements

The three-way table below was measured at the pre-interning revision (commits 1–2). Re-measured 2026-06-12 with the interning revision, same cluster, same-session A/B against the pre-interning build: retained-heap growth 0→400 routes +11.2 MB (interning) vs +23.4 MB (pre-interning control re-run) at identical capture cadence, and steady state at 400 routes 79.0–79.5 MB vs 81.6–82.1 MB (3 captures each, identical workload history) — the gains hold; interning is not worse. Per-translation allocation churn is identical (~21 MB per proxy→xds rebuild on both builds; interning's one clone per unique RouteOption per pass does not register in profiles). Single per-phase captures carry ~±10 MB run-to-run noise (the same pre-interning image measured 60.0 MB and 74.1 MB at 400 routes on consecutive days), so growth deltas and same-session comparisons are the meaningful numbers; the original table remains structurally valid. All of these numbers reflect interning enabled (GG_ROUTE_OPTION_INTERNING=true); since the flag now defaults off, the out-of-the-box build matches the main/pre-interning baseline by construction.

Three images from the same toolchain, reproducer from the issue (1 RouteOption with staged transformations, 2-listener Gateway, N HTTPRoutes via extensionRef) in kind, capturing /debug/pprof/heap?gc=1 (forced GC ⇒ retained heap only) per phase:

retained heap	base (main @ 86ea356)	#11247 only	this PR
0 routes	50.8 MB	44.4 MB	45.7 MB
400 routes	74.7 MB	69.8 MB	60.0 MB
growth 0→400	+23.9 MB	+25.4 MB (no change)	+14.4 MB (−40%)

alloc_space (cumulative, same workload)	base	#11247 only	this PR
translateGatewayHTTPRouteRule	122.5 MB	85.5 MB	19.5 MB (6.3×)
GetRouteOptionForRouteRule	106.0 MB	71.5 MB	13.0 MB (8.2×)
ShallowMergeRouteOptions (merge clone)	37.0 MB	eliminated	eliminated
RouteOption.DeepCopyObject (cached-client copies)	53.5 MB	56.0 MB (unchanged)	eliminated

With #11247 alone, the merge clone disappears but RouteOption.DeepCopyObject (the cached client copying per rule) stays, and retained growth is unchanged — each route retains its own private copy via the shared pointers. With both, all routes referencing the same RouteOption share one copy.

Micro-benchmark (merge_benchmark_test.go), per-route cost of seeding merged options from a RouteOption carrying a 1,500-message transformation template:

	bytes/route	allocs/route	time/route
deep clone (before)	203,676	2,015	65 µs
shallow copy (after)	384	1	0.38 µs

Note on the e2e numbers: the retained-heap delta on main is capped by a separate pre-existing path — solo-kit's in-memory Proxy client deep-clones the whole translated Proxy for the persistence/debug snapshot, which re-expands shared options per route. The 1.20.x user profiles show translation output retained directly through the merge frames, so the retained-heap win on the LTS line should be substantially larger than −40%. The whole-Proxy snapshot clone is a worthwhile follow-up.

Mutation safety (the concern raised on the issue)

The in-place suggestion floated earlier on the issue was correctly called out as unsafe. The audit of every consumer of merged route options (this repo and solo-projects) found that "nothing downstream mutates nested options" was not a safe bet — it was already false in three places:

• the transformation plugin's escapeCharacters write-back (OSS, fixed in commit 3 — fires for any RouteOption carrying a transformation template, i.e. exactly the workload this PR targets);
• the headermodifier plugin's in-place HeaderManipulation writes (OSS, fixed in commit 4 — reachable via delegation);
• the enterprise portal plugin's in-place staged-transformations merge (fixed in the solo-projects companion PR).

With the flag off (default), the per-route deep copy keeps the informer cache protected exactly as on main today, so none of these mutators can reach it. With the flag on, interning means a future mutator of this class contaminates one pass's output for the routes sharing that RouteOption — a visible, self-healing bug — instead of silently corrupting the informer cache for every consumer until resync. The two known OSS mutators (commits 3–4) are fixed unconditionally regardless of the flag. The remaining (much weaker) rule for plugin authors, relevant only when interning is enabled: don't mutate nested messages reachable from merged route options; reassign top-level fields instead.

Enforcement, not just documentation:

• lookups must pass UnsafeDisableDeepCopy (pinned by recording-client specs);
• the merged result must share sub-messages across route rules but never with the client's objects (pointer-identity specs);
• a stale interned copy must never be served after a RouteOption update (update-mid-pass spec);
• a mutation guard runs the full attachment/merge/override flow asserting the client-returned RouteOptions are never modified — verified to fail when a nested mutation is introduced;
• mutation-safety specs for the transformation and headermodifier fixes, each watched failing against the old code.

Concurrency: the query is per-pass and route plugins run sequentially within a pass; -race over the gateway2 translator tree, routeoptions, proxy_syncer, and glooutils is clean.

Minor deliberate behavior changes (neither observable with real-world resources):

• extensionRef lookups no longer default an empty HTTPRoute namespace to "default" (HTTPRoutes are namespaced; two tests relied on the quirk and now set a namespace).
• ErrTypesNotEqual can no longer occur, so all attachment-lookup errors now set the ResolvedRefs condition.

Testing steps

• All affected suites green: projects/gloo/pkg/utils, routeoptions + query, headermodifier, transformation (the 9 specs that shell out to an envoy binary for config validation don't run on the dev machine — unchanged set, covered by CI), httproute, listener, full projects/gateway2/... (incl. the 50 golden-file translator specs covering all delegation RouteOptions merge cases), Edge translator (projects/gateway/pkg/translator, shares merge.go).
• E2E in kind, re-run with the interning revision: 400 routes / 800 translated routes — Gateway accepted, all routes programmed, transformations verified on the wire (x-served-by/x-cache-status injected, inja extractor reading request headers, headerManipulation stripping headers), zero errors in gloo logs. Also verified interning freshness end-to-end: patching the shared RouteOption propagates to all referencing routes on the next sync (interned copy replaced on resourceVersion change).
• Independent pre-existing finding from the re-run (affects pre-interning build equally, observed via in-place image A/B): the kube-gateway proxy syncer can enter a state where it rebuilds the xds snapshot ~2/sec indefinitely with unchanged output (~21 MB alloc per rebuild). Not caused or worsened by this PR; will file separately.

Notes for reviewers

• The key design question from the earlier revision — "what stops a plugin from corrupting the informer cache through the shared options?" — is now answered structurally: the cache is unreachable from translation output. The audit that motivated this (and commits 3–4) is the strongest argument that a comment-only contract would not have held.
• Backports (affected users are on 1.20.x): needs controller-runtime ≥ 0.17 on the target branch (for UnsafeDisableDeepCopy as a GetOption) and the portal companion on the matching enterprise LTS branch; commits 3–4 should be backported with it.
• Supersedes Avoid per-route deep-clone of RouteOptions during translation #11246 and Fix/routeoptions-shallow-copy-oom #11247 if accepted (this branch includes Fix/routeoptions-shallow-copy-oom #11247's commit verbatim; Avoid per-route deep-clone of RouteOptions during translation #11246 is an alternative copy-on-write approach in the query).

Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works

🤖 Generated with Claude Code

[solo-changelog-bot]

Issues linked to changelog:
https://github.com/solo-io/solo-projects/issues/8802

[github-actions]

Visit the preview URL for this PR (updated for commit 2e14c25):

https://gloo-edge--pr11264-will-routeoptions-oo-kdy5vnw3.web.app

_{(expires Tue, 23 Jun 2026 01:58:32 GMT)}

_{🔥 via Firebase Hosting GitHub Action 🌎}

_{Sign: 77c2b86e287749579b7ff9cadb81e099042ef677}