Description
The policy engine rebuilds every route's policy chain on every xDS update. Because xDS is State-of-the-World, any artifact change (deploy/redeploy/delete of any REST API, LLM proxy, etc.) pushes the full snapshot, and HandlePolicyChainUpdate unconditionally calls buildPolicyChain → registry.GetInstance → each policy's GetPolicy factory for every policy on every route. So redeploying API B re-instantiates the policy instances of unrelated APIs A, C, and so on.
This hurts stateful policies that keep per-instance state across requests. The clearest example is the backend-JWT policy (policies/backend-jwt), which caches generated upstream JWTs so an upcoming client request can reuse an already-signed token instead of re-signing. Because GetPolicy is re-invoked on every unrelated deployment today, the policy cannot simply hold that cache on its instance — it had to be engineered around the behavior:
GetPolicy returns a process-wide singleton (var ins) rather than a fresh instance, so the token cache and key cache survive redeploys (the code comment states it is "called on each API deployment").- The token cache is a single shared cache across all APIs, with every token-shaping field folded into the cache key for implicit invalidation.
ensureTokenCache guards against rebuilding the cache except on a genuine cacheMaxSize change.
That complexity exists solely because instances are torn down and rebuilt on every unrelated snapshot. Any policy wanting a natural per-instance cache (token caches, warm lookups, connection pools, rate-limit windows) faces the same churn.
Fix
Reconcile each snapshot against the last-applied set on the policy-engine side: rebuild only changed/new routes, reuse the existing chain and its instances for unchanged routes, and drop removed routes. Change detection uses an order-sensitive canonical-JSON signature of each route's behavioral config (excluding volatile metadata), fails safe to rebuild, and emits per-route decision logs. Implemented in 46d8ed6f2.
Outcome: redeploying one API no longer re-runs other APIs' GetPolicy factories, so policies can keep per-instance caches and state across unrelated deployments — and workarounds like the backend-JWT process-wide singleton are no longer needed for correctness.
Version
No response
Related Issue
No response
Description
The policy engine rebuilds every route's policy chain on every xDS update. Because xDS is State-of-the-World, any artifact change (deploy/redeploy/delete of any REST API, LLM proxy, etc.) pushes the full snapshot, and
HandlePolicyChainUpdateunconditionally callsbuildPolicyChain→registry.GetInstance→ each policy'sGetPolicyfactory for every policy on every route. So redeploying API B re-instantiates the policy instances of unrelated APIs A, C, and so on.This hurts stateful policies that keep per-instance state across requests. The clearest example is the backend-JWT policy (
policies/backend-jwt), which caches generated upstream JWTs so an upcoming client request can reuse an already-signed token instead of re-signing. BecauseGetPolicyis re-invoked on every unrelated deployment today, the policy cannot simply hold that cache on its instance — it had to be engineered around the behavior:GetPolicyreturns a process-wide singleton (var ins) rather than a fresh instance, so the token cache and key cache survive redeploys (the code comment states it is "called on each API deployment").ensureTokenCacheguards against rebuilding the cache except on a genuinecacheMaxSizechange.That complexity exists solely because instances are torn down and rebuilt on every unrelated snapshot. Any policy wanting a natural per-instance cache (token caches, warm lookups, connection pools, rate-limit windows) faces the same churn.
Fix
Reconcile each snapshot against the last-applied set on the policy-engine side: rebuild only changed/new routes, reuse the existing chain and its instances for unchanged routes, and drop removed routes. Change detection uses an order-sensitive canonical-JSON signature of each route's behavioral config (excluding volatile metadata), fails safe to rebuild, and emits per-route decision logs. Implemented in 46d8ed6f2.
Outcome: redeploying one API no longer re-runs other APIs'
GetPolicyfactories, so policies can keep per-instance caches and state across unrelated deployments — and workarounds like the backend-JWT process-wide singleton are no longer needed for correctness.Version
No response
Related Issue
No response