Performance Optimization Plan for Policy Engine ExtProc Processing

[renuka-fernando]

Summary

This proposal defines a phased optimization plan for the Policy Engine's ExternalProcessorServer.Process() function -- the hot path that handles every HTTP request flowing through the WSO2 API Platform gateway. The plan targets reduced per-request heap allocations, eliminated redundant computation, and improved throughput by introducing benchmarks, fixing duplicate work, adding object pooling, and implementing zero-allocation patterns for CEL evaluation and metadata construction.

Motivation

Problem Statement

The Policy Engine sits on the critical data path between Envoy and upstream services. Every HTTP request triggers a bidirectional gRPC stream that allocates multiple objects on the heap, parses protobuf metadata, evaluates CEL conditions, and constructs response messages -- all from scratch on every request.

Specific issues identified through code analysis:

1. No performance baseline exists. There are zero benchmark tests in the codebase. Without benchmarks, there is no way to measure current performance, detect regressions, or validate optimizations.

2. Redundant computation on every request. The extractRouteMetadata() function is called twice per request (once at extproc.go:143 in handleProcessingPhase and again at extproc.go:331 inside initializeExecutionContext). Each call runs prototext.Unmarshal to parse route metadata from protobuf text format, involving a string to []byte conversion and full proto parsing.

3. Excessive per-request heap allocations. Every request allocates at minimum 10+ objects that are immediately garbage-collected: PolicyExecutionContext, SharedContext, RequestContext, ResponseContext, headers maps (doubled due to normalization copy), Metadata maps, Body structs, CEL evaluation maps (~20 keys each), executor result slices, and protobuf response messages. None of these objects are pooled with sync.Pool.

4. CEL evaluation overhead even when unused. The EvaluateRequestCondition() function in internal/pkg/cel/evaluator.go creates a map[string]interface{} with approximately 20 entries on every invocation. If no policies in a chain have execution conditions, this work is entirely wasted.

5. Double header map copy. buildRequestContext() creates a map[string][]string from Envoy headers, then policy.NewHeaders() (in sdk/gateway/policy/v1alpha/headers.go:108) iterates the entire map again to normalize keys to lowercase into a new map. The same pattern applies to buildResponseContext().

Who Benefits

• Platform operators: Lower CPU and memory consumption per request means higher throughput on the same hardware, reduced infrastructure cost, and better tail latency under load.
• API consumers: Reduced processing latency in the policy engine directly translates to faster API response times.
• Gateway developers: Benchmarks provide a regression safety net and a clear framework for evaluating future changes.

Why Now

The Policy Engine is feature-complete for its initial release and is entering production-readiness hardening. Performance characteristics should be established and optimized before production traffic scales up. The absence of any benchmarks means there is currently no way to detect performance regressions introduced by new features or refactors. Additionally, several of the identified issues (duplicate function calls, double header copy) are low-risk fixes that deliver immediate gains.

Detailed Design

Overview

The optimization plan is structured in four phases, ordered by risk and impact. Phase 0 establishes measurement infrastructure. Phase 1 eliminates redundant computation with no behavioral change. Phase 2 adds early-exit paths to skip unnecessary work. Phase 3 introduces sync.Pool-based object reuse. Phase 4 explores advanced zero-allocation patterns. Each phase can be implemented, reviewed, and shipped independently.

Changes Required

Component	File/Area	Change Description
Policy Engine	internal/kernel/extproc.go	Remove duplicate extractRouteMetadata() call; pass result into initializeExecutionContext()
Policy Engine	internal/kernel/extproc.go	Add route metadata cache (sync.Map or LRU) keyed by route name to avoid per-request prototext.Unmarshal
Policy Engine	internal/kernel/extproc.go	Skip PolicyExecutionContext creation when no policy chain exists; reduce skipAllProcessing() allocations
Policy Engine	internal/kernel/execution_context.go	Build headers with lowercase keys in the initial loop, eliminating the second map copy in NewHeaders()
Policy Engine	internal/kernel/execution_context.go	Add sync.Pool for PolicyExecutionContext, RequestContext, ResponseContext, and SharedContext
Policy Engine	internal/executor/chain.go	Pre-compute span name strings on PolicySpec at config time; check span.IsRecording() before formatting
Policy Engine	internal/executor/chain.go	Pool RequestExecutionResult and ResponseExecutionResult slices
Policy Engine	internal/registry/chain.go	Add HasExecutionConditions bool flag computed at chain build time
Policy Engine	internal/pkg/cel/evaluator.go	Skip CEL context map creation when chain has no execution conditions; implement custom CEL Activation interface
Policy Engine	internal/kernel/analytics.go	Pre-build template structpb.Struct for analytics metadata; use proto Clone instead of fresh construction
Policy Engine	internal/kernel/translator.go	Reduce structpb allocations in buildDynamicMetadata() via template cloning
SDK	sdk/gateway/policy/v1alpha/headers.go	Modify NewHeaders() to accept pre-normalized keys directly, removing the internal re-normalization loop
Policy Engine	internal/kernel/*_test.go (new)	Benchmark tests for Process, extractRouteMetadata, buildRequestContext, buildResponseContext, CEL evaluation, policy chain execution, response translation

Detailed Optimization Plan

Phase 0: Establish Baseline with Benchmarks

Goal: Create measurement infrastructure so every subsequent phase can prove its value.

Work items:

1. Write Go benchmark tests (*_test.go with Benchmark* functions) for:
   • Process function end-to-end (using a mock gRPC stream)
   • extractRouteMetadata (with realistic Envoy attributes)
   • buildRequestContext (with typical header counts: 10, 25, 50 headers)
   • buildResponseContext (with typical header counts)
   • CEL condition evaluation (EvaluateRequestCondition, EvaluateResponseCondition)
   • Policy chain execution (ExecuteRequestPolicies, ExecuteResponsePolicies)
   • Response translation (TranslateRequestHeadersActions, TranslateResponseHeadersActions)
2. Capture baseline metrics: ns/op, B/op, allocs/op using go test -bench -benchmem
3. Set up CPU and memory profiling (pprof) integration for targeted analysis
4. Document baseline numbers in the benchmark test file comments

Expected impact: No runtime impact. Provides the foundation for measuring all subsequent improvements.

---

Phase 1: Eliminate Redundant Computation

Goal: Remove duplicate calls, unnecessary parsing, and repeated allocations with no behavioral change.

1.1 Fix duplicate extractRouteMetadata() call

Currently in handleProcessingPhase (extproc.go:143):

routeMetadata := s.extractRouteMetadata(req)
rm := s.initializeExecutionContext(ctx, req, execCtx)

And initializeExecutionContext (extproc.go:331) calls extractRouteMetadata again:

func (s *ExternalProcessorServer) initializeExecutionContext(...) *RouteMetadata {
    routeMetadata := s.extractRouteMetadata(req) // DUPLICATE
    ...
}

Fix: Change initializeExecutionContext to accept a RouteMetadata parameter instead of extracting it internally.

1.2 Cache parsed route metadata by route name

The extractRouteMetadata() function at extproc.go:425 calls prototext.Unmarshal([]byte(metadataStr), &envoyMetadata) on every request. Route metadata for a given route name is static (it changes only when an API is redeployed). A sync.Map or size-bounded LRU cache keyed by route name eliminates this per-request parsing entirely.

1.3 Pre-compute tracing span names at config time

In chain.go:85:

_, span := c.tracer.Start(traceCtx, fmt.Sprintf(constants.SpanPolicyRequestFormat, spec.Name), ...)

The fmt.Sprintf allocates a new string per-policy per-request. Pre-compute these at config time and store them on PolicySpec (e.g., spec.RequestSpanName, spec.ResponseSpanName).

1.4 Pre-compute HasExecutionConditions flag

Add a HasExecutionConditions bool field to PolicyChain in internal/registry/chain.go, computed at chain build time. When false, the executor can skip all CEL evaluation logic entirely.

1.5 Build headers with lowercase keys directly

In buildRequestContext() (execution_context.go:318):

headersMap[key] = append(headersMap[key], value)

Change to:

headersMap[strings.ToLower(key)] = append(headersMap[strings.ToLower(key)], value)

Then modify NewHeaders(headersMap) in the SDK to skip the normalization loop entirely (breaking change accepted), since all callers will now provide pre-normalized keys. This eliminates the second map allocation. Apply the same pattern to buildResponseContext().

Expected impact:

• Eliminates 1 full prototext.Unmarshal + proto allocation per request (item 1.1)
• Eliminates prototext.Unmarshal on all subsequent requests for cached routes (item 1.2)
• Eliminates 1 fmt.Sprintf allocation per policy per phase (item 1.3)
• Eliminates CEL evaluation overhead for chains without conditions (item 1.4)
• Eliminates 1 full map copy per request phase for headers (item 1.5, saves ~2 map allocations per request)
• Estimated: 30-50% reduction in allocs/op for the common case

---

Phase 2: Skip Unnecessary Work

Goal: Add early-exit paths so requests that do not need certain processing skip it entirely.

2.1 Skip PolicyExecutionContext creation for routes without policies

When GetPolicyChainForKey() returns nil, the current code still calls extractRouteMetadata() and constructs a RouteMetadata. In skipAllProcessing() (extproc.go:356-384), it calls extractMetadataFromRouteMetadata() and structpb.NewStruct() to build dynamic metadata even though no policies ran.

Fix: Pre-build a minimal skip response with only route-level analytics that avoids structpb.NewStruct() when the metadata map is empty or has only string values (use direct structpb.NewStringValue construction instead).

2.2 Skip CEL evaluation context when no conditions exist

When chain.HasExecutionConditions is false (from Phase 1.4), the entire condition evaluation block in ExecuteRequestPolicies and ExecuteResponsePolicies can be skipped. This avoids the map[string]interface{} allocation with ~20 entries per CEL evaluation.

2.3 Skip structpb allocations in skipAllProcessing()

When all analytics metadata values are strings, use direct structpb.NewStringValue() construction instead of going through structpb.NewStruct() which validates and converts each value.

2.4 Check span recording state before formatting attributes

The code already does this in several places (if span.IsRecording()), but the span name string is computed via fmt.Sprintf before the span is created (chain.go:85). After Phase 1.3 pre-computes these strings, this is resolved. Additionally, verify all attribute-setting code paths are guarded.

Expected impact:

• Eliminates PolicyExecutionContext + SharedContext + map allocations for no-policy routes (item 2.1)
• Eliminates ~20-entry map allocation per CEL evaluation when conditions are absent (item 2.2)
• Reduces structpb overhead for skip responses (item 2.3)
• Estimated: Additional 10-20% reduction in allocs/op for routes with no conditions

---

Phase 3: Object Pooling with sync.Pool

Goal: Reuse per-request objects across requests to reduce GC pressure.

3.1 Pool PolicyExecutionContext

Create a sync.Pool for PolicyExecutionContext. After each request completes (stream ends), reset the context fields and return it to the pool. The Reset() method must nil out all pointer fields and clear all maps.

var execCtxPool = sync.Pool{
    New: func() interface{} {
        return &PolicyExecutionContext{
            analyticsMetadata: make(map[string]interface{}, 8),
        }
    },
}

3.2 Pool RequestContext and ResponseContext

Similar pools for request and response contexts. These hold headers maps, body references, and shared context pointers.

3.3 Pool header maps

Pre-allocate map[string][]string with a typical capacity (e.g., 16 entries) and return to pool after use.

3.4 Pool CEL evaluation context maps

When CEL evaluation is needed, get a pre-allocated map[string]interface{} from a pool, populate it, evaluate, then clear and return.

3.5 Pool executor result slices

Pool RequestExecutionResult and ResponseExecutionResult with pre-allocated Results slices.

Expected impact:

• Significant reduction in GC pressure under sustained load
• Estimated: 40-60% reduction in B/op (bytes allocated per operation)
• Estimated: 20-30% reduction in GC pause frequency at high request rates
• Most impactful under high concurrency where GC is the bottleneck

Risk: Pool misuse (returning objects with stale references) can cause data races or memory corruption. Each pooled type needs a thorough Reset() method and tests verifying clean state.

---

Phase 4: Advanced Optimizations

Goal: Squeeze out remaining allocation overhead for maximum throughput.

4.1 Custom CEL Activation interface

The CEL library supports custom interpreter.Activation implementations. Instead of building a map[string]interface{} with ~20 entries, implement an Activation that reads directly from RequestContext or ResponseContext fields. This is a zero-allocation approach to CEL evaluation.

type RequestActivation struct {
    ctx   *policy.RequestContext
    phase string
}

func (a *RequestActivation) ResolveName(name string) (interface{}, bool) {
    switch name {
    case "request.Method":
        return a.ctx.Method, true
    case "request.Path":
        return a.ctx.Path, true
    // ... etc
    }
    return nil, false
}

4.2 Template-based structpb construction for analytics

Pre-build a structpb.Struct template at config time with the fixed fields (API name, version, context, etc.) and use proto.Clone() per request, then fill in dynamic values. This avoids constructing the nested struct tree from scratch each time.

4.3 Faster request ID generation

Replace uuid.New().String() (which uses crypto/rand) with a faster alternative like github.com/rs/xid or github.com/oklog/ulid when cryptographic uniqueness is not required. Alternatively, use uuid.NewString() which avoids the intermediate [16]byte to string conversion in newer UUID library versions.

4.4 Proto message pooling for response construction

Pool extprocv3.ProcessingResponse, extprocv3.HeadersResponse, and extprocv3.CommonResponse protobuf messages. These follow predictable structures and can be reset and reused.

Expected impact:

• CEL activation adapter: eliminates the largest per-evaluation allocation (~20-entry map)
• Template structpb: reduces analytics metadata construction to a clone + field set
• Faster IDs: reduces latency on the first request phase when no x-request-id header exists
• Estimated: Additional 15-25% reduction in allocs/op beyond Phase 3

Compatibility

Question	Answer	Details
Backwards compatible?	Not required	Breaking changes to internal interfaces and SDK functions are accepted.
Requires migration?	No	No data model or configuration changes. Custom policy authors may need to recompile against the updated SDK if NewHeaders() signature or Headers interface changes.
Breaking changes?	Yes (accepted)	NewHeaders() in the SDK will be modified to skip normalization (callers must provide lowercase keys). Internal interfaces such as PolicyExecutionContext, executor interfaces, and CEL evaluator signatures may change to support pooling and pre-computed fields. Adding new fields to PolicySpec (e.g., RequestSpanName) and PolicyChain (e.g., HasExecutionConditions) changes their struct layout. All breaking changes are internal to the gateway module or SDK and do not affect external APIs or configuration schemas.

Migration Steps

Custom policy authors who import the SDK will need to recompile after the NewHeaders() change. No code changes are required on their end since the function signature remains the same -- only the contract changes (keys must already be lowercase, which Envoy headers are by specification).

References

• Envoy External Processing Filter Documentation - Defines the ext_proc protocol this code implements
• Go sync.Pool Documentation - Standard library object pool used in Phase 3
• CEL-Go Custom Activation - Interface for zero-allocation CEL evaluation in Phase 4
• Go Benchmark Testing - Go standard benchmark framework used in Phase 0
• gateway/policy-engine/internal/kernel/extproc.go - Main Process function and extractRouteMetadata
• gateway/policy-engine/internal/kernel/execution_context.go - PolicyExecutionContext and context builders
• gateway/policy-engine/internal/executor/chain.go - Policy chain execution with CEL evaluation
• gateway/policy-engine/internal/pkg/cel/evaluator.go - CEL evaluator with per-request map allocation
• gateway/policy-engine/internal/kernel/translator.go - Response translation with structpb construction
• gateway/policy-engine/internal/kernel/analytics.go - Analytics metadata building
• gateway/policy-engine/internal/registry/chain.go - PolicyChain struct definition
• sdk/gateway/policy/v1alpha/headers.go - Headers normalization and NewHeaders constructor