Skip to content

HTTP/3 + proxyProtocol ClientTrafficPolicy generates invalid QUIC listener and causes continuous LDS NACKs #9798

Description

@zero734kr

Description:

When http3 and proxyProtocol are enabled together on the same ClientTrafficPolicy, Envoy Gateway generates a QUIC listener containing the PROXY protocol listener filter.

Envoy rejects the generated LDS update with:

Didn't find a registered implementation for 'envoy.filters.listener.proxy_protocol' with type URL: 'envoy.extensions.filters.listener.proxy_protocol.v3.ProxyProtocol'

This results in continuous LDS NACKs from every Envoy proxy.

The affected listener in this case is: envoy-gateway-system/korlark/https-api-quic

TCP HTTPS traffic continues working because Envoy retains the last valid LDS configuration, so the problem can be easy to miss from Gateway/HTTPRoute status and normal HTTP/2 traffic alone.

In my deployment, both Envoy replicas continuously NACK the same Listener resource. xds_nack_total reached 164 for each replica, with 39 additional NACKs per replica during the most recent one-hour window.

Expected behavior:

Envoy Gateway should not generate a QUIC listener configuration that its supported Envoy version rejects.

Possible expected behaviors would be:

  1. Reject this ClientTrafficPolicy combination during validation with a clear status/error.
  2. Disable HTTP/3 for the affected listener and report the reason, similar to the existing handling for unsupported HTTP/3 + downstream client TLS validation combinations.
  3. Generate a valid QUIC listener without the incompatible listener filter, if that is semantically supported.

The current behavior allows an apparently valid policy to reach xDS generation and then repeatedly fail at the Envoy dataplane.

Repro steps:

Apply a Gateway with an HTTPS listener and the following
ClientTrafficPolicy:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
  name: proxy-protocol-http3
spec:
  http3: {}
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: example
  proxyProtocol:
    optional: true

The underlying LoadBalancer in my environment uses PROXY protocol for the TCP data path.

After reconciliation, inspect the Envoy Gateway controller logs:

kubectl logs -n envoy-gateway-system deploy/envoy-gateway \
  --since=1h | grep -Ei 'nack|error_detail|reject|listener'

Also inspect:

sum by (nodeID, typeURL) (
  increase(xds_nack_total[1h])
)

In my deployment, both Envoy replicas continuously NACK the generated QUIC Listener.

HTTP/1.1 and HTTP/2 traffic through the TCP listener remain functional.

Environment:

  • Envoy Gateway: v1.9.0
  • Envoy Proxy: v1.39.0 (distroless-v1.39.0)
  • Gateway API: v1.6.1
  • Kubernetes: v1.36.1
  • Platform: Vultr Kubernetes Engine (VKE)
  • Envoy Gateway controller replicas: 1
  • Envoy Proxy replicas: 2
  • LoadBalancer: Vultr Load Balancer with PROXY protocol enabled

Relevant policy:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
  name: korlark-proxy-protocol
spec:
  http3: {}
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: korlark
  proxyProtocol:
    optional: true

Logs:

Both Envoy replicas report the same error:

error xds cache/snapshotcache.go:396 Envoy rejected the last update for type type.googleapis.com/envoy.config.listener.v3.Listener on node envoy-envoy-gateway-system-korlark-245bed12-54d658d9b9-zv55k with code 13 and message Error adding/updating listener(s) envoy-gateway-system/korlark/https-api-quic: Didn't find a registered implementation for 'envoy.filters.listener.proxy_protocol' with type URL: 'envoy.extensions.filters.listener.proxy_protocol.v3.ProxyProtocol'

error xds cache/snapshotcache.go:396 Envoy rejected the last update for type type.googleapis.com/envoy.config.listener.v3.Listener on node envoy-envoy-gateway-system-korlark-245bed12-54d658d9b9-p575h with code 13 and message Error adding/updating listener(s) envoy-gateway-system/korlark/https-api-quic: Didn't find a registered implementation for 'envoy.filters.listener.proxy_protocol' with type URL: 'envoy.extensions.filters.listener.proxy_protocol.v3.ProxyProtocol'

Observed metrics:

increase(xds_nack_total[1h])

{nodeID="envoy-envoy-gateway-system-korlark-245bed12-54d658d9b9-p575h",
 typeURL="type.googleapis.com/envoy.config.listener.v3.Listener"} 39

{nodeID="envoy-envoy-gateway-system-korlark-245bed12-54d658d9b9-zv55k",
 typeURL="type.googleapis.com/envoy.config.listener.v3.Listener"} 39

Cumulative values:

xds_nack_total{nodeID="...-p575h",
  typeURL="type.googleapis.com/envoy.config.listener.v3.Listener"} 164

xds_nack_total{nodeID="...-zv55k",
  typeURL="type.googleapis.com/envoy.config.listener.v3.Listener"} 164

The NACKs are deterministic and occur identically on both dataplane replicas.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions