Proxy Serves Partial Content on Large Requests

[reed-hanger]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

I recently updated my Kubernetes nodes from Bottlerocket 1.57 (Linux kernel 6.12.73) to 1.60 (Linux kernel 6.12.79), and I noticed web requests flowing through our Kong Ingress Controller consistently returned net::ERR_HTTP2_PROTOCOL_ERROR 206 (Partial Content) on larger files (+100KB). Basically, the client kept receiving only part of a JavaScript file. This occurred with the following kong helm chart versions:
3.2.0
3.0.2
2.42.0

Through troubleshooting with AWS Support, we determined the web server has no issues sending the file to kong, but part of the file repeatedly gets stuck in the receive buffer (Recv-Q), and Kong will send a TCP ZeroWindow flag back to the web server. We believe the issue stems from some kernel level changes documented here. I'm surprised to see this is not already a major issue reported here, so I assume there's something unique in our environment or kong config that's causing this problem to surface. AWS support responded with the message below after I sent them some tests where I repeatedly ran ss -tin on the kong pod (via an ephemeral container with some network tools) and node when requesting a large file:

1. The receive window (rcv_wnd) drops from a healthy value to 2560 (sample 11) — this is consistent with aggressive receive window clamping behaviour. Notably, rcv_ssthresh (82764) remains well above the data in flight, suggesting the window reduction may not be driven by normal backpressure alone. This pattern aligns with the window_clamp regression we identified from the upstream kernel commit a2cbb1603943.

2. The full file (117573 bytes) does arrive (bytes_received: 117829), but 56389 bytes remain stuck in the receive buffer (Recv-Q). The application (Kong proxy) is not draining the buffer to forward the data to the downstream client.

3. The connection enters a deadlocked state where no further progress is made — the upstream has finished sending, but Kong cannot complete forwarding.

[+] From the tcp_mem output:

91521 122029 183042 (pages = ~357MB / 476MB / 715MB)

This confirms the system is not under global TCP memory pressure.

[+] From the node-level capture (node_ss.txt), the host connections (Cilium health probes on port 4240) are healthy with normal window sizes. This further confirms the issue is specific to application-level pod traffic transiting through the kernel's TCP stack.

The new data strengthens our working hypothesis: the kernel appears to be reducing rcv_wnd to near-zero values during the initial data burst, which may be triggering a cascading stall in Kong's proxy buffering logic. Since this behaviour is present on kernel 6.12.79 (BR 1.60) but NOT on kernel 6.12.73 (BR 1.57), it is consistent with a change in TCP receive window management introduced between these two kernel versions. We need the Bottlerocket kernel team to definitively validate whether the window_clamp regression applies here.

There are my values for our kong chart:

ingressController:
  installCRDs: false
certificates:
  enabled: true
  proxy:
    enabled: true
    issuer: letsencrypt-prod
    commonName: "${cluster_lob}-${cluster_env}.${aws_projects_root_domain}"
    dnsNames:
    - "${cluster_lob}-${cluster_env}.${aws_projects_root_domain}"
    - "*.${cluster_lob}-${cluster_env}.${aws_projects_root_domain}"
    - "*.k8s-${cluster_name}.${cluster_lob}-${cluster_env}.${aws_projects_root_domain}"
  admin:
    enabled: false
  portal:
    enabled: false
  cluster:
    enabled: false
autoscaling:
  enabled: true
  minReplicas: 3
  maxReplicas: 9
podDisruptionBudget:
  enabled: true
  maxUnavailable: 1
manager:
  enabled: false
portal:
  enabled: false
portalapi:
  enabled: false
env:
  trusted_ips: ${aws_internal_nlb_ips}
  real_ip_header: x-forwarded-for
  proxy_access_log: "off"
proxy:
  type: NodePort
  http:
    nodePort: 30001
    parameters:
    - proxy_protocol
  tls:
    nodePort: 30002
    parameters:
    - proxy_protocol
    - http2
updateStrategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 100%
    maxUnavailable: 0%
topologySpreadConstraints:
- topologyKey: topology.kubernetes.io/zone
  maxSkew: 1
  whenUnsatisfiable: DoNotSchedule
  labelSelector:
    matchLabels:
      app.kubernetes.io/component: app
      app.kubernetes.io/instance: kong
      app.kubernetes.io/name: kong

I found that adding these values to the helm chart helped with this issue:

env:
  nginx_proxy_proxy_buffering: "on"
  nginx_proxy_proxy_buffer_size: "128k"
  nginx_proxy_proxy_buffers: "8 256k"
  nginx_proxy_proxy_busy_buffers_size: "256k"

However, I continued to intermittently receive partial responses on very large requests (+5MB). I'm not sure of the full implications of these settings, so I'm hesitant to increase the proxy buffer sizes more.

I was wondering if there were any suggestions on the kong side to sidestep or fix this issue. I tested disabling the proxy buffer (nginx_proxy_proxy_buffering: "off") entirely, but that did not help. I can upload packet captures of the kong and web server when a request is made if that helps.

Expected Behavior

No response

Steps To Reproduce

Kong Ingress Controller version

Tested on the following versions of the kong (not ingress) helm chart:
3.2.0
3.0.2
2.42.0

Kubernetes version

This occurred in both AWS EKS 1.33 and 1.35. I did more extensive testing on 1.35 with Bottlerocket versions...
1.60.0-c1f9ba0c

Anything else?

No response