Memory regression in KIC 3.5: ~60% RAM increase and GC fails to release memory to OS (compared to 3.3)

[javascripterzzeerrr]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

After upgrading Kong Ingress Controller from v3.3 to v3.5.11, we observed a critical memory regression. The total RAM consumption (sys_bytes) of the leader KIC pod increased by approximately 50% (from 1.84 GB to 2.94 GB).

I used the /metrics KIC endpoint by 10255 port to collect Go runtime metrics. Analysis of the data reveals two root causes:

1. The Garbage Collector in v3.5.11 practically stopped releasing idle memory back to the OS. By comparing go_memstats_heap_released_bytes to go_memstats_heap_idle_bytes, only 0.6% of the idle heap was released in v3.5.11, compared to 50% in v3.3.

2. The memory footprint per cached object has increased by ~28% (from ~215 bytes/object in v3.3 to ~276 bytes/object in v3.5.11) - go_memstats_heap_alloc_bytes / go_memstats_heap_objects (however, I am not sure that it is correct to compare this way)

Expected Behavior

Memory consumption and GC behavior should be comparable to v3.3. The Go runtime should actively release idle heap memory back to the OS. The internal data structures representing Kubernetes objects should not significantly increase in size.

Steps To Reproduce

KIC v3.3 - Stable baseline
go_memstats_sys_bytes: 1.84 GB
go_memstats_heap_alloc_bytes: 1.19 GB
go_memstats_heap_objects: 5,542,030
go_memstats_heap_idle_bytes: 552 MB
go_memstats_heap_released_bytes: 283 MB (51% of idle memory successfully returned to OS)
go_version: go1.23.0

KIC v3.5 - Memory regression
go_memstats_sys_bytes: 2.94 GB
go_memstats_heap_alloc_bytes: 1.84 GB
go_memstats_heap_objects: 6,654,745
go_memstats_heap_idle_bytes: 978 MB
go_memstats_heap_released_bytes: 6 MB (Only 0.6% of idle memory returned to OS)
go_version: go1.23.0

Kong Ingress Controller version

KIC Version: 3.5.11 (upgraded from 3.3.0)
Kong Gateway Version: 3.9, 3.8 (we have checked both)

DB-less mode

Kubernetes version

v1.33.1

Anything else?

No response

[javascripterzzeerrr]

Good afternoon, there are updates:

We noticed that Kong Ingress Controller on our cluster might not be cleaning up memory actively enough, since the cluster is large and has many objects. We started using the GOMEMLIMIT env var, which helped - memory consumption now doesn't significantly exceed the GOMEMLIMIT threshold, however, CPU load has slightly increased.

Helm Chart limitation: Configuring this via values.yaml (ingressController.env) doesn't work because the chart prepends CONTROLLER_, resulting in CONTROLLER_GOMEMLIMIT (which Go ignores). I had to manually edit the Deployment's env section to set GOMEMLIMIT correctly. Native support for Go runtime variables without the prefix is needed until this regression is fixed.

[pmalek]

Hi @javascripterzzeerrr,

Thanks for creating the issue.

At this moment I can't comment on the actual issue itself as this requires some investigation but you can define non prefix environment variables with the chart by using the customEnv value: https://github.com/Kong/charts/blob/15c4389787ea2393fa000758b9ed12ef205b7e6d/charts/kong/values.yaml#L620-L623

LMK if this works.

[javascripterzzeerrr]

Yeah, thanks a lot, it works
https://github.com/Kong/charts/blob/15c4389787ea2393fa000758b9ed12ef205b7e6d/charts/kong/values.yaml#L620-L623