Support Kubernetes 1.34+ on DigitalOcean - #403
Conversation
What: adds a memory request/limit to Haproxy. Why: Starting with Kubernetes 1.34, DigitalOcean kills Haproxy (and also Reticulum and Photomnemonic, for some reason) shortly after startup because it thinks it's using too much memory. Adding the limit to Haproxy seems to fix everything (possibly because it increases the QoS Class of the pod from "BestEffort" to "Burstable", though I'm not sure why this fixes the Reticulum and Photomnemonic pods as they're still "BestEffort"). The extremely low request and extremely high limit should essentially preserve the previous behavior and make it so it just uses whatever memory it needs (which is normally around 30-40 Mi to begin with). Notes: It is unclear how much memory Haproxy really needs, but the default example has it with a minimum of 2Gi (and this was apparently in response to a bug: haproxytech/kubernetes-ingress@ff82484). Since that's much higher than we usually see it with Hubs, giving it an essentially unlimited amount to preserve the previous behavior seems reasonable. The memory used by the Reticulum pod has increased dramatically (from beginning with a few hundred Mi to beginning with around seventeen hundred Mi), but it's unclear why.
|
Thank you for the pull request. ROADMAP STATUS: This pull request isn't currently on any roadmap. Updates will be conveyed here as its place on/off a roadmap changes. You can view the roadmaps here: Roadmaps Google Drive folder. For more information on how the roadmaps work, see our roadmaps policy on GitHub. |
DougReeder
left a comment
There was a problem hiding this comment.
This doesn't break anything using versions
Client Version: v1.34.2
Kustomize Version: v5.7.1
Server Version: v1.33.12
so we could implement it as-is, but I'd like to set a memory resource request of 65Mi, as after restart + 2 users logged in, I see the HAProxy memory useage as 62Mi.
|
What advantage does increasing the minimum memory allocated have? Currently with 2 users logged into my instance HAProxy is using 58Mi. update: actually about 2.5 users because Hubs Bot was connected |
|
If you want to go with something like 60 MI, I won’t argue, but if we have a memory request, it should mean something. |
What?
Adds a memory request/limit to Haproxy.
Why?
Starting with Kubernetes 1.34, DigitalOcean kills Haproxy (and also Reticulum and Photomnemonic, for some reason) shortly after startup because it thinks it's using too much memory. Adding the limit to Haproxy seems to fix everything (possibly because it increases the QoS Class of the pod from "BestEffort" to "Burstable", though I'm not sure why this fixes the Reticulum and Photomnemonic pods as they're still "BestEffort"). The extremely low request and extremely high limit should essentially preserve the previous behavior and make it so it just uses whatever memory it needs (which is normally around 30-40 Mi to begin with).
Examples
N/A
How to test
npm run gen-hcce(remember to create a backup of hcce.yaml first if you have customized it).- --default-sslline in hcce.yamlkubectl delete deployment --all -n hccekubectl delete pods --all -n hccenpm run applyDocumentation of functionality
This PR preserves the previous behavior, so no documentation update is needed.
Known limitations
This doesn't add memory requests/limits to Reticulum, Photomnemonic, or any other pods.
Alternative implementations considered
Adding more constrained memory requests/limits, but it is unclear how much memory Haproxy really needs, and the default example has it with a minimum of 2Gi (and this was apparently in response to a bug: haproxytech/kubernetes-ingress@ff82484). Since that's much higher than we usually see it with Hubs, giving it an essentially unlimited amount to preserve the previous behavior seemed better than more constrained settings.
Adding requests/limits to other pods, but this isn't required to fix the issue and it would be more complicated, and potentially require changes to allow auto-calculation based on a user provided value for the total memory of their instance (which would also necessitate a docs update).
Open questions
None.
Additional details or related context
The memory used by the Reticulum pod has increased dramatically (from beginning with a few hundred Mi to beginning with around seventeen hundred Mi), but it's unclear why.