Skip to content

Support Kubernetes 1.34+ on DigitalOcean - #403

Open
Exairnous wants to merge 1 commit into
Hubs-Foundation:hotfix/2.1.0from
Exairnous:add-haproxy-memory-request-limit
Open

Support Kubernetes 1.34+ on DigitalOcean#403
Exairnous wants to merge 1 commit into
Hubs-Foundation:hotfix/2.1.0from
Exairnous:add-haproxy-memory-request-limit

Conversation

@Exairnous

Copy link
Copy Markdown
Member

What?

Adds a memory request/limit to Haproxy.

Why?

Starting with Kubernetes 1.34, DigitalOcean kills Haproxy (and also Reticulum and Photomnemonic, for some reason) shortly after startup because it thinks it's using too much memory. Adding the limit to Haproxy seems to fix everything (possibly because it increases the QoS Class of the pod from "BestEffort" to "Burstable", though I'm not sure why this fixes the Reticulum and Photomnemonic pods as they're still "BestEffort"). The extremely low request and extremely high limit should essentially preserve the previous behavior and make it so it just uses whatever memory it needs (which is normally around 30-40 Mi to begin with).

Examples

N/A

How to test

  1. Regenerate your hcce.yaml file with npm run gen-hcce (remember to create a backup of hcce.yaml first if you have customized it).
  2. Comment out the - --default-ssl line in hcce.yaml
  3. Run these commands to completely restart your instance:
    • kubectl delete deployment --all -n hcce
    • kubectl delete pods --all -n hcce
    • npm run apply
  4. Visit your instance and see that everything works.
  5. If you are using DigitalOcean and/or on a version of Kubernetes before 1.34, you should see no difference with these changes.
  6. If you are using DigitalOcean and/or on a version of Kubernetes of 1.34+, you should see that without these changes, your instance won't work and will be stuck with pods in a CrashLoopBackOff state due to being continually killed for being out of memory; however, with these changes you should see that things work normally.

Documentation of functionality

This PR preserves the previous behavior, so no documentation update is needed.

Known limitations

This doesn't add memory requests/limits to Reticulum, Photomnemonic, or any other pods.

Alternative implementations considered

Adding more constrained memory requests/limits, but it is unclear how much memory Haproxy really needs, and the default example has it with a minimum of 2Gi (and this was apparently in response to a bug: haproxytech/kubernetes-ingress@ff82484). Since that's much higher than we usually see it with Hubs, giving it an essentially unlimited amount to preserve the previous behavior seemed better than more constrained settings.

Adding requests/limits to other pods, but this isn't required to fix the issue and it would be more complicated, and potentially require changes to allow auto-calculation based on a user provided value for the total memory of their instance (which would also necessitate a docs update).

Open questions

None.

Additional details or related context

The memory used by the Reticulum pod has increased dramatically (from beginning with a few hundred Mi to beginning with around seventeen hundred Mi), but it's unclear why.

What: adds a memory request/limit to Haproxy.

Why: Starting with Kubernetes 1.34, DigitalOcean kills Haproxy (and also Reticulum and Photomnemonic, for some reason) shortly after startup because it thinks it's using too much memory.  Adding the limit to Haproxy seems to fix everything (possibly because it increases the QoS Class of the pod from "BestEffort" to "Burstable", though I'm not sure why this fixes the Reticulum and Photomnemonic pods as they're still "BestEffort").  The extremely low request and extremely high limit should essentially preserve the previous behavior and make it so it just uses whatever memory it needs (which is normally around 30-40 Mi to begin with).

Notes:

It is unclear how much memory Haproxy really needs, but the default example has it with a minimum of 2Gi (and this was apparently in response to a bug: haproxytech/kubernetes-ingress@ff82484).  Since that's much higher than we usually see it with Hubs, giving it an essentially unlimited amount to preserve the previous behavior seems reasonable.

The memory used by the Reticulum pod has increased dramatically (from beginning with a few hundred Mi to beginning with around seventeen hundred Mi), but it's unclear why.
@github-actions

Copy link
Copy Markdown

Thank you for the pull request.

ROADMAP STATUS: This pull request isn't currently on any roadmap. Updates will be conveyed here as its place on/off a roadmap changes.

You can view the roadmaps here: Roadmaps Google Drive folder.

For more information on how the roadmaps work, see our roadmaps policy on GitHub.

@DougReeder DougReeder left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't break anything using versions
Client Version: v1.34.2
Kustomize Version: v5.7.1
Server Version: v1.33.12

so we could implement it as-is, but I'd like to set a memory resource request of 65Mi, as after restart + 2 users logged in, I see the HAProxy memory useage as 62Mi.

@Exairnous

Exairnous commented Aug 26, 2026

Copy link
Copy Markdown
Member Author

What advantage does increasing the minimum memory allocated have? Currently with 2 users logged into my instance HAProxy is using 58Mi.

update: actually about 2.5 users because Hubs Bot was connected

@DougReeder

Copy link
Copy Markdown
Member

If you want to go with something like 60 MI, I won’t argue, but if we have a memory request, it should mean something.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants