Refactor/eks/managed nodes/system reserved and kube reserved

[wcarlsen]

Describe your changes

This is a stacked PR depending on #2380 to be merged first.

We remove variables:

• kube_reserved_cpu (kubelet reserved CPU) since, this doesn't scale with max_pods. The scaling is calculated in this fashion (6% of the first core + 1% of the second core + 0.5% of the third and fourth core + 0.25% of the remainding cores.
• kube_memory (kubelet reserved memory) scales with Max Pods, but EKS doesn't handle overwrites to to Max Pods in terms of reserved memory. The calculation is as following 255Mi + (11Mi * Max_pods).
• I'm also removing systemReserved completely, since AWS EKS team doesn't set because it requires a bunch of system knowledge see this link for Kubernetes docs on the topic https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/#general-guidelines.
• Migrated instance_type from launch template to node group to better support mixed instance types.
• Added check block to throw a warning when instance types with lower than 110 max pods

Issue ticket number and link

Checklist before requesting a review

• I have tested changes in my sandbox
• I have added the needed changes in the test/integration folder to apply my changes in QA. Read the guide on adding environment variables in QA
• I have rebased the code to master (or merged in the latest from master)

Is it a new release?

• Apply a release tag release:(major|minor|patch), following semantic versioning in this guide or norelease if there is no changes to the Terraform code

[DFDS-Snyk]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues
✅	Code Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

[wcarlsen]

After speaking with @SEQUOIIA and @rifisdfds we decided to:

• make max_pods a constant of 110 which is also the upper limit recommended by Kubernetes docs https://kubernetes.io/docs/setup/best-practices/cluster-large/
• test the current implementation with a node crasher script and if it fails try to set systemReserved.memory to a fixed value

[wcarlsen]

This PR fails when being rolled back to master branch and not on applying feature branch see https://github.com/dfds/infrastructure-modules/actions/runs/28049163804/job/83038273166. The issue on master comes from a change on this PR migrating instance_type from launch_template resource to node_group resource to improve support for mixed instance_types. When it rolls back to master it cannot because there no changes really happening to the launch_template, but the instance_type parameter isn't set on the nodegroup and then it fails.

So the failing check on this PR is not from applying the changes, it is from the rollback.

[samidbb]

@wcarlsen would you please update the description to reference the docs mentioning about the the calculation in regards to scaling of kubelet reserved CPU ?

[wcarlsen]

@wcarlsen would you please update the description to reference the docs mentioning about the the calculation in regards to scaling of kubelet reserved CPU ?

Sure thing. It is pretty hard to find and it is a combination of a lot of unofficial links and some AWS Batch documentation for AWS EKS. Basically AWS uses the same calculation for CPU as GKE and AKS and on memory they diverge.

If I find some good links I will add them, but any changes to this branch breaks master when rolling back. So it will have to wait until after tomorrow.

You probably have to be more specific with what you mean with docs? Is in a comment in the code or in the description of this PR or somewhere else?

[wcarlsen]

@wcarlsen would you please update the description to reference the docs mentioning about the the calculation in regards to scaling of kubelet reserved CPU ?

Sure thing. It is pretty hard to find and it is a combination of a lot of unofficial links and some AWS Batch documentation for AWS EKS. Basically AWS uses the same calculation for CPU as GKE and AKS and on memory they diverge.

If I find some good links I will add them, but any changes to this branch breaks master when rolling back. So it will have to wait until after tomorrow.

You probably have to be more specific with what you mean with docs? Is in a comment in the code or in the description of this PR or somewhere else?

Here is the links from AWS Batch on EKS kubeReserved:

• CPU https://docs.aws.amazon.com/batch/latest/userguide/memory-cpu-batch-eks.html#memory-cpu-batch-eks-node-cpu-reservations
• Memory https://docs.aws.amazon.com/batch/latest/userguide/memory-cpu-batch-eks.html#memory-cpu-batch-eks-node-memory-reservations

and here are links to the official go code setting these values:

• Memory https://github.com/awslabs/amazon-eks-ami/blob/d17a9d345efadc98bc0d9183391fe950a5b21ae6/nodeadm/internal/kubelet/config.go#L442-L443
• CPU https://github.com/awslabs/amazon-eks-ami/blob/d17a9d345efadc98bc0d9183391fe950a5b21ae6/nodeadm/internal/kubelet/config.go#L410

[wcarlsen]

This one mixes in different concept so replacing it with this PR #2395