diff --git a/src/content/docs/aws/services/eks.mdx b/src/content/docs/aws/services/eks.mdx index 3bc78b58..c6545425 100644 --- a/src/content/docs/aws/services/eks.mdx +++ b/src/content/docs/aws/services/eks.mdx @@ -6,6 +6,7 @@ persistence: supported with limitations --- import FeatureCoverage from "../../../../components/feature-coverage/FeatureCoverage"; +import { Tabs, TabItem, Steps } from '@astrojs/starlight/components'; ## Introduction @@ -447,6 +448,195 @@ The `ls-secret-tls` secret is created in the `default` namespace. If your ingress and services are residing in a custom namespace, it is essential to copy the secret to that custom namespace to make use of it. ::: +## Self-managed nodes and Karpenter + +In addition to [managed node groups](#creating-a-managed-node-group), LocalStack supports self-managed worker nodes: EC2 instances that join an EKS cluster on their own using the standard EKS bootstrap user-data. +This is the same mechanism that [Karpenter](https://karpenter.sh/) relies on to provision capacity, so you can run the full Karpenter node lifecycle (SSM AMI lookup, EC2 Fleet provisioning, node bootstrap, and scale-up/scale-down) against a local cluster without any LocalStack-specific configuration. + +When an EC2 instance is launched with EKS bootstrap user-data, LocalStack parses it, connects the instance container to the cluster's internal network, and starts a [k3s](https://k3s.io/) agent inside it. +The instance then registers as a worker node, exactly as a real EKS node would appear to the control plane. + +Two AMI families are supported: + +- **Amazon Linux 2023 (AL2023)**: configuration is carried as a multipart MIME `application/node.eks.aws` NodeConfig document (used by `nodeadm` and Karpenter). +- **Bottlerocket**: configuration is carried as a plain TOML document under `[settings.kubernetes]`. + +LocalStack reads the same set of fields from both formats and propagates them to the registered node, including node labels, taints, topology metadata (region, zone, instance type), and the provider ID. This matches what real EKS nodes look like from the cluster's perspective. + +:::note +Self-managed nodes use the embedded k3d-backed provider (`EKS_K8S_PROVIDER=k3s`), which is the default. +The walkthrough below runs entirely between Docker containers, so it also works on macOS where direct host-to-instance networking is not available. +::: + +### Resolving a node AMI + +Self-managed nodes must be launched from an EKS-optimized AMI. +LocalStack resolves the standard EKS AMI [SSM public parameters](https://docs.aws.amazon.com/eks/latest/userguide/retrieve-ami-id.html) to a k3s-backed image, so you can look them up the same way Karpenter does. + + + +```bash title="Resolve AL2023 AMI" +awslocal ssm get-parameter \ + --name /aws/service/eks/optimized-ami/1.35/amazon-linux-2023/x86_64/standard/recommended/image_id \ + --query 'Parameter.Value' --output text +``` + +```bash title="Output" +ami-eks-k3d-1.35-amd64-standard +``` + + +```bash title="Resolve Bottlerocket AMI" +awslocal ssm get-parameter \ + --name /aws/service/bottlerocket/aws-k8s-1.35/x86_64/latest/image_id \ + --query 'Parameter.Value' --output text +``` + +```bash title="Output" +ami-eks-k3d-1.35-amd64 +``` + + + +The walkthrough below assumes a cluster named `cluster1` that is already `ACTIVE` (see [Create an embedded Kubernetes cluster](#create-an-embedded-kubernetes-cluster)). + +### Joining a self-managed node + + + + + + 1. Create a user-data file containing a `NodeConfig` document. At minimum, `spec.cluster.name` must match the name of your EKS cluster, which LocalStack uses to resolve which cluster the node should join. Optionally, you can set kubelet configuration and node labels: + + ```text title="al2023-userdata.txt" + MIME-Version: 1.0 + Content-Type: multipart/mixed; boundary="//" + + --// + Content-Type: application/node.eks.aws + + apiVersion: node.eks.aws/v1alpha1 + kind: NodeConfig + spec: + cluster: + name: cluster1 + kubelet: + config: + maxPods: 20 + registerWithTaints: + - key: dedicated + value: gpu + effect: NoSchedule + flags: + - '--node-labels=role=worker,env=demo' + --//-- + ``` + + 2. Launch an EC2 instance using the resolved AL2023 AMI and the user-data file: + + ```bash title="Launch AL2023 node" + awslocal ec2 run-instances \ + --image-id ami-eks-k3d-1.35-amd64-standard \ + --count 1 \ + --instance-type t3.medium \ + --user-data file://./al2023-userdata.txt + ``` + + + + + + + 1. Bottlerocket carries its bootstrap configuration as a plain TOML document. Create a user-data file with a `[settings.kubernetes]` section, setting `cluster-name` to your cluster: + + ```toml title="bottlerocket-userdata.toml" + [settings.kubernetes] + cluster-name = "cluster1" + max-pods = 30 + + [settings.kubernetes.node-labels] + "role" = "worker" + "env" = "demo" + + [settings.kubernetes.node-taints] + "dedicated" = ["gpu:NoSchedule"] + ``` + + 2. Launch an EC2 instance using the resolved Bottlerocket AMI: + + ```bash title="Launch Bottlerocket node" + awslocal ec2 run-instances \ + --image-id ami-eks-k3d-1.35-amd64 \ + --count 1 \ + --instance-type m5.large \ + --user-data file://./bottlerocket-userdata.toml + ``` + + + + +### Verifying the node + +Point `kubectl` at the cluster and list the nodes: + +```bash +awslocal eks update-kubeconfig --name cluster1 && \ + kubectl config use-context arn:aws:eks:us-east-1:000000000000:cluster/cluster1 +``` + +After a few seconds, the self-managed instance registers and becomes `Ready` alongside the cluster's control-plane node. +The node name is the EC2 instance ID: + +```bash +kubectl get nodes +``` + +```bash title="Output" +NAME STATUS ROLES AGE VERSION +i-8a2eb615ddf838df7 Ready 33s v1.35.5+k3s1 +k3d-cluster1-c2fad0d2-server-0 Ready control-plane 2m v1.35.5+k3s1 +``` + +You can confirm that the configuration from the user-data was applied to the node: + +```bash +kubectl get node i-8a2eb615ddf838df7 \ + -o jsonpath='{.spec.providerID}{"\n"}{.spec.taints}{"\n"}{.status.allocatable.pods}{"\n"}' +``` + +```bash title="Output" +aws:///us-east-1a/i-8a2eb615ddf838df7 +[{"effect":"NoSchedule","key":"dedicated","value":"gpu"}] +20 +``` + +### Supported configuration fields + +LocalStack reads the following fields from the node bootstrap user-data and reflects them on the registered Kubernetes node. +The AL2023 and Bottlerocket columns show where each value comes from in the respective format: + +| Behaviour | AL2023 (`NodeConfig`) | Bottlerocket (TOML) | +|:-----------------------------------|:---------------------------------------|:------------------------------------------| +| Target cluster | `spec.cluster.name` | `cluster-name` | +| Cluster DNS | `kubelet.config.clusterDNS` | `cluster-dns-ip` | +| Max pods | `kubelet.config.maxPods` | `max-pods` | +| Eviction thresholds | `kubelet.config.evictionHard` | `[settings.kubernetes.eviction-hard]` | +| Node taints | `kubelet.config.registerWithTaints` | `[settings.kubernetes.node-taints]` | +| Node labels | `kubelet.flags` (`--node-labels`) | `[settings.kubernetes.node-labels]` | + +In addition, LocalStack always emits the topology labels `topology.kubernetes.io/region`, `topology.kubernetes.io/zone`, and `node.kubernetes.io/instance-type` (derived from the instance's placement and type), as well as the provider ID in the form `aws:////`. + +:::note +Fields such as `apiServerEndpoint` and `certificateAuthority` are required when bootstrapping against real EKS, but are resolved internally by LocalStack and can be omitted from the user-data. +::: + +### Using Karpenter + +Because self-managed nodes rely only on standard EC2 and EKS bootstrap behaviour, [Karpenter](https://karpenter.sh/) can drive node provisioning against a local cluster without any LocalStack-specific changes. +Karpenter looks up node AMIs through the SSM parameters described above, launches instances via EC2 Fleet with the appropriate NodeConfig or Bottlerocket user-data, and the new instances join the cluster as described. + +To try this out, follow the upstream [Getting started with Karpenter](https://karpenter.sh/docs/getting-started/getting-started-with-karpenter/) guide, pointing the AWS endpoints at LocalStack. + ## Use an existing Kubernetes installation You can also access the EKS API using your existing local Kubernetes installation.