Add support for vpc.controlPlaneOnPrivateSubnets (#8793)

* Add support for vpc.controlPlaneOnPrivateSubnets

When eksctl creates a VPC, it passes both the public and the private subnets
to the EKS API, so the control plane's cross-account ENIs are placed in public
subnets as well. Restricting them to private subnets afterwards requires
`eksctl utils update-cluster-vpc-config`, which calls the EKS API directly and
leaves the cluster's CloudFormation stack out of sync.

Add a `vpc.controlPlaneOnPrivateSubnets` field so that only the private subnets
are passed to the EKS API at creation time. Public subnets are still created and
used for NAT gateways and internet-facing load balancers, so this does not make
the cluster fully private.

The field is honoured for eksctl-created VPCs (IPv4 and IPv6, including
fully-private clusters) and for pre-existing VPCs. It is rejected when combined
with `vpc.controlPlaneSubnetIDs`, and when the configured private subnets do not
cover at least two availability zones, which EKS requires. That last check is
best-effort: subnets given only by ID have their zone resolved from EC2 after
validation runs, so they are allowed through and rejected by the EKS API instead.
Outposts are exempt, since the control plane there is already private-only.

`eksctl utils update-cluster-vpc-config` rejects the field rather than silently
ignoring it, and points at `vpc.controlPlaneSubnetIDs` instead.

Control plane subnet selection is now built through a single helper. autoMode is
passed into it rather than derived, so that it keeps applying only to VPCs that
eksctl creates: the pre-existing VPC path has never restricted the control plane
for Auto Mode clusters, and deriving it would have changed that behaviour for
configurations that do not use the new field.

Issue #8792

Signed-off-by: guessi <guessi@gmail.com>

* Address review feedback on controlPlaneOnPrivateSubnets

- Count distinct availability zones in c.AvailabilityZones when
  validating controlPlaneOnPrivateSubnets for eksctl-created VPCs,
  instead of assuming SetSubnets guarantees per-zone coverage.
  Duplicate zones collapse into a single private subnet in
  vpc.SetSubnets, and validateAvailabilityZones permits duplicates,
  so this was previously unchecked.
- Rework the update-cluster-vpc-config tests for
  controlPlaneOnPrivateSubnets to assert against
  cmdutils.NewUpdateClusterVPCLoader(...).Load() directly instead of
  executing the full command, which reached AWS via
  NewProviderForExistingCluster.
- Change eksctl utils update-cluster-vpc-config to warn and ignore
  vpc.controlPlaneOnPrivateSubnets instead of returning a hard error,
  consistent with how eksctl update nodegroup handles other
  unsupported fields in a config file.
- Update the userdocs claim that eksctl validates AZ coverage before
  creating anything to note the pre-existing-subnets-by-ID exception,
  where AZs are resolved from EC2 after validation runs.

* Don't reject controlPlaneOnPrivateSubnets when AZs are auto-selected

validateControlPlaneOnPrivateSubnets ran before eks.SetAvailabilityZones, so
when availabilityZones was left unset it saw an empty c.AvailabilityZones and
rejected the config with 'got 0 ([])', even though eksctl always selects
distinct zones itself on that path. Only enforce the distinct-zone count when
availabilityZones is explicitly set, and flip the test that had locked in the
rejection to expect success instead.

Document the auto-selection exception in the userdocs alongside the existing
pre-existing-subnets-by-ID one, as a nested list since the two exceptions no
longer read clearly as a single sentence.

Signed-off-by: guessi <guessi@gmail.com>

* Assert the update-cluster-vpc-config warning and document it

The loader test only asserted that no error was returned, which passes just as
well if the warn-and-ignore behaviour is dropped entirely. Capture logger output
and assert the warning text, and assert the field is not translated into
controlPlaneSubnetIDs. Verified the assertion fails when the warning is changed.

Document that the field only takes effect at cluster creation time and is warned
about and ignored by eksctl utils update-cluster-vpc-config, which the code did
but the docs did not mention.

Signed-off-by: guessi <guessi@gmail.com>

---------

Signed-off-by: guessi <guessi@gmail.com>

Author: guessi

examples/49-control-plane-on-private-subnets.yaml

@@ -0,0 +1,29 @@
+# An example config for restricting the EKS control plane's cross-account ENIs to private
+# subnets at cluster creation time.
+# To create the cluster, run `eksctl create cluster -f 49-control-plane-on-private-subnets.yaml`
+#
+# Public subnets are still created and used for the NAT gateways and for internet-facing load
+# balancers; only the subnets passed to the EKS API are restricted. Requires at least two
+# private subnets across at least two availability zones.
+
+apiVersion: eksctl.io/v1alpha5
+kind: ClusterConfig
+metadata:
+  name: cluster-49
+  region: us-west-2
+
+availabilityZones:
+  - us-west-2a
+  - us-west-2b
+
+vpc:
+  controlPlaneOnPrivateSubnets: true
+  nat:
+    gateway: HighlyAvailable
+  clusterEndpoints:
+    publicAccess: true
+    privateAccess: true
+
+managedNodeGroups:
+  - name: mng1
+    privateNetworking: true

pkg/apis/eksctl.io/v1alpha5/assets/schema.json

@@ -1181,6 +1181,11 @@
           "description": "controls how the control plane routes egress traffic. Valid values: \"AWS_MANAGED\" (default), \"CUSTOMER_ROUTED\"",
           "x-intellij-html-description": "controls how the control plane routes egress traffic. Valid values: &quot;AWS<em>MANAGED&quot; (default), &quot;CUSTOMER</em>ROUTED&quot;"
         },
+        "controlPlaneOnPrivateSubnets": {
+          "type": "boolean",
+          "description": "restricts the control plane (the cross-account ENIs that EKS places in the cluster subnets) to private subnets only, excluding public subnets. It applies both when eksctl creates the VPC and when a pre-existing VPC is used. Cannot be combined with ControlPlaneSubnetIDs. Requires at least two private subnets spanning at least two availability zones, which must have NAT or the relevant VPC endpoints for nodes to reach the API server.",
+          "x-intellij-html-description": "restricts the control plane (the cross-account ENIs that EKS places in the cluster subnets) to private subnets only, excluding public subnets. It applies both when eksctl creates the VPC and when a pre-existing VPC is used. Cannot be combined with ControlPlaneSubnetIDs. Requires at least two private subnets spanning at least two availability zones, which must have NAT or the relevant VPC endpoints for nodes to reach the API server."
+        },
         "controlPlaneSecurityGroupIDs": {
           "items": {
             "type": "string"
@@ -1277,6 +1282,7 @@
         "clusterEndpoints",
         "publicAccessCIDRs",
         "controlPlaneSubnetIDs",
+        "controlPlaneOnPrivateSubnets",
         "controlPlaneSecurityGroupIDs",
         "controlPlaneEgressMode"
       ],

pkg/apis/eksctl.io/v1alpha5/types.go

@@ -927,6 +927,12 @@ func (c *ClusterConfig) IsControlPlaneOnOutposts() bool {
 	return c.Outpost != nil && c.Outpost.ControlPlaneOutpostARN != ""
 }
 
+// IsControlPlaneOnPrivateSubnets returns true if the control plane's cross-account ENIs
+// should be restricted to private subnets only.
+func (c *ClusterConfig) IsControlPlaneOnPrivateSubnets() bool {
+	return c.VPC != nil && IsEnabled(c.VPC.ControlPlaneOnPrivateSubnets)
+}
+
 // GetOutpost returns the Outpost info.
 func (c *ClusterConfig) GetOutpost() *Outpost {
 	return c.Outpost

pkg/apis/eksctl.io/v1alpha5/validation.go

@@ -18,6 +18,7 @@ import (
 	"github.com/kris-nova/logger"
 
 	corev1 "k8s.io/api/core/v1"
+	"k8s.io/apimachinery/pkg/util/sets"
 	"k8s.io/apimachinery/pkg/util/validation"
 	kubeletapis "k8s.io/kubelet/pkg/apis"
 
@@ -501,6 +502,10 @@ func (c *ClusterConfig) ValidateVPCConfig() error {
 		return errors.New("only one of vpc.securityGroup and vpc.controlPlaneSecurityGroupIDs can be specified")
 	}
 
+	if err := c.validateControlPlaneOnPrivateSubnets(); err != nil {
+		return err
+	}
+
 	if (c.VPC.IPv6Cidr != "" || c.VPC.IPv6Pool != "") && !c.IPv6Enabled() {
 		return fmt.Errorf("Ipv6Cidr and Ipv6CidrPool are only supported when IPFamily is set to IPv6")
 	}
@@ -554,6 +559,79 @@ func (c *ClusterConfig) ValidateVPCConfig() error {
 	return nil
 }
 
+// validateControlPlaneOnPrivateSubnets validates vpc.controlPlaneOnPrivateSubnets against
+// the rest of the VPC configuration.
+func (c *ClusterConfig) validateControlPlaneOnPrivateSubnets() error {
+	if !IsEnabled(c.VPC.ControlPlaneOnPrivateSubnets) {
+		return nil
+	}
+
+	if len(c.VPC.ControlPlaneSubnetIDs) > 0 {
+		return errors.New("only one of vpc.controlPlaneSubnetIDs and vpc.controlPlaneOnPrivateSubnets can be specified")
+	}
+
+	// The control plane is already restricted to private subnets on Outposts, where a
+	// single subnet in a single zone is expected, so the checks below do not apply.
+	if c.IsControlPlaneOnOutposts() {
+		return nil
+	}
+
+	// Subnets are nil when eksctl creates the VPC. Private subnets are then derived from
+	// availabilityZones by vpc.SetSubnets, which runs after validation and keys private
+	// subnets by zone name, so duplicate zones collapse into a single subnet.
+	// validateAvailabilityZones only checks the count of c.AvailabilityZones and explicitly
+	// permits duplicates, so distinct zones must be counted here instead.
+	//
+	// availabilityZones is optional: when it is left unset, eksctl selects the zones itself
+	// in eks.SetAvailabilityZones, which runs after validation, so c.AvailabilityZones is
+	// still empty here. Auto-selection always yields distinct zones, so there is nothing to
+	// validate on that path.
+	if c.VPC.Subnets == nil {
+		if len(c.AvailabilityZones) > 0 {
+			if azs := sets.New(c.AvailabilityZones...); azs.Len() < MinRequiredAvailabilityZones {
+				return fmt.Errorf("vpc.controlPlaneOnPrivateSubnets requires at least %d distinct availability zones, got %d (%v)", MinRequiredAvailabilityZones, azs.Len(), c.AvailabilityZones)
+			}
+		}
+		return nil
+	}
+
+	if numPrivate := len(c.VPC.Subnets.Private); numPrivate < MinRequiredSubnets {
+		return fmt.Errorf("vpc.controlPlaneOnPrivateSubnets requires at least %d private subnets, got %d", MinRequiredSubnets, numPrivate)
+	}
+
+	if azs := distinctSubnetAZs(c.VPC.Subnets.Private); len(azs) < MinRequiredAvailabilityZones {
+		return fmt.Errorf("vpc.controlPlaneOnPrivateSubnets requires private subnets in at least %d availability zones, got %d (%v)", MinRequiredAvailabilityZones, len(azs), azs)
+	}
+
+	return nil
+}
+
+// distinctSubnetAZs returns the unique availability zones covered by the given subnets.
+// A subnet's zone is taken from its AZ field, falling back to the mapping key, which is an
+// AZ name in the common form.
+//
+// This is best-effort: subnets given only by ID have their zone resolved from EC2 later, so
+// their real zone is unknown here and the mapping key is used instead. Such a configuration
+// is allowed through and is rejected by the EKS API if the subnets turn out to share a zone.
+// The check is deliberately permissive rather than risk rejecting a valid pre-existing VPC.
+func distinctSubnetAZs(subnets AZSubnetMapping) []string {
+	seen := make(map[string]struct{}, len(subnets))
+	azs := make([]string, 0, len(subnets))
+	for key, spec := range subnets {
+		az := spec.AZ
+		if az == "" {
+			az = key
+		}
+		if _, ok := seen[az]; ok {
+			continue
+		}
+		seen[az] = struct{}{}
+		azs = append(azs, az)
+	}
+	slices.Sort(azs)
+	return azs
+}
+
 func (c *ClusterConfig) unsupportedVPCCNIAddonVersion() (bool, error) {
 	for _, addon := range c.Addons {
 		if addon.Name == VPCCNIAddon {

pkg/apis/eksctl.io/v1alpha5/validation_test.go

@@ -1534,6 +1534,142 @@ var _ = Describe("ClusterConfig validation", func() {
 			})
 		})
 
+		Context("controlPlaneOnPrivateSubnets", func() {
+			privateSubnets := func(azs ...string) api.AZSubnetMapping {
+				m := api.NewAZSubnetMapping()
+				for i, az := range azs {
+					m.Set(fmt.Sprintf("subnet-alias-%d", i), api.AZSubnetSpec{
+						ID: fmt.Sprintf("subnet-%d", i),
+						AZ: az,
+					})
+				}
+				return m
+			}
+
+			When("it is enabled and eksctl creates the VPC with two distinct availability zones", func() {
+				It("does not reject the config, since subnets are derived from availabilityZones later", func() {
+					cfg.VPC.Subnets = nil
+					cfg.AvailabilityZones = []string{"us-west-2a", "us-west-2b"}
+					cfg.VPC.ControlPlaneOnPrivateSubnets = api.Enabled()
+					err = cfg.ValidateVPCConfig()
+					Expect(err).NotTo(HaveOccurred())
+				})
+			})
+
+			When("it is enabled and eksctl creates the VPC with a duplicated availability zone", func() {
+				It("returns an error, since the duplicate collapses into a single private subnet", func() {
+					cfg.VPC.Subnets = nil
+					cfg.AvailabilityZones = []string{"us-west-2a", "us-west-2a"}
+					cfg.VPC.ControlPlaneOnPrivateSubnets = api.Enabled()
+					err = cfg.ValidateVPCConfig()
+					Expect(err).To(MatchError("vpc.controlPlaneOnPrivateSubnets requires at least 2 distinct availability zones, got 1 ([us-west-2a us-west-2a])"))
+				})
+			})
+
+			When("it is enabled and eksctl creates the VPC without availability zones set", func() {
+				It("does not reject the config, since eksctl selects distinct zones itself later", func() {
+					cfg.VPC.Subnets = nil
+					cfg.AvailabilityZones = nil
+					cfg.VPC.ControlPlaneOnPrivateSubnets = api.Enabled()
+					err = cfg.ValidateVPCConfig()
+					Expect(err).NotTo(HaveOccurred())
+				})
+			})
+
+			When("it is enabled with two private subnets across two AZs", func() {
+				It("does not return an error", func() {
+					cfg.VPC.Subnets = &api.ClusterSubnets{
+						Private: privateSubnets("us-west-2a", "us-west-2b"),
+					}
+					cfg.VPC.ControlPlaneOnPrivateSubnets = api.Enabled()
+					err = cfg.ValidateVPCConfig()
+					Expect(err).NotTo(HaveOccurred())
+				})
+			})
+
+			When("it is enabled together with controlPlaneSubnetIDs", func() {
+				It("returns an error", func() {
+					cfg.VPC.ControlPlaneOnPrivateSubnets = api.Enabled()
+					cfg.VPC.ControlPlaneSubnetIDs = []string{"subnet-1234", "subnet-5678"}
+					err = cfg.ValidateVPCConfig()
+					Expect(err).To(MatchError("only one of vpc.controlPlaneSubnetIDs and vpc.controlPlaneOnPrivateSubnets can be specified"))
+				})
+			})
+
+			When("it is enabled but the VPC has no private subnets", func() {
+				It("returns an error instead of silently using public subnets", func() {
+					cfg.VPC.Subnets = &api.ClusterSubnets{
+						Public: privateSubnets("us-west-2a", "us-west-2b"),
+					}
+					cfg.VPC.ControlPlaneOnPrivateSubnets = api.Enabled()
+					err = cfg.ValidateVPCConfig()
+					Expect(err).To(MatchError("vpc.controlPlaneOnPrivateSubnets requires at least 2 private subnets, got 0"))
+				})
+			})
+
+			When("it is enabled with only one private subnet", func() {
+				It("returns an error", func() {
+					cfg.VPC.Subnets = &api.ClusterSubnets{
+						Private: privateSubnets("us-west-2a"),
+					}
+					cfg.VPC.ControlPlaneOnPrivateSubnets = api.Enabled()
+					err = cfg.ValidateVPCConfig()
+					Expect(err).To(MatchError("vpc.controlPlaneOnPrivateSubnets requires at least 2 private subnets, got 1"))
+				})
+			})
+
+			When("it is enabled with two private subnets in the same AZ", func() {
+				It("returns an error, since EKS requires two availability zones", func() {
+					cfg.VPC.Subnets = &api.ClusterSubnets{
+						Private: privateSubnets("us-west-2a", "us-west-2a"),
+					}
+					cfg.VPC.ControlPlaneOnPrivateSubnets = api.Enabled()
+					err = cfg.ValidateVPCConfig()
+					Expect(err).To(MatchError("vpc.controlPlaneOnPrivateSubnets requires private subnets in at least 2 availability zones, got 1 ([us-west-2a])"))
+				})
+			})
+
+			When("private subnets are given only by ID", func() {
+				It("allows the config through, since their zones are resolved from EC2 later", func() {
+					subnets := api.NewAZSubnetMapping()
+					subnets.Set("alias-a", api.AZSubnetSpec{ID: "subnet-aaa"})
+					subnets.Set("alias-b", api.AZSubnetSpec{ID: "subnet-bbb"})
+					cfg.VPC.ID = "vpc-123"
+					cfg.VPC.Subnets = &api.ClusterSubnets{Private: subnets}
+					cfg.VPC.ControlPlaneOnPrivateSubnets = api.Enabled()
+					err = cfg.ValidateVPCConfig()
+					Expect(err).NotTo(HaveOccurred())
+				})
+			})
+
+			When("one private subnet is keyed by AZ and another repeats that AZ explicitly", func() {
+				It("returns an error", func() {
+					subnets := api.NewAZSubnetMapping()
+					subnets.Set("us-west-2a", api.AZSubnetSpec{ID: "subnet-aaa"})
+					subnets.Set("alias-b", api.AZSubnetSpec{ID: "subnet-bbb", AZ: "us-west-2a"})
+					cfg.VPC.ID = "vpc-123"
+					cfg.VPC.Subnets = &api.ClusterSubnets{Private: subnets}
+					cfg.VPC.ControlPlaneOnPrivateSubnets = api.Enabled()
+					err = cfg.ValidateVPCConfig()
+					Expect(err).To(MatchError("vpc.controlPlaneOnPrivateSubnets requires private subnets in at least 2 availability zones, got 1 ([us-west-2a])"))
+				})
+			})
+
+			When("it is enabled on Outposts", func() {
+				It("does not enforce the multi-AZ requirement", func() {
+					cfg.VPC.Subnets = &api.ClusterSubnets{
+						Private: privateSubnets("us-west-2a"),
+					}
+					cfg.VPC.ControlPlaneOnPrivateSubnets = api.Enabled()
+					cfg.Outpost = &api.Outpost{
+						ControlPlaneOutpostARN: "arn:aws:outposts:us-west-2:1234:outpost/op-1234",
+					}
+					err = cfg.ValidateVPCConfig()
+					Expect(err).NotTo(HaveOccurred())
+				})
+			})
+		})
+
 		Context("ipv6 CIDRs", func() {
 			When("IPv6Cidr or IPv6CidrPool is provided and ipv6 is not set", func() {
 				It("returns an error", func() {

pkg/apis/eksctl.io/v1alpha5/vpc.go

@@ -178,6 +178,14 @@ type (
 		// ControlPlaneSubnetIDs configures the subnets for the control plane.
 		// +optional
 		ControlPlaneSubnetIDs []string `json:"controlPlaneSubnetIDs,omitempty"`
+		// ControlPlaneOnPrivateSubnets restricts the control plane (the cross-account ENIs
+		// that EKS places in the cluster subnets) to private subnets only, excluding public
+		// subnets. It applies both when eksctl creates the VPC and when a pre-existing VPC
+		// is used. Cannot be combined with ControlPlaneSubnetIDs. Requires at least two
+		// private subnets spanning at least two availability zones, which must have NAT or
+		// the relevant VPC endpoints for nodes to reach the API server.
+		// +optional
+		ControlPlaneOnPrivateSubnets *bool `json:"controlPlaneOnPrivateSubnets,omitempty"`
 		// ControlPlaneSecurityGroupIDs configures the security groups for the control plane.
 		// +optional
 		ControlPlaneSecurityGroupIDs []string `json:"controlPlaneSecurityGroupIDs,omitempty"`