Skip to content

Commit 85b7ce9

Browse files
feat(cluster): Add controlPlaneEgressMode support (#249)
Description of changes: EKS launched [controlPlaneEgressMode](https://aws.amazon.com/about-aws/whats-new/2026/06/amazon-eks-customer-routed-control-plane-egress), which lets customers route control plane egress traffic through their own VPC (`CUSTOMER_ROUTED`) instead of AWS managed networking (`AWS_MANAGED`). This adds support for configuring and updating it on the `Cluster` resource. Supersedes #228, which I've taken over. The `updateControlPlaneEgressMode` handler is Cavaughn's work and he's credited as co-author; the rest has been rebased onto `main` and reworked. #228 can be closed once this merges. ### Why the field was ignored, and what unblocks it The SDK bump to v1.91.0 (#245) deliberately added three `ignore.field_paths` entries for this field, with the note *"ignore it until the controller reconciles it as its own update type"*. The reason: EKS accepts only one type of update per `UpdateClusterConfig` call, and because `newVpcConfigRequest` populates every VPC field, the egress mode was riding along on endpoint-access and subnet updates and EKS rejected those with `Only one type of update can be allowed`. This PR satisfies that precondition, so the ignore entries are removed and the field is generated normally. Each VPC helper now sends only its own fields. ### Commits 1. **Generate the field** — remove the three ignore entries, add `late_initialize`, regenerate. EKS defaults the mode to `AWS_MANAGED` and returns it on read even when the spec omits it, so without `late_initialize` the controller treats the returned default as drift against a nil spec value and never converges. Regenerated with code-generator `v0.63.0` against `aws_service_sdk_version` `v1.91.0`. 2. **Reconcile it as its own update type** — a dedicated `customUpdate` branch and handler. The endpoint-access and subnet/security-group helpers clear the egress mode; the new handler clears everything else. The desired value is nil-guarded before dereference (clearing the field previously panicked). Whether a transition is legal is EKS's rule, not the controller's, so the request is sent as-is and any refusal surfaces as the service's own error. 3. **e2e coverage** — one test for reconcile behaviour, one for the create path. ### A note on the unsupported transition EKS refuses to move a cluster from `CUSTOMER_ROUTED` back to `AWS_MANAGED`: ``` InvalidParameterException: Update of ControlPlaneEgressMode from CUSTOMER_ROUTED to AWS_MANAGED is not supported ``` That code isn't in the resource's `terminal_codes`, so it's reported as recoverable and retried indefinitely. I deliberately did **not** encode the transition rule in the controller — that would break the moment AWS relaxes it, and it silently skips a call the service is the authority on. Making this terminal is a `terminal_codes` decision and is left out of this PR. Worth noting `InvalidParameterException` is broad on this resource (a rejected `publicAccessCidrs` value returns it too, and also retries forever today), so it may be worth adding regardless — happy to do it here if reviewers prefer. ### Testing - `go build ./...` and `go test ./...` pass; `gofmt` clean. - Generated output is reproducible: `ack-workspace build eks` (code-generator `v0.63.0`) produces the committed files, including the `delta.go` comparison and the `!= ""` guard on the read path. This exact commit was deployed to a live EKS cluster and each behaviour exercised end to end: | Behaviour | Result | |---|---| | Create with the field omitted | `late_initialize` adopts `AWS_MANAGED`; Synced with no drift and no API calls | | Clear the field from the spec | No panic, no terminal condition; late-init re-adopts the observed value | | Endpoint-access-only update, egress mode set | `EndpointAccessUpdate` succeeds; params carry only endpoint-access fields, no `ControlPlaneEgressMode`. No `Only one type of update can be allowed` rejection | | `AWS_MANAGED` -> `CUSTOMER_ROUTED` | `ControlPlaneEgressUpdate` succeeds carrying only `ControlPlaneEgressMode`; endpoint access untouched | | `CUSTOMER_ROUTED` -> `AWS_MANAGED` | Refused by EKS; surfaces as `ACK.Recoverable` with the message above, cluster left unchanged, no update record created | | Create directly with `CUSTOMER_ROUTED` | Cluster reaches ACTIVE in that mode, Synced, and `list_updates` is empty — the value was carried by `CreateCluster`, not a follow-up update | One gap reviewers should know about: **the e2e tests have not been run under pytest.** They are verified by syntax and structure only, and the behaviours they assert were validated by driving the same sequences against a live cluster by hand. I did not have a bootstrapped e2e environment. The create-path check used public subnets from a dev-account VPC rather than the e2e bootstrap VPC, so subnet IDs differ, but the topology is the same (public subnets with an internet gateway route) and EKS accepted `CUSTOMER_ROUTED` at create there. Also note `test/e2e/requirements.txt` pins only `acktest`, and the botocore it resolves does not yet model `controlPlaneEgressMode` — botocore silently drops response members it doesn't know, so AWS-side assertions on it go through a helper that skips just that comparison rather than failing for an unrelated reason. A botocore floor in `requirements.txt` would let CI make the stronger assertion. All AWS-side values quoted above were read with an SDK build that does model the field. By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
1 parent f450a4e commit 85b7ce9

14 files changed

Lines changed: 329 additions & 28 deletions

File tree

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,13 @@
11
ack_generate_info:
2-
build_date: "2026-08-28T19:00:54Z"
2+
build_date: "2026-09-02T17:21:51Z"
33
build_hash: 399b047701458765c3dd8e9dd663ce2ef0337c05
4-
go_version: go1.26.5
4+
go_version: go1.26.0
55
version: v0.63.0
6-
api_directory_checksum: c9071729170aba1679a65bc5c006b89bb0dbed4a
6+
api_directory_checksum: 212ccb6a04ed51d790966d4b2549421657ca0388
77
api_version: v1alpha1
88
aws_service_sdk_version: v1.91.0
99
generator_config_info:
10-
file_checksum: ed98c39b3dc48ec34f4f462b64aa350d2fea98af
10+
file_checksum: 8044a2785c40dab32dcc01f65f21bf717bf2fe13
1111
original_file_name: generator.yaml
1212
last_modification:
1313
reason: API generation

apis/v1alpha1/generator.yaml

Lines changed: 2 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -199,6 +199,8 @@ resources:
199199
pre_delete_include: true
200200
ControlPlaneScalingConfig.Tier:
201201
late_initialize: {}
202+
ResourcesVpcConfig.ControlPlaneEgressMode:
203+
late_initialize: {}
202204
# Control plane component config fields. The backend injects tier defaults
203205
# into these on the read path, so late_initialize prevents the controller
204206
# from treating those defaults as drift and reconciling forever. Each
@@ -690,15 +692,6 @@ ignore:
690692
- CreateCapabilityInput.ClientRequestToken
691693
- CreateAddonInput.NamespaceConfig
692694
- CreateAddonOutput.Addon.NamespaceConfig
693-
# ControlPlaneEgressMode (new in the v1.91.0 model) is returned by EKS on read
694-
# (e.g. "AWS_MANAGED") and gets folded into the ResourcesVpcConfig update
695-
# request. EKS rejects an UpdateClusterConfig that carries an endpoint-access
696-
# change together with controlPlaneEgressMode ("Only one type of update can be
697-
# allowed"), breaking the existing VPC endpoint-access update. Ignore it until
698-
# the controller reconciles it as its own update type.
699-
- CreateClusterInput.ResourcesVpcConfig.ControlPlaneEgressMode
700-
- CreateClusterOutput.Cluster.ResourcesVpcConfig.ControlPlaneEgressMode
701-
- DescribeClusterOutput.Cluster.ResourcesVpcConfig.ControlPlaneEgressMode
702695
# The v1.91.0 SDK model also adds unrelated new fields to the Cluster
703696
# (Outpost/local-cluster etcd placement) and Nodegroup (managed-nodegroup warm
704697
# pools) shapes. These are separate EKS capabilities that need their own

apis/v1alpha1/types.go

Lines changed: 6 additions & 4 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

apis/v1alpha1/zz_generated.deepcopy.go

Lines changed: 10 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

config/crd/bases/eks.services.k8s.aws_clusters.yaml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -383,6 +383,8 @@ spec:
383383
can specify up to five security groups. However, we recommend that you use
384384
a dedicated security group for your cluster control plane.
385385
properties:
386+
controlPlaneEgressMode:
387+
type: string
386388
endpointPrivateAccess:
387389
type: boolean
388390
endpointPublicAccess:

generator.yaml

Lines changed: 2 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -199,6 +199,8 @@ resources:
199199
pre_delete_include: true
200200
ControlPlaneScalingConfig.Tier:
201201
late_initialize: {}
202+
ResourcesVpcConfig.ControlPlaneEgressMode:
203+
late_initialize: {}
202204
# Control plane component config fields. The backend injects tier defaults
203205
# into these on the read path, so late_initialize prevents the controller
204206
# from treating those defaults as drift and reconciling forever. Each
@@ -690,15 +692,6 @@ ignore:
690692
- CreateCapabilityInput.ClientRequestToken
691693
- CreateAddonInput.NamespaceConfig
692694
- CreateAddonOutput.Addon.NamespaceConfig
693-
# ControlPlaneEgressMode (new in the v1.91.0 model) is returned by EKS on read
694-
# (e.g. "AWS_MANAGED") and gets folded into the ResourcesVpcConfig update
695-
# request. EKS rejects an UpdateClusterConfig that carries an endpoint-access
696-
# change together with controlPlaneEgressMode ("Only one type of update can be
697-
# allowed"), breaking the existing VPC endpoint-access update. Ignore it until
698-
# the controller reconciles it as its own update type.
699-
- CreateClusterInput.ResourcesVpcConfig.ControlPlaneEgressMode
700-
- CreateClusterOutput.Cluster.ResourcesVpcConfig.ControlPlaneEgressMode
701-
- DescribeClusterOutput.Cluster.ResourcesVpcConfig.ControlPlaneEgressMode
702695
# The v1.91.0 SDK model also adds unrelated new fields to the Cluster
703696
# (Outpost/local-cluster etcd placement) and Nodegroup (managed-nodegroup warm
704697
# pools) shapes. These are separate EKS capabilities that need their own

helm/crds/eks.services.k8s.aws_clusters.yaml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -383,6 +383,8 @@ spec:
383383
can specify up to five security groups. However, we recommend that you use
384384
a dedicated security group for your cluster control plane.
385385
properties:
386+
controlPlaneEgressMode:
387+
type: string
386388
endpointPrivateAccess:
387389
type: boolean
388390
endpointPublicAccess:

pkg/resource/cluster/delta.go

Lines changed: 7 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

pkg/resource/cluster/hook.go

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -300,6 +300,19 @@ func (rm *resourceManager) customUpdate(
300300
return returnClusterUpdating(updatedRes)
301301
}
302302

303+
if delta.DifferentAt("Spec.ResourcesVPCConfig.ControlPlaneEgressMode") &&
304+
desired.ko.Spec.ResourcesVPCConfig != nil &&
305+
desired.ko.Spec.ResourcesVPCConfig.ControlPlaneEgressMode != nil {
306+
if err := rm.updateControlPlaneEgressMode(ctx, desired); err != nil {
307+
awsErr, ok := extractAWSError(err)
308+
if ok && awsErr.Code == "ResourceInUseException" {
309+
return nil, requeueAfterAsyncUpdate()
310+
}
311+
return nil, err
312+
}
313+
return returnClusterUpdating(updatedRes)
314+
}
315+
303316
// Handle access configuration updates
304317
if delta.DifferentAt("Spec.AccessConfig") {
305318
if err := rm.updateAccessConfig(ctx, desired); err != nil {
@@ -611,6 +624,7 @@ func (rm *resourceManager) updateConfigResourcesVPCConfigPublicAndPrivateAccess(
611624
// publicAccessCidrs
612625
input.ResourcesVpcConfig.SubnetIds = nil
613626
input.ResourcesVpcConfig.SecurityGroupIds = nil
627+
input.ResourcesVpcConfig.ControlPlaneEgressMode = ""
614628

615629
_, err = rm.sdkapi.UpdateClusterConfig(ctx, input)
616630
rm.metrics.RecordAPICall("UPDATE", "UpdateClusterConfig", err)
@@ -637,6 +651,7 @@ func (rm *resourceManager) updateConfigResourcesVPCConfigSubnetsAndSecurityGroup
637651
input.ResourcesVpcConfig.EndpointPublicAccess = nil
638652
input.ResourcesVpcConfig.EndpointPrivateAccess = nil
639653
input.ResourcesVpcConfig.PublicAccessCidrs = nil
654+
input.ResourcesVpcConfig.ControlPlaneEgressMode = ""
640655

641656
_, err = rm.sdkapi.UpdateClusterConfig(ctx, input)
642657
rm.metrics.RecordAPICall("UPDATE", "UpdateClusterConfig", err)
@@ -904,3 +919,30 @@ func extractAWSError(err error) (awsErr *smithy.GenericAPIError, ok bool) {
904919
}
905920
return nil, false
906921
}
922+
923+
func (rm *resourceManager) updateControlPlaneEgressMode(
924+
ctx context.Context,
925+
r *resource,
926+
) (err error) {
927+
rlog := ackrtlog.FromContext(ctx)
928+
exit := rlog.Trace("rm.updateControlPlaneEgressMode")
929+
defer exit(err)
930+
input := &svcsdk.UpdateClusterConfigInput{
931+
Name: r.ko.Spec.Name,
932+
ResourcesVpcConfig: rm.newVpcConfigRequest(r),
933+
}
934+
935+
input.ResourcesVpcConfig.SubnetIds = nil
936+
input.ResourcesVpcConfig.SecurityGroupIds = nil
937+
input.ResourcesVpcConfig.EndpointPublicAccess = nil
938+
input.ResourcesVpcConfig.EndpointPrivateAccess = nil
939+
input.ResourcesVpcConfig.PublicAccessCidrs = nil
940+
941+
_, err = rm.sdkapi.UpdateClusterConfig(ctx, input)
942+
rm.metrics.RecordAPICall("UPDATE", "UpdateClusterConfig", err)
943+
if err != nil {
944+
return err
945+
}
946+
947+
return nil
948+
}

pkg/resource/cluster/manager.go

Lines changed: 11 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)