Describe the bug
Updating a ResolverEndpoint's spec.ipAddresses from one set of subnets to another can permanently strand one of the old IP associations — the endpoint ends up with 4 IPs (3 correct new + 1 stale old) instead of 3. This is 100% reproducible in both directions of a full subnet swap, and does not reliably self-heal — the stale IP can persist indefinitely until some unrelated future spec change happens to reference the same subnet again.
Steps to reproduce
Update a ResolverEndpoint from 3 subnets (A) to 3 different subnets (B) in a single spec change (endpointSubnetRefs). Controller: route53resolver.services.k8s.aws-1.8.1.
What happens:
Controller callsAssociateResolverEndpointIpAddressfor all 3 new subnets.
Immediately after, callsDisassociateResolverEndpointIpAddressfor all 3 old subnets — with no wait for the new IPs to reachATTACHEDstate.
2 of the 3 disassociates succeed; the 3rd — consistently whichever is last in iteration order — fails:
BerrorCode: InvalidRequestException
errorMessage: [RSLVR-00506] Can't remove the only attached IP address from resolver endpoint.
Confirmed via CloudTrail across multiple endpoints, and reproduced symmetrically in both directions of the same subnet swap (A→B and B→A) — always the last old IP processed.
The controller doesn't retry this rejection within the reconcile — it just surfaces the error and moves on. The stale IP staysATTACHEDindefinitely; it is only cleaned up if somelaterspec change happens to recompute a diff that includes it in the removal set with enough headroom to succeed. If a later change instead happens to reference that same subnet again, the stale IP gets silently absorbed as "already satisfying desired" and is never properly disassociated/reassociated at all.
Root cause:
Race between association and disassociation — the new IPs haven't reached ATTACHED yet when the last old IP's removal is attempted, so AWS's own endpoint-can't-go-to-zero-attached-IPs safety check (RSLVR-00506) rejects it. Code path: pkg/resource/resolver_endpoint/hook.go, SyncIPAddresses/GetIPAddressDifference.
Impact:
Persistent AWS resource drift (an extra live IP/ENI on the endpoint) with no bounded recovery path — not a fixed retry/backoff, but an incidental side effect of whatever future spec change happens to touch the same field. Confirmed on synthetic test resources; currently verifying whether it also strands IPs on real, adopted (non-controller-created) ResolverEndpoints, which would mean production DNS-forwarding infrastructure can be left with unintended stale IPs indefinitely.
Describe the bug
Updating a ResolverEndpoint's spec.ipAddresses from one set of subnets to another can permanently strand one of the old IP associations — the endpoint ends up with 4 IPs (3 correct new + 1 stale old) instead of 3. This is 100% reproducible in both directions of a full subnet swap, and does not reliably self-heal — the stale IP can persist indefinitely until some unrelated future spec change happens to reference the same subnet again.
Steps to reproduce
Update a ResolverEndpoint from 3 subnets (A) to 3 different subnets (B) in a single spec change (endpointSubnetRefs). Controller: route53resolver.services.k8s.aws-1.8.1.
What happens:
Controller callsAssociateResolverEndpointIpAddressfor all 3 new subnets.
Immediately after, callsDisassociateResolverEndpointIpAddressfor all 3 old subnets — with no wait for the new IPs to reachATTACHEDstate.
2 of the 3 disassociates succeed; the 3rd — consistently whichever is last in iteration order — fails:
BerrorCode: InvalidRequestException
errorMessage: [RSLVR-00506] Can't remove the only attached IP address from resolver endpoint.
Confirmed via CloudTrail across multiple endpoints, and reproduced symmetrically in both directions of the same subnet swap (A→B and B→A) — always the last old IP processed.
The controller doesn't retry this rejection within the reconcile — it just surfaces the error and moves on. The stale IP staysATTACHEDindefinitely; it is only cleaned up if somelaterspec change happens to recompute a diff that includes it in the removal set with enough headroom to succeed. If a later change instead happens to reference that same subnet again, the stale IP gets silently absorbed as "already satisfying desired" and is never properly disassociated/reassociated at all.
Root cause:
Race between association and disassociation — the new IPs haven't reached ATTACHED yet when the last old IP's removal is attempted, so AWS's own endpoint-can't-go-to-zero-attached-IPs safety check (RSLVR-00506) rejects it. Code path: pkg/resource/resolver_endpoint/hook.go, SyncIPAddresses/GetIPAddressDifference.
Impact:
Persistent AWS resource drift (an extra live IP/ENI on the endpoint) with no bounded recovery path — not a fixed retry/backoff, but an incidental side effect of whatever future spec change happens to touch the same field. Confirmed on synthetic test resources; currently verifying whether it also strands IPs on real, adopted (non-controller-created) ResolverEndpoints, which would mean production DNS-forwarding infrastructure can be left with unintended stale IPs indefinitely.