Describe the bug
UserPool in cognitoidentityprovider-controller exposes two spec fields that both map to the same set of AWS tags, and ACK's own injected tags land in one of them but are compared against the other. The result is a delta on Spec.UserPoolTags that recurs on every reconcile and that no update can resolve.
The two surfaces:
| Field |
Origin |
Who touches it |
spec.tags |
Synthetic, from generator.yaml: fields.Tags.from: {operation: TagResource, path: Tags} |
All of ACK's tag machinery — EnsureTags, FilterSystemTags, mirrorAWSTags (pkg/resource/user_pool/manager.go), and SyncTags → TagResource/UntagResource (pkg/resource/user_pool/hook.go) |
spec.userPoolTags |
Native CreateUserPool / DescribeUserPool member |
Populated from the API response in sdkFind/sdkCreate; sent back verbatim in the create and update payloads |
The loop:
- The runtime merges its default tags (
services.k8s.aws/controller-version, services.k8s.aws/namespace) into Spec.Tags, and sdkCreate writes them to the pool with TagResource (pkg/resource/user_pool/sdk.go:881).
DescribeUserPool returns those tags, and sdkFind copies them into Spec.UserPoolTags (sdk.go:445-448) — not into Spec.Tags.
- The user never set
spec.userPoolTags, so desired is nil while latest holds two entries. The generated comparison is a length check first, so it fires unconditionally (pkg/resource/user_pool/delta.go:463-469):
if len(a.ko.Spec.UserPoolTags) != len(b.ko.Spec.UserPoolTags) {
delta.Add("Spec.UserPoolTags", a.ko.Spec.UserPoolTags, b.ko.Spec.UserPoolTags)
}
- Nothing converges it.
mirrorAWSTags — the mechanism that exists precisely to keep AWS-injected tags out of the diff — only reads and writes Spec.Tags, and UserPoolTags has no late_initialize.
- Because the delta is at
Spec.UserPoolTags rather than Spec.Tags, the tags-only short circuit in sdkUpdate does not fire:
if !delta.DifferentExcept("Spec.Tags") {
return desired, nil
}
So a real UpdateUserPool call is issued on every reconcile, for every UserPool, forever — carrying the nil desired UserPoolTags (sdk.go:1542), which changes nothing.
Steps to reproduce
- Install
cognitoidentityprovider-controller (reproduced on main at v1.4.1 — runtime v0.62.0, code-generator v0.62.1) with a short resync so the loop is observable, e.g. reconcile.defaultResyncPeriod=60.
- Create a
UserPool with no tags at all:
apiVersion: cognitoidentityprovider.services.k8s.aws/v1alpha1
kind: UserPool
metadata:
name: ack-test-up-neither
namespace: ack-system
spec:
name: ack-test-up-neither
- Stream the controller log and count the diff events. Note that
kubectl logs --since= is unreliable here — the container log rotates aggressively enough to make a recurring diff look like a one-off, so stream continuously:
kubectl -n ack-system logs -f deploy -l app.kubernetes.io/instance=ack-cognitoidentityprovider-controller \
| grep 'desired resource state has changed'
Observed over ~7 minutes at a 60s resync across 8 pools: 56 diff events, every one on Spec.UserPoolTags, 7 per pool — i.e. one per resource per reconcile, with no convergence.
delta events: 56
BY PATH:
56 Spec.UserPoolTags
SAMPLE:
Spec.UserPoolTags [ack-test-up-neither] A=None
B={'services.k8s.aws/controller-version': ..., 'services.k8s.aws/namespace': ...}
Expected outcome
A UserPool that has converged issues no further UpdateUserPool calls. ACK's own injected tags should not appear as a user-visible drift, which is what mirrorAWSTags already guarantees for spec.tags.
Actual outcome
ACK.ResourceSynced does reach True, so the resource is not stuck — this is a silent no-op update loop rather than a hang. The costs are a permanent UpdateUserPool call per pool per resync (a throttling risk at scale) and a diff that never clears, which makes drift on this resource impossible to read.
Candidate fixes
Listing these for maintainer input rather than picking one, since two of the three change the CRD surface:
- Suppress the duplicate:
ignore.field_paths: [UserPool.UserPoolTags]. spec.tags is the ACK-conventional surface and the only one wired to TagResource/UntagResource, so the native member is redundant. Removes a field that currently does work on create, so it is breaking for anyone setting spec.userPoolTags directly.
- Make the native member the managed one — point the tag config at
UserPoolTags and drop the synthetic Tags field. Arguably the more honest model, but breaking for the spec.tags users who exist today.
late_initialize on UserPoolTags so the mirrored values are absorbed into the spec. Cheapest and non-breaking, but leaves two tag surfaces that can disagree, and a user removing a tag from spec.userPoolTags would not converge.
Option 1 looks right to us, but the choice is really about which surface the project wants to keep.
Environment
- Kubernetes version: 1.34 (EKS Auto Mode)
- Using EKS: yes, 1.34
- AWS service targeted: Cognito Identity Provider (
cognitoidentityprovider-controller v1.4.1, ACK runtime v0.62.0, code-generator v0.62.1)
Notes
Found while validating cross-resource references on UserPool (aws-controllers-k8s/cognitoidentityprovider-controller#54, and previously in the closed #50). It is unrelated to that change and pre-existing: pkg/resource/user_pool/delta.go is byte-identical between that branch and main, the Tags entry in generator.yaml is untouched, and the diff fires on pools that use no references at all. Filing it here so it can be fixed on its own terms.
Same class of problem as the older #1939 / #2000 tag-driven re-reconcile reports, though milder — here the resource still reports Synced=True.
Describe the bug
UserPoolincognitoidentityprovider-controllerexposes two spec fields that both map to the same set of AWS tags, and ACK's own injected tags land in one of them but are compared against the other. The result is a delta onSpec.UserPoolTagsthat recurs on every reconcile and that no update can resolve.The two surfaces:
spec.tagsgenerator.yaml:fields.Tags.from: {operation: TagResource, path: Tags}EnsureTags,FilterSystemTags,mirrorAWSTags(pkg/resource/user_pool/manager.go), andSyncTags→TagResource/UntagResource(pkg/resource/user_pool/hook.go)spec.userPoolTagsCreateUserPool/DescribeUserPoolmembersdkFind/sdkCreate; sent back verbatim in the create and update payloadsThe loop:
services.k8s.aws/controller-version,services.k8s.aws/namespace) intoSpec.Tags, andsdkCreatewrites them to the pool withTagResource(pkg/resource/user_pool/sdk.go:881).DescribeUserPoolreturns those tags, andsdkFindcopies them intoSpec.UserPoolTags(sdk.go:445-448) — not intoSpec.Tags.spec.userPoolTags, so desired isnilwhile latest holds two entries. The generated comparison is a length check first, so it fires unconditionally (pkg/resource/user_pool/delta.go:463-469):mirrorAWSTags— the mechanism that exists precisely to keep AWS-injected tags out of the diff — only reads and writesSpec.Tags, andUserPoolTagshas nolate_initialize.Spec.UserPoolTagsrather thanSpec.Tags, the tags-only short circuit insdkUpdatedoes not fire:So a real
UpdateUserPoolcall is issued on every reconcile, for everyUserPool, forever — carrying thenildesiredUserPoolTags(sdk.go:1542), which changes nothing.Steps to reproduce
cognitoidentityprovider-controller(reproduced onmainat v1.4.1 — runtime v0.62.0, code-generator v0.62.1) with a short resync so the loop is observable, e.g.reconcile.defaultResyncPeriod=60.UserPoolwith no tags at all:kubectl logs --since=is unreliable here — the container log rotates aggressively enough to make a recurring diff look like a one-off, so stream continuously:Observed over ~7 minutes at a 60s resync across 8 pools: 56 diff events, every one on
Spec.UserPoolTags, 7 per pool — i.e. one per resource per reconcile, with no convergence.Expected outcome
A
UserPoolthat has converged issues no furtherUpdateUserPoolcalls. ACK's own injected tags should not appear as a user-visible drift, which is whatmirrorAWSTagsalready guarantees forspec.tags.Actual outcome
ACK.ResourceSynceddoes reachTrue, so the resource is not stuck — this is a silent no-op update loop rather than a hang. The costs are a permanentUpdateUserPoolcall per pool per resync (a throttling risk at scale) and a diff that never clears, which makes drift on this resource impossible to read.Candidate fixes
Listing these for maintainer input rather than picking one, since two of the three change the CRD surface:
ignore.field_paths: [UserPool.UserPoolTags].spec.tagsis the ACK-conventional surface and the only one wired toTagResource/UntagResource, so the native member is redundant. Removes a field that currently does work on create, so it is breaking for anyone settingspec.userPoolTagsdirectly.UserPoolTagsand drop the syntheticTagsfield. Arguably the more honest model, but breaking for thespec.tagsusers who exist today.late_initializeonUserPoolTagsso the mirrored values are absorbed into the spec. Cheapest and non-breaking, but leaves two tag surfaces that can disagree, and a user removing a tag fromspec.userPoolTagswould not converge.Option 1 looks right to us, but the choice is really about which surface the project wants to keep.
Environment
cognitoidentityprovider-controllerv1.4.1, ACK runtime v0.62.0, code-generator v0.62.1)Notes
Found while validating cross-resource references on
UserPool(aws-controllers-k8s/cognitoidentityprovider-controller#54, and previously in the closed #50). It is unrelated to that change and pre-existing:pkg/resource/user_pool/delta.gois byte-identical between that branch andmain, theTagsentry ingenerator.yamlis untouched, and the diff fires on pools that use no references at all. Filing it here so it can be fixed on its own terms.Same class of problem as the older #1939 / #2000 tag-driven re-reconcile reports, though milder — here the resource still reports
Synced=True.