Skip to content

bug: cognitoidentityprovider UserPool has a perpetual Spec.UserPoolTags delta from ACK's own injected tags #3012

Description

@gustavodiaz7722

Describe the bug

UserPool in cognitoidentityprovider-controller exposes two spec fields that both map to the same set of AWS tags, and ACK's own injected tags land in one of them but are compared against the other. The result is a delta on Spec.UserPoolTags that recurs on every reconcile and that no update can resolve.

The two surfaces:

Field Origin Who touches it
spec.tags Synthetic, from generator.yaml: fields.Tags.from: {operation: TagResource, path: Tags} All of ACK's tag machinery — EnsureTags, FilterSystemTags, mirrorAWSTags (pkg/resource/user_pool/manager.go), and SyncTagsTagResource/UntagResource (pkg/resource/user_pool/hook.go)
spec.userPoolTags Native CreateUserPool / DescribeUserPool member Populated from the API response in sdkFind/sdkCreate; sent back verbatim in the create and update payloads

The loop:

  1. The runtime merges its default tags (services.k8s.aws/controller-version, services.k8s.aws/namespace) into Spec.Tags, and sdkCreate writes them to the pool with TagResource (pkg/resource/user_pool/sdk.go:881).
  2. DescribeUserPool returns those tags, and sdkFind copies them into Spec.UserPoolTags (sdk.go:445-448) — not into Spec.Tags.
  3. The user never set spec.userPoolTags, so desired is nil while latest holds two entries. The generated comparison is a length check first, so it fires unconditionally (pkg/resource/user_pool/delta.go:463-469):
if len(a.ko.Spec.UserPoolTags) != len(b.ko.Spec.UserPoolTags) {
    delta.Add("Spec.UserPoolTags", a.ko.Spec.UserPoolTags, b.ko.Spec.UserPoolTags)
}
  1. Nothing converges it. mirrorAWSTags — the mechanism that exists precisely to keep AWS-injected tags out of the diff — only reads and writes Spec.Tags, and UserPoolTags has no late_initialize.
  2. Because the delta is at Spec.UserPoolTags rather than Spec.Tags, the tags-only short circuit in sdkUpdate does not fire:
if !delta.DifferentExcept("Spec.Tags") {
    return desired, nil
}

So a real UpdateUserPool call is issued on every reconcile, for every UserPool, forever — carrying the nil desired UserPoolTags (sdk.go:1542), which changes nothing.

Steps to reproduce

  1. Install cognitoidentityprovider-controller (reproduced on main at v1.4.1 — runtime v0.62.0, code-generator v0.62.1) with a short resync so the loop is observable, e.g. reconcile.defaultResyncPeriod=60.
  2. Create a UserPool with no tags at all:
apiVersion: cognitoidentityprovider.services.k8s.aws/v1alpha1
kind: UserPool
metadata:
  name: ack-test-up-neither
  namespace: ack-system
spec:
  name: ack-test-up-neither
  1. Stream the controller log and count the diff events. Note that kubectl logs --since= is unreliable here — the container log rotates aggressively enough to make a recurring diff look like a one-off, so stream continuously:
kubectl -n ack-system logs -f deploy -l app.kubernetes.io/instance=ack-cognitoidentityprovider-controller \
  | grep 'desired resource state has changed'

Observed over ~7 minutes at a 60s resync across 8 pools: 56 diff events, every one on Spec.UserPoolTags, 7 per pool — i.e. one per resource per reconcile, with no convergence.

delta events: 56
BY PATH:
    56  Spec.UserPoolTags
SAMPLE:
  Spec.UserPoolTags  [ack-test-up-neither]  A=None
                     B={'services.k8s.aws/controller-version': ..., 'services.k8s.aws/namespace': ...}

Expected outcome

A UserPool that has converged issues no further UpdateUserPool calls. ACK's own injected tags should not appear as a user-visible drift, which is what mirrorAWSTags already guarantees for spec.tags.

Actual outcome

ACK.ResourceSynced does reach True, so the resource is not stuck — this is a silent no-op update loop rather than a hang. The costs are a permanent UpdateUserPool call per pool per resync (a throttling risk at scale) and a diff that never clears, which makes drift on this resource impossible to read.

Candidate fixes

Listing these for maintainer input rather than picking one, since two of the three change the CRD surface:

  1. Suppress the duplicate: ignore.field_paths: [UserPool.UserPoolTags]. spec.tags is the ACK-conventional surface and the only one wired to TagResource/UntagResource, so the native member is redundant. Removes a field that currently does work on create, so it is breaking for anyone setting spec.userPoolTags directly.
  2. Make the native member the managed one — point the tag config at UserPoolTags and drop the synthetic Tags field. Arguably the more honest model, but breaking for the spec.tags users who exist today.
  3. late_initialize on UserPoolTags so the mirrored values are absorbed into the spec. Cheapest and non-breaking, but leaves two tag surfaces that can disagree, and a user removing a tag from spec.userPoolTags would not converge.

Option 1 looks right to us, but the choice is really about which surface the project wants to keep.

Environment

  • Kubernetes version: 1.34 (EKS Auto Mode)
  • Using EKS: yes, 1.34
  • AWS service targeted: Cognito Identity Provider (cognitoidentityprovider-controller v1.4.1, ACK runtime v0.62.0, code-generator v0.62.1)

Notes

Found while validating cross-resource references on UserPool (aws-controllers-k8s/cognitoidentityprovider-controller#54, and previously in the closed #50). It is unrelated to that change and pre-existing: pkg/resource/user_pool/delta.go is byte-identical between that branch and main, the Tags entry in generator.yaml is untouched, and the diff fires on pools that use no references at all. Filing it here so it can be fixed on its own terms.

Same class of problem as the older #1939 / #2000 tag-driven re-reconcile reports, though milder — here the resource still reports Synced=True.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

kind/bugCategorizes issue or PR as related to a bug.service/cognitoidentityproviderIndicates issues or PRs that are related to cognitoidentityprovider-controller.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions