Skip to content

DBClusters stuck in 'Reset master credentials' reconcile loop when 'enableIAMDatabaseAuthentication' is updated without 'applyImmediately' #2327

Description

@fbuchmeier-abi

What happened?

When applying a change to a dbclusters.rds.aws.crossplane.io that is not applied immediately by default and applyImmediately is set to false in the dbcluster, the managed resource will be stuck in a reconcile loop resetting the master credentials on every interval.

I can see that similar issues already existed for:

How can we reproduce it?

  1. Deploy an Example DBCluster with applyImmediately=false and enableIAMDatabaseAuthentication=false:

    apiVersion: v1
    kind: Secret
    metadata:
      name: example-rds-cluster-iam-auth-root
      namespace: default
    type: Opaque
    stringData:
      password: j2qDttjE2ia5LrKhGD+N
    ---
    apiVersion: rds.aws.crossplane.io/v1alpha1
    kind: DBCluster
    metadata:
      name: example-rds-cluster-iam-auth-v2ljv
    spec:
      deletionPolicy: Delete
      forProvider:
        applyImmediately: false
        backupRetentionPeriod: 1
        copyTagsToSnapshot: true
        databaseName: exampledb
        dbClusterParameterGroupName: parameterGroupName
        dbSubnetGroupName: example.com
        deletionProtection: false
        enableCloudwatchLogsExports:
        - audit
        enableIAMDatabaseAuthentication: false
        engine: aurora-mysql
        engineMode: provisioned
        engineVersion: 8.0.mysql_aurora.3.10.3
        finalDBSnapshotIdentifier: ""
        masterUserPasswordSecretRef:
          key: password
          name: example-rds-cluster-iam-auth-root
          namespace: default
        masterUsername: master
        port: 3306
        region: eu-central-1
        skipFinalSnapshot: true
        storageEncrypted: true
        tags: []
      managementPolicies:
      - '*'
      providerConfigRef:
        name: aws-provider-config
      writeConnectionSecretToRef:
        name: example-rds-cluster-iam-auth-v2ljv-rds-cluster-managed-resource
        namespace: crossplane-system
  2. Deploy the same cluster again, this time with:

    spec:
      enableIAMDatabaseAuthentication: true

    Be sure to keep applyImmediately: false

  3. Wait until the cluster has been provisioned and check the status in AWS

    aws rds describe-events --source-identifier example-rds-cluster-iam-auth-123 --source-type db-cluster --region eu-central-1
    {
        "Events": [
            {
                "SourceIdentifier": "example-rds-cluster-iam-auth-123",
                "SourceType": "db-cluster",
                "Message": "DB cluster created as a new cluster",
                "EventCategories": [
                    "creation"
                ],
                "Date": "2026-08-14T12:22:42.364000+00:00",
                "SourceArn": "arn:aws:rds:eu-central-1:accountID:cluster:example-rds-cluster-iam-auth-123"
            },
            {
                "SourceIdentifier": "example-rds-cluster-iam-auth-123",
                "SourceType": "db-cluster",
                "Message": "Reset master credentials",
                "EventCategories": [
                    "configuration change"
                ],
                "Date": "2026-08-14T12:39:48.785000+00:00",
                "SourceArn": "arn:aws:rds:eu-central-1:accountID:cluster:example-rds-cluster-iam-auth-123"
            },
            {
                "SourceIdentifier": "example-rds-cluster-iam-auth-123",
                "SourceType": "db-cluster",
                "Message": "Reset master credentials",
                "EventCategories": [
                    "configuration change"
                ],
                "Date": "2026-08-14T12:40:07.086000+00:00",
                "SourceArn": "arn:aws:rds:eu-central-1:accountID:cluster:example-rds-cluster-iam-auth-123"
            },
    ...
            {
                "SourceIdentifier": "example-rds-cluster-iam-auth-123",
                "SourceType": "db-cluster",
                "Message": "Reset master credentials",
                "EventCategories": [
                    "configuration change"
                ],
                "Date": "2026-08-14T13:13:55.979000+00:00",
                "SourceArn": "arn:aws:rds:eu-central-1:accountID:cluster:example-rds-cluster-iam-auth-123"
            }
        ]
    }
    

What environment did it happen in?

Crossplane version: xpkg.crossplane.io/crossplane/crossplane:v2.3.4
AWS provider version: xpkg.upbound.io/crossplane-contrib/provider-aws:v0.58.1
Kubernetes version: v1.35.6-eks-bca9cf6
Kubernetes distribution: EKS

Please let me know if I can support you in troubleshooting this issue of if any additional information is required to reproduce it.

Thank you very much for your time!

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions