OCPBUGS-98220: fix(cpo,hcco): prevent premature KAS convergence during KMS key rotation [release-4.22]

[csrwng]

What this PR does / why we need it:

KMS key rotation fails because IsDeploymentReady returns true while old KAS pods are still terminating. Kubernetes deployment status fields explicitly exclude terminating pods ("non-terminating"), so all status counters look healthy while pods running the old encryption config are still serving requests. This causes ShouldPromoteTargetKey to fire prematurely, rewriting the encryption config and triggering another KAS rollout — resulting in 3 KAS pods with different encryption configs running simultaneously.

This PR:

• Adds a hasTerminatingPods check to KASDeploymentConvergedWithEncryptionConfig that lists pods matching the deployment's selector and returns false if any have DeletionTimestamp set
• Adds an encryption-config-hash annotation to the KAS pod template so convergence can verify the deployment rolled out with the correct config
• Passes client.Reader to the convergence function from both the CPO and HCCO callers

The fix fails closed — if listing pods errors, convergence is not reported.

This is the release-4.22 backport of #8970.

Which issue(s) this PR fixes:

Fixes OCPBUGS-98086

Special notes for your reviewer:

• Tested with HA control plane on AKS with two full KMS key rotations
• Both rotations completed successfully with no premature state advancement
• The hasTerminatingPods helper uses the deployment's own Spec.Selector to find its pods — no hardcoded labels

Checklist:

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: d9aabfc3-4504-4213-b04a-4cb0f06b47cf

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: csrwng

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [csrwng]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[csrwng]

/jira cherry-pick OCPBUGS-98086

[openshift-ci-robot]

@csrwng: Jira Issue OCPBUGS-98086 has been cloned as Jira Issue OCPBUGS-98220. Will retitle bug to link to clone.
/retitle OCPBUGS-98220: fix(cpo,hcco): prevent premature KAS convergence during KMS key rotation [release-4.22]

Details

In response to this:

/jira cherry-pick OCPBUGS-98086

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@csrwng: This pull request references Jira Issue OCPBUGS-98220, which is invalid:

• release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
• expected dependent Jira Issue OCPBUGS-98086 to be in one of the following states: MODIFIED, ON_QA, VERIFIED, but it is POST instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

What this PR does / why we need it:

KMS key rotation fails because IsDeploymentReady returns true while old KAS pods are still terminating. Kubernetes deployment status fields explicitly exclude terminating pods ("non-terminating"), so all status counters look healthy while pods running the old encryption config are still serving requests. This causes ShouldPromoteTargetKey to fire prematurely, rewriting the encryption config and triggering another KAS rollout — resulting in 3 KAS pods with different encryption configs running simultaneously.

This PR:

• Adds a hasTerminatingPods check to KASDeploymentConvergedWithEncryptionConfig that lists pods matching the deployment's selector and returns false if any have DeletionTimestamp set
• Adds an encryption-config-hash annotation to the KAS pod template so convergence can verify the deployment rolled out with the correct config
• Passes client.Reader to the convergence function from both the CPO and HCCO callers

The fix fails closed — if listing pods errors, convergence is not reported.

This is the release-4.22 backport of #8970.

Which issue(s) this PR fixes:

Fixes OCPBUGS-98086

Special notes for your reviewer:

• Tested with HA control plane on AKS with two full KMS key rotations
• Both rotations completed successfully with no premature state advancement
• The hasTerminatingPods helper uses the deployment's own Spec.Selector to find its pods — no hardcoded labels

Checklist:

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[codecov]

Codecov Report

❌ Patch coverage is 54.76190% with 38 lines in your changes missing coverage. Please review.
✅ Project coverage is 35.86%. Comparing base (aa2d83b) to head (250dd74).
⚠️ Report is 2 commits behind head on release-4.22.

Files with missing lines	Patch %	Lines
support/secretencryption/encryptionconfig.go	54.54%	17 Missing and 3 partials ⚠️
...ontrollers/hostedcontrolplane/v2/kas/deployment.go	0.00%	8 Missing ⚠️
...lers/hostedcontrolplane/v2/kas/secretencryption.go	64.28%	4 Missing and 1 partial ⚠️
...goperator/controllers/reencryption/reencryption.go	72.22%	5 Missing ⚠️

Additional details and impacted files

@@               Coverage Diff                @@
##           release-4.22    #8972      +/-   ##
================================================
+ Coverage         35.85%   35.86%   +0.01%     
================================================
  Files               774      774              
  Lines             94736    94791      +55     
================================================
+ Hits              33969    33999      +30     
- Misses            57986    58007      +21     
- Partials           2781     2785       +4     

Files with missing lines	Coverage Δ
...lers/hostedcontrolplane/v2/kas/secretencryption.go	53.07% <64.28%> (-0.53%)	⬇️
...goperator/controllers/reencryption/reencryption.go	69.34% <72.22%> (+0.31%)	⬆️
...ontrollers/hostedcontrolplane/v2/kas/deployment.go	25.71% <0.00%> (-0.15%)	⬇️
support/secretencryption/encryptionconfig.go	76.04% <54.54%> (-18.19%)	⬇️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ahitacat]

/verified by ahitacat

I have verify this test using the following annotation:
hypershift.openshift.io/control-plane-operator-image: quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator-4-22:on-pr-250dd743772fd0f9582359151d6dcf4cd85d3f48

To verify this works I have update twice the key version (on completion of the first one):

Create a cluster
Update activeKey version
Wait for completion
Update activeKey version (to a new one)
During the second update I can verified that the kas pods are recreating without issue:

NAME                              READY   STATUS        RESTARTS   AGE
kube-apiserver-7d7d474fc-4g572    7/7     Running       0          8m14s
kube-apiserver-7d7d474fc-bl4kg    5/7     Terminating   0          6m34s
kube-apiserver-7dd69b59d7-27z8s   0/7     Pending       0          96s
kube-apiserver-7dd69b59d7-62jnf   7/7     Running       0          3m7s
Once it finished the second update the history of the activeKey in the hostedCluster RC looks like this:

history:
      - completionTime: "2026-07-13T08:36:00Z"
        from:
          fingerprint: 1f7cf75a26047f7f4bb0a59c57d47796c44e84cfb04ab126b4ca91531d3d82b3
          provider: Azure
        startedTime: "2026-07-13T08:25:29Z"
        state: Completed
        to:
          fingerprint: 722926100ce3aaf2de826a6fd08ac19fb1ee667d514f7822b8218c503e9a24ee
          provider: Azure
      - completionTime: "2026-07-13T08:13:43Z"
        from:
          fingerprint: 99ad754c2a0576b6f6a0f13af601ebc750e7d56d653f904e99042e392a3e3fc7
          provider: Azure
        startedTime: "2026-07-13T07:59:47Z"
        state: Completed
        to:
          fingerprint: 1f7cf75a26047f7f4bb0a59c57d47796c44e84cfb04ab126b4ca91531d3d82b3
          provider: Azure

With this we can see it succeeded of re-encrypt the second time with a new version. And cluster is healthy.

[openshift-ci-robot]

@ahitacat: This PR has been marked as verified by ahitacat.

Details

In response to this:

/verified by ahitacat

I have verify this test using the following annotation:
hypershift.openshift.io/control-plane-operator-image: quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator-4-22:on-pr-250dd743772fd0f9582359151d6dcf4cd85d3f48

To verify this works I have update twice the key version (on completion of the first one):

Create a cluster
Update activeKey version
Wait for completion
Update activeKey version (to a new one)
During the second update I can verified that the kas pods are recreating without issue:

NAME                              READY   STATUS        RESTARTS   AGE
kube-apiserver-7d7d474fc-4g572    7/7     Running       0          8m14s
kube-apiserver-7d7d474fc-bl4kg    5/7     Terminating   0          6m34s
kube-apiserver-7dd69b59d7-27z8s   0/7     Pending       0          96s
kube-apiserver-7dd69b59d7-62jnf   7/7     Running       0          3m7s
Once it finished the second update the history of the activeKey in the hostedCluster RC looks like this:

history:
     - completionTime: "2026-07-13T08:36:00Z"
       from:
         fingerprint: 1f7cf75a26047f7f4bb0a59c57d47796c44e84cfb04ab126b4ca91531d3d82b3
         provider: Azure
       startedTime: "2026-07-13T08:25:29Z"
       state: Completed
       to:
         fingerprint: 722926100ce3aaf2de826a6fd08ac19fb1ee667d514f7822b8218c503e9a24ee
         provider: Azure
     - completionTime: "2026-07-13T08:13:43Z"
       from:
         fingerprint: 99ad754c2a0576b6f6a0f13af601ebc750e7d56d653f904e99042e392a3e3fc7
         provider: Azure
       startedTime: "2026-07-13T07:59:47Z"
       state: Completed
       to:
         fingerprint: 1f7cf75a26047f7f4bb0a59c57d47796c44e84cfb04ab126b4ca91531d3d82b3
         provider: Azure

With this we can see it succeeded of re-encrypt the second time with a new version. And cluster is healthy.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[jparrill]

/jira refresh

[openshift-ci-robot]

@jparrill: This pull request references Jira Issue OCPBUGS-98220, which is invalid:

• release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
• expected dependent Jira Issue OCPBUGS-98086 to be in one of the following states: MODIFIED, ON_QA, VERIFIED, but it is ASSIGNED instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[jparrill]

/lgtm

[csrwng]

/hold
/pipeline required

[csrwng]

/test e2e-aws
/test e2e-aks

[openshift-ci]

@csrwng: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.