OCPBUGS-97705: fix(azure): skip KMS validation for private Key Vaults on ARO HCP

[vsolanki12]

Summary

• Skip ValidAzureKMSConfig HTTPS validation for ARO HCP clusters with KeyVaultAccess: Private
• CPO cannot reach private Key Vault endpoints — KAS pods access them through the private router (HAProxy TCP passthrough via hostAlias)
• Short-circuit to ValidAzureKMSConfig: True with message deferring validation to runtime, matching the existing self-managed Azure pattern

Bug

OCPBUGS-97705

validateAzureKMSConfig() loads managed identity credentials and then attempts direct HTTPS validation against the Key Vault URL. For private Key Vaults, this fails with 403 ForbiddenByConnection because CPO has no network path to the private endpoint.

Verification

Tested on a live ARO HCP cluster on AKS (OCP 5.0.0-ec.3) with custom CPO image containing the fix:

$ oc get hostedcontrolplane -n clusters-vsolanki-management-hc -o jsonpath='{.status.conditions[?(@.type=="ValidAzureKMSConfig")]}' | python3 -m json.tool
{
    "lastTransitionTime": "2026-07-09T06:12:33Z",
    "message": "KMS configuration accepted; Key Vault access is validated at runtime through the private router",
    "observedGeneration": 4,
    "reason": "AsExpected",
    "status": "True",
    "type": "ValidAzureKMSConfig"
}

Test plan

• Unit test TestValidateAzureKMSConfig_PrivateKeyVault added and passing
• make test passes
• make verify passes
• Verified on live ARO HCP cluster with private Key Vault KMS config

Summary by CodeRabbit

• Bug Fixes
  • Azure KMS validation now correctly succeeds for ARO clusters using a private Key Vault endpoint.
  • The KMS status condition is marked healthy when access is verified through the private router, avoiding unnecessary validation failures.

[openshift-merge-bot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[openshift-ci-robot]

@vsolanki12: This pull request references Jira Issue OCPBUGS-97705, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (5.0.0) matches configured target version for branch (5.0.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

• Skip ValidAzureKMSConfig HTTPS validation for ARO HCP clusters with KeyVaultAccess: Private
• CPO cannot reach private Key Vault endpoints — KAS pods access them through the private router (HAProxy TCP passthrough via hostAlias)
• Short-circuit to ValidAzureKMSConfig: True with message deferring validation to runtime, matching the existing self-managed Azure pattern

Bug

OCPBUGS-97705

validateAzureKMSConfig() loads managed identity credentials and then attempts direct HTTPS validation against the Key Vault URL. For private Key Vaults, this fails with 403 ForbiddenByConnection because CPO has no network path to the private endpoint.

Verification

Tested on a live ARO HCP cluster on AKS (OCP 5.0.0-ec.3) with custom CPO image containing the fix:

$ oc get hostedcontrolplane -n clusters-vsolanki-management-hc -o jsonpath='{.status.conditions[?(@.type=="ValidAzureKMSConfig")]}' | python3 -m json.tool
{
   "lastTransitionTime": "2026-07-09T06:12:33Z",
   "message": "KMS configuration accepted; Key Vault access is validated at runtime through the private router",
   "observedGeneration": 4,
   "reason": "AsExpected",
   "status": "True",
   "type": "ValidAzureKMSConfig"
}

Test plan

• Unit test TestValidateAzureKMSConfig_PrivateKeyVault added and passing
• make test passes
• make verify passes
• Verified on live ARO HCP cluster with private Key Vault KMS config

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

📝 Walkthrough

Walkthrough

The validateAzureKMSConfig function in the HostedControlPlane controller now includes an early-success path for Azure Red Hat OpenShift (ARO) deployments using private Key Vault endpoints. When this condition is detected, the ValidAzureKMSConfig status condition is set to True with a message noting runtime validation via the private router, bypassing the existing Key Vault DNS suffix resolution and encryption test logic. A corresponding unit test, TestValidateAzureKMSConfig_PrivateKeyVault, was added to verify this new behavior.

🚥 Pre-merge checks | ✅ 11

✅ Passed checks (11 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	Added test name is static (TestValidateAzureKMSConfig_PrivateKeyVault); no Ginkgo-style titles or dynamic identifiers were introduced.
Test Structure And Quality	✅ Passed	PASS: The new unit test is single-purpose, uses t.Setenv cleanup, has no cluster waits, and asserts the new ARO/private-KeyVault branch correctly.
Topology-Aware Scheduling Compatibility	✅ Passed	The PR only changes Azure KMS validation and adds a unit test; it introduces no pod affinity, node selectors, replicas, PDBs, or other topology-sensitive scheduling logic.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	No new Ginkgo e2e tests were added; the change is a Go unit test and no IPv4 or external-connectivity assumptions were found.
No-Weak-Crypto	✅ Passed	The PR only adds an Azure KMS early-return and a unit test; I found no MD5/SHA1/DES/RC4/3DES/Blowfish/ECB, custom crypto, or secret/token comparisons in the changed code.
Container-Privileges	✅ Passed	PR only changes Go controller/test logic; no pod/K8s manifests or privilege fields (privileged, hostPID, hostNetwork, hostIPC, SYS_ADMIN, allowPrivilegeEscalation, runAsUser) were added.
No-Sensitive-Data-In-Logs	✅ Passed	No new log statements were added; the change only sets a status condition and test data, with no secrets/PII/internal hostnames logged.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: skipping Azure KMS validation for private Key Vaults on ARO HCP.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: vsolanki12
Once this PR has been reviewed and has the lgtm label, please assign jparrill for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

@vsolanki12: This pull request references Jira Issue OCPBUGS-97705, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (5.0.0) matches configured target version for branch (5.0.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

Summary

• Skip ValidAzureKMSConfig HTTPS validation for ARO HCP clusters with KeyVaultAccess: Private
• CPO cannot reach private Key Vault endpoints — KAS pods access them through the private router (HAProxy TCP passthrough via hostAlias)
• Short-circuit to ValidAzureKMSConfig: True with message deferring validation to runtime, matching the existing self-managed Azure pattern

Bug

OCPBUGS-97705

validateAzureKMSConfig() loads managed identity credentials and then attempts direct HTTPS validation against the Key Vault URL. For private Key Vaults, this fails with 403 ForbiddenByConnection because CPO has no network path to the private endpoint.

Verification

Tested on a live ARO HCP cluster on AKS (OCP 5.0.0-ec.3) with custom CPO image containing the fix:

$ oc get hostedcontrolplane -n clusters-vsolanki-management-hc -o jsonpath='{.status.conditions[?(@.type=="ValidAzureKMSConfig")]}' | python3 -m json.tool
{
   "lastTransitionTime": "2026-07-09T06:12:33Z",
   "message": "KMS configuration accepted; Key Vault access is validated at runtime through the private router",
   "observedGeneration": 4,
   "reason": "AsExpected",
   "status": "True",
   "type": "ValidAzureKMSConfig"
}

Test plan

• Unit test TestValidateAzureKMSConfig_PrivateKeyVault added and passing
• make test passes
• make verify passes
• Verified on live ARO HCP cluster with private Key Vault KMS config

Summary by CodeRabbit

• Bug Fixes
• Azure KMS validation now correctly succeeds for ARO clusters using a private Key Vault endpoint.
• The KMS status condition is marked healthy when access is verified through the private router, avoiding unnecessary validation failures.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 43.55%. Comparing base (3fc211a) to head (a9d251f).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8965      +/-   ##
==========================================
+ Coverage   43.53%   43.55%   +0.02%     
==========================================
  Files         771      771              
  Lines       95753    95811      +58     
==========================================
+ Hits        41683    41732      +49     
- Misses      51172    51179       +7     
- Partials     2898     2900       +2     

Files with missing lines	Coverage Δ
...ostedcontrolplane/hostedcontrolplane_controller.go	46.48% <100.00%> (+0.75%)	⬆️

... and 2 files with indirect coverage changes

Flag	Coverage Δ
cmd-support	37.22% <ø> (+0.02%)	⬆️
cpo-hostedcontrolplane	45.62% <100.00%> (+0.11%)	⬆️
cpo-other	45.10% <ø> (ø)
hypershift-operator	53.65% <ø> (ø)
other	32.08% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[openshift-ci]

@vsolanki12: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[hypershift-jira-solve-ci]

I have all the information needed. Here is the analysis:

Test Failure Analysis Complete

Job Information

• Prow Job: Red Hat Konflux / hypershift-operator-main-on-pull-request
• Build ID: hypershift-operator-main-on-pull-request-s6w8t
• PR: OCPBUGS-97705: fix(azure): skip KMS validation for private Key Vaults on ARO HCP #8965 — OCPBUGS-97705: fix(azure): skip KMS validation for private Key Vaults on ARO HCP
• Pipeline Namespace: crt-redhat-acm-tenant
• Started: 2026-07-09T07:01:35Z
• Completed: 2026-07-09T07:14:06Z

Test Failure Analysis

Error

task coverity-availability-check has the status "TaskRunImagePullFailed":
the step "coverity-availability-check" in TaskRun "hyp523d0aba87019c29e0e353b6502b6b30-coverity-availability-check"
failed to pull the image "". The pod errored with the message:
"Back-off pulling image "quay.io/konflux-ci/konflux-test:v1.4.49@sha256:74899c7a3dde27548e1ad9c665055bfef56c227251bd9224885f4bbd7addebd9":
ErrImagePull: copying system image from manifest list: reading blob
sha256:56b2ec53eb90753f4259bc1fd55c9b03d2aeb15f32351977d3513cdfc735b78c:
fetching blob: received unexpected HTTP status: 503 Service Unavailable."

Summary

This failure is not caused by the PR's code changes. The Konflux pipeline task coverity-availability-check failed because Quay.io returned a transient 503 Service Unavailable error while pulling the image quay.io/konflux-ci/konflux-test:v1.4.49. All 16 other tasks in the pipeline (including build-images, all security scans, and SBOM generation) succeeded. The two other Konflux pipelines on this PR (control-plane-operator-main-on-pull-request and hypershift-release-mce-50-on-pull-request) also completed successfully. The image is confirmed accessible now (HTTP 200), proving the outage was transient.

Root Cause

The root cause is a transient Quay.io registry outage (HTTP 503) that occurred during the coverity-availability-check task's image pull at approximately 07:13 UTC on 2026-07-09.

Specifically, the Konflux pipeline runner attempted to pull the image quay.io/konflux-ci/konflux-test:v1.4.49 (pinned to digest sha256:74899c7a...). While resolving the manifest list succeeded, the subsequent blob fetch for layer sha256:56b2ec53... received an HTTP 503 from Quay.io's blob storage backend. After the initial failure, Kubernetes entered ErrImagePull → ImagePullBackOff retry, but the task timed out before the registry recovered.

Key facts confirming this is infrastructure flakiness, not a code issue:

1. The image quay.io/konflux-ci/konflux-test:v1.4.49 is not defined by this PR — it is a Konflux platform image used for the Coverity static analysis check.
2. The image currently returns HTTP 200 when queried, confirming the outage was transient.
3. All 16 other tasks in this same pipeline succeeded, including build-images which compiled the PR's Go code.
4. The other two Konflux pipelines on this PR (control-plane-operator-main and hypershift-release-mce-50) completed successfully.
5. The coverity-availability-check task is a security scanning step, not a build or test step — it has no relation to the Azure KMS validation changes in this PR.

Recommendations

1. Rerun the failed pipeline — This is a transient infrastructure issue. Retrigger the hypershift-operator-main-on-pull-request pipeline by pushing an empty commit or using the Konflux UI to re-run the pipeline. The image is accessible now.

2. No code changes required — The PR's changes (skipping KMS validation for private Key Vaults on ARO HCP) are not implicated. The control-plane-operator-main-on-pull-request pipeline, which builds and tests the same code, succeeded.

3. If the failure recurs, it may indicate a broader Quay.io availability issue. Check status.quay.io and the Konflux team's communication channels for known outages.

Evidence

Evidence	Detail
Failed Task	coverity-availability-check (1 of 17 tasks in pipeline)
Task Status	TaskRunImagePullFailed
Failed Image	quay.io/konflux-ci/konflux-test:v1.4.49@sha256:74899c7a3dde27548e1ad9c665055bfef56c227251bd9224885f4bbd7addebd9
Error Type	HTTP 503 Service Unavailable during blob fetch
Failed Blob	sha256:56b2ec53eb90753f4259bc1fd55c9b03d2aeb15f32351977d3513cdfc735b78c
Image Status Now	HTTP 200 (accessible — confirms transient outage)
Other 16 Tasks	All succeeded (init, clone-repository, prefetch-dependencies, build-images, build-image-index, deprecated-base-image-check, sast-snyk-check, sast-unicode-check, clair-scan, clamav-scan, ecosystem-cert-preflight-checks, push-dockerfile, rpms-signature-scan, apply-tags, sast-shell-check, show-sbom)
control-plane-operator pipeline	✅ Succeeded (07:09:30 UTC)
hypershift-release-mce-50 pipeline	✅ Succeeded (07:09:33 UTC)
PR Code Relation	None — coverity-availability-check is a Konflux platform security scan task, unrelated to the Azure KMS changes

---