Skip to content

OCPBUGS-98571: Skip Azure topology LB scope override for ARO HCP IngressController#8992

Open
bryan-cox wants to merge 2 commits into
openshift:mainfrom
bryan-cox:OCPBUGS-98571
Open

OCPBUGS-98571: Skip Azure topology LB scope override for ARO HCP IngressController#8992
bryan-cox wants to merge 2 commits into
openshift:mainfrom
bryan-cox:OCPBUGS-98571

Conversation

@bryan-cox

@bryan-cox bryan-cox commented Jul 13, 2026

Copy link
Copy Markdown
Member

What this PR does / why we need it:

PR #7821 (CNTRLPLANE-2172) added code in the HCCO's NewIngressParams() to force InternalLoadBalancer scope on the guest-cluster IngressController for Azure clusters with Private or PublicAndPrivate topology. This was correct for self-managed Azure but also affected ARO HCP clusters, which use shared ingress and should keep ExternalLoadBalancer scope (or defer to the annotation set by Cluster Service).

When CS sets topology=PublicAndPrivate on ARO HCP clusters (as required by the enhancement openshift/enhancements#1985), the HCCO creates the IngressController with loadBalancerScope=Internal. This causes:

  • The ingress operator creates a per-cluster internal Azure LB (10.0.0.5)
  • *.apps DNS points to this private IP instead of the shared ingress public IP
  • Application routes are unreachable externally

This PR adds an IsAroHCPByHCP guard so the topology-based scope override only applies to self-managed Azure. ARO HCP clusters now respect the IngressControllerLoadBalancerScope annotation propagated from the HostedCluster, matching the pattern already used in router.go.

The fix commit is authored by @zgalor who identified the root cause and wrote the initial fix in #8993.

Which issue(s) this PR fixes:

Fixes https://issues.redhat.com/browse/OCPBUGS-98571

Special notes for your reviewer:

Root cause analysis

There is exactly one code path that sets the IngressController LB scope — params.go:50-54. The fix is a one-line guard addition.

Empirical validation on a live ARO HCP cluster

The fix was validated on @zgalor's ARO HCP cluster (zg-fix-ingress):

  1. azureAuthenticationConfigType = ManagedIdentities on the HCP — confirmed IsAroHCPByHCP returns true
  2. MANAGED_SERVICE = ARO-HCP set on the HCCO deployment
  3. After overriding the HCCO image with the fix and deleting the IngressController, it was recreated with scope: External

Why the initial fix attempt appeared to fail

@zgalor's initial fix attempt (PR #8993) appeared not to work for two compounding reasons:

  1. Day-2 skip: ReconcileDefaultIngressController in reconcile.go:18-20 skips reconciliation when the IngressController already exists (ResourceVersion != ""). Deleting the IngressController is required.
  2. CPO ≠ HCCO image: @zgalor built and deployed a custom CPO image, but the params.go code runs inside the HCCO, which is a separate Deployment. The CPO sets the HCCO image from HOSTED_CLUSTER_CONFIG_OPERATOR_IMAGE (set by the hypershift-operator to cpo.Image), not from its own container image. Patching the CPO container doesn't change the HCCO. Once the HCCO image was explicitly overridden, the fix worked.

Why the guard is correct

  • Self-managed Azure uses AzureAuthenticationTypeWorkloadIdentitiesIsAroHCPByHCP returns false → topology override fires as before (no regression)
  • ARO HCP uses AzureAuthenticationTypeManagedIdentitiesIsAroHCPByHCP returns true → topology override is skipped → scope comes from annotation (lines 38-40) or defaults to ExternalLoadBalancer
  • This matches the guard pattern used in router.go:61: !azureutil.IsAroHCPByHCP(hcp)

Note on CodeRabbit review comment (PR #8993)

CodeRabbit suggested also gating the annotation path (IngressControllerLoadBalancerScope) on !IsAroHCPByHCP. This is incorrect — the annotation is the mechanism Cluster Service uses to control the scope. Blocking it for ARO HCP would break CS's ability to configure private ingress when needed. The unit test "When ARO HCP has annotation set to Internal it should respect it" explicitly validates this behavior.

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Summary by CodeRabbit

  • Bug Fixes
    • Corrected Azure ingress load balancer behavior for ARO hosted control planes on private and public-and-private Azure topologies, ensuring they use an external load balancer.
  • Tests
    • Expanded ingress parameter tests to cover ARO-specific load balancer scope selection for the affected Azure topologies.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci-robot openshift-ci-robot added jira/severity-critical Referenced Jira bug's severity is critical for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jul 13, 2026
@openshift-ci-robot

openshift-ci-robot commented Jul 13, 2026

Copy link
Copy Markdown

@bryan-cox: This pull request references Jira Issue OCPBUGS-98571, which is invalid:

  • expected the bug to target the "5.0.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

What this PR does / why we need it:

PR #7821 (CNTRLPLANE-2172) added code in the HCCO's NewIngressParams() to force InternalLoadBalancer scope on the guest-cluster IngressController for Azure clusters with Private or PublicAndPrivate topology. This was correct for self-managed Azure but also affected ARO HCP clusters, which use shared ingress and should keep ExternalLoadBalancer scope (or defer to the annotation set by Cluster Service).

When CS sets topology=PublicAndPrivate on ARO HCP clusters (as required by the enhancement openshift/enhancements#1985), the HCCO creates the IngressController with loadBalancerScope=Internal. This causes:

  • The ingress operator creates a per-cluster internal Azure LB (10.0.0.5)
  • *.apps DNS points to this private IP instead of the shared ingress public IP
  • Application routes are unreachable externally

This PR adds an IsAroHCPByHCP guard so the topology-based scope override only applies to self-managed Azure. ARO HCP clusters now respect the IngressControllerLoadBalancerScope annotation propagated from the HostedCluster, matching the pattern already used in router.go.

Which issue(s) this PR fixes:

Fixes https://issues.redhat.com/browse/OCPBUGS-98571

Special notes for your reviewer:

Root cause analysis

There is exactly one code path that sets the IngressController LB scope — params.go:50-54. The fix is a one-line guard addition.

The bug reporter attempted the same fix but observed no change because ReconcileDefaultIngressController in reconcile.go:18-20 skips reconciliation when the IngressController already exists (ResourceVersion != ""). The fix is correct for new cluster creation. Existing clusters with the wrong scope would need their IngressController deleted to trigger re-creation.

Why the guard is correct

  • Self-managed Azure uses AzureAuthenticationTypeWorkloadIdentitiesIsAroHCPByHCP returns false → topology override fires as before (no regression)
  • ARO HCP uses AzureAuthenticationTypeManagedIdentitiesIsAroHCPByHCP returns true → topology override is skipped → scope comes from annotation (lines 38-40) or defaults to ExternalLoadBalancer
  • This matches the guard pattern used in router.go:61: !azureutil.IsAroHCPByHCP(hcp)

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor
📝 Walkthrough

Walkthrough

Azure ingress parameter generation now excludes ARO hosted control planes from internal load balancer selection for private and public-and-private topologies. Table-driven tests verify external load balancer scope for both ARO Azure topologies using managed identities.

Suggested reviewers: enxebre, sdminonne

🚥 Pre-merge checks | ✅ 11
✅ Passed checks (11 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed All added/modified test titles are static string literals; no generated IDs, timestamps, IPs, or other dynamic values appear.
Test Structure And Quality ✅ Passed The added cases are pure table-driven unit tests, each validates one behavior, and there are no cluster ops, waits, or cleanup gaps.
Topology-Aware Scheduling Compatibility ✅ Passed Only ingress LB-scope logic and tests changed; no pod scheduling, affinity, node selectors, replicas, or PDBs were added.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed Only a unit test file changed; no new Ginkgo e2e tests, IPv4-only assumptions, or external connectivity requirements were added.
No-Weak-Crypto ✅ Passed Touched files only adjust Azure ingress LB scope and tests; no weak crypto, custom crypto, or secret comparisons are present.
Container-Privileges ✅ Passed PR only changes Go code and unit tests; no container/K8s manifests or privilege/securityContext settings were modified.
No-Sensitive-Data-In-Logs ✅ Passed No new logging statements were added; the PR only changes Azure load balancer scope logic and unit tests, so it doesn't expose sensitive data in logs.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the change: it skips the Azure topology load balancer scope override for ARO HCP IngressController.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@openshift-ci openshift-ci Bot added the area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release label Jul 13, 2026
@openshift-ci openshift-ci Bot requested review from enxebre and sdminonne July 13, 2026 17:03
@openshift-ci

openshift-ci Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bryan-cox

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 13, 2026
@codecov

codecov Bot commented Jul 13, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 43.80%. Comparing base (65839bb) to head (9f47b30).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #8992   +/-   ##
=======================================
  Coverage   43.79%   43.80%           
=======================================
  Files         772      772           
  Lines       96045    96046    +1     
=======================================
+ Hits        42067    42071    +4     
+ Misses      51063    51061    -2     
+ Partials     2915     2914    -1     
Files with missing lines Coverage Δ
...igoperator/controllers/resources/ingress/params.go 87.75% <100.00%> (+6.50%) ⬆️
Flag Coverage Δ
cmd-support 37.43% <ø> (ø)
cpo-hostedcontrolplane 45.91% <ø> (ø)
cpo-other 45.14% <100.00%> (+0.02%) ⬆️
hypershift-operator 54.07% <ø> (ø)
other 32.12% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

zgalor and others added 2 commits July 13, 2026 13:39
…logy

PR openshift#7821 (CNTRLPLANE-2172) added code to force InternalLoadBalancer
scope on the guest-cluster IngressController for Azure clusters with
Private or PublicAndPrivate topology. This was correct for self-managed
Azure but also affected ARO HCP clusters, which use shared ingress and
should keep ExternalLoadBalancer scope (or defer to the annotation set
by Cluster Service).

Add an IsAroHCPByHCP guard so the topology-based scope override only
applies to self-managed Azure. ARO HCP clusters now respect the
IngressControllerLoadBalancerScope annotation propagated from the
HostedCluster, matching the pattern already used in router.go.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Cover the ARO HCP guard added in the previous commit:
- PublicAndPrivate topology → ExternalLoadBalancer (not Internal)
- Private topology → ExternalLoadBalancer (shared ingress)
- Annotation override to Internal is still respected

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@openshift-ci-robot openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jul 13, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@bryan-cox: This pull request references Jira Issue OCPBUGS-98571, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.0.0) matches configured target version for branch (5.0.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)
Details

In response to this:

What this PR does / why we need it:

PR #7821 (CNTRLPLANE-2172) added code in the HCCO's NewIngressParams() to force InternalLoadBalancer scope on the guest-cluster IngressController for Azure clusters with Private or PublicAndPrivate topology. This was correct for self-managed Azure but also affected ARO HCP clusters, which use shared ingress and should keep ExternalLoadBalancer scope (or defer to the annotation set by Cluster Service).

When CS sets topology=PublicAndPrivate on ARO HCP clusters (as required by the enhancement openshift/enhancements#1985), the HCCO creates the IngressController with loadBalancerScope=Internal. This causes:

  • The ingress operator creates a per-cluster internal Azure LB (10.0.0.5)
  • *.apps DNS points to this private IP instead of the shared ingress public IP
  • Application routes are unreachable externally

This PR adds an IsAroHCPByHCP guard so the topology-based scope override only applies to self-managed Azure. ARO HCP clusters now respect the IngressControllerLoadBalancerScope annotation propagated from the HostedCluster, matching the pattern already used in router.go.

The fix commit is authored by @zgalor who identified the root cause and wrote the initial fix in #8993.

Which issue(s) this PR fixes:

Fixes https://issues.redhat.com/browse/OCPBUGS-98571

Special notes for your reviewer:

Root cause analysis

There is exactly one code path that sets the IngressController LB scope — params.go:50-54. The fix is a one-line guard addition.

Empirical validation on a live ARO HCP cluster

The fix was validated on @zgalor's ARO HCP cluster (zg-fix-ingress):

  1. azureAuthenticationConfigType = ManagedIdentities on the HCP — confirmed IsAroHCPByHCP returns true
  2. MANAGED_SERVICE = ARO-HCP set on the HCCO deployment
  3. After overriding the HCCO image with the fix and deleting the IngressController, it was recreated with scope: External

Why the initial fix attempt appeared to fail

@zgalor's initial fix attempt (PR #8993) appeared not to work for two compounding reasons:

  1. Day-2 skip: ReconcileDefaultIngressController in reconcile.go:18-20 skips reconciliation when the IngressController already exists (ResourceVersion != ""). Deleting the IngressController is required.
  2. CPO ≠ HCCO image: @zgalor built and deployed a custom CPO image, but the params.go code runs inside the HCCO, which is a separate Deployment. The CPO sets the HCCO image from HOSTED_CLUSTER_CONFIG_OPERATOR_IMAGE (set by the hypershift-operator to cpo.Image), not from its own container image. Patching the CPO container doesn't change the HCCO. Once the HCCO image was explicitly overridden, the fix worked.

Why the guard is correct

  • Self-managed Azure uses AzureAuthenticationTypeWorkloadIdentitiesIsAroHCPByHCP returns false → topology override fires as before (no regression)
  • ARO HCP uses AzureAuthenticationTypeManagedIdentitiesIsAroHCPByHCP returns true → topology override is skipped → scope comes from annotation (lines 38-40) or defaults to ExternalLoadBalancer
  • This matches the guard pattern used in router.go:61: !azureutil.IsAroHCPByHCP(hcp)

Note on CodeRabbit review comment (PR #8993)

CodeRabbit suggested also gating the annotation path (IngressControllerLoadBalancerScope) on !IsAroHCPByHCP. This is incorrect — the annotation is the mechanism Cluster Service uses to control the scope. Blocking it for ARO HCP would break CS's ability to configure private ingress when needed. The unit test "When ARO HCP has annotation set to Internal it should respect it" explicitly validates this behavior.

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot

Copy link
Copy Markdown

@bryan-cox: This pull request references Jira Issue OCPBUGS-98571, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.0.0) matches configured target version for branch (5.0.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
Details

In response to this:

What this PR does / why we need it:

PR #7821 (CNTRLPLANE-2172) added code in the HCCO's NewIngressParams() to force InternalLoadBalancer scope on the guest-cluster IngressController for Azure clusters with Private or PublicAndPrivate topology. This was correct for self-managed Azure but also affected ARO HCP clusters, which use shared ingress and should keep ExternalLoadBalancer scope (or defer to the annotation set by Cluster Service).

When CS sets topology=PublicAndPrivate on ARO HCP clusters (as required by the enhancement openshift/enhancements#1985), the HCCO creates the IngressController with loadBalancerScope=Internal. This causes:

  • The ingress operator creates a per-cluster internal Azure LB (10.0.0.5)
  • *.apps DNS points to this private IP instead of the shared ingress public IP
  • Application routes are unreachable externally

This PR adds an IsAroHCPByHCP guard so the topology-based scope override only applies to self-managed Azure. ARO HCP clusters now respect the IngressControllerLoadBalancerScope annotation propagated from the HostedCluster, matching the pattern already used in router.go.

The fix commit is authored by @zgalor who identified the root cause and wrote the initial fix in #8993.

Which issue(s) this PR fixes:

Fixes https://issues.redhat.com/browse/OCPBUGS-98571

Special notes for your reviewer:

Root cause analysis

There is exactly one code path that sets the IngressController LB scope — params.go:50-54. The fix is a one-line guard addition.

Empirical validation on a live ARO HCP cluster

The fix was validated on @zgalor's ARO HCP cluster (zg-fix-ingress):

  1. azureAuthenticationConfigType = ManagedIdentities on the HCP — confirmed IsAroHCPByHCP returns true
  2. MANAGED_SERVICE = ARO-HCP set on the HCCO deployment
  3. After overriding the HCCO image with the fix and deleting the IngressController, it was recreated with scope: External

Why the initial fix attempt appeared to fail

@zgalor's initial fix attempt (PR #8993) appeared not to work for two compounding reasons:

  1. Day-2 skip: ReconcileDefaultIngressController in reconcile.go:18-20 skips reconciliation when the IngressController already exists (ResourceVersion != ""). Deleting the IngressController is required.
  2. CPO ≠ HCCO image: @zgalor built and deployed a custom CPO image, but the params.go code runs inside the HCCO, which is a separate Deployment. The CPO sets the HCCO image from HOSTED_CLUSTER_CONFIG_OPERATOR_IMAGE (set by the hypershift-operator to cpo.Image), not from its own container image. Patching the CPO container doesn't change the HCCO. Once the HCCO image was explicitly overridden, the fix worked.

Why the guard is correct

  • Self-managed Azure uses AzureAuthenticationTypeWorkloadIdentitiesIsAroHCPByHCP returns false → topology override fires as before (no regression)
  • ARO HCP uses AzureAuthenticationTypeManagedIdentitiesIsAroHCPByHCP returns true → topology override is skipped → scope comes from annotation (lines 38-40) or defaults to ExternalLoadBalancer
  • This matches the guard pattern used in router.go:61: !azureutil.IsAroHCPByHCP(hcp)

Note on CodeRabbit review comment (PR #8993)

CodeRabbit suggested also gating the annotation path (IngressControllerLoadBalancerScope) on !IsAroHCPByHCP. This is incorrect — the annotation is the mechanism Cluster Service uses to control the scope. Blocking it for ARO HCP would break CS's ability to configure private ingress when needed. The unit test "When ARO HCP has annotation set to Internal it should respect it" explicitly validates this behavior.

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Summary by CodeRabbit

  • Bug Fixes
  • Corrected Azure ingress load balancer behavior for ARO hosted control planes on private and public-and-private Azure topologies, ensuring they use an external load balancer.
  • Tests
  • Expanded ingress parameter tests to cover ARO-specific load balancer scope selection for the affected Azure topologies.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@bryan-cox

Copy link
Copy Markdown
Member Author

/verified later @zgalor

@openshift-ci-robot openshift-ci-robot added verified-later verified Signifies that the PR passed pre-merge verification criteria labels Jul 13, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@bryan-cox: This PR has been marked to be verified later by @zgalor.

Details

In response to this:

/verified later @zgalor

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@bryan-cox

Copy link
Copy Markdown
Member Author

/test all

@bryan-cox

Copy link
Copy Markdown
Member Author

/pipeline required

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-v2-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws
/test e2e-v2-gke

@zgalor

zgalor commented Jul 13, 2026

Copy link
Copy Markdown

/verfied by @zgalor

  • Validated on OCP 4.22.3 with topology=PublicAndPrivate on a personal dev environment
  • Baseline cluster (no fix): -internal LB created at 10.0.0.5, IngressController scope: Internal
  • Fix cluster: after overriding the HCCO image (not just the CPO image) with this fix and deleting the IngressController, the HCCO recreated it with scope: External
  • Note: initial testing only overrode the CPO image which did not work — the HCCO is a separate deployment pulled from the release image, so the fix must be applied to the HCCO image specifically

@csrwng csrwng added the lgtm Indicates that a PR is ready to be merged. label Jul 13, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Tests from second stage were triggered manually. Pipeline can be controlled only manually, until HEAD changes. Use command to trigger second stage.

@csrwng csrwng added the acknowledge-critical-fixes-only Indicates if the issuer of the label is OK with the policy. label Jul 13, 2026
@cwbotbot

cwbotbot commented Jul 13, 2026

Copy link
Copy Markdown

Test Results

e2e-aws

Failed Tests

Total failed tests: 3

  • TestCreateCluster
  • TestCreateCluster/ValidateHostedCluster
  • TestCreateCluster/ValidateHostedCluster/EnsureNoCrashingPods

e2e-aks

@bryan-cox

Copy link
Copy Markdown
Member Author

/test e2e-aws

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

/retest-required

Remaining retests: 0 against base HEAD 65839bb and 2 for PR HEAD 9f47b30 in total

@openshift-ci

openshift-ci Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

@bryan-cox: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws 9f47b30 link true /test e2e-aws

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@hypershift-jira-solve-ci

Copy link
Copy Markdown
Contributor

I now have all the evidence I need. Let me produce the final report:

Test Failure Analysis Complete

Job Information

  • Prow Job: pull-ci-openshift-hypershift-main-e2e-aws
  • Build ID: 2076827834034163712
  • Target: e2e-aws
  • PR: #8992 — OCPBUGS-98571: Skip Azure topology LB scope override for ARO HCP IngressController
  • Test Results: 577 total, 544 passed, 3 failed, 30 skipped

Test Failure Analysis

Error

util.go:829: Container cluster-image-registry-operator in pod
cluster-image-registry-operator-5b78fbd885-cw7ps has a restartCount > 0 (1)
--- FAIL: TestCreateCluster/ValidateHostedCluster/EnsureNoCrashingPods (0.03s)

Summary

The TestCreateCluster test failed because its sub-test EnsureNoCrashingPods detected that the cluster-image-registry-operator container in the hosted cluster create-cluster-m8wgr had a restartCount of 1. The EnsureNoCrashingPods check enforces a default crash toleration of 0 restarts for containers not in its allowlist, and cluster-image-registry-operator is not in the allowlist. This single restart in one hosted cluster caused a cascading failure: EnsureNoCrashingPodsValidateHostedClusterTestCreateCluster. All 13 other hosted clusters' EnsureNoCrashingPods checks passed. This is a flaky, pre-existing issue unrelated to the PR's changes, which only modify Azure ARO HCP IngressController logic in params.go — code that is never executed in this AWS-platform e2e test.

Root Cause

The cluster-image-registry-operator container restarted once during the lifecycle of the create-cluster-m8wgr hosted cluster. This is a transient, non-deterministic event — a single restart during initial cluster bring-up can be caused by timing-dependent conditions such as:

  1. Dependency race during cluster initialization: The image-registry-operator may start before all its dependencies (e.g., API server, internal registry storage) are fully available, causing it to crash once and then recover successfully.
  2. Leader election contention: If the operator lost a leader election during startup, it could restart. However, the test code does check for "election lost" in previous container logs and skips those — so this is less likely unless the log format didn't match.
  3. Transient resource pressure: Brief resource contention on the node during the initial hosted cluster rollout could cause an OOMKill or readiness probe failure.

Why this is unrelated to PR #8992:

  • PR OCPBUGS-98571: Skip Azure topology LB scope override for ARO HCP IngressController #8992 modifies exactly 2 files: control-plane-operator/controllers/hostedcontrolplane/ingress/params.go and control-plane-operator/controllers/hostedcontrolplane/ingress/params_test.go
  • The change adds a guard !azureutil.IsAroHCPByHCP(hcp) to skip the internal LB scope override for ARO HCP clusters — this code path is only entered when hcp.Spec.Platform.Type == hyperv1.AzurePlatform
  • This test runs on AWS platform, so the Azure code path is never executed
  • The cluster-image-registry-operator is an OpenShift core component, not part of the ingress parameter logic modified by this PR

Why only one hosted cluster was affected:
14 separate hosted clusters ran EnsureNoCrashingPods — only the create-cluster-m8wgr cluster had this restart. This randomness is characteristic of a transient/flaky condition, not a code regression.

Recommendations
  1. Retry the job — This is a flaky failure unrelated to the PR. Run /test e2e-aws to re-trigger.
  2. Consider adding cluster-image-registry-operator to podCrashTolerations — Other operators with known transient restart behavior (e.g., aws-ebs-csi-driver-controller, gcp-cloud-controller-manager) already have a toleration of 1. Adding "cluster-image-registry-operator": 1 would prevent this flake from blocking unrelated PRs.
  3. File a tracking issue — If this flake is recurring, file an issue against openshift/cluster-image-registry-operator to investigate why it occasionally restarts once during hosted cluster bring-up.
Evidence
Evidence Detail
Failing test TestCreateCluster/ValidateHostedCluster/EnsureNoCrashingPods
Failing container cluster-image-registry-operator in pod cluster-image-registry-operator-5b78fbd885-cw7ps
Restart count 1 (toleration: 0 — not in allowlist)
Hosted cluster e2e-clusters-qkwwf/create-cluster-m8wgr
Other clusters affected 0 of 13 — all other EnsureNoCrashingPods checks passed
Test results 544 passed / 3 failed / 30 skipped out of 577 total
Failure cascade EnsureNoCrashingPods (root) → ValidateHostedClusterTestCreateCluster
PR #8992 changes Azure-only: params.go + params_test.go (ARO HCP IngressController LB scope)
Test platform AWS (PR changes only affect AzurePlatform code path)
PR relevance None — Azure ingress params code is not executed on AWS platform

@zgalor

zgalor commented Jul 14, 2026

Copy link
Copy Markdown

/test e2e-aws

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

acknowledge-critical-fixes-only Indicates if the issuer of the label is OK with the policy. approved Indicates a PR has been approved by an approver from all required OWNERS files. area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release jira/severity-critical Referenced Jira bug's severity is critical for the branch this PR is targeting. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. verified Signifies that the PR passed pre-merge verification criteria verified-later

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants