OCPBUGS-98461: requeue CRR on transiently unavailable resources

[jparrill]

Summary

• Change requeueSoon from false to true in 6 return paths across ensureOldSignerCertificateRevoked and ensureNewSignerCertificatePropagated where a resource is transiently nil

Fixes

• https://redhat.atlassian.net/browse/OCPBUGS-98461

Root Cause

CertificateRevocationRequest operations stall at PreviousCertificatesRevoked=False for 10+ minutes (3/8 runs, Jul 8-10, most frequent CI failure). #8563 fixed one instance of this class of bug (stale trust bundle cache returning requeueSoon=false), but three other return paths in each function have the same issue:

Line	Condition	Before	After
:690, :1011	secretForSignerClass not found	requeueSoon=false	requeueSoon=true
:698, :1018	signer secret nil	requeueSoon=false	requeueSoon=true
:717, :1029	trust bundle ConfigMap nil	requeueSoon=false	requeueSoon=true

When any of these resources is transiently unavailable (being re-created, not yet propagated), the controller returned needsWork=true, requeueSoon=false — waiting for a watch event that might never come. With requeueSoon=true, it retries on the next synthetic requeue.

Frequency

3/8 runs in Jul 8-10 window. Most frequent issue in e2e-aws-ovn. One CRR completed in 2min while a parallel one in the same run stalled for 10min+ — consistent with a transient resource availability race.

Test plan

• Unit test: "When trust bundle ConfigMap is transiently unavailable during revocation it should requeue"
• go test ./control-plane-pki-operator/certificaterevocationcontroller/ — all pass
• make lint — 0 issues
• make verify — clean (except expected uncommitted files)

/cc @csrwng

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes

  • Improved certificate revocation processing when required certificate or trust-bundle data is temporarily unavailable.
  • Reconciliation now automatically retries instead of stopping prematurely.

• Tests

  • Added coverage for retry behavior when trust-bundle information is temporarily unavailable during certificate revocation.

[openshift-merge-bot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

[openshift-ci]

@jparrill: GitHub didn't allow me to request PR reviews from the following users: cewong.

Note that only openshift members and repo collaborators can review this PR, and authors cannot review their own PRs.

Details

In response to this:

Summary

• Change requeueSoon from false to true in 6 return paths across ensureOldSignerCertificateRevoked and ensureNewSignerCertificatePropagated where a resource is transiently nil

Fixes

• https://redhat.atlassian.net/browse/OCPBUGS-98461

Root Cause

CertificateRevocationRequest operations stall at PreviousCertificatesRevoked=False for 10+ minutes (3/8 runs, Jul 8-10, most frequent CI failure). #8563 fixed one instance of this class of bug (stale trust bundle cache returning requeueSoon=false), but three other return paths in each function have the same issue:

Line	Condition	Before	After
:690, :1011	secretForSignerClass not found	requeueSoon=false	requeueSoon=true
:698, :1018	signer secret nil	requeueSoon=false	requeueSoon=true
:717, :1029	trust bundle ConfigMap nil	requeueSoon=false	requeueSoon=true

When any of these resources is transiently unavailable (being re-created, not yet propagated), the controller returned needsWork=true, requeueSoon=false — waiting for a watch event that might never come. With requeueSoon=true, it retries on the next synthetic requeue.

Frequency

3/8 runs in Jul 8-10 window. Most frequent issue in e2e-aws-ovn. One CRR completed in 2min while a parallel one in the same run stalled for 10min+ — consistent with a transient resource availability race.

Test plan

• Unit test: "When trust bundle ConfigMap is transiently unavailable during revocation it should requeue"
• go test ./control-plane-pki-operator/certificaterevocationcontroller/ — all pass
• make lint — 0 issues
• make verify — clean (except expected uncommitted files)

/cc @csrwng @CEWong

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[coderabbitai]

📝 Walkthrough

Walkthrough

The certificate revocation controller now returns requeue=true when signer certificate data or aggregated trust-bundle ConfigMaps are unavailable during propagation and revocation checks. Existing completion and error behavior remains unchanged. A new test covers revocation with a missing trust-bundle ConfigMap and verifies that reconciliation requeues without an error.

Suggested reviewers: sdminonne

🚥 Pre-merge checks | ✅ 10 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Test Structure And Quality	⚠️ Warning	The new subtest is single-purpose and well-scoped, but it uses Expect(err).ToNot(HaveOccurred()) with no failure message.	Add a diagnostic message to the error assertion (e.g. Expect(err).ToNot(HaveOccurred(), "processing should requeue when trust bundle is missing")).

✅ Passed checks (10 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	The added test title is a static string; it contains no timestamps, IDs, or other run-to-run values.
Topology-Aware Scheduling Compatibility	✅ Passed	Only requeue logic and a unit test changed; no node selectors, affinities, spread constraints, replicas, or PDB/scheduling code were introduced.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	No Ginkgo e2e tests added. PR adds standard Go unit tests (func Test, t.Run) to certificaterevocationcontroller_test.go, which are not subject to this check.
No-Weak-Crypto	✅ Passed	Touched code only changes requeue paths and tests; no MD5/SHA1/DES/RC4/3DES/Blowfish/ECB or secret-comparison logic was added.
Container-Privileges	✅ Passed	PR modifies only Go controller code without introducing container privilege settings, hostPID, hostNetwork, hostIPC, SYS_ADMIN capabilities, root execution, or allowPrivilegeEscalation configurations.
No-Sensitive-Data-In-Logs	✅ Passed	The PR only changes requeue flags and adds a nil-CM test; no new logging or sensitive-data exposure was introduced.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: requeueing CRRs when prerequisite resources are transiently unavailable.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jparrill

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [jparrill]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

@jparrill: This pull request references Jira Issue OCPBUGS-98461, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (5.0.0) matches configured target version for branch (5.0.0)
• bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

• Change requeueSoon from false to true in 6 return paths across ensureOldSignerCertificateRevoked and ensureNewSignerCertificatePropagated where a resource is transiently nil

Fixes

• https://redhat.atlassian.net/browse/OCPBUGS-98461

Root Cause

CertificateRevocationRequest operations stall at PreviousCertificatesRevoked=False for 10+ minutes (3/8 runs, Jul 8-10, most frequent CI failure). #8563 fixed one instance of this class of bug (stale trust bundle cache returning requeueSoon=false), but three other return paths in each function have the same issue:

Line	Condition	Before	After
:690, :1011	secretForSignerClass not found	requeueSoon=false	requeueSoon=true
:698, :1018	signer secret nil	requeueSoon=false	requeueSoon=true
:717, :1029	trust bundle ConfigMap nil	requeueSoon=false	requeueSoon=true

When any of these resources is transiently unavailable (being re-created, not yet propagated), the controller returned needsWork=true, requeueSoon=false — waiting for a watch event that might never come. With requeueSoon=true, it retries on the next synthetic requeue.

Frequency

3/8 runs in Jul 8-10 window. Most frequent issue in e2e-aws-ovn. One CRR completed in 2min while a parallel one in the same run stalled for 10min+ — consistent with a transient resource availability race.

Test plan

• Unit test: "When trust bundle ConfigMap is transiently unavailable during revocation it should requeue"
• go test ./control-plane-pki-operator/certificaterevocationcontroller/ — all pass
• make lint — 0 issues
• make verify — clean (except expected uncommitted files)

/cc @csrwng @CEWong

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes

• Improved certificate revocation processing when required certificate or trust-bundle data is temporarily unavailable.

• Reconciliation now automatically retries instead of stopping prematurely.

• Tests

• Added coverage for retry behavior when trust-bundle information is temporarily unavailable during certificate revocation.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[jparrill]

/test aws-e2e

[codecov]

Codecov Report

❌ Patch coverage is 16.66667% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 43.81%. Comparing base (d241190) to head (9fa0cbe).
⚠️ Report is 11 commits behind head on main.

Files with missing lines	Patch %	Lines
...ationcontroller/certificaterevocationcontroller.go	16.66%	5 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #8997   +/-   ##
=======================================
  Coverage   43.80%   43.81%           
=======================================
  Files         772      772           
  Lines       96100    96100           
=======================================
+ Hits        42097    42103    +6     
+ Misses      51084    51080    -4     
+ Partials     2919     2917    -2     

Files with missing lines	Coverage Δ
...ationcontroller/certificaterevocationcontroller.go	56.67% <16.66%> (+0.67%)	⬆️

Flag	Coverage Δ
cmd-support	37.45% <ø> (ø)
cpo-hostedcontrolplane	45.90% <ø> (ø)
cpo-other	45.13% <ø> (ø)
hypershift-operator	54.07% <ø> (ø)
other	32.18% <16.66%> (+0.06%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[coderabbitai]

🧹 Nitpick comments (1)

control-plane-pki-operator/certificaterevocationcontroller/certificaterevocationcontroller_test.go (1)

2029-2042: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Cover the remaining changed requeue branches.

This test covers only the missing trust bundle during old-signer revocation. Verify that tests also cover missing signer data and missing trust bundles during new-signer propagation, plus a missing current signer during revocation; add focused subtests if they are not already covered.

As per coding guidelines, unit test any code changes and additions.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@control-plane-pki-operator/certificaterevocationcontroller/certificaterevocationcontroller_test.go`
around lines 2029 - 2042, Extend the certificate revocation controller tests
around processCertificateRevocationRequest to cover each remaining requeue
branch: missing signer data, missing trust bundle during new-signer propagation,
and missing current signer during revocation. Add focused subtests using the
existing revoked CRR, secret, controller, and expectation helpers, asserting no
error and requeue=true for each transiently unavailable dependency.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In
`@control-plane-pki-operator/certificaterevocationcontroller/certificaterevocationcontroller_test.go`:
- Around line 2029-2042: Extend the certificate revocation controller tests
around processCertificateRevocationRequest to cover each remaining requeue
branch: missing signer data, missing trust bundle during new-signer propagation,
and missing current signer during revocation. Add focused subtests using the
existing revoked CRR, secret, controller, and expectation helpers, asserting no
error and requeue=true for each transiently unavailable dependency.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: d2b5c30b-21e6-49dd-9aaf-78481e22e58b

📥 Commits

Reviewing files that changed from the base of the PR and between d241190 and 9fa0cbe.

📒 Files selected for processing (2)

• control-plane-pki-operator/certificaterevocationcontroller/certificaterevocationcontroller.go
• control-plane-pki-operator/certificaterevocationcontroller/certificaterevocationcontroller_test.go

[openshift-ci]

@jparrill: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[openshift-ci-robot]

@jparrill: This pull request references Jira Issue OCPBUGS-98461, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (5.0.0) matches configured target version for branch (5.0.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

Summary

• Change requeueSoon from false to true in 6 return paths across ensureOldSignerCertificateRevoked and ensureNewSignerCertificatePropagated where a resource is transiently nil

Fixes

• https://redhat.atlassian.net/browse/OCPBUGS-98461

Root Cause

CertificateRevocationRequest operations stall at PreviousCertificatesRevoked=False for 10+ minutes (3/8 runs, Jul 8-10, most frequent CI failure). #8563 fixed one instance of this class of bug (stale trust bundle cache returning requeueSoon=false), but three other return paths in each function have the same issue:

Line	Condition	Before	After
:690, :1011	secretForSignerClass not found	requeueSoon=false	requeueSoon=true
:698, :1018	signer secret nil	requeueSoon=false	requeueSoon=true
:717, :1029	trust bundle ConfigMap nil	requeueSoon=false	requeueSoon=true

When any of these resources is transiently unavailable (being re-created, not yet propagated), the controller returned needsWork=true, requeueSoon=false — waiting for a watch event that might never come. With requeueSoon=true, it retries on the next synthetic requeue.

Frequency

3/8 runs in Jul 8-10 window. Most frequent issue in e2e-aws-ovn. One CRR completed in 2min while a parallel one in the same run stalled for 10min+ — consistent with a transient resource availability race.

Test plan

• Unit test: "When trust bundle ConfigMap is transiently unavailable during revocation it should requeue"
• go test ./control-plane-pki-operator/certificaterevocationcontroller/ — all pass
• make lint — 0 issues
• make verify — clean (except expected uncommitted files)

/cc @csrwng

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes

• Improved certificate revocation processing when required certificate or trust-bundle data is temporarily unavailable.

• Reconciliation now automatically retries instead of stopping prematurely.

• Tests

• Added coverage for retry behavior when trust-bundle information is temporarily unavailable during certificate revocation.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[ironcladlou]

/lgtm

[openshift-merge-bot]

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-v2-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws
/test e2e-v2-gke

[hypershift-jira-solve-ci]

Now I have all the evidence. Let me compile the final report:

Test Failure Analysis Complete

Job Information

• Prow Job: pull-ci-openshift-hypershift-main-e2e-aws-upgrade-hypershift-operator
• Build ID: 2077042723524710400
• Target: e2e-aws-upgrade-hypershift-operator
• Test: TestUpgradeHyperShiftOperator/ValidateHostedCluster/EnsureNoCrashingPods
• PR: OCPBUGS-98461: requeue CRR on transiently unavailable resources #8997 (OCPBUGS-98461: requeue CRR on transiently unavailable resources)

Test Failure Analysis

Error

Container cluster-network-operator in pod cluster-network-operator-cdf5cbc9f-f94kp has a restartCount > 0 (1)

Summary

The EnsureNoCrashingPods subtest failed because the cluster-network-operator (CNO) container in the hosted control plane namespace experienced a single transient restart (restartCount=1). The CNO crashed at startup during initial cluster bring-up due to a TLS handshake timeout when attempting to connect to the Kubernetes API server (Get "https://172.29.0.1:443/api?timeout=32s": net/http: TLS handshake timeout). The container successfully recovered one second later and was running healthy when the test checked it. This is a flaky infrastructure issue unrelated to the PR changes, which only modify the PKI certificate revocation controller.

Root Cause

The cluster-network-operator container started at 15:28:21Z during hosted cluster bring-up and immediately tried to create a client connection to the Kubernetes API server at 172.29.0.1:443. The TLS handshake timed out after ~10 seconds, causing a fatal error at 15:28:31Z (main.go:103: failed to apply cluster TLS profile: failed to create CNO client: failed create new cluster client for cluster management: Get "https://172.29.0.1:443/api?timeout=32s": net/http: TLS handshake timeout). The container exited with code 255 and was automatically restarted by kubelet at 15:28:33Z, after which it ran successfully.

This is a transient infrastructure timing issue during hosted cluster initialization. The API server (or its TLS termination) was not yet fully ready when the CNO first attempted to connect. This is a known pattern in HyperShift hosted clusters where control plane components start concurrently and race against API server readiness.

The EnsureNoCrashingPods test has zero tolerance for restarts by default for the cluster-network-operator — it is not listed in the podCrashTolerations map (defined in test/e2e/util/util.go, lines 119-150) which grants elevated thresholds to components like ingress-operator (20), dns-operator (5), and aws-ebs-csi-driver-controller (1). The two available exemptions — certificate rotation (for kube-controller-manager only) and leader election failure (requires "election lost" in previous container logs) — also do not apply, because the CNO crash was caused by a TLS handshake timeout, not leader election loss.

This failure is unrelated to PR #8997, which only modifies control-plane-pki-operator/certificaterevocationcontroller/certificaterevocationcontroller.go and its test file. Those changes fix stalled CertificateRevocationRequest operations by setting requeueSoon=true on transiently unavailable resources — completely orthogonal to the CNO startup race.

Recommendations

1. Retry/retest: This is a transient infrastructure flake. Re-running the job should pass since the CNO TLS timeout is a race condition during cluster bring-up, not a deterministic failure.

2. Consider adding cluster-network-operator to podCrashTolerations: The CNO is subject to the same startup-race conditions as other control plane components that already have elevated crash tolerances. Adding "cluster-network-operator": 1 to the podCrashTolerations map in test/e2e/util/util.go would prevent this class of transient TLS-timeout restarts from causing false test failures, similar to how aws-ebs-csi-driver-controller, network-node-identity, and gcp-cloud-controller-manager are already granted a threshold of 1.

3. No changes needed in PR OCPBUGS-98461: requeue CRR on transiently unavailable resources #8997: The PR's changes to the certificate revocation controller are completely unrelated to this failure.

Evidence

Evidence	Detail
Failed test	TestUpgradeHyperShiftOperator/ValidateHostedCluster/EnsureNoCrashingPods
Failing pod	cluster-network-operator-cdf5cbc9f-f94kp in namespace e2e-clusters-j8t64-ho-upgrade-mdxhz
Container	cluster-network-operator — restartCount=1
Crash time	15:28:31Z (10s after start at 15:28:21Z)
Recovery time	15:28:33Z (1s after crash)
Exit code	255
Fatal error	F0714 15:28:31.797406 main.go:103] failed to apply cluster TLS profile: failed to create CNO client: failed create new cluster client for cluster management: Get "https://172.29.0.1:443/api?timeout=32s": net/http: TLS handshake timeout
Default crash toleration	0 (CNO not in podCrashTolerations map)
Leader election exemption	Not applicable — previous logs contain TLS timeout, not "election lost"
Certificate rotation exemption	Not applicable — only for kube-controller-manager-* pods
PR #8997 files changed	control-plane-pki-operator/certificaterevocationcontroller/certificaterevocationcontroller.go, *_test.go (unrelated to CNO)
Other exempted pod	capi-provider-5b9d98d64f-rp7nh manager container — exempted via leader election failure detection
Prow artifacts	artifacts/e2e-aws-upgrade-hypershift-operator/hypershift-aws-run-e2e-nested/

---