OCPBUGS-98213: fix router component ordering to prevent missing HAProxy backends#8971
Conversation
|
Pipeline controller notification For optional jobs, comment This repository is configured in: LGTM mode |
openshift-ci-robot
added
jira/valid-reference
|
@hypershift-jira-solve-ci[bot]: This pull request references Jira Issue OCPBUGS-98213, which is valid. The bug has been moved to the POST state. 3 validation(s) were run on this bug
The bug has been updated to refer to the pull request using the external bug tracker. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
@hypershift-jira-solve-ci[bot]: This pull request references Jira Issue OCPBUGS-98213, which is valid. 3 validation(s) were run on this bug
DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthrough
Sequence Diagram(s)sequenceDiagram
participant Reconciler as HostedControlPlaneReconciler
participant Router as routerv2 component
participant API as Kubernetes Route API
participant HCP as HostedControlPlane
Reconciler->>Router: register component with dependencies
Router->>HCP: inspect HCP router mode and platform
Router->>API: list Routes in HCP namespace
API-->>Router: return Route objects
Router->>Router: check expected names and spec.host readiness
Router-->>Reconciler: permit or block reconciliation
🚥 Pre-merge checks | ✅ 10 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (10 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
openshift-ci
Bot
added
area/control-plane-operator
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller.go (1)
279-285: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winConsider
WithDependenciesinstead of positional ordering.Comment-guarded slice ordering is fragile: any future insertion/reordering in
registerComponents, or a move away from strictly sequential reconciliation ofr.components, silently reintroduces this bug. This framework already has an explicit ordering primitive for exactly this case —WithDependencies(...), which blocks a component's reconciliation until its declared dependencies are ready — used elsewhere (e.g.clusterpolicy,dnsoperator,konnectivity_agentall callWithDependencies(oapiv2.ComponentName)/kasv2.ComponentName). Declaringrouterv2.NewComponent().WithDependencies(ignitionserverv2.ComponentName, metricsproxyv2.ComponentName)(if the framework's builder API supports appending dependencies) would make the ordering contract explicit and self-enforcing rather than implicit in registration order.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller.go` around lines 279 - 285, The router registration in registerComponents is relying on fragile slice order to wait for ignition-server and metrics-proxy, which can be broken by future reordering. Update routerv2.NewComponent to declare an explicit dependency on the components that create Route objects using the framework’s WithDependencies API, so router reconciliation is blocked until those dependencies are ready. Use the existing component names in this controller (for example ignitionserverv2.ComponentName and metricsproxyv2.ComponentName) to make the ordering contract explicit and self-enforcing.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In
`@control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller.go`:
- Around line 279-285: The router registration in registerComponents is relying
on fragile slice order to wait for ignition-server and metrics-proxy, which can
be broken by future reordering. Update routerv2.NewComponent to declare an
explicit dependency on the components that create Route objects using the
framework’s WithDependencies API, so router reconciliation is blocked until
those dependencies are ready. Use the existing component names in this
controller (for example ignitionserverv2.ComponentName and
metricsproxyv2.ComponentName) to make the ordering contract explicit and
self-enforcing.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 8b89eabd-9f22-4e18-a641-6d02aef5df25
📒 Files selected for processing (2)
control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller.gocontrol-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller_test.go
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #8971 +/- ##
==========================================
+ Coverage 43.53% 43.81% +0.28%
==========================================
Files 771 772 +1
Lines 95798 96106 +308
==========================================
+ Hits 41707 42113 +406
+ Misses 51192 51076 -116
- Partials 2899 2917 +18
... and 24 files with indirect coverage changes
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
bryan-cox
left a comment
bryan-cox
left a comment
There was a problem hiding this comment.
Review
How Reconciliation Actually Works
reconcileInfrastructure()runs first (line 1084) — creates KAS routes, konnectivity routes, and OAuth routes viainfra/infra.goAdmitHCPManagedRoutes()runs next (line 1201) — sets host/status on HCP-managed routesfor _, c := range r.components(line 1263) — iterates sequentially, calling each component'sReconcile(), which callsupdate(), which applies manifests via server-side apply
Does component order matter?
Partially. The router's adaptConfig (v2/router/config.go:43) does:
cpContext.Client.List(cpContext, routeList, client.InNamespace(cpContext.HCP.Namespace))cpContext.Client is r.Client (line 1247) — the controller-runtime cached client backed by informer caches. When ignition-server's Reconcile() creates a Route via server-side apply, the Route hits etcd, but the informer cache update from the watch event is async. So ordering the router after route-creating components gives the informer more time to process the watch event, making it more likely the routes are visible — but it's not a hard guarantee.
What routes does the router care about?
The router's adaptConfig handles: KAS (3 routes), ignition, konnectivity, OAuth (3 routes), metrics_forwarder, and metrics_proxy.
- KAS, konnectivity, OAuth routes are created in
reconcileInfrastructure(), which runs before the component loop. These are not affected by component ordering at all. - Only ignition-server and metrics-proxy create routes via the component framework (
v2/assets/ignition-server/route.yamlandv2/assets/metrics-proxy/route.yaml). These are the ones affected by this PR.
The route-creating component list is complete
The PR correctly identifies ignition-server and metrics-proxy as the only two route-creating components in the framework. Confirmed by:
- Only two
route.yamlassets exist underv2/assets/ - No other components reference
routev1.Routeoutside of router itself
Test quality
The test is well-structured:
- Uses exported
ComponentNameconstants - Properly sets up the reconciler with required mocks
- Verifies position ordering in the actual component slice
- Only tests AWS platform, but the route-creating components aren't platform-conditional so this is fine
Issues
-
The PR description overstates the fix. It says this "eliminates the race window" — it doesn't. It reduces the window by giving the informer cache more time to sync, but the cache update is still async. A fully robust fix would use
WithDependencies()on the router component (the framework supports this — seemetricsproxyv2which already usesWithDependencies(endpointresolverv2.ComponentName)), or use an uncached read inadaptConfig. -
The 6-line inline comment is verbose. 2 lines would suffice:
// Router must come after route-creating components (ignition-server, metrics-proxy) // so their routes exist in the cache when adaptConfig generates the HAProxy config.
-
Not a blocker, but worth noting: the router already handles missing routes gracefully — if a route doesn't exist, it just doesn't get a backend entry. The next reconcile pass will pick it up. So this is a first-reconcile latency optimization, not a hard correctness bug.
Verdict
Conditional ACK — the ordering change is a reasonable improvement and the test is good. But the PR description should be corrected to not claim it "eliminates" the race, and the comment should be shortened.
|
/test address-review-comments |
|
Review agent triggered. View job |
hypershift-jira-solve-ci
Bot
force-pushed
the
fix-OCPBUGS-98213
branch
from
b092cf4 to
6f72168
Compare
|
Done. Addressed both items:
Good point on #3 — the router does handle missing routes gracefully via subsequent reconcile passes, so this is indeed a first-reconcile latency optimization rather than a hard correctness fix. AI-assisted response via Claude Code |
|
@hypershift-jira-solve-ci[bot]: This pull request references Jira Issue OCPBUGS-98213, which is valid. The bug has been moved to the POST state. 3 validation(s) were run on this bug
DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Was https://redhat.atlassian.net/browse/OCPBUGS-98213?focusedCommentId=17546910 taken into account when generating this PR? |
|
/test address-review-comments |
|
Review agent triggered. View job |
|
@hypershift-jira-solve-ci[bot]: This pull request references Jira Issue OCPBUGS-98213, which is valid. 3 validation(s) were run on this bug
DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Addressed review feedback:
The positional ordering test is preserved as a secondary safety net — AI-assisted response via Claude Code |
|
/restructure-commits |
|
🤖 Restructuring commits: workflow run |
hypershift-jira-solve-ci
Bot
force-pushed
the
fix-OCPBUGS-98213
branch
from
20f0e73 to
eb42c61
Compare
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: bryan-cox, hypershift-jira-solve-ci[bot] The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
|
/test e2e-kubevirt-aws-ovn-reduced |
|
/lgtm |
|
/test e2e-kubevirt-aws-ovn-reduced |
|
Tests from second stage were triggered manually. Pipeline can be controlled only manually, until HEAD changes. Use command to trigger second stage. |
|
/retest |
|
/hold |
openshift-ci
Bot
added
the
do-not-merge/hold
|
/hold doesn't play nice with private key vaults. hypershift/support/controlplane-component/status.go Lines 45 to 49 in 65839bb |
…AS dependency The ignition-server only needs the management cluster KAS (via its service account token), not the hosted cluster KAS. Including it in the implicit KAS dependency creates a deadlock on private clusters with KMS encryption (e.g. Azure ARO-HCP with private Key Vault): Router → ignition-server → KAS → Router (for KMS connectivity) By excluding ignition-server from the KAS dependency, its deployment and route are created independently, unblocking the router, which in turn provides the network path KAS needs to reach the KMS endpoint. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
New changes are detected. LGTM label has been removed. |
openshift-ci
Bot
added
the
area/hypershift-operator
|
/test e2e-aws |
| var aroBaseHCPRouterRouteNames = []string{ | ||
| "kube-apiserver-internal", | ||
| "konnectivity-server", | ||
| "oauth-internal", |
There was a problem hiding this comment.
oauth-internal isn't enabled on all clusters. We should be using util.HCPOAuthEnabled(hcp) to determine if we should append this route as required.
There was a problem hiding this comment.
Done. oauth-internal is now conditionally appended only when HCPOAuthEnabled(hcp) returns true, matching how the infra reconciler decides whether to create the route (commit 00e36f3).
AI-assisted response via Claude Code
| "karpenter", | ||
| "router", | ||
| "ignition-server", | ||
| ) |
There was a problem hiding this comment.
we also need ignition-server-proxy as the router controlplanecomponent relies on it
apiVersion: hypershift.openshift.io/v1beta1
kind: ControlPlaneComponent
metadata:
creationTimestamp: "2026-07-13T22:38:50Z"
generation: 1
name: router
namespace: ocm-arohcppers-2rhhk4qkam2sn29s7cvv7vo6obkq6n2v-s8w6z3e3c8j3a0h
resourceVersion: "6027863"
uid: 51365be9-cb76-4e0c-acdf-6cefeeb5e425
spec: {}
status:
conditions:
- lastTransitionTime: "2026-07-13T22:38:58Z"
message: Deployment.apps "router" not found
reason: NotFound
status: "False"
type: Available
- lastTransitionTime: "2026-07-13T22:38:58Z"
message: Service "ignition-server-proxy" not found
reason: ReconciliationError
status: "False"
type: RolloutComplete
observedGeneration: 1
resources:
- group: ""
kind: ConfigMap
name: router
- group: policy
kind: PodDisruptionBudget
name: routerThere was a problem hiding this comment.
Done. Added ignition-server-proxy to componentsExcludedFromKubeAPIServerDependency alongside ignition-server (commit d565a6e). The proxy only needs the management cluster KAS (via its service account), not the hosted KAS, so excluding it breaks the same deadlock.
AI-assisted response via Claude Code
|
/test address-review-comments |
|
Review agent triggered. View job |
| "karpenter-operator", | ||
| "karpenter", | ||
| "router", | ||
| "ignition-server", |
There was a problem hiding this comment.
| "ignition-server", | |
| "ignition-server", | |
| "ignition-server-proxy", |
There was a problem hiding this comment.
Done. Applied the suggestion — ignition-server-proxy is now excluded from the implicit KAS dependency (commit d565a6e).
AI-assisted response via Claude Code
|
/test address-review-comments |
|
Review agent triggered. View job |
The oauth-internal route is only created by the infra reconciler when OAuth is enabled (HCPOAuthEnabled). When OAuth is disabled (e.g. OIDC authentication), the route is never created, causing the ARO router predicate to wait forever — a deadlock. Move oauth-internal from the unconditional base list to a conditional append gated on HCPOAuthEnabled, matching how the infra reconciler decides whether to create the route. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
/test address-review-comments |
|
Review agent triggered. View job |
hypershift-jira-solve-ci
Bot
force-pushed
the
fix-OCPBUGS-98213
branch
from
a9f8697 to
00e36f3
Compare
…icit KAS dependency The ignition-server-proxy component does not need the hosted KAS — it only proxies to the ignition-server, which is already excluded. Including it in the implicit KAS dependency creates the same deadlock as ignition-server on private clusters with KMS encryption: the router waits for the ignition-server-proxy Service to exist, but ignition- server-proxy waits for KAS, which waits for the router. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
/test e2e-aws |
|
@hypershift-jira-solve-ci[bot]: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
I now have all the evidence needed. Here is the final report: Test Failure Analysis CompleteJob Information
Test Failure AnalysisErrorSummaryThe Root CauseThe failure is a flaky AWS infrastructure teardown timeout, unrelated to the PR. The The teardown timed out after ~24 minutes (
This is a race condition between the hosted cluster destruction and AWS's asynchronous resource cleanup. Notably, PR #8971 unrelatedness confirmed: The PR modifies Recommendations
Evidence
|
6 participants
OCPBUGS-98213: fix router component ordering to prevent missing HAProxy backends
What this PR does / why we need it:
The router's
adaptConfigfunction lists existing Route objects to generate the HAProxy ConfigMap. When the router was registered early in the component list (position #8), components that create Routes — ignition-server (#35) and metrics-proxy (#38) — had not yet run during the first reconcile pass. This caused the initial HAProxy config to miss ignition and metrics-proxy backends, routing ignition requests to the KAS default backend and failing NodePool ignition.This PR:
metricsproxyv2(the last route-creating component) so that all Routes exist when the router config is generated.ignition-serverandmetrics-proxyas explicitWithDependencieson the router component, blocking its reconciliation until both dependencies are Available and RolloutComplete. This provides a stronger guarantee than positional ordering alone, though the informer cache update from watch events is still async.TestRouterComponentComesAfterRouteCreatingComponentsto verify the router is always registered after ignition-server and metrics-proxy, preventing future regressions from component reordering.Which issue(s) this PR fixes:
Fixes https://redhat.atlassian.net/browse/OCPBUGS-98213
Special notes for your reviewer:
The fix uses two complementary mechanisms: positional ordering in
registerComponents(belt) andWithDependenciesvia the component framework (suspenders).WithDependenciesensures the router's reconciliation is skipped until ignition-server and metrics-proxy report Available + RolloutComplete, giving the informer cache time to reflect their routes.Checklist:
Always review AI generated responses prior to use.
Generated with Claude Code via openshift-developer plugin
Summary by CodeRabbit