do not merge: OCPBUGS-98213 Verification

[bennerv]

What

Help verify router pod doesn't get deployed until all route are able to be rendered in the router configmap.

Why

Testing

• Build and push OCPBUGS-98213: fix router component ordering to prevent missing HAProxy backends openshift/hypershift#8971 out of band to ACR
• FDescribe on private key vault test so only that e2e runs.
• Look at kusto for the run in question
  • Check events to validate there's only ever a single replicaset of the private-router pod in the hostedcluster namespace
  • Check mgmt-agent logs and kube-apiserver logs to validate the configmap is created, then updated later with the ignition route, then the router pod is created

[bennerv]

/hold will not merge.

[Copilot]

Pull request overview

This PR is a verification-oriented change to help confirm HyperShift router behavior (router pod should not be deployed until routes can be rendered into the router ConfigMap). It does this by (1) forcing a single focused E2E scenario and (2) adding mgmt-agent watchers to emit Kubernetes event timelines (router ConfigMap lifecycle).

Changes:

• Focuses the “Private KeyVault” cluster-create E2E and pins it to an explicitly resolved OCP 4.23 candidate install version, plus an out-of-band CPO image override tag.
• Adds a ConfigMapWatcher in mgmt-agent to log router ConfigMap create/update/delete events (via a field-selector-scoped informer).
• Wires the new watcher into mgmt-agent startup under leader election and adds basic unit tests.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File	Description
test/e2e/cluster_create_private_kv.go	Focused E2E test + explicit version resolution + experimental tag to override CPO image
mgmt-agent/pkg/controller/cmwatcher.go	New informer-backed ConfigMap watcher that logs router ConfigMap events
mgmt-agent/pkg/controller/cmwatcher_test.go	Unit tests for the new watcher/logging helper
mgmt-agent/cmd/options.go	Wires ConfigMap watcher/informers into mgmt-agent lifecycle under leader election

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.

[bennerv]

KQL queries to show that there is a single revision of router pod replicasets during installation. Can use these for cluster installs if we know the hostedcluster namespace.

let hcNamespace="";
cluster('https://hcp-dev-us-2.eastus2.kusto.windows.net').database('ServiceLogs').kubeAudit
| where timestamp between (datetime(2026-07-13T15:43:01Z) .. datetime(2026-07-14T16:06:00Z))
| where resourceId has 'ci01-j8862720-mgmt-1'
| where stage == 'ResponseComplete'
| where tostring(objectRef.namespace) == 'ocm-arohcpci01-2rhbkb9hj5tgfjeb1avcm9ncp7upjvb1-e1j5q9p7g7s1v2z'
| where objectRef.resource in("deployments", "replicasets")
| project timestamp=requestReceivedTimestamp,
    verb,
    resource = tostring(objectRef.resource),
    name = tostring(objectRef.name),
    userName = tostring(user.username),
    responseCode = toint(responseStatus.code)
| order by timestamp asc

cluster('https://hcp-dev-us-2.eastus2.kusto.windows.net').database('ServiceLogs').kubernetesEvents
| where timestamp between (datetime(2026-07-13T15:43:01Z) .. datetime(2026-07-13T16:06:00Z))
| where cluster in ('ci01-j8862720-svc', 'ci01-j8862720-mgmt-1')
| where eventNamespace == hcNamespace
| where objectKind == 'Deployment' and reason == 'ScalingReplicaSet'
| extend firstSeen = coalesce(firstSeen, todatetime(log.event_time))
| project firstSeen, objectName, message
| order by firstSeen asc

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bennerv

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• test/OWNERS [bennerv]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 3 comments.

[openshift-ci]

@bennerv: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/verify	5b31334	link	true	/test verify
ci/prow/e2e-parallel	5b31334	link	true	/test e2e-parallel

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.