Skip to content

ROSAENG-60886: Migrate from EKS Auto Mode to OSS Karpenter - #698

Open
theautoroboto wants to merge 1 commit into
openshift-online:mainfrom
theautoroboto:remove-auto-mode-v2
Open

ROSAENG-60886: Migrate from EKS Auto Mode to OSS Karpenter#698
theautoroboto wants to merge 1 commit into
openshift-online:mainfrom
theautoroboto:remove-auto-mode-v2

Conversation

@theautoroboto

@theautoroboto theautoroboto commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Replaces EKS Auto Mode with OSS Karpenter for MC and RC workload node provisioning
  • Karpenter installed via Helm during ECS bootstrap, adopted by ArgoCD via the eks-nodepool chart
  • Node pools configured with FIPS-compliant AL2023 AMIs and CriticalAddonsOnly tolerations for bootstrap nodes
  • Moves kms:CreateGrant from node role to Karpenter controller role
  • Adds Karpenter controller IAM role and SQS interruption queue via Terraform
  • Waits for hypershift Application to be Healthy before ECS bootstrap exits
  • Fixes hypershift-install Job: kubectl not in hypershift-operator image; replaced with curl against the in-cluster API server for CRD polling and external-dns patching
  • Fixes monitoring bootstrap failure: tls.enabled must be disabled alongside admissionWebhooks.enabled to avoid missing TLS secret mount crash
  • Adds AWS Load Balancer Controller with TargetGroupBinding API version corrections
  • Updates monitoring e2e to use explicit ginkgo invocation with 15m timeout (Makefile target was timing out at 5m)

Test plan

  • /test on-demand-e2e — passing
  • Confirm mc01-bootstrap shows === Prometheus Operator CRDs present — proceeding with hypershift install ===
  • Confirm monitoring Application reaches Healthy
  • Confirm hypershift Application reaches Healthy
  • Confirm hosted cluster provisioning succeeds end-to-end

🤖 Generated with Claude Code

@openshift-ci openshift-ci Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 23, 2026
@openshift-ci

openshift-ci Bot commented Jul 23, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign typeid for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai

coderabbitai Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

The PR migrates EKS provisioning to OSS Karpenter, adds Karpenter-aware bootstrap and validation flows, introduces AWS Load Balancer Controller infrastructure, updates Kubernetes integrations and Argo CD drift handling, and expands design and module documentation.

Changes

Karpenter platform migration

Layer / File(s) Summary
EKS infrastructure and IAM
terraform/modules/eks-cluster/*, terraform/config/*
Adds conditional Karpenter IAM, IRSA, interruption queues, EventBridge rules, bootstrap nodes, addons, outputs, and ECS wiring.
Bootstrap orchestration and node manifests
terraform/modules/ecs-bootstrap/*, argocd/config/*/eks-nodepool/*, argocd/config/management-cluster/hypershift/*
Installs Karpenter, applies EC2NodeClass and NodePool resources, prewarms capacity, and updates HyperShift and Argo CD bootstrap behavior.
Load balancer controller and Argo CD integration
terraform/modules/aws-load-balancer-controller/*, argocd/config/regional-cluster/aws-load-balancer-controller/*, config/templates/*, deploy/*/applicationset.yaml, argocd/config/**/*targetgroupbinding.yaml
Adds controller IAM and Pod Identity resources, updates TargetGroupBinding APIs, and configures Argo CD ignore rules.
Validation and operational tooling
scripts/validate-*.sh, scripts/buildspec/*, ci/*, terraform/config/pipeline-*
Adds AWS and Kubernetes validation scripts and increases bootstrap timing allowances.
Architecture and module documentation
docs/design/*, terraform/modules/*/README.md, docs/README.md
Documents Karpenter architecture, bootstrap sequencing, IAM choices, and module usage.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related PRs

Suggested reviewers: makdaam, typeid

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

@theautoroboto

Copy link
Copy Markdown
Contributor Author

/test on-demand-e2e

@theautoroboto
theautoroboto marked this pull request as draft July 23, 2026 16:41
@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 23, 2026
@theautoroboto

Copy link
Copy Markdown
Contributor Author

/test on-demand-e2e

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 12

🧹 Nitpick comments (1)
terraform/modules/eks-cluster/iam.tf (1)

262-289: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick win

Remove the unused instance-profile write permissions. EC2NodeClass.spec.instanceProfile is used in both nodeclass templates, and this policy already assumes the profile is pre-created; keep iam:GetInstanceProfile, iam:ListInstanceProfiles, and iam:PassRole, but drop iam:CreateInstanceProfile, iam:TagInstanceProfile, iam:AddRoleToInstanceProfile, iam:DeleteInstanceProfile, and iam:RemoveRoleFromInstanceProfile. terraform/modules/eks-cluster/iam.tf:262-289

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@terraform/modules/eks-cluster/iam.tf` around lines 262 - 289, Remove the
IAMInstanceProfileCreate and IAMInstanceProfileModify statements from the
policy, eliminating CreateInstanceProfile, TagInstanceProfile,
AddRoleToInstanceProfile, DeleteInstanceProfile, and
RemoveRoleFromInstanceProfile permissions. Preserve the existing
iam:GetInstanceProfile, iam:ListInstanceProfiles, and iam:PassRole permissions.

Source: Path instructions

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@argocd/config/management-cluster/eks-nodepool/templates/00-nodeclass.yaml`:
- Around line 7-8: Update the amiSelectorTerms configuration in the NodeClass to
use the bottlerocket-fips@latest alias instead of bottlerocket@latest,
preserving the existing AMI selector structure.

In `@argocd/config/regional-cluster/eks-nodepool/templates/00-nodeclass.yaml`:
- Around line 1-18: Update the EC2NodeClass resource named fips so
amiSelectorTerms selects the approved FIPS Bottlerocket AMI instead of the
standard bottlerocket@latest alias. Preserve the existing FIPS pool contract and
leave the cluster, subnet, security group, and metadata settings unchanged.

In `@argocd/config/shared/argocd/values.yaml`:
- Around line 172-185: Update the redis-ha configuration block in values.yaml to
explicitly set redis-ha.enabled to true, and verify its HA replica settings
remain aligned with the intended Argo CD HA configuration while preserving the
existing tolerations.

In `@argocd/config/shared/storageclass/templates/gp3.yaml`:
- Line 7: Replace the existing gp3 StorageClass rather than mutating its
immutable provisioner: add a new StorageClass with a distinct name using
ebs.csi.aws.com, update management and regional consumers/defaulting to
reference it, and retain the old class until the new driver is available before
retiring it.

In `@docs/design/fips-eks-compute.md`:
- Around line 77-79: Revise the compliance statement in the Bottlerocket
workload description to scope it to node-level FIPS readiness and FIPS-validated
cryptographic modules. Remove the implication that this alone satisfies FedRAMP
High/Moderate requirements for the cluster or customer-bearing workloads, unless
the statement explicitly references the broader required control set.

In `@docs/design/karpenter-node-provisioning.md`:
- Around line 27-32: Update the rationale in
docs/design/karpenter-node-provisioning.md:27-32 to remove the claim that Pod
Identity requires a separate admission webhook, retaining only the bootstrap and
compatibility reasons for choosing IRSA. Also update
docs/design/zoa-trusted-actions.md:816 to state that IRSA remains supported and
compare Pod Identity as an option that primarily avoids per-cluster OIDC
provider management; do not describe IRSA as deprecated.

In `@docs/design/rate-limiting-architecture.md`:
- Line 242: Update the Provisioned Concurrency note so each struck-through value
uses its own matched Markdown delimiters: `~~200 req/s~~` for traffic and
`~~$30-45/month~~` for cost. Keep the surrounding explanation unchanged.

In `@scripts/validate-rc-aws.sh`:
- Around line 135-151: Update the node-group validation flow around the
ng_desired and ng_ready queries to run these detail queries only when the node
group is confirmed ACTIVE, matching the guarded structure in validate-mc-aws.sh.
Preserve safe fallback handling for AWS or jq failures so missing or unavailable
node groups do not abort the script or leave ng_ready empty before the numeric
comparison.

In `@terraform/modules/aws-load-balancer-controller/README.md`:
- Around line 14-16: Update the upstream recommended policy URL in the README
IAM role description to reference version v2.17.1, matching the versions used by
iam.tf and Chart.yaml.

In `@terraform/modules/ecs-bootstrap/README.md`:
- Line 77: Update the karpenter_version entry in the Inputs table to document
the module’s actual default of "1.13.0", matching the default defined by the
karpenter_version variable.

In `@terraform/modules/eks-cluster/README.md`:
- Line 102: Update the ami_kms_key_arn documentation row in the module README so
kms:CreateGrant is described as granted only to the Karpenter controller role,
while preserving the existing kms:Decrypt role description.

In `@terraform/modules/eks-cluster/variables.tf`:
- Around line 84-88: Update the ami_kms_key_arn variable description to
accurately state that setting it adds the implemented KMS permissions to the
Karpenter controller role only, specifically kms:CreateGrant and
kms:DescribeKey, and remove claims about kms:Decrypt or node-role permissions.

---

Nitpick comments:
In `@terraform/modules/eks-cluster/iam.tf`:
- Around line 262-289: Remove the IAMInstanceProfileCreate and
IAMInstanceProfileModify statements from the policy, eliminating
CreateInstanceProfile, TagInstanceProfile, AddRoleToInstanceProfile,
DeleteInstanceProfile, and RemoveRoleFromInstanceProfile permissions. Preserve
the existing iam:GetInstanceProfile, iam:ListInstanceProfiles, and iam:PassRole
permissions.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 0775ea04-2597-48b3-ad3f-5c3825efa1a2

📥 Commits

Reviewing files that changed from the base of the PR and between f16e9fa and 98def0f.

📒 Files selected for processing (60)
  • Makefile
  • argocd/config/management-cluster/eks-nodepool/templates/00-nodeclass.yaml
  • argocd/config/management-cluster/eks-nodepool/templates/10-nodepool.yaml
  • argocd/config/management-cluster/eks-nodepool/values.yaml
  • argocd/config/management-cluster/hypershift/templates/05-job.yaml
  • argocd/config/management-cluster/monitoring/values.yaml
  • argocd/config/regional-cluster/aws-load-balancer-controller/Chart.yaml
  • argocd/config/regional-cluster/aws-load-balancer-controller/values.yaml
  • argocd/config/regional-cluster/eks-nodepool/templates/00-nodeclass.yaml
  • argocd/config/regional-cluster/eks-nodepool/templates/10-nodepool.yaml
  • argocd/config/regional-cluster/eks-nodepool/values.yaml
  • argocd/config/regional-cluster/loki/templates/targetgroupbinding.yaml
  • argocd/config/regional-cluster/platform-api/templates/targetgroupbinding.yaml
  • argocd/config/regional-cluster/thanos/templates/targetgroupbinding.yaml
  • argocd/config/shared/argocd/values.yaml
  • argocd/config/shared/storageclass/templates/gp3.yaml
  • ci/ephemeral-provider/__init__.py
  • config/templates/argocd-bootstrap/applicationset.yaml.j2
  • deploy/ephemeral/us-east-1/argocd-bootstrap-management-cluster/applicationset.yaml
  • deploy/ephemeral/us-east-1/argocd-bootstrap-regional-cluster/applicationset.yaml
  • deploy/integration/us-east-1/argocd-bootstrap-management-cluster/applicationset.yaml
  • deploy/integration/us-east-1/argocd-bootstrap-regional-cluster/applicationset.yaml
  • docs/README.md
  • docs/design/fips-eks-compute.md
  • docs/design/fully-private-eks-bootstrap.md
  • docs/design/gitops-cluster-configuration.md
  • docs/design/karpenter-node-provisioning.md
  • docs/design/logging-platform.md
  • docs/design/rate-limiting-architecture.md
  • docs/design/thanos-metrics-infrastructure.md
  • docs/design/zoa-trusted-actions.md
  • scripts/buildspec/provision-infra-mc.sh
  • scripts/buildspec/register.sh
  • scripts/validate-mc-aws.sh
  • scripts/validate-mc-k8s.sh
  • scripts/validate-rc-aws.sh
  • scripts/validate-rc-k8s.sh
  • terraform/config/management-cluster/main.tf
  • terraform/config/pipeline-management-cluster/main.tf
  • terraform/config/pipeline-regional-cluster/main.tf
  • terraform/config/regional-cluster/imports.sh
  • terraform/config/regional-cluster/main.tf
  • terraform/modules/aws-load-balancer-controller/README.md
  • terraform/modules/aws-load-balancer-controller/iam.tf
  • terraform/modules/aws-load-balancer-controller/main.tf
  • terraform/modules/aws-load-balancer-controller/outputs.tf
  • terraform/modules/aws-load-balancer-controller/variables.tf
  • terraform/modules/aws-load-balancer-controller/versions.tf
  • terraform/modules/ecs-bootstrap/README.md
  • terraform/modules/ecs-bootstrap/main.tf
  • terraform/modules/ecs-bootstrap/variables.tf
  • terraform/modules/eks-cluster/README.md
  • terraform/modules/eks-cluster/data.tf
  • terraform/modules/eks-cluster/iam.tf
  • terraform/modules/eks-cluster/locals.tf
  • terraform/modules/eks-cluster/main.tf
  • terraform/modules/eks-cluster/outputs.tf
  • terraform/modules/eks-cluster/variables.tf
  • terraform/modules/eks-cluster/versions.tf
  • terraform/modules/hyperfleet-infrastructure/amazonmq.tf

Comment thread argocd/config/shared/argocd/values.yaml
Comment thread argocd/config/shared/storageclass/templates/gp3.yaml
Comment thread docs/design/fips-eks-compute.md
Comment thread scripts/validate-rc-aws.sh Outdated
Comment thread terraform/modules/aws-load-balancer-controller/README.md
Comment thread terraform/modules/ecs-bootstrap/README.md
Comment thread terraform/modules/eks-cluster/README.md Outdated
Comment thread terraform/modules/eks-cluster/variables.tf
@openshift-ci openshift-ci Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 23, 2026
@theautoroboto

Copy link
Copy Markdown
Contributor Author

/test on-demand-e2e

2 similar comments
@theautoroboto

Copy link
Copy Markdown
Contributor Author

/test on-demand-e2e

@theautoroboto

Copy link
Copy Markdown
Contributor Author

/test on-demand-e2e

@theautoroboto
theautoroboto marked this pull request as ready for review July 23, 2026 19:51
@openshift-ci openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 23, 2026
@theautoroboto

Copy link
Copy Markdown
Contributor Author

/test on-demand-e2e

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/design/zoa-trusted-actions.md`:
- Line 825: Update the IRSA rationale in the trusted-actions design section to
remove the claim that IRSA is deprecated. State instead that the platform
standardizes on EKS Pod Identity for new workload service accounts and avoids
per-cluster OIDC provider management, while preserving the existing comparison
context.

In `@scripts/validate-mc-k8s.sh`:
- Around line 138-184: Qualify all HyperShift NodePool kubectl resource
references in the NodePool validation block: update the list command, per-object
get commands, and process-substitution command to use
nodepools.hypershift.openshift.io, while preserving the existing counting and
readiness checks.

In `@terraform/modules/eks-cluster/iam.tf`:
- Around line 395-409: Add an aws:SourceArn condition to the AllowEventBridge
statement in aws_sqs_queue_policy.karpenter_interruption, restricting
sqs:SendMessage to the Karpenter EventBridge rule ARN(s); optionally also
constrain aws:SourceAccount. Reuse the existing Karpenter EventBridge rule
resource symbols and preserve the queue policy’s enablement and permissions.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: ece5d3d8-316a-4df0-98b5-a9949422062f

📥 Commits

Reviewing files that changed from the base of the PR and between 98def0f and 9f79938.

📒 Files selected for processing (57)
  • Makefile
  • argocd/config/management-cluster/eks-nodepool/templates/00-nodeclass.yaml
  • argocd/config/management-cluster/eks-nodepool/templates/10-nodepool.yaml
  • argocd/config/management-cluster/eks-nodepool/values.yaml
  • argocd/config/management-cluster/hypershift/templates/05-job.yaml
  • argocd/config/management-cluster/monitoring/values.yaml
  • argocd/config/regional-cluster/aws-load-balancer-controller/Chart.yaml
  • argocd/config/regional-cluster/aws-load-balancer-controller/values.yaml
  • argocd/config/regional-cluster/eks-nodepool/templates/00-nodeclass.yaml
  • argocd/config/regional-cluster/eks-nodepool/templates/10-nodepool.yaml
  • argocd/config/regional-cluster/eks-nodepool/values.yaml
  • argocd/config/regional-cluster/loki/templates/targetgroupbinding.yaml
  • argocd/config/regional-cluster/platform-api/templates/targetgroupbinding.yaml
  • argocd/config/regional-cluster/thanos/templates/targetgroupbinding.yaml
  • argocd/config/shared/argocd/values.yaml
  • argocd/config/shared/storageclass/templates/gp3.yaml
  • ci/ephemeral-provider/__init__.py
  • config/templates/argocd-bootstrap/applicationset.yaml.j2
  • deploy/ephemeral/us-east-1/argocd-bootstrap-management-cluster/applicationset.yaml
  • deploy/ephemeral/us-east-1/argocd-bootstrap-regional-cluster/applicationset.yaml
  • deploy/integration/us-east-1/argocd-bootstrap-management-cluster/applicationset.yaml
  • deploy/integration/us-east-1/argocd-bootstrap-regional-cluster/applicationset.yaml
  • docs/README.md
  • docs/design/fips-eks-compute.md
  • docs/design/fully-private-eks-bootstrap.md
  • docs/design/karpenter-node-provisioning.md
  • docs/design/logging-platform.md
  • docs/design/thanos-metrics-infrastructure.md
  • docs/design/zoa-trusted-actions.md
  • scripts/buildspec/provision-infra-mc.sh
  • scripts/buildspec/register.sh
  • scripts/validate-mc-aws.sh
  • scripts/validate-mc-k8s.sh
  • scripts/validate-rc-aws.sh
  • scripts/validate-rc-k8s.sh
  • terraform/config/management-cluster/main.tf
  • terraform/config/pipeline-management-cluster/main.tf
  • terraform/config/pipeline-regional-cluster/main.tf
  • terraform/config/regional-cluster/imports.sh
  • terraform/config/regional-cluster/main.tf
  • terraform/modules/aws-load-balancer-controller/README.md
  • terraform/modules/aws-load-balancer-controller/iam.tf
  • terraform/modules/aws-load-balancer-controller/main.tf
  • terraform/modules/aws-load-balancer-controller/outputs.tf
  • terraform/modules/aws-load-balancer-controller/variables.tf
  • terraform/modules/aws-load-balancer-controller/versions.tf
  • terraform/modules/ecs-bootstrap/README.md
  • terraform/modules/ecs-bootstrap/main.tf
  • terraform/modules/ecs-bootstrap/variables.tf
  • terraform/modules/eks-cluster/README.md
  • terraform/modules/eks-cluster/data.tf
  • terraform/modules/eks-cluster/iam.tf
  • terraform/modules/eks-cluster/locals.tf
  • terraform/modules/eks-cluster/main.tf
  • terraform/modules/eks-cluster/outputs.tf
  • terraform/modules/eks-cluster/variables.tf
  • terraform/modules/eks-cluster/versions.tf
🚧 Files skipped from review as they are similar to previous changes (46)
  • argocd/config/regional-cluster/platform-api/templates/targetgroupbinding.yaml
  • terraform/modules/aws-load-balancer-controller/versions.tf
  • argocd/config/regional-cluster/eks-nodepool/values.yaml
  • argocd/config/regional-cluster/eks-nodepool/templates/10-nodepool.yaml
  • argocd/config/management-cluster/eks-nodepool/values.yaml
  • argocd/config/management-cluster/eks-nodepool/templates/10-nodepool.yaml
  • ci/ephemeral-provider/init.py
  • terraform/config/pipeline-regional-cluster/main.tf
  • terraform/modules/eks-cluster/locals.tf
  • terraform/config/pipeline-management-cluster/main.tf
  • terraform/config/management-cluster/main.tf
  • Makefile
  • config/templates/argocd-bootstrap/applicationset.yaml.j2
  • terraform/modules/aws-load-balancer-controller/main.tf
  • argocd/config/regional-cluster/aws-load-balancer-controller/values.yaml
  • docs/design/logging-platform.md
  • deploy/integration/us-east-1/argocd-bootstrap-management-cluster/applicationset.yaml
  • deploy/ephemeral/us-east-1/argocd-bootstrap-management-cluster/applicationset.yaml
  • argocd/config/shared/storageclass/templates/gp3.yaml
  • terraform/modules/aws-load-balancer-controller/variables.tf
  • docs/README.md
  • terraform/modules/aws-load-balancer-controller/outputs.tf
  • docs/design/thanos-metrics-infrastructure.md
  • terraform/modules/eks-cluster/data.tf
  • terraform/modules/aws-load-balancer-controller/README.md
  • argocd/config/management-cluster/monitoring/values.yaml
  • deploy/integration/us-east-1/argocd-bootstrap-regional-cluster/applicationset.yaml
  • terraform/modules/eks-cluster/versions.tf
  • argocd/config/regional-cluster/aws-load-balancer-controller/Chart.yaml
  • terraform/modules/ecs-bootstrap/variables.tf
  • deploy/ephemeral/us-east-1/argocd-bootstrap-regional-cluster/applicationset.yaml
  • argocd/config/regional-cluster/thanos/templates/targetgroupbinding.yaml
  • terraform/modules/eks-cluster/variables.tf
  • terraform/config/regional-cluster/main.tf
  • argocd/config/shared/argocd/values.yaml
  • argocd/config/regional-cluster/loki/templates/targetgroupbinding.yaml
  • terraform/modules/eks-cluster/outputs.tf
  • scripts/validate-mc-aws.sh
  • terraform/modules/eks-cluster/main.tf
  • terraform/modules/aws-load-balancer-controller/iam.tf
  • docs/design/fips-eks-compute.md
  • terraform/modules/ecs-bootstrap/main.tf
  • scripts/validate-rc-aws.sh
  • docs/design/karpenter-node-provisioning.md
  • argocd/config/management-cluster/hypershift/templates/05-job.yaml
  • terraform/modules/eks-cluster/README.md

Comment thread docs/design/zoa-trusted-actions.md
Comment thread scripts/validate-mc-k8s.sh Outdated
Comment thread terraform/modules/eks-cluster/iam.tf
@theautoroboto theautoroboto changed the title Migrate from EKS Auto Mode to OSS Karpenter ROSAENG-60886: Migrate from EKS Auto Mode to OSS Karpenter Jul 24, 2026
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 24, 2026
@openshift-ci-robot

openshift-ci-robot commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

@theautoroboto: This pull request references ROSAENG-60886 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the epic to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Summary

  • Replaces EKS Auto Mode with OSS Karpenter for MC node provisioning (original migration commit)
  • Moves kms:CreateGrant from node role to Karpenter controller role
  • Waits for hypershift Application to be Healthy before ECS bootstrap exits
  • Fixes hypershift-install Job CRD wait: kubectl is not in the hypershift-operator image; replaced with curl against the in-cluster API server
  • Fixes monitoring bootstrap failure: admissionWebhooks.enabled: false skips the certgen Job but tls.enabled defaults to true, crashing PrometheusOperator on the missing TLS secret mount; now sets both to false

Root causes fixed (from CI run 2080263806742695936)

Job CRD wait: The hypershift-install Job pod uses quay.io/cbusse_openshift/hypershift-operator:aws-iam-auth which does not include kubectl. The original CRD wait loop silently failed every iteration, the Job timed out, and hypershift never installed. The ECS bootstrap (which runs on a different container with kubectl) could see the CRDs fine — the Job pod could not.

monitoring TLS: kube-prometheus-stack 72.6.2 gates the tls-secret volume mount on prometheusOperator.tls.enabled (not admissionWebhooks.enabled). Disabling admission webhooks stops the certgen Job from creating the secret, while the Deployment still tries to mount it → pod crash → monitoring Degraded → hypershift sync blocks.

Test plan

  • Trigger on-demand-e2e on this PR
  • Confirm mc01-bootstrap CodeBuild log shows === Prometheus Operator CRDs present — proceeding with hypershift install ===
  • Confirm monitoring Application reaches Healthy
  • Confirm hypershift Application reaches Healthy
  • Confirm hosted cluster provisioning succeeds end-to-end

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features
  • Added OSS Karpenter-based node provisioning with FIPS-capable nodes, interruption handling, and pre-warm validation.
  • Added AWS Load Balancer Controller support and updated load-balancing integrations.
  • Added comprehensive management and regional cluster health-validation scripts.
  • Bug Fixes
  • Improved bootstrap reliability with longer timeouts, dependency waits, and clearer diagnostics.
  • Reduced false OutOfSync reports for controller-managed resources.
  • Updated storage provisioning and TargetGroupBinding APIs.
  • Documentation
  • Added Karpenter architecture guidance and updated cluster provisioning documentation.
  • Chores
  • Increased infrastructure pipeline and registration timeouts for longer-running deployments.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@theautoroboto

Copy link
Copy Markdown
Contributor Author

/test on-demand-e2e

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@ci/ephemeral-provider/orchestrator.py`:
- Around line 233-244: Update the docstring describing the override-directory
handling to reflect the logic around load_and_merge: region YAML files in
config/ephemeral are replaced, while defaults.yaml is retained and merged with
the override to preserve existing environment settings.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 966663f8-4dcf-4747-9d6a-905bda0cf917

📥 Commits

Reviewing files that changed from the base of the PR and between c22c4dc and 0588478.

📒 Files selected for processing (1)
  • ci/ephemeral-provider/orchestrator.py

Comment thread ci/ephemeral-provider/orchestrator.py
@theautoroboto

Copy link
Copy Markdown
Contributor Author

/test on-demand-e2e

@openshift-ci openshift-ci Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 24, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
argocd/config/shared/argocd/templates/sre-targetgroupbinding.yaml (1)

2-2: 📐 Maintainability & Code Quality | 🔵 Trivial

Validate the shared rendered configuration.

Run Helm lint and validate rendered output for both management and regional rendering contexts before merge.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@argocd/config/shared/argocd/templates/sre-targetgroupbinding.yaml` at line 2,
Validate the shared rendered configuration for sre-targetgroupbinding by running
Helm lint and checking the rendered output in both management and regional
rendering contexts before merging.

Source: Coding guidelines

argocd/config/regional-cluster/grafana/templates/sre-targetgroupbinding.yaml (1)

2-2: 📐 Maintainability & Code Quality | 🔵 Trivial

Run Helm lint and validate rendered configuration for every changed chart.

  • argocd/config/regional-cluster/grafana/templates/sre-targetgroupbinding.yaml#L2-L2: lint and render the Grafana chart.
  • argocd/config/regional-cluster/loki/templates/targetgroupbinding.yaml#L2-L2: lint and render the Loki chart.
  • argocd/config/regional-cluster/loki/templates/targetgroupbinding.yaml#L20-L20: validate the query-frontend rendering.
  • argocd/config/regional-cluster/loki/templates/targetgroupbinding.yaml#L38-L38: validate the SRE query-frontend rendering.
  • argocd/config/regional-cluster/monitoring/templates/sre-targetgroupbinding.yaml#L2-L2: lint and render the monitoring chart.
  • argocd/config/shared/argocd/templates/sre-targetgroupbinding.yaml#L2-L2: validate both management and regional renderings.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@argocd/config/regional-cluster/grafana/templates/sre-targetgroupbinding.yaml`
at line 2, Run Helm lint and validate rendered output for every affected chart:
Grafana at
argocd/config/regional-cluster/grafana/templates/sre-targetgroupbinding.yaml:2-2;
Loki at
argocd/config/regional-cluster/loki/templates/targetgroupbinding.yaml:2-2,
including query-frontend at :20-20 and SRE query-frontend at :38-38; monitoring
at
argocd/config/regional-cluster/monitoring/templates/sre-targetgroupbinding.yaml:2-2;
and both management and regional renderings for
argocd/config/shared/argocd/templates/sre-targetgroupbinding.yaml:2-2. Fix any
lint or rendering errors exposed by these validations.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In
`@argocd/config/regional-cluster/grafana/templates/sre-targetgroupbinding.yaml`:
- Line 2: Run Helm lint and validate rendered output for every affected chart:
Grafana at
argocd/config/regional-cluster/grafana/templates/sre-targetgroupbinding.yaml:2-2;
Loki at
argocd/config/regional-cluster/loki/templates/targetgroupbinding.yaml:2-2,
including query-frontend at :20-20 and SRE query-frontend at :38-38; monitoring
at
argocd/config/regional-cluster/monitoring/templates/sre-targetgroupbinding.yaml:2-2;
and both management and regional renderings for
argocd/config/shared/argocd/templates/sre-targetgroupbinding.yaml:2-2. Fix any
lint or rendering errors exposed by these validations.

In `@argocd/config/shared/argocd/templates/sre-targetgroupbinding.yaml`:
- Line 2: Validate the shared rendered configuration for sre-targetgroupbinding
by running Helm lint and checking the rendered output in both management and
regional rendering contexts before merging.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 09a325f2-c261-44a8-a248-39f9ce5e3f17

📥 Commits

Reviewing files that changed from the base of the PR and between 0588478 and e56c6be.

📒 Files selected for processing (5)
  • argocd/config/regional-cluster/grafana/templates/sre-targetgroupbinding.yaml
  • argocd/config/regional-cluster/loki/templates/targetgroupbinding.yaml
  • argocd/config/regional-cluster/monitoring/templates/sre-targetgroupbinding.yaml
  • argocd/config/regional-cluster/thanos/templates/targetgroupbinding.yaml
  • argocd/config/shared/argocd/templates/sre-targetgroupbinding.yaml
🚧 Files skipped from review as they are similar to previous changes (1)
  • argocd/config/regional-cluster/thanos/templates/targetgroupbinding.yaml

@openshift-ci openshift-ci Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 24, 2026
@theautoroboto

Copy link
Copy Markdown
Contributor Author

/test on-demand-e2e

3 similar comments
@theautoroboto

Copy link
Copy Markdown
Contributor Author

/test on-demand-e2e

@theautoroboto

Copy link
Copy Markdown
Contributor Author

/test on-demand-e2e

@theautoroboto

Copy link
Copy Markdown
Contributor Author

/test on-demand-e2e

@theautoroboto

Copy link
Copy Markdown
Contributor Author

/test on-demand-e2e

6 similar comments
@theautoroboto

Copy link
Copy Markdown
Contributor Author

/test on-demand-e2e

@theautoroboto

Copy link
Copy Markdown
Contributor Author

/test on-demand-e2e

@theautoroboto

Copy link
Copy Markdown
Contributor Author

/test on-demand-e2e

@theautoroboto

Copy link
Copy Markdown
Contributor Author

/test on-demand-e2e

@theautoroboto

Copy link
Copy Markdown
Contributor Author

/test on-demand-e2e

@theautoroboto

Copy link
Copy Markdown
Contributor Author

/test on-demand-e2e

@theautoroboto

Copy link
Copy Markdown
Contributor Author

/test on-demand-e2e

2 similar comments
@theautoroboto

Copy link
Copy Markdown
Contributor Author

/test on-demand-e2e

@theautoroboto

Copy link
Copy Markdown
Contributor Author

/test on-demand-e2e

@coderabbitai coderabbitai Bot mentioned this pull request Jul 30, 2026
theautoroboto added a commit to theautoroboto/rosa-hyperfleet that referenced this pull request Jul 30, 2026
validate-rc-k8s.sh and validate-mc-k8s.sh (and their AWS counterparts)
require kubectl access to private EKS endpoints that Prow cannot reach.
Remove all four scripts and the k8s validation block in ci/e2e-tests.sh
that called them, fixing the validate=1 CI failure on PR openshift-online#698.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@theautoroboto

Copy link
Copy Markdown
Contributor Author

/test on-demand-e2e

theautoroboto added a commit to theautoroboto/rosa-hyperfleet that referenced this pull request Jul 30, 2026
validate-rc-k8s.sh and validate-mc-k8s.sh (and their AWS counterparts)
require kubectl access to private EKS endpoints that Prow cannot reach.
Remove all four scripts and the k8s validation block in ci/e2e-tests.sh
that called them, fixing the validate=1 CI failure on PR openshift-online#698.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@theautoroboto

Copy link
Copy Markdown
Contributor Author

/test on-demand-e2e

@theautoroboto

Copy link
Copy Markdown
Contributor Author

/retest

1 similar comment
@theautoroboto

Copy link
Copy Markdown
Contributor Author

/retest

@theautoroboto

Copy link
Copy Markdown
Contributor Author

/test check-docs

1 similar comment
@theautoroboto

Copy link
Copy Markdown
Contributor Author

/test check-docs

@theautoroboto

Copy link
Copy Markdown
Contributor Author

Closing in favour of a new PR — CI integration broke after history squash (force-push rewrote all commit SHAs). Same branch, clean history.

@theautoroboto

Copy link
Copy Markdown
Contributor Author

/test on-demand-e2e

Replaces EKS Auto Mode with OSS Karpenter for workload node
provisioning on both Regional and Management Clusters. Karpenter is
installed via Helm during bootstrap and adopted by ArgoCD via the
eks-nodepool chart. Node pools are configured with FIPS-compliant
AL2023 AMIs and CriticalAddonsOnly tolerations for bootstrap nodes.

Key changes:
- Remove EKS Auto Mode configuration; add Karpenter Helm install to
  ECS bootstrap with idempotency guard
- Add Karpenter controller IAM role and SQS interruption queue via
  Terraform; move KMS CreateGrant to Karpenter controller role
- Add EC2NodeClass and NodePool resources to eks-nodepool chart
- Fix hypershift-install Job shell and add verbose exit code logging
- Fix CSI Secret / Prometheus CRD chicken-and-egg in hypershift-install
- Update monitoring test to use explicit ginkgo invocation with 15m
  timeout (Makefile target was timing out at 5m)
- Add .claude/worktrees/ to .gitignore

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@theautoroboto

Copy link
Copy Markdown
Contributor Author

/test on-demand-e2e

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants