Autonode - #44
Merged
Merged
Conversation
Adds enable_layer_openshift_ai with full operator install chain: NFD -> NVIDIA GPU Operator -> Red Hat OpenShift AI (DataScienceCluster). Includes S3 bucket + IAM for RHOAI data connections, comprehensive GPU instance type reference, and fix for coalesce() regression on HCP/Classic cluster outputs. Made-with: Cursor
…Serve prereqs - Switch OpenShift AI operator channel from stable to fast-3.x for v3+ - Add Service Mesh and Serverless operator installs as KServe prerequisites - Add KnativeServing CR for serverless inference - Annotate RHOAI service accounts with IRSA role ARN for S3 access - Create S3 data connection secret for RHOAI workbenches - Expand README with Day 2 setup, S3 access, and GPU verification docs - Update example tfvars with component overrides and GPU machine pool examples Made-with: Cursor
RHOAI v3+ no longer requires S3 for model serving — OCI images and PVC are the recommended storage backends. S3 is now opt-in, only needed when datasciencepipelines component is Managed. - Default openshift_ai_create_s3 to false across all environments - Update README with v3 storage table (OCI/PVC/S3 by use case) - Add KServe examples for OCI, PVC, and S3 model serving - Update air-gapped guidance to recommend OCI modelcar images - Note pipelines component requires S3 in DSC component table Made-with: Cursor
…prereqs RHOAI v3+ uses KServe RawDeployment (Headed) mode which eliminates the need for Service Mesh and Serverless operators as prerequisites. This simplifies the install from 6 operators to 3. - Upgrade DSCInitialization and DataScienceCluster templates to v2 API - Remove Service Mesh and Serverless operator install (Stage 3 removed) - Update component list: remove modelmeshserving, codeflare, datasciencepipelines; add aipipelines, nim, trainer, feastoperator, llamastackoperator, mlflowoperator, sparkoperator - Update README architecture diagram and component table - Update variable descriptions across all environments Made-with: Cursor
These components exist in ODH upstream main but are not yet in the RHOAI 3.x CRD schema. Removes them from the template, defaults, variable descriptions, and documentation. Made-with: Cursor
RHOAI 3.x on fast-3.x channel uses v1 API, not v2. The v2 API with aipipelines/nim/trainer is ODH upstream only. - Revert DSC and DSCI templates to v1 API - Remove kserve.serving subfield (original bug, removed in v3 CRD) - Restore v1 components: datasciencepipelines, modelmeshserving, codeflare - Keep feastoperator and llamastackoperator (confirmed in v3 CRD) - Revert variable descriptions and defaults to match v1 schema - Service Mesh auto-managed by RHOAI operator, not installed separately Made-with: Cursor
RHOAI v3 admission webhook rejects Managed state for deprecated components. ModelMesh and CodeFlare are deprecated in v3. Made-with: Cursor
Made-with: Cursor
Drop modelmeshserving, codeflare, kueue from template — these are deprecated in RHOAI v3 and the operator manages their defaults. Only declare components we actively configure. Made-with: Cursor
The RHOAI IAM role is now created whenever the OpenShift AI layer is enabled, independent of the S3 toggle. S3 bucket and its IAM policy are conditional on create_s3. A new ECR push/pull policy is attached when ecr_repository_arn is provided, allowing RHOAI service accounts to publish OCI model images to ECR via IRSA. Made-with: Cursor
The ECR repository ARN isn't known until apply, so it can't drive count. Use a separate create_ecr_policy boolean derived from the static var.create_ecr flag instead. Made-with: Cursor
The role ARN isn't known until apply so it can't drive for_each. Use a new openshift_ai_create_irsa boolean derived from the static enable_layer_openshift_ai flag instead. Made-with: Cursor
- Add availability_zone field to HCP machine pools with az_subnet_map resolution, enabling pools to target specific AZs for instance types with limited availability (e.g., g7e only in us-east-1b/1d) - Update GPU reference: g7e uses NVIDIA RTX PRO 6000 Blackwell (96GB GDDR7, native FP4/FP8) not L40S - Document AZ constraints for GPU instance types in OpenShift AI README with lookup commands and targeting examples - Byron dev env: multi-AZ, m6i.4xlarge workers, Graviton monitoring pool with tolerations, full observability stack Made-with: Cursor
Update default and example versions across commercial environments and examples to the latest 4.20 patch. GovCloud remains on 4.16.x EUS. Made-with: Cursor
Add single_nat_gateway variable (default null = auto) so multi-AZ deployments can opt into a shared NAT gateway instead of 1-per-AZ, saving ~$64/month for dev/lab clusters at the cost of AZ resilience. Made-with: Cursor
- README: mention shared NAT option in cost tables and features - BYO-VPC: add NAT Gateway Topology section with cost/resilience table - Machine Pools HCP: add AZ targeting example for GPU pools, update GPU instance reference with g7e/g6e/p5, document availability_zone and az_subnet_map inputs Made-with: Cursor
openshift-logging and openshift-operators-redhat exist on fresh clusters. Switch from kubernetes_namespace_v1 (which errors on existing resources) to kubectl_manifest with server_side_apply for idempotent namespace creation. Made-with: Cursor
CLO 6.x observability.openshift.io/v1 API no longer accepts spec.collector.type — Vector is the only collector and the field was removed from the schema. Made-with: Cursor
The cert-manager-openshift-routes controller uses --enable-leader-election which requires get/create/update/patch on leases in the cert-manager namespace. Add a Role and RoleBinding granting this to the cert-manager service account. Made-with: Cursor
…_pools type The environment variables.tf machine_pools type was missing these fields, causing Terraform to silently drop them. The module-level type had them but values never reached it. Made-with: Cursor
kubernetes_secret_v1 data field already handles base64 encoding. The explicit base64encode() calls caused double encoding, resulting in Loki resolving sts.dXMtZWFzdC0x.amazonaws.com instead of sts.us-east-1.amazonaws.com. Made-with: Cursor
Add modules for ROSA HCP AutoNode private preview with Spot instance support, enabling Karpenter-based node autoscaling as an alternative to static ROSA machine pools. New modules: - modules/cluster/autonode/ - Karpenter IAM role with OIDC trust, controller policy (EC2/IAM/SSM/SQS/Pricing), ec2:CreateTags on control-plane-operator, subnet discovery tags, and optional ECR pull - modules/cluster/autonode-pool/ - Karpenter NodePool CRDs via kubectl_manifest with machine_pools-style variable format, Spot/ on-demand capacity type, skip_k8s_destroy gating, and auto-filtering of restricted kubernetes.io labels Changes: - .gitignore: exclude environments/stage-hcp/ (private preview testing) - rosa-hcp module: ignore_changes for audit_log_arn to prevent provider host resolution failure when audit logging is enabled via CLI Made-with: Cursor
- LokiStack: add configurable per-tenant ingestion rate and burst size to prevent 429 Too Many Requests when running multiple operators. Defaults to 10 MB/s rate, 20 MB burst. Refactor inline YAML to use the shared lokistack template with node placement support. - autonode-pool: add consolidation_policy field (default WhenEmptyOrUnderutilized) to control Karpenter node disruption behavior Made-with: Cursor
Made-with: Cursor # Conflicts: # modules/gitops-layers/operator/outputs.tf # modules/gitops-layers/resources/main.tf # modules/gitops-layers/resources/outputs.tf
- Make taint value optional in autonode-pool module (defaults to empty) - Add instance_types, limits, weight, expire_after fields for advanced Karpenter NodePool configuration - Support simple pool definitions (just name + instance_type) without requiring labels or taints - Wire autonode module and autonode_pools into commercial-hcp environment with cluster_properties passthrough for shard targeting - Add autonode outputs (role ARN, enable command, pool names) to commercial-hcp Made-with: Cursor
Karpenter v1 API requires consolidateAfter in the disruption block. Defaults to "30s" so pools consolidate shortly after becoming eligible. Made-with: Cursor
Shows simple, multi-type, GPU, and on-demand fallback pool examples with consolidateAfter, limits, weight, and expire_after usage. Made-with: Cursor
Add comprehensive AutoNode documentation with Red Hat's official Technology Preview disclaimer, making it clear that AutoNode-enabled clusters are for testing only and not production-supported. - docs/AUTONODE.md: full deployment guide, pool reference, known limitations, and teardown instructions - modules/cluster/autonode/README.md: IAM module inputs/outputs - modules/cluster/autonode-pool/README.md: NodePool module reference - examples/autonode.tfvars: TP warning header - examples/README.md: TP callout on autonode section Made-with: Cursor
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
…ings Suppress 4 false-positive checkov failures: - CKV_AWS_290/CKV_AWS_355 on Karpenter IAM policy (AWS APIs require Resource="*") - CKV_AWS_355 on ECR pull policy (ecr:GetAuthorizationToken requires Resource="*") - CKV_AWS_184 on EFS (moved annotation inside resource block for module-call visibility) Made-with: Cursor
The inline #checkov:skip annotations are not reliably suppressed in SARIF output used by GitHub code scanning. Add CKV_AWS_184 (EFS CMK), CKV_AWS_290 (Karpenter write actions), and CKV_AWS_355 (Karpenter/ECR Resource="*") to the config-level skip-check list with justifications. Made-with: Cursor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Technology Preview of Karpenter/Autonode for ROSA HCP in US-EAST-1 NOT FOR PRODUCTION USE. NOT FOR PRODUCTION USE.