Skip to content

ROSAENG-62527: pass --no-wait to vpc delete when fire-and-forget is set - #174

Merged
openshift-merge-bot[bot] merged 1 commit into
openshift-online:mainfrom
cdoan1:ROSAENG-62527-plumb-nowait-2
Jul 27, 2026
Merged

ROSAENG-62527: pass --no-wait to vpc delete when fire-and-forget is set#174
openshift-merge-bot[bot] merged 1 commit into
openshift-online:mainfrom
cdoan1:ROSAENG-62527-plumb-nowait-2

Conversation

@cdoan1

@cdoan1 cdoan1 commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

When ROSA_REGIONAL_TEARDOWN_FIRE_AND_FORGET=true (pre-merge e2e), pass --no-wait to rosactl cluster-vpc delete to skip the ~20 min VPC CloudFormation stack deletion wait. Nightly e2e runs still wait for full teardown.

Description

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update
  • Refactoring (no functional changes)
  • CI/CD or tooling change

Testing

  • Unit tests pass (make test)
  • Integration tests pass (if applicable)
  • Manual verification completed

Checklist

  • My code follows the project's coding conventions
  • I have updated documentation as needed
  • I have added tests that prove my fix/feature works
  • All new and existing tests pass

Summary by CodeRabbit

  • Tests
    • Improved end-to-end cleanup coverage for cluster VPC deletion.
    • Added configurable fire-and-forget regional teardown for OIDC, VPC, and IAM cleanup.
    • When enabled, cleanup now logs a notice and starts async deletion per stack, skipping the remaining synchronous delete steps for that stack.
    • Refined the teardown command execution to build an explicit CLI argument list for more reliable automated invocation, while preserving existing retry/backoff behavior.

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 24, 2026
@openshift-ci-robot

openshift-ci-robot commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

@cdoan1: This pull request references ROSAENG-62527 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "5.0.0" version, but no target version was set.

Details

In response to this:

When ROSA_REGIONAL_TEARDOWN_FIRE_AND_FORGET=true (pre-merge e2e), pass --no-wait to rosactl cluster-vpc delete to skip the ~20 min VPC CloudFormation stack deletion wait. Nightly e2e runs still wait for full teardown.

Description

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update
  • Refactoring (no functional changes)
  • CI/CD or tooling change

Testing

  • Unit tests pass (make test)
  • Integration tests pass (if applicable)
  • Manual verification completed

Checklist

  • My code follows the project's coding conventions
  • I have updated documentation as needed
  • I have added tests that prove my fix/feature works
  • All new and existing tests pass

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 24, 2026
@coderabbitai

coderabbitai Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

Regional teardown tests conditionally launch asynchronous OIDC, VPC, and IAM deletions through rosactl and skip synchronous cleanup. The IAM path marks cleanup complete, while the deletion helper explicitly constructs its CLI arguments.

Changes

Regional teardown cleanup

Layer / File(s) Summary
Fire-and-forget deletion flow
test/e2e-cli/cluster_test.go
The helper builds explicit rosactl arguments. OIDC, VPC, and IAM cleanup branches invoke asynchronous deletion and return early when ROSA_REGIONAL_TEARDOWN_FIRE_AND_FORGET is enabled; only IAM sets cleanupCompleted before returning.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

Suggested reviewers: typeid, jmelis, slopezz

🚥 Pre-merge checks | ✅ 9 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Ai-Attribution ⚠️ Warning The commit uses Co-Authored-By: Claude Opus 4.6, but AI assistance requires Red Hat Assisted-by or Generated-by trailers. Replace the AI-related Co-Authored-By trailer with the appropriate Assisted-by or Generated-by trailer, per repo policy.
✅ Passed checks (9 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately captures the main behavior change: fire-and-forget teardown now passes --no-wait to VPC deletion.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
No-Weak-Crypto ✅ Passed Changed code only alters rosactl delete args/cleanup flow; no weak ciphers, custom crypto, or secret/token comparisons found.
Container-Privileges ✅ Passed PR only changes Go E2E test logic; no container/K8s manifest privilege settings were added or modified.
No-Sensitive-Data-In-Logs ✅ Passed New logs only emit stack names, cluster name, and start errors; no passwords, tokens, PII, internal hostnames, or customer data are introduced.
No-Hardcoded-Secrets ✅ Passed No hardcoded secrets found in the diff; added literals are CLI args and env-flag checks only.
No-Injection-Vectors ✅ Passed The PR only adds exec.Command with an arg slice; no SQL, shell=True, eval/exec-on-data, yaml.load, os.system, or DOM sinks were introduced.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 golangci-lint (2.12.2)

level=error msg="[linters_context] typechecking error: pattern ./...: directory prefix . does not contain main module or its selected dependencies"


Comment @coderabbitai help to get the list of available commands.

@typeid typeid left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See below.

@typeid typeid left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue: When ROSA_REGIONAL_TEARDOWN_FIRE_AND_FORGET is set, --no-wait is only plumbed into the cluster-vpc delete spec (and the fireAndForgetInfraDelete DeferCleanup helper). The cluster-oidc and cluster-iam delete specs still run synchronously and will block.

Two options to fix:

  1. Plumb --no-wait into the OIDC and IAM delete specs the same way it's done for VPC.
  2. (preferred) When fire-and-forget is set, just call fireAndForgetInfraDelete for all stacks and skip the ordered delete specs entirely — that helper already does the right thing (--no-wait + Process.Release for every stack). This would be simpler and match the "fire and forget" intent more directly.

@cdoan1
cdoan1 force-pushed the ROSAENG-62527-plumb-nowait-2 branch from f2bb51b to c8cb882 Compare July 24, 2026 18:48

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/e2e-cli/cluster_test.go`:
- Around line 828-833: The fire-and-forget teardown path currently marks cleanup
complete without confirming all async deletes launched. Update
fireAndForgetInfraDelete to return or aggregate cmd.Start errors for every
requested stack, and set cleanupCompleted only when all launches succeed;
otherwise preserve the later VPC/IAM cleanup paths.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift-online/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 2e39a11b-1e80-4618-886e-d0ede569eaf0

📥 Commits

Reviewing files that changed from the base of the PR and between f2bb51b and c8cb882.

📒 Files selected for processing (1)
  • test/e2e-cli/cluster_test.go

@cdoan1

cdoan1 commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator Author

/override ci/prow/lint ci/prow/unit ci/prow/verify

@openshift-ci

openshift-ci Bot commented Jul 24, 2026

Copy link
Copy Markdown

@cdoan1: Overrode contexts on behalf of cdoan1: ci/prow/lint, ci/prow/unit, ci/prow/verify

Details

In response to this:

/override ci/prow/lint ci/prow/unit ci/prow/verify

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@cdoan1

cdoan1 commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator Author

/test

@cdoan1

cdoan1 commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator Author

/test rosa-regionality-compatibility-e2e

@openshift-ci

openshift-ci Bot commented Jul 24, 2026

Copy link
Copy Markdown

@cdoan1: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/rosa-regionality-compatibility-e2e c8cb882 link true /test rosa-regionality-compatibility-e2e

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@cdoan1

cdoan1 commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator Author

e2e failure:

⏺ This failure is also not related to our changes. It's a race condition in the VPC setup step — the test created a VPC stack asynchronously but the subsequent cluster-vpc list ran before CloudFormation had registered the
stack:

Expected
: No cluster VPC stacks found.
to contain substring
: e2e-1784927901

The stack creation returned "Stack is being created asynchronously" but the list command ran immediately and found nothing. This is a pre-existing timing issue in the vpc-list setup spec (line 314), not in the cleanup specs
we modified. The 19 skipped tests are all downstream specs that depend on a successfully created VPC.

All other test suites passed (Platform API, ZOA, Platform Monitoring). A re-run should get past this transient CloudFormation propagation delay.

@typeid

typeid commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

I'd suggest a slightly different approach here — opened cdoan1#1 with the change.

Instead of having the oidc test launch all three deletes and the vpc/iam tests skip on a shared flag, each cleanup test should independently check ROSA_REGIONAL_TEARDOWN_FIRE_AND_FORGET and fire its own --no-wait delete. My initial thought was that launching everything at once makes more sense for fire-and-forget, but in Ginkgo's Ordered container model it creates awkward cross-test coupling — one test silently takes responsibility for other tests' resources, and the remaining tests exist only to skip. Having each test own its own resource is more idiomatic and easier to reason about.

@cdoan1
cdoan1 force-pushed the ROSAENG-62527-plumb-nowait-2 branch from c8cb882 to 84a8ffb Compare July 27, 2026 14:54

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
test/e2e-cli/cluster_test.go (1)

305-306: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Re-enable the disabled cluster list checks. test/e2e-cli/cluster_test.go:305-306 and :462 are still XIt, so the VPC/OIDC list assertions never run; switch them back to It and keep the VPC polling if the create remains async.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e-cli/cluster_test.go` around lines 305 - 306, Re-enable both disabled
cluster list checks by changing the XIt declarations at
test/e2e-cli/cluster_test.go:305-306 and test/e2e-cli/cluster_test.go:462 to It;
retain the existing VPC polling in the “should be able to list the cluster-vpc
and find that cluster in the list” test because create-vpc remains asynchronous.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/e2e-cli/cluster_test.go`:
- Around line 836-841: Update the fire-and-forget branches in
test/e2e-cli/cluster_test.go at 836-841, 854-859, and 872-877 so each call to
fireAndForgetInfraDelete dispatches exactly one stack: cluster-oidc at 836-841,
cluster-vpc at 854-859, and cluster-iam at 872-877; leave the surrounding
cleanupCompleted and return behavior unchanged.

---

Outside diff comments:
In `@test/e2e-cli/cluster_test.go`:
- Around line 305-306: Re-enable both disabled cluster list checks by changing
the XIt declarations at test/e2e-cli/cluster_test.go:305-306 and
test/e2e-cli/cluster_test.go:462 to It; retain the existing VPC polling in the
“should be able to list the cluster-vpc and find that cluster in the list” test
because create-vpc remains asynchronous.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift-online/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 40d63811-24e3-4f99-96ec-63054311dfdf

📥 Commits

Reviewing files that changed from the base of the PR and between c8cb882 and 84a8ffb.

📒 Files selected for processing (1)
  • test/e2e-cli/cluster_test.go

@cdoan1
cdoan1 force-pushed the ROSAENG-62527-plumb-nowait-2 branch from 84a8ffb to 8c1a106 Compare July 27, 2026 15:09

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/e2e-cli/cluster_test.go`:
- Around line 836-841: Track asynchronous cleanup dispatch separately for each
required infrastructure stack instead of setting suite-wide cleanupCompleted
after one launch. In test/e2e-cli/cluster_test.go at lines 836-841, 854-859, and
872-877, record successful dispatch for cluster-oidc, cluster-vpc, and
cluster-iam respectively; propagate any process-start failure, and suppress the
DeferCleanup safety net only when every required delete has been successfully
launched.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift-online/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 071b394a-9e79-4fd5-85da-8cf71f5b4ff0

📥 Commits

Reviewing files that changed from the base of the PR and between 84a8ffb and 8c1a106.

📒 Files selected for processing (1)
  • test/e2e-cli/cluster_test.go

Comment thread test/e2e-cli/cluster_test.go
When ROSA_REGIONAL_TEARDOWN_FIRE_AND_FORGET=true (pre-merge e2e),
pass --no-wait to rosactl cluster-vpc delete to skip the ~20 min
VPC CloudFormation stack deletion wait. Nightly e2e runs still wait
for full teardown.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@cdoan1
cdoan1 force-pushed the ROSAENG-62527-plumb-nowait-2 branch from 8c1a106 to c3abb75 Compare July 27, 2026 15:16

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/e2e-cli/cluster_test.go`:
- Around line 82-83: Update fireAndForgetInfraDelete so its delete command does
not always include --no-wait. Add an explicit mode or separate synchronous
cleanup path for the DeferCleanup caller, ensuring safety-net and nightly
teardown wait for infrastructure deletion while normal fire-and-forget cleanup
remains asynchronous.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift-online/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: c591c9ac-7aa0-4c84-b725-54b1940cc47a

📥 Commits

Reviewing files that changed from the base of the PR and between 8c1a106 and c3abb75.

📒 Files selected for processing (1)
  • test/e2e-cli/cluster_test.go

Comment thread test/e2e-cli/cluster_test.go

@typeid typeid left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/test rosa-regionality-compatiblity-e2e
/lgtm
/approve

@openshift-ci

openshift-ci Bot commented Jul 27, 2026

Copy link
Copy Markdown

@typeid: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test images
/test lint
/test rosa-regionality-compatibility-e2e
/test unit
/test verify

Use /test all to run the following jobs that were automatically triggered:

pull-ci-openshift-online-rosa-hyperfleet-api-main-images
pull-ci-openshift-online-rosa-hyperfleet-api-main-lint
pull-ci-openshift-online-rosa-hyperfleet-api-main-unit
pull-ci-openshift-online-rosa-hyperfleet-api-main-verify
Details

In response to this:

/test rosa-regionality-compatiblity-e2e
/lgtm
/approve

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jul 27, 2026
@openshift-ci

openshift-ci Bot commented Jul 27, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cdoan1, typeid

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@typeid

typeid commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

Thank you for this @cdoan1 - it will be incredibly helpful!

@openshift-merge-bot
openshift-merge-bot Bot merged commit d8bf579 into openshift-online:main Jul 27, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants