ROSAENG-62805 | fix(test): fail-fast destroy when uninstall times out

[amandahla]

PR Summary

Fail-fast cluster destroy when uninstall does not complete: skip OIDC provider and prepared-resource cleanup so stuck uninstalls surface one clear error instead of cascading IAM/OIDC noise. The destroy step (and job) still fail.

Detailed Description of the Issue

On rosa-hcp-advanced-f3, destroy sometimes times out waiting for uninstall (~60m). The harness then still attempted OIDC provider deletion and DestroyResources(), which fail because the cluster is still present — producing secondary errors and follow-on destroy-post failures (see ROSAENG-62643 plan item 3).

Related Issues and PRs

• Jira: ROSAENG-62805 (sub-task of ROSAENG-62643)
• Fixes: N/A
• Related PR(s): N/A
• Related design/docs: N/A

Type of Change

• fix - resolves an incorrect behavior or bug.
• test - adds or updates tests only.

Previous Behavior

After uninstall wait failure, destroy still ran OIDC cleanup and always ran DestroyResources(), cascading noisy secondary failures.

Behavior After This Change

• destroyCluster() returns (errors, clusterRemoved)
• Uninstall wait failure → return immediately (no OIDC delete)
• Destroy() skips prepared resource cleanup when clusterRemoved == false
• Destroy still fails the step (job stays red on stuck uninstall)
• Unit coverage for skipPreparedResourceCleanup

This does not root-cause hung uninstall (OCM/Hive); it only improves failure signal.

How to Test (Step-by-Step)

Preconditions

N/A for unit path; e2e destroy timeout is environment-dependent.

Test Steps

1. Review tests/utils/handler/cluster_handler.go
2. go test ./tests/utils/handler/
3. make pre-push-checks

Expected Results

Handler tests pass; on uninstall timeout, logs show skip of prepared resource cleanup and a single destroy error path.

Proof of the Fix

• make fmt / make fmt-check / make lint OK
• CodeRabbit local (-t committed): 0 findings
• make pre-push-checks OK (standalone); push hook OK on retry after unrelated pkg/ocm flake
• Evidence job (failure mode): https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-rosa-master-e2e-rosa-hcp-advanced-f3/2079982247187845120

Breaking Changes

• No breaking changes

Developer Verification Checklist

• Commit subject/title follows [JIRA-TICKET] | [TYPE]: <MESSAGE>.
• Related docs updated or N/A.
• Structure tests / command args updated or N/A.
• Local verification run (make pre-push-checks).
• Risks/limitations noted (does not fix hung uninstall; aws: Refactor init code into its own file. #4 destroy-post skip is separate).

Made with Cursor

Summary by CodeRabbit

• Bug Fixes

  • Improved cluster deletion handling when removal is incomplete or the cluster no longer exists.
  • Prevented follow-up resource cleanup when cluster uninstallation has not finished, reducing the risk of cleanup errors.
  • Ensured cleanup continues appropriately after successful cluster removal.

• Tests

  • Added coverage for conditional resource cleanup during cluster deletion.

[coderabbitai]

📝 Walkthrough

Walkthrough

Cluster destruction now returns both errors and whether the cluster was fully removed. The flow treats missing clusters as removed, waits for uninstall completion before deleting the OIDC provider, and skips prepared-resource cleanup when uninstall does not complete. Tests cover the cleanup-skip predicate.

Suggested reviewers: olucasfreitas

🚥 Pre-merge checks | ✅ 15

✅ Passed checks (15 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is specific, concise, and matches the main change: fail-fast destroy behavior when uninstall times out.
Description check	✅ Passed	The description follows the template well and includes the required context, impact, testing, proof, and verification sections.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	Added Ginkgo titles are static and deterministic; no timestamps, IDs, hostnames, or generated values appear in test names.
Test Structure And Quality	✅ Passed	PASS: The new Ginkgo specs are isolated pure-function tests with one assertion each; no cluster setup, waits, or cleanup are needed, and they match local test style.
Microshift Test Compatibility	✅ Passed	New Ginkgo specs only cover pure helper logic; they reference no MicroShift-unsupported APIs, namespaces, or cluster features.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	Added specs are unit tests for a pure helper and contain no SNO-relevant multi-node or topology assumptions.
Topology-Aware Scheduling Compatibility	✅ Passed	Only destroy-path test utilities changed; no manifests/controllers/scheduling constraints were added, and no anti-affinity, spread, or nodeSelector logic appears in the patch.
Ote Binary Stdout Contract	✅ Passed	No process-level stdout writes were added; the only new output is logrus via GinkgoWriter, and the new tests are pure Ginkgo specs.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	New Ginkgo tests only exercise pure helper functions; no IPv4 literals, IP parsing, host URLs, or external connectivity are present.
No-Weak-Crypto	✅ Passed	Changed handler files add only destroy-flow logic; no weak crypto APIs, custom crypto, or secret comparisons were introduced.
Container-Privileges	✅ Passed	PR only changes Go handler/test files; no manifests or pod specs were modified, and touched files contain no privileged/hostNetwork/allowPrivilegeEscalation settings.
No-Sensitive-Data-In-Logs	✅ Passed	Changed logs are generic and only include cluster IDs; no passwords, tokens, PII, hostnames, or customer data appear in new log paths.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/utils/handler/cluster_handler_test.go`:
- Around line 48-56: Replace the direct skipPreparedResourceCleanup assertions
with a focused mocked-handler test around Destroy(). Configure destroyCluster()
to report clusterRemoved=false, invoke Destroy(), and assert the
prepared-resource cleanup method DestroyResources() is not called. Keep the test
scoped to the uninstall-failure behavior rather than restating the helper’s
boolean inversion.

In `@tests/utils/handler/cluster_handler.go`:
- Around line 1232-1248: Wrap the errors returned by the cluster cleanup flow
with operation and cluster context using fmt.Errorf and %w. Update the
errDeleteCluster return in DeleteCluster and the err return from
WaitForClusterPassUninstalled to preserve their original causes while
identifying the failed operation and clusterID.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: ab49c7b4-8360-4615-9447-c11b12646078

📥 Commits

Reviewing files that changed from the base of the PR and between d042fb5 and 08aefcb.

📒 Files selected for processing (2)

• tests/utils/handler/cluster_handler.go
• tests/utils/handler/cluster_handler_test.go

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: amandahla

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• tests/OWNERS [amandahla]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

@amandahla: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/govulncheck	08aefcb	link	false	/test govulncheck

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.