Skip to content

Distinguish transient from terminal condition failures - #83

Open
jcpowermac wants to merge 1 commit into
openshift:mainfrom
jcpowermac:finding-3-transient-failed-condition
Open

Distinguish transient from terminal condition failures#83
jcpowermac wants to merge 1 commit into
openshift:mainfrom
jcpowermac:finding-3-transient-failed-condition

Conversation

@jcpowermac

@jcpowermac jcpowermac commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Reconcile previously set every handler error to reason=Failed, even when the underlying cause was transient (e.g. cluster operators still stabilizing or an upgrade in progress) and the operator was about to retry and succeed on its own. Consumers watching for a terminal Failed reason had no way to tell "needs intervention" apart from "will resolve itself."

Add a transientError wrapper and a Retrying reason so preflight can mark known-transient conditions and have Reconcile record the right reason automatically.

Summary by CodeRabbit

  • New Features

    • Improved migration progress and readiness reporting with clearer rollout details and diagnostics.
    • Destination folders and ownership tags are now created safely without duplication.
    • Final readiness now requires stable operators, converged configuration, and sustained successful checks.
  • Bug Fixes

    • Added validation and warnings for vSphere CSI and Storage operator management states.
    • Temporary upgrade and operator health issues now trigger retries instead of permanent failures.
    • Improved status updates to preserve concurrent changes and prevent stale results.
  • Tests

    • Expanded coverage for storage validation, transient errors, retries, and status handling.

@openshift-ci
openshift-ci Bot requested review from AnnaZivkovic and vr4manta July 28, 2026 18:38
@openshift-ci

openshift-ci Bot commented Jul 28, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jcpowermac

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 28, 2026
@coderabbitai

coderabbitai Bot commented Jul 28, 2026

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f35f5107-ccca-4abf-8863-0eb39c7d4721

📥 Commits

Reviewing files that changed from the base of the PR and between 519081d and bcd714e.

📒 Files selected for processing (3)
  • internal/controller/preflight.go
  • internal/controller/preflight_test.go
  • internal/controller/vmwarecloudfoundationmigration_controller.go

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.


Walkthrough

The migration controller now classifies transient failures separately from permanent errors. It validates storage management state, performs idempotent destination setup, tracks rollout from cluster state, requires sustained readiness, bounds diagnostics, and merges status updates against the latest resource.

Changes

Migration reliability and lifecycle handling

Layer / File(s) Summary
Transient errors and storage preflight
api/v1alpha1/vmwarecloudfoundationmigration_types.go, internal/controller/errors.go, internal/controller/errors_test.go, internal/controller/preflight.go, internal/controller/preflight_test.go
Adds ReasonRetrying, transient-error classification, CSI and Storage management-state validation, warning handling, and related tests.
Destination setup and workload rollout
internal/controller/vmwarecloudfoundationmigration_controller.go
Adds idempotent folder and ownership-tag creation. Derives rollout progress from cluster state, reports bounded diagnostics, and removes source MachineSets after worker removal.
Stable readiness and status persistence
internal/controller/vmwarecloudfoundationmigration_controller.go
Requires six spaced successful readiness observations. Checks operator stability, MachineConfigPool convergence, and non-target vCenter removal. Merges status changes with concurrent updates and retries conflicts.

Estimated code review effort: 4 (Complex) | ~60 minutes

Merge Risk: ⚪ Minimal · up to bcd71

The change distinguishes transient condition failures from terminal failures so retryable situations can report an appropriate status; no actionable merge-blocking risk remains beyond normal checks and review.

Sequence Diagram(s)

sequenceDiagram
  participant MigrationController
  participant ClusterState
  participant vSphere
  MigrationController->>ClusterState: read worker and rollout state
  ClusterState-->>MigrationController: return readiness and generation data
  MigrationController->>vSphere: create missing folders and ownership tags
  vSphere-->>MigrationController: return destination objects
  MigrationController->>ClusterState: remove source MachineSets after worker removal
  MigrationController->>ClusterState: verify stable operators and MachineConfigPools
Loading

Suggested reviewers: vr4manta, annazivkovic

🚥 Pre-merge checks | ✅ 14 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 42.86% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 7 functions across 5 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (14 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the primary change: distinguishing transient condition failures from terminal failures.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed No changed Ginkgo test title exists. The pull request adds standard Go tests with static Test... names and t.Run(tt.name) subtests whose names come from fixed string literals. The modified preflig…
Test Structure And Quality ✅ Passed PASS. The pull request adds standard testing tests, not Ginkgo Describe/It blocks. The added tests use table-driven t.Run cases for one related behavior each, include diagnostic t.Fatalf mes…
Microshift Test Compatibility ✅ Passed No new Ginkgo e2e tests were added. The pull request adds internal/controller/errors_test.go and modifies internal/controller/preflight_test.go; both use Go testing.T and contain no It, `Descr…
Single Node Openshift (Sno) Test Compatibility ✅ Passed No new Ginkgo e2e tests were added. The only changed test files use standard Go testing with Test... and t.Run, and the diff adds no Ginkgo declarations or imports. The SNO multi-node compatibil…
Topology-Aware Scheduling Compatibility ✅ Passed PASS: The pull request introduces no topology-related scheduling constraint. The exact diff changes a condition-reason constant, transient-error handling, two preflight error classifications, tests, a…
Ote Binary Stdout Contract ✅ Passed No OTE stdout contract violation was introduced. The diff changes API constants, controller error handling, preflight logic, and unit tests only. It adds no stdout writer, logging configuration, or su…
Ipv6 And Disconnected Network Test Compatibility ✅ Passed PASS: The pull request adds no new Ginkgo e2e tests. The changed tests are Go testing unit tests in internal/controller, and the only simulator networking uses a local in-process govmomi server. T…
No-Weak-Crypto ✅ Passed PASS: The pull request adds only condition-reason and transient-error handling. The added lines contain no MD5, SHA1, DES, RC4, Blowfish, ECB, custom cryptography, or secret/token comparison. A non-ve…
Container-Privileges ✅ Passed PASS. The pull request changes only Go source and test files. It does not change any container or Kubernetes manifest. The repository's existing manager manifest sets runAsNonRoot: true and `allowPr…
No-Sensitive-Data-In-Logs ✅ Passed No sensitive-data logging was introduced. The patch adds a ReasonRetrying constant and an error wrapper. The two new runtime error messages contain only fixed retry guidance and cluster operator nam…
Full details: Stable And Deterministic Test Names

Explanation

No changed Ginkgo test title exists. The pull request adds standard Go tests with static Test... names and t.Run(tt.name) subtests whose names come from fixed string literals. The modified preflight test also uses fixed table-name strings. No added title contains a pod suffix, timestamp, UUID, node name, random namespace, IP address, or other run-dependent value.

Full details: Test Structure And Quality

Explanation

PASS. The pull request adds standard testing tests, not Ginkgo Describe/It blocks. The added tests use table-driven t.Run cases for one related behavior each, include diagnostic t.Fatalf messages, add no Eventually or Consistently calls, and create no cluster-scoped resources. The existing simulator setup uses defer cleanup for the model, server, and sessions. The changed test files add no setup or cleanup issue under the stated Ginkgo-specific check.

Full details: Microshift Test Compatibility

Explanation

No new Ginkgo e2e tests were added. The pull request adds internal/controller/errors_test.go and modifies internal/controller/preflight_test.go; both use Go testing.T and contain no It, Describe, Context, or When declarations. The existing test/e2e files are unchanged, so the MicroShift API compatibility check is not applicable.

Full details: Single Node Openshift (Sno) Test Compatibility

Explanation

No new Ginkgo e2e tests were added. The only changed test files use standard Go testing with Test... and t.Run, and the diff adds no Ginkgo declarations or imports. The SNO multi-node compatibility check is therefore not applicable.

Full details: Topology-Aware Scheduling Compatibility

Explanation

PASS: The pull request introduces no topology-related scheduling constraint. The exact diff changes a condition-reason constant, transient-error handling, two preflight error classifications, tests, and the controller's condition-reason selection. It does not add or modify pod affinity, topology spread, replica scheduling, node selectors or affinity, tolerations, taints, or PDBs. Existing MachineSet replica logic is unchanged and therefore is not pull-request-caused.

Full details: Ote Binary Stdout Contract

Explanation

No OTE stdout contract violation was introduced. The diff changes API constants, controller error handling, preflight logic, and unit tests only. It adds no stdout writer, logging configuration, or suite setup. Existing suite messages use GinkgoWriter, which the check allows, and the repository contains no OTE binary entry point.

Full details: Ipv6 And Disconnected Network Test Compatibility

Explanation

PASS: The pull request adds no new Ginkgo e2e tests. The changed tests are Go testing unit tests in internal/controller, and the only simulator networking uses a local in-process govmomi server. The diff adds no IPv4 literals, IPv4-only parsing, IPv6-unsafe URL construction, public hosts, registry pulls, or external URLs.

Full details: No-Weak-Crypto

Explanation

PASS: The pull request adds only condition-reason and transient-error handling. The added lines contain no MD5, SHA1, DES, RC4, Blowfish, ECB, custom cryptography, or secret/token comparison. A non-vendor scan also found no weak-crypto or constant-time comparison patterns. Existing vendor references are not introduced by this pull request.

Full details: Container-Privileges

Explanation

PASS. The pull request changes only Go source and test files. It does not change any container or Kubernetes manifest. The repository's existing manager manifest sets runAsNonRoot: true and allowPrivilegeEscalation: false; no introduced privileged: true, host namespace, or SYS_ADMIN setting is present.

Full details: No-Sensitive-Data-In-Logs

Explanation

No sensitive-data logging was introduced. The patch adds a ReasonRetrying constant and an error wrapper. The two new runtime error messages contain only fixed retry guidance and cluster operator names. The Reconcile change selects the condition reason but retains the existing status and event messages. No new log call records passwords, tokens, API keys, PII, session IDs, hostnames, or customer data.

  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
internal/controller/vmwarecloudfoundationmigration_controller.go (1)

163-163: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Add a controller-level test for the new reason mapping.

Please ensure a handler returning a transient error persists Status=False, ReasonRetrying, and the original message, while a plain error still persists ReasonFailed. This guards the end-to-end wiring beyond the existing classifier unit tests.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@internal/controller/vmwarecloudfoundationmigration_controller.go` at line
163, Add a controller-level test covering the status update path around
setCondition: verify a handler returning a transient error persists
Status=False, ReasonRetrying, and the original error message, and verify a plain
error persists ReasonFailed. Exercise the controller wiring end to end rather
than only testing the error classifier.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@internal/controller/vmwarecloudfoundationmigration_controller.go`:
- Line 163: Add a controller-level test covering the status update path around
setCondition: verify a handler returning a transient error persists
Status=False, ReasonRetrying, and the original error message, and verify a plain
error persists ReasonFailed. Exercise the controller wiring end to end rather
than only testing the error classifier.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: bfe3b7b3-11d4-4398-88be-941d2656146e

📥 Commits

Reviewing files that changed from the base of the PR and between 5944945 and ca9317c.

📒 Files selected for processing (6)
  • api/v1alpha1/vmwarecloudfoundationmigration_types.go
  • internal/controller/errors.go
  • internal/controller/errors_test.go
  • internal/controller/preflight.go
  • internal/controller/preflight_test.go
  • internal/controller/vmwarecloudfoundationmigration_controller.go

@jcpowermac
jcpowermac force-pushed the finding-3-transient-failed-condition branch from ca9317c to 519081d Compare August 10, 2026 19:28
@coderabbitai

coderabbitai Bot commented Aug 10, 2026

Copy link
Copy Markdown

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

Reconcile previously set every handler error to reason=Failed, even
when the underlying cause was transient (e.g. cluster operators still
stabilizing or an upgrade in progress) and the operator was about to
retry and succeed on its own. Consumers watching for a terminal
Failed reason had no way to tell "needs intervention" apart from
"will resolve itself."

Add a transientError wrapper and a Retrying reason so preflight can
mark known-transient conditions and have Reconcile record the right
reason automatically.
@jcpowermac
jcpowermac force-pushed the finding-3-transient-failed-condition branch from 519081d to bcd714e Compare August 31, 2026 14:26
@openshift-ci

openshift-ci Bot commented Aug 31, 2026

Copy link
Copy Markdown

@jcpowermac: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant