Skip to content

OCPBUGS-98700: Azure: delete bootstrap ignition storage during bootstrap destroy#10701

Open
chdeshpa-hue wants to merge 1 commit into
openshift:mainfrom
chdeshpa-hue:fix/azure-bootstrap-storage-leak
Open

OCPBUGS-98700: Azure: delete bootstrap ignition storage during bootstrap destroy#10701
chdeshpa-hue wants to merge 1 commit into
openshift:mainfrom
chdeshpa-hue:fix/azure-bootstrap-storage-leak

Conversation

@chdeshpa-hue

@chdeshpa-hue chdeshpa-hue commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Summary

  • On Azure IPI (CAPI) installs, the installer creates a storage account ({infraID-no-hyphens}sa) to host bootstrap.ign, but never deletes it during bootstrap destroy. The account can persist for the life of the cluster with sensitive bootstrap material.
  • Regression from CORS-2525: Terraform-era fix (#7642) was reverted (#7822 / OCPBUGS-24995). CAPI reimplemented creation but not deletion.
  • Extend existing PostDestroy to delete the ignition container and, if no other containers remain (e.g. OKD/CVM vhd image gallery), delete the storage account. Azure Stack excluded. Best-effort (warn, do not fail install). Idempotent on 404.

Fix details

  1. Delete ignition container (always safe)
  2. List remaining containers; skip SA delete if any remain
  3. Conditionally delete storage account
  4. Treat ResourceNotFound/404 as success
  5. Skip Azure Stack (different storage SDK)

Pattern: AWS PostDestroy S3 ignition cleanup; GCP equivalent in #8489 (OCPBUGS-33681).

What this does NOT change

  • Existing SSH/NAT/konnectivity cleanup in PostDestroy
  • Full destroy cluster (RG deletion still cleans up)
  • Azure Stack storage path

Test plan

  • gofmt / go vet clean on pkg/infrastructure/azure/...
  • Darwin/arm64 and Linux/amd64 builds of openshift-install
  • Pre-fix cluster: confirmed storage account {infraID}sa remains after install complete
  • Post-fix install: confirm logs show ignition container (+ SA) deleted after bootstrap destroy
  • OKD/CVM path: ignition deleted, SA retained when vhd container remains
  • Azure Stack path: storage cleanup skipped

Build evidence

$ go version
go version go1.25.8 darwin/arm64
$ gofmt -d pkg/infrastructure/azure/azure.go
(clean)
$ go vet ./pkg/infrastructure/azure/...
(clean)
$ go build -o bin/openshift-install-azure-fix ./cmd/openshift-install/
(success)
$ GOOS=linux GOARCH=amd64 go build -o bin/openshift-install-azure-fix-linux ./cmd/openshift-install/
(success)

Fixes: https://issues.redhat.com/browse/OCPBUGS-98700

Made with Cursor

Summary by CodeRabbit

  • Bug Fixes
    • Improved cleanup of bootstrap storage resources after infrastructure deletion.
    • Automatically removes empty bootstrap storage accounts when applicable.
    • Cleanup now safely handles resources that have already been removed.

The installer creates a storage account to host bootstrap.ign during
Azure IPI installs but never deletes it during bootstrap destroy.
This leaves sensitive bootstrap data (CA certs, auth tokens) in a
network-accessible storage account for the lifetime of the cluster.

Extend PostDestroy to delete the ignition container and, if no other
containers remain (e.g. vhd for OKD/confidential VM image gallery),
delete the entire storage account. Azure Stack is excluded (uses a
different storage SDK). Cleanup is best-effort: failures log a warning
but do not block install completion.

Co-authored-by: Cursor <cursoragent@cursor.com>
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci-robot openshift-ci-robot added jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jul 21, 2026
@openshift-ci-robot

openshift-ci-robot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

@chdeshpa-hue: This pull request references Jira Issue OCPBUGS-98700, which is invalid:

  • expected the bug to target the "5.0.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

  • On Azure IPI (CAPI) installs, the installer creates a storage account ({infraID-no-hyphens}sa) to host bootstrap.ign, but never deletes it during bootstrap destroy. The account can persist for the life of the cluster with sensitive bootstrap material.
  • Regression from CORS-2525: Terraform-era fix (#7642) was reverted (#7822 / OCPBUGS-24995). CAPI reimplemented creation but not deletion.
  • Extend existing PostDestroy to delete the ignition container and, if no other containers remain (e.g. OKD/CVM vhd image gallery), delete the storage account. Azure Stack excluded. Best-effort (warn, do not fail install). Idempotent on 404.

Fix details

  1. Delete ignition container (always safe)
  2. List remaining containers; skip SA delete if any remain
  3. Conditionally delete storage account
  4. Treat ResourceNotFound/404 as success
  5. Skip Azure Stack (different storage SDK)

Pattern: AWS PostDestroy S3 ignition cleanup; GCP equivalent in #8489 (OCPBUGS-33681).

What this does NOT change

  • Existing SSH/NAT/konnectivity cleanup in PostDestroy
  • Full destroy cluster (RG deletion still cleans up)
  • Azure Stack storage path

Test plan

  • gofmt / go vet clean on pkg/infrastructure/azure/...
  • Darwin/arm64 and Linux/amd64 builds of openshift-install
  • Pre-fix cluster: confirmed storage account {infraID}sa remains after install complete
  • Post-fix install: confirm logs show ignition container (+ SA) deleted after bootstrap destroy
  • OKD/CVM path: ignition deleted, SA retained when vhd container remains
  • Azure Stack path: storage cleanup skipped

Build evidence

$ go version
go version go1.25.8 darwin/arm64
$ gofmt -d pkg/infrastructure/azure/azure.go
(clean)
$ go vet ./pkg/infrastructure/azure/...
(clean)
$ go build -o bin/openshift-install-azure-fix ./cmd/openshift-install/
(success)
$ GOOS=linux GOARCH=amd64 go build -o bin/openshift-install-azure-fix-linux ./cmd/openshift-install/
(success)

Fixes: https://issues.redhat.com/browse/OCPBUGS-98700

Made with Cursor

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown
📝 Walkthrough

Walkthrough

Azure Provider.PostDestroy now cleans up bootstrap ignition storage on non-StackCloud environments. The new helper deletes the ignition container, removes an empty storage account, and treats Azure 404 responses as non-fatal.

Changes

Azure bootstrap cleanup

Layer / File(s) Summary
Post-destroy bootstrap storage cleanup
pkg/infrastructure/azure/azure.go
Provider.PostDestroy invokes bootstrap cleanup outside StackCloud environments; deleteBootstrapIgnition removes the ignition container and empty storage account, while isNotFoundError handles Azure 404 responses idempotently.

Estimated code review effort: 3 (Moderate) | ~20 minutes

🚥 Pre-merge checks | ✅ 15
✅ Passed checks (15 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the main Azure bootstrap-destroy cleanup change.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed Only pkg/infrastructure/azure/azure.go changed; no *_test.go files or Ginkgo titles were added or modified.
Test Structure And Quality ✅ Passed No Ginkgo test code was changed; this PR only modifies pkg/infrastructure/azure/azure.go, so the test-quality check is not applicable.
Microshift Test Compatibility ✅ Passed Only pkg/infrastructure/azure/azure.go changed; no new Ginkgo e2e tests or MicroShift-sensitive APIs were added.
Single Node Openshift (Sno) Test Compatibility ✅ Passed No new Ginkgo e2e tests were added; only pkg/infrastructure/azure/azure.go changed, so SNO-specific test compatibility is not applicable.
Topology-Aware Scheduling Compatibility ✅ Passed Only pkg/infrastructure/azure/azure.go changed; the diff adds Azure storage cleanup in PostDestroy, with no manifests, controllers, or scheduling constraints.
Ote Binary Stdout Contract ✅ Passed PASS: The PR only touches pkg/infrastructure/azure; it adds no stdout writes in main/init/TestMain/suite setup, and installer main already suppresses klog/stdout.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed No new Ginkgo e2e tests were added; the only changed file is pkg/infrastructure/azure/azure.go and it contains no It/Describe/Context/When blocks or external connectivity assumptions.
No-Weak-Crypto ✅ Passed PASS: The changed Azure code only adds storage cleanup and 404 handling; no weak algorithms, custom crypto, or non-constant-time secret comparisons were introduced.
Container-Privileges ✅ Passed PR only changes Azure Go cleanup logic; no privileged, hostPID/Network/IPC, SYS_ADMIN, or allowPrivilegeEscalation settings were added.
No-Sensitive-Data-In-Logs ✅ Passed New logs only mention storage account/container/resource-group names; no passwords, tokens, PII, or session IDs are logged.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 golangci-lint (2.12.2)

Error: can't load config: unsupported version of the configuration: "" See https://golangci-lint.run/docs/product/migration-guide for migration instructions
The command is terminated due to an error: can't load config: unsupported version of the configuration: "" See https://golangci-lint.run/docs/product/migration-guide for migration instructions


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@openshift-ci

openshift-ci Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign jhixson74 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci
openshift-ci Bot requested review from jhixson74 and rna-afk July 21, 2026 02:12
@chdeshpa-hue

Copy link
Copy Markdown
Contributor Author

/jira refresh

@openshift-ci-robot openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jul 21, 2026
@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@chdeshpa-hue: This pull request references Jira Issue OCPBUGS-98700, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.0.0) matches configured target version for branch (5.0.0)
  • bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)
Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/infrastructure/azure/azure.go`:
- Around line 931-932: Update the container-listing error branch in the helper
used by PostDestroy to return the encountered error instead of nil, preserving
the existing successful return path. Ensure PostDestroy continues to handle the
propagated error for best-effort cleanup.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 535bc5a7-4392-41a9-bdb9-5068013338a7

📥 Commits

Reviewing files that changed from the base of the PR and between 91e9633 and f797579.

📒 Files selected for processing (1)
  • pkg/infrastructure/azure/azure.go

Comment on lines +931 to +932
return nil
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Propagate the error when listing containers fails.

Returning nil here swallows the error and falsely reports success to the caller. Since PostDestroy safely traps errors from this helper to perform a best-effort cleanup without failing the overall process, returning the error directly ensures it is properly propagated and logged by the caller. As per path instructions, never ignore error returns.

♻️ Proposed fix to propagate the error
-			logrus.Warnf("Failed to list containers in storage account %s, skipping account deletion: %v", storageAccountName, err)
-			return nil
+			return fmt.Errorf("failed to list containers in storage account %s: %w", storageAccountName, err)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
return nil
}
return fmt.Errorf("failed to list containers in storage account %s: %w", storageAccountName, err)
}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/infrastructure/azure/azure.go` around lines 931 - 932, Update the
container-listing error branch in the helper used by PostDestroy to return the
encountered error instead of nil, preserving the existing successful return
path. Ensure PostDestroy continues to handle the propagated error for
best-effort cleanup.

Source: Path instructions

@openshift-ci

openshift-ci Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

@chdeshpa-hue: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants