Skip to content

✨ Allow forced detachment of a host from Ironic#2955

Open
CrystalChun wants to merge 3 commits intometal3-io:mainfrom
CrystalChun:force-detachment
Open

✨ Allow forced detachment of a host from Ironic#2955
CrystalChun wants to merge 3 commits intometal3-io:mainfrom
CrystalChun:force-detachment

Conversation

@CrystalChun
Copy link
Copy Markdown

@CrystalChun CrystalChun commented Jan 30, 2026

What this PR does / why we need it:
Introduces the force argument for detaching. If set to true, it would bypass whichever status the host is currently in, abort the action in Ironic, and force the host's detachment.

Note: there is a soft dependency on the new feature in Ironic that allows the abort action on deploying nodes. Without it, deprovisioning will be triggered first, which won't preserve the exact host's state.

Finally, I have prepared e2e tests for this change, but they ended up being timing-dependent. To fix this dependency, I need direct access to Ironic Node record, which is going to greatly increase the already large size of this change. The tests will be proposed separately.

Fixes #2923

Checklist:

  • Documentation has been updated, if necessary.
  • Unit tests have been added, if necessary.
  • E2E tests have been added, if necessary.
  • Integration tests have been added, if necessary.

@metal3-io-bot metal3-io-bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 30, 2026
@metal3-io-bot
Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign rozzii for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@metal3-io-bot metal3-io-bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jan 30, 2026
@metal3-io-bot
Copy link
Copy Markdown
Contributor

Hi @CrystalChun. Thanks for your PR.

I'm waiting for a metal3-io member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@metal3-io-bot metal3-io-bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Jan 30, 2026
@dtantsur
Copy link
Copy Markdown
Member

dtantsur commented Feb 2, 2026

/ok-to-test

@metal3-io-bot metal3-io-bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Feb 2, 2026
@dtantsur
Copy link
Copy Markdown
Member

dtantsur commented Feb 2, 2026

@CrystalChun while I'm testing something locally, could you squash your commits and remove references to RH Jira from the messages?

@CrystalChun CrystalChun changed the title ✨ Allow forced detachment of a host from Ironic WIP: ✨ Allow forced detachment of a host from Ironic Feb 2, 2026
@CrystalChun
Copy link
Copy Markdown
Author

@CrystalChun while I'm testing something locally, could you squash your commits and remove references to RH Jira from the messages?

Yes, squashed my commits and removed the RH Jira mentions from the commit message and the description of this PR!

@metal3-io-bot metal3-io-bot added the needs-rebase Indicates that a PR cannot be merged because it has merge conflicts with HEAD. label Feb 19, 2026
@metal3-io-bot metal3-io-bot removed the needs-rebase Indicates that a PR cannot be merged because it has merge conflicts with HEAD. label Feb 23, 2026
@metal3-io-bot metal3-io-bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Mar 2, 2026
@dtantsur
Copy link
Copy Markdown
Member

dtantsur commented Mar 2, 2026

@CrystalChun I've provided a draft implementation, needs testing now (also together with soon-to-be-merged Ironic change https://review.opendev.org/c/openstack/ironic/+/973279)

@dtantsur dtantsur marked this pull request as ready for review March 4, 2026 15:48
@dtantsur dtantsur changed the title WIP: ✨ Allow forced detachment of a host from Ironic ✨ Allow forced detachment of a host from Ironic Mar 6, 2026
@metal3-io-bot metal3-io-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 6, 2026
@dtantsur dtantsur force-pushed the force-detachment branch 3 times, most recently from 72ec11b to e086f2b Compare March 6, 2026 15:43
@dtantsur dtantsur force-pushed the force-detachment branch 3 times, most recently from 611558f to 13bb2e2 Compare March 11, 2026 13:50
@dtantsur
Copy link
Copy Markdown
Member

/hold

Despite 2 seconds sleep, one of the tests still did not reach "deploying" before forced detachment was applied.

@metal3-io-bot metal3-io-bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 11, 2026
@metal3-io-bot metal3-io-bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Mar 12, 2026
@dtantsur
Copy link
Copy Markdown
Member

/hold cancel

I had to drop the e2e tests for now to avoid blocking the entire feature. There are signs that they would pass, but they're timing-dependent, which is pretty bad. I'll propose them separately.

@metal3-io-bot metal3-io-bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 12, 2026
@Rozzii
Copy link
Copy Markdown
Member

Rozzii commented Mar 12, 2026

I will take a look soon, for some reason this PR have not received the 2reviewer automatically.
/cc @Rozzii

@metal3-io-bot metal3-io-bot requested a review from Rozzii March 12, 2026 11:14
Comment thread internal/controller/metal3.io/host_state_machine.go
@Rozzii Rozzii moved this to BMO WIP in Metal3 - Roadmap Mar 30, 2026
@Rozzii
Copy link
Copy Markdown
Member

Rozzii commented Apr 8, 2026

In my review queue now, I would like to merge this before 0.13 release @Sunnatillo @smoshiur1237 please keep an eye on this PR.

@Rozzii Rozzii added this to the BMO - v0.13 milestone Apr 8, 2026
@dtantsur dtantsur mentioned this pull request Apr 10, 2026
4 tasks
@Rozzii Rozzii requested a review from Copilot April 14, 2026 11:10
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds support for forced BareMetalHost detachment so operators can request immediate detachment even when the host is mid-operation, with Ironic-side logic attempting to abort in-progress workflows when possible.

Changes:

  • Extend the Provisioner interface and controller detachment flow to accept a force boolean.
  • Implement forced-detach behavior in the Ironic provisioner by reusing deletion logic and issuing abort/fallback transitions depending on Ironic microversion support.
  • Add unit tests for Ironic forced-detach behavior and introduce an API annotation argument (force) to trigger forced detachment.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
pkg/provisioner/provisioner.go Updates the provisioner interface to accept a force flag for detachment.
pkg/provisioner/ironic/ironic.go Refactors deletion into realDelete(..., force) and wires Detach(..., force) to use it, including abort/deprovision logic by state.
pkg/provisioner/ironic/delete_test.go Adds unit test coverage for forced detachment state handling and updates existing detach calls for the new signature.
pkg/provisioner/ironic/clients/features.go Adds detection for deployment-abort support and selects microversion 1.110 when available.
pkg/provisioner/fixture/fixture.go Updates fixture provisioner to implement the new Detach(ctx, force) signature.
pkg/provisioner/demo/demo.go Updates demo provisioner to implement the new Detach(ctx, force) signature.
internal/controller/metal3.io/host_state_machine_test.go Updates mock provisioner to match the new Detach(ctx, force) signature.
internal/controller/metal3.io/host_state_machine.go Enables forced detach via detached annotation JSON args and adjusts delete initiation behavior for detached hosts.
internal/controller/metal3.io/baremetalhost_controller.go Plumbs the force flag through reconciler detachment handling.
apis/metal3.io/v1alpha1/baremetalhost_types.go Adds force to DetachedAnnotationArguments for annotation-driven forced detachment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread internal/controller/metal3.io/host_state_machine.go Outdated
Comment thread pkg/provisioner/ironic/clients/features.go
Comment thread pkg/provisioner/ironic/ironic.go Outdated
Comment thread pkg/provisioner/ironic/delete_test.go
Comment on lines 305 to +316
if hasDetachedAnnotation(hsm.Host) {
// Only allow detaching hosts in Provisioned/ExternallyProvisioned/Ready/Available states
// Only allow detaching hosts in Provisioned/ExternallyProvisioned/Ready/Available states unless forced
switch info.host.Status.Provisioning.State {
case metal3api.StateProvisioned, metal3api.StateExternallyProvisioned, metal3api.StateReady, metal3api.StateAvailable:
return hsm.Reconciler.detachHost(ctx, hsm.Provisioner, info)
return hsm.Reconciler.detachHost(ctx, hsm.Provisioner, info, false)
case metal3api.StateDeleting:
// No point in detaching a host that is being deleted already
default:
if hasForceDetachAnnotation(hsm.Host) {
info.log.Info("forcing detach of host", "provisioningState", info.host.Status.Provisioning.State)
return hsm.Reconciler.detachHost(ctx, hsm.Provisioner, info, true)
}
Copy link

Copilot AI Apr 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new forced-detach path (hasForceDetachAnnotation + passing force=true into detachHost) is not covered by the existing host state machine unit tests. Add a unit test that sets the detached annotation payload to JSON with {"force":true} while the host is in a non-detachable provisioning state (e.g. StateProvisioning) and asserts the reconciler calls Provisioner.Detach(..., true) and transitions OperationalStatus to Detached appropriately.

Copilot uses AI. Check for mistakes.
@metal3-io-bot metal3-io-bot added the needs-rebase Indicates that a PR cannot be merged because it has merge conflicts with HEAD. label Apr 17, 2026
CrystalChun and others added 3 commits April 21, 2026 16:03
Adds Force as DetachedAnnotationArgument to API.
Detach in all states if force is set to true by aborting the currently
running process.

Co-Authored-By: Dmitry Tantsur <dtantsur@protonmail.com>
Signed-off-by: CrystalChun <cchun@redhat.com>
Signed-off-by: Dmitry Tantsur <dtantsur@protonmail.com>
The deletion logic does not need to be aware of which states are valid
for detachment. If the operationalStatus is detached, the host can be
deleted immediately.

Signed-off-by: Dmitry Tantsur <dtantsur@protonmail.com>
Signed-off-by: Dmitry Tantsur <dtantsur@protonmail.com>
@metal3-io-bot metal3-io-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed needs-rebase Indicates that a PR cannot be merged because it has merge conflicts with HEAD. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Apr 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

Status: BMO WIP

Development

Successfully merging this pull request may close these issues.

[RFE] Forced detachment of a host from Ironic

6 participants