automatedCleaning : add Ironic erase_devices and erase_devices_express

[jaxklag]

Hello,

The metal3 automatedcleaning documentation indicates that erase_devices is still not implemented :

Note: Ironic supports full data removal, which is not currently exposed in Metal3.

Would it be possible to implement the "erase_devices_express" and even "erase_devices" provided by Ironic ?

As to me erase_devices_express is the best compromise but erase_devices still need to be implemented to let the choice to the user the way he wants to wipe disks.
Also if data disks do not support hardware assisted erasure.

As asked by @alanmeadows, be able to choose which disks (those who don't support hardware assisted erasure) will be wiped in each modes would be very usefull to.

Regards,

Originally posted by @jaxklag in #32

The BMH resources could be :

Automated cleaning

apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: example-host
spec:
  automatedCleaningMode: express | deep | metadata
  bootMACAddress: 00:8a:b6:8e:ac:b8
  bmc:
    address: ipmi://192.168.111.1:6230
    credentialsName: example-node-bmc-secret
  online: true

• automatedCleaningMode: express # for erase_devices_express in automated cleaning
• automatedCleaningMode: deep # for erase_devices in automated cleaning

Manual cleaning

manualCleaningMode with capability to use the 3 erasure mode with their priority and if possible the disks associated which each erasure mode.

This lets the user choose which modes he wants to execute thanks to priority and on which disks.

apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: example-host
spec:
  manualCleaningMode:
     express:
         priority: 95
     deep:
         priority: 80
         data-disks: # list of data disk which do not support hardware assisted erasure. Avoid to clean twice data disk which support hardware assisted erasure.
             - disk id 1
             - disk id 2
      metadata:
         priority: 0 # allow to disable metadata cleaning to avoid disk to be cleaned twice. express fall back to metadata if disk does not support hardware assisted erasure
  bootMACAddress: 00:8a:b6:8e:ac:b8
  bmc:
    address: ipmi://192.168.111.1:6230
    credentialsName: example-node-bmc-secret
  online: true

[dtantsur]

The automated cleaning part looks like a valid feature request to me. We may need to extend Ironic API to be able to configure this per Node rather than globally.

I'm also fully behind supporting a list of disks not to clean.

I'm rather unconvinced by the manual cleaning part. The proposed design does not seem very consistent with how other manual cleaning actions are done. I'm also not quite sure what you're trying to achieve here. The manual cleaning part will definitely need someone to write a design proposal.

[dtantsur]

automatedCleaningMode: express # for erase_devices_express in automated cleaning
automatedCleaningMode: deep # for erase_devices in automated cleaning

I vote for calling them expressOnly (to clarify that fully secure erase may not happen) and full (deep is fine but I usually hear people say "full erasure" in this context).

[Rozzii]

/triage accepted
So half of this proposal looks fine as @dtantsur mentioned but we will need some clarification on the manual cleaing and also a proper design doc.
There was also an other proposal about adding yet an other cleaning mode #2589 so please make sure that you take this into consideration also.

[jaxklag]

automatedCleaningMode: express # for erase_devices_express in automated cleaning
automatedCleaningMode: deep # for erase_devices in automated cleaning

I vote for calling them expressOnly (to clarify that fully secure erase may not happen) and full (deep is fine but I usually hear people say "full erasure" in this context).

Like you want ;)

It doesn't matter what name is used, as long as it's clearly explained in the documentation.

[jaxklag]

/triage accepted So half of this proposal looks fine as @dtantsur mentioned but we will need some clarification on the manual cleaing and also a proper design doc.
There was also an other proposal about adding yet an other cleaning mode #2589 so please make sure that you take this into consideration also.

I'm going to "try" to explain better and deeper.

I misunderstood the Ironic cleaning documentation.
I thought i must use manual cleaning to provide some advanced cleaning options like priorities, data disk to clean etc ...
This is why i proposed the manulaCleaning definition.

I found that in automated cleaning steps, we could use runbooks to have more granularity in the cleaning process.
And even use "hybrid" mode using runbooks if defined and automatedcleaining else.

See Ironic different cleaning steps
Configuring automated cleaning with runbooks

Hybrid (‘hybrid’)

This uses a runbook-based cleaning method if a cleaning runbook is configured for the node being cleaned. In this mode, if there is not a runbook configured for cleaning Ironic will fall-back to autogenerating cleaning steps.

Starting with the 2025.2/Flamingo release, operators can configure Ironic to use runbooks for automated cleaning instead of relying on autogenerated steps. This provides more control over the cleaning process and ensures consistency across nodes.

In fact, the capabilities described in manual cleaning could be implemented into automated cleaning using runbooks to offer more options to the user.

A point to consider before going forward :

Ironic documentation indicates that using runbook, execute only steps defined into the runbook. So if BMO use runbooks to enhance automated cleaning capabilities, what about other cleaning tasks like raid config, bios config and also all task automatically added by Ironic in the automatic mode ?

If runbook are not incompatible the way how BMO works today,

I think the data model must be changed to offer all the capabilities provided by Ironic cleaning.
Since an option to exclude some data disks is used, runbook are required and cleaning process become an advanced cleaning process.

I don't know how the data disk to include or exclude are transmitted to IPA for in-band cleaning.
Moreover, i don't know how ironic ipa is able to cummunicate directrly with data disks to cryptographically erase data.

To support hardware-assisted erase, your IPA must include a hardware manager that knows how to invoke secure erase commands (e.g., NVMe sanitize, ATA SECURITY ERASE).

For out-of-band cleaning, i don't find in the spec how to indicate Ironic interface the data disk we want to exclude or include.

What i know is that Redfish Idrac allow to specify data disk for cryptographic erasure. One API call for each data disk.

curl -skL -u user:pass -X POST https://10.9.0.4:443/redfish/v1/Systems/System.Embedded.1/Storage/BOSS.SL.12-1/Drives/Disk.Direct.0-0:BOSS.SL.12-1/Actions/Drive.SecureErase -H "Content-Type: application/json" -d '{}'

But for hardware assisted erasure (it's the disk drive which wipes its data) for compatible drives, iDrac redfish doesn't allow to specify data disks.

curl -skL -u user:pass -X POST https://10.9.0.4:443/redfish/v1/Dell/Managers/iDRAC.Embedded.1/DellLCService/Actions/DellLCService.SystemErase -H "Content-Type: application/json" -d '{"Component": ["OverwritePD"]}'

# Storage Components
"OverwritePD" – Overwrites data on non-encrypted drives.
"CryptographicErasePD" – Cryptographic erase for SED/ISE/NVMe/NVDIMM.
"PERCNVCache" – Clears PERC controller cache.
"vFlash" – Erases vFlash partitions.
"NonVolatileMemory" – Erases NVDIMM data.

1 - Choose the Ironic cleaning steps. There are 3 steps available. The key could be "automatedCleaningStep" of type "string". This steps is transmited to Ironic in the API call.

• autogenerated
• runbook (be carefull. If no runbook is provided to Ironic, cleaning process fails)
• hybrid (could be the default value and makes "automatedCleaningStep key" optional for the user.

2 - Choose their storage cleaning options defined in a key named "automatedCleaningOptions" of type "list of dict". Mandatory key.
Options are the cleaning mode

• erase_devices_metadata
• erase_devices
• erase_devices_express # use hardwaare assisted erasure if drive support it
• crypto # delete the encryption key for supported drives via the BMC interface of the baremetal server
• disabled (this value doesn't exist into Ironic. Equivalent ot Metal3 automatedCleaningMode: disabled) Could be set in another key.

Priority-based ordering does not apply with runbooks; steps execute in the order specified in the runbook.

So priority is useless with runbooks ! Usefull for manual cleaning but we don't need manual cleaning.

As Ironic documentation, "crypto" value could be equivalent to this cleaning step for Ironic

# erase_pattern – Dictionary of disk type and corresponding erase pattern to be used to perform specific out-of-band sanitize disk erase. Supported values are, for “hdd”: (“overwrite”, “crypto”, “zero”), for “ssd”: (“block”, “crypto”, “zero”). Default pattern is: {“hdd”: “overwrite”, “ssd”: “block”}.

[
  {
    "interface": "management",
    "step": "erase_devices",
    "args": {
      "erase_pattern": {
        "hdd": "overwrite",
        "ssd": "crypto"
      }
    }
  }
]

Data disk hints

It could be great to have more hints to be able to detect if drives are compatible with :

• Hardware assisted erasure
• SED (Self Encryption Drives) => Data encryption is done at rest by the storage controller and the disk drive (Both must be compatible.) For exemple Perc Storage controller and some SED compatible disk drives. Encryption is transparent for the customer and no encryption config is needed on the OS. Additionnal Benefit to offload encryption on the storage controller and not on the CPU.

A drive can be SED compatible but not hardware assisted erasure compatible.
So if it's SED, simply erase the cryptographic key to make the data unavailable.
If not compatible SED or if user also wants to erase data on SED disks, use hardware assisted erasure to wipe the data in only a few minutes for several To. Wipe also some disk area and disk caches unavailable at the OS layer. More secure.
Else, use traditionnal process to securely erase the data rewriting data on disk.

Here are some use cases :

automatedcleaning using Autogenerated cleaning steps from Ironic.

All disk are selected. No way to exclude some of them since autogenerated cleaning steps.

Steps are collected from hardware interfaces and ordered from higher to lower priority, where a larger integer is a higher priority. In case of a conflict between priorities across interfaces, the following resolution order is used: Power, Management, Deploy, BIOS, and RAID interfaces.

apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: example-host
spec:
  automatedCleaningStep: autogenerated |hybrid => It's equivalent since no advanced cleaning options defined
  automatedCleaningOptions:
    - storageCleaningOptions: expressOnly | full | metadata

automatedcleaning using hybrid cleaning steps from Ironic.

This uses a runbook-based cleaning method if a cleaning runbook is configured for the node being cleaned. In this mode, if there is not a runbook configured for cleaning Ironic will fall-back to autogenerating cleaning steps..

This mode allows :

• to use several cleaning modes.
• optionnaly define the disks to be included or excluded for that mode (how to do that ?). Hints could be use to select data disks. Some limitation with out-of-band cleaning.
• to deactivate some cleaning mode. Using runbook, only defined steps are executed. So to disable metadata cleaning, do not specify it.

apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: example-host
spec:
  automatedCleaningStep: hybrid
  automatedCleaningOptions:
    - storageCleaningOptions: crypto # delete the cryptographic key used by storage controllers and disk drive to encrypt data (out-of-band only)
       DeviceHints:
          SED: true
    - storageCleaningOptions: expressOnly # use hardware assisted data erasure else metadata erasure
       DeviceHints:
          hardwareAssistedErasure: true
          SED: false # expressOnly do not delete the encryption key. Must use storageCleaningOptions: crypto. Valuer could also be "true" if user wants to delete data on disk even for SED drives.
    - storageCleaningOptions: full # rewrite data on disk for non SED and non hardware assietd compatible drives.
       DeviceHints:
          hardwareAssistedErasure: false
          SED: false

Other proposal : #2589

I think it's not possible to exclude some disk. So disk must be explicitely defined via hints.
After the server was provisionned and initially cleaned, user can change its resource like below :

apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: example-host
spec:
  automatedCleaningStep: hybrid
  automatedCleaningOptions:
    - storageCleaningOptions: metadata
       DeviceHints:
         wwn: "www disk1" # Only wipe OS disk. Not data disks
    - storageCleaningOptions: metadata
       DeviceHints:
         wwn: "www disk2" # Only wipe OS disk. Not data disks