Non-deterministic static IP assignment failure in DHCP-less IPA boot

[aydinhus]

Environment

• Component: Metal3 / Ironic (IPA - Ironic Python Agent)
• Network configuration method: preprovisioningNetworkDataName (static IP, no DHCP)
• Reproduced on: Multiple physically separate environments, different deployments, and different hardware vendors

---

Problem Description

In a DHCP-less environment, static network parameters are passed via preprovisioningNetworkDataName. Both the inspection and provisioning phases perform their own IPA boot, and the failure can occur independently in either or both — with no correlation between them. A successful inspection does not predict a successful provisioning boot, and vice versa.

The failure is non-deterministic: retrying with identical configuration can succeed or fail unpredictably on each attempt.

---

Steps to Reproduce

1. Set up a DHCP-less environment.
2. Configure preprovisioningNetworkDataName with a valid static network secret (IP, gateway, DNS, etc.).
3. Trigger provisioning and observe IPA boot logs across multiple attempts for both phases.
4. Observe that static network configuration is applied inconsistently, independently in each phase, with no apparent pattern.

---

Kernel Parameters Attempted

The following extraKernelParams were tested in various combinations:

ipa-inspection-dhcp-all-interfaces=0
ipa-inspection-dhcp-wait-timeout=0
ipa-hardware-initialization-delay=30

None produced consistent results.

---

Workaround

Shortened relevant timeout values to ~6 minutes to allow faster retry cycles. Does not address the root cause.

---

Expected Behavior

Static network configuration should be reliably applied on every IPA boot — in both inspection and provisioning phases.

---

Actual Behavior

Static network configuration is applied intermittently on a per-boot basis. The per-phase independence of failures suggests a race condition or timing issue within the IPA network initialization sequence, rather than an issue with how network data is passed from Metal3/Ironic.

---

Additional Context

• Observed across multiple physically separate environments, different deployments, and different hardware vendors — ruling out environment- or hardware-specific causes.
• Any insights into the ordering or timing of network configuration application during IPA boot would be appreciated.

[metal3-io-bot]

This issue is currently awaiting triage.
If Metal3.io contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.
The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[dtantsur]

We'll really need full ramdisk logs (including glean, networkmanager, etc) for both a successful and a failed run to triage. Would you be able to provide them?

/triage needs-information

[aydinhus]

Thanks for the response. I'd like to collect these logs, but I'm hitting a challenge in the failure case:

• Ironic's automatic log collection will likely only capture timeout errors since the ramdisk never reaches back to Ironic.
• I can inject an SSH key, but in the failure case the node never gets an IP, so SSH won't be reachable. Is there a way to inject a user into the ramdisk without building a custom image, so I can log in via console? Or do you have a recommended approach for collecting ramdisk logs when the node fails to get an IP?

I can see logs streaming on the console during the boot. I think capturing them fully via IPMI SOL with ipa-debug=1 should be doable — would that be useful for triage?

[dtantsur]

Is there a way to inject a user into the ramdisk without building a custom image

I'm afraid you'll need a custom image: https://docs.openstack.org/diskimage-builder/latest/elements/devuser/README.html

[dtantsur]

I'm putting the topic of debugging IPA without networking on the agenda for the next Ironic team gathering in October: https://etherpad.opendev.org/p/ironic-ptg-2027.1