One failed PXE exchange is enough to hang a machine permanently. It takes a power cycle by hand to recover, and nothing in kezio reports why.
What happens
UEFI firmware that does not complete PXE moves to the next entry in its BootOrder. On OVMF the auto-created entries run PXEv4, PXEv6, HTTPv4, HTTPv6 - so the machine's next attempt is UEFI HTTP Boot, and it identifies itself accordingly:
vendor class: HTTPClient:Arch:00016:UNDI:003001
bootd answers that with the options it has for PXE:
bootfile name: shimx64.efi
next server: 192.0.2.2
An HTTP Boot client cannot use a TFTP server and a bare filename - it wants a URL. So it discards the offer, sends another DHCPDISCOVER, gets the same answer, and loops. It never returns to PXE on its own, because firmware only walks BootOrder forward.
Observed
Deploying 30 machines at once, 4 of 30 (13%) ended up in this loop. All four showed the same pattern, here for one of them across its last four exchanges:
4 DHCPDISCOVER
4 DHCPOFFER
4 vendor class: HTTPClient:Arch:00016:UNDI:003001
4 bootfile name: shimx64.efi
They sat in Inspecting indefinitely - the agent never booted, so nothing registered. Clearing and setting spec.online (a cold restart, which sends firmware back to the first BootOrder entry) recovered all four.
At ten machines this hit one machine; at thirty it hit four. It tracks how many machines contend for the same DHCP exchange, which is exactly the situation a site-wide rollout creates.
Why this is worth fixing rather than avoiding
Staggering the deploys makes it rarer, which is the wrong shape of fix: the trigger is a lost or slow first exchange, and a busy segment, a slow switch port or a retransmit will do it. The consequence is not a slow boot but a machine that waits for a human forever.
The pieces already exist
config/bootserver/README.md documents BOOT_EFI_DIR and GET /boot/http/<name>, which serve the signed shimx64.efi/grubx64.efi over HTTP, for UEFI HTTP Boot, and boot-artifacts-init-patch.yaml already fetches both binaries into that directory. What is missing is the DHCP side: bootd never tells an HTTP Boot client where that is.
dnsmasq can answer both from one config by matching the client architecture:
dhcp-match=set:httpclient,option:client-arch,16
dhcp-boot=tag:httpclient,http://<bootd-address>/boot/http/shimx64.efi
Done this way the fallback stops being a dead end and becomes what firmware intends it to be - a second path that works. That is strictly better than making the first path fail less often.
Notes
- Seen in lease mode (
Subnet.spec.dhcp.mode: lease). Proxy mode needs checking too: config/bootd/README.md already records that dnsmasq's proxyDHCP engine only answers PXEClient, so the same client gets nothing there.
- Arch 16 is x64 UEFI HTTP; arch 15 is x64 UEFI HTTP with an alternate encoding on some firmware, and a future non-x86-64 architecture would add its own, the same way
dhcp-match on option 93 already handles the TFTP side.
- Recovering by hand: clear
spec.online, wait for the machine to stop, set it again.
One failed PXE exchange is enough to hang a machine permanently. It takes a power cycle by hand to recover, and nothing in kezio reports why.
What happens
UEFI firmware that does not complete PXE moves to the next entry in its BootOrder. On OVMF the auto-created entries run PXEv4, PXEv6, HTTPv4, HTTPv6 - so the machine's next attempt is UEFI HTTP Boot, and it identifies itself accordingly:
bootd answers that with the options it has for PXE:
An HTTP Boot client cannot use a TFTP server and a bare filename - it wants a URL. So it discards the offer, sends another DHCPDISCOVER, gets the same answer, and loops. It never returns to PXE on its own, because firmware only walks BootOrder forward.
Observed
Deploying 30 machines at once, 4 of 30 (13%) ended up in this loop. All four showed the same pattern, here for one of them across its last four exchanges:
They sat in
Inspectingindefinitely - the agent never booted, so nothing registered. Clearing and settingspec.online(a cold restart, which sends firmware back to the first BootOrder entry) recovered all four.At ten machines this hit one machine; at thirty it hit four. It tracks how many machines contend for the same DHCP exchange, which is exactly the situation a site-wide rollout creates.
Why this is worth fixing rather than avoiding
Staggering the deploys makes it rarer, which is the wrong shape of fix: the trigger is a lost or slow first exchange, and a busy segment, a slow switch port or a retransmit will do it. The consequence is not a slow boot but a machine that waits for a human forever.
The pieces already exist
config/bootserver/README.mddocumentsBOOT_EFI_DIRandGET /boot/http/<name>, which serve the signedshimx64.efi/grubx64.efiover HTTP, for UEFI HTTP Boot, andboot-artifacts-init-patch.yamlalready fetches both binaries into that directory. What is missing is the DHCP side: bootd never tells an HTTP Boot client where that is.dnsmasq can answer both from one config by matching the client architecture:
Done this way the fallback stops being a dead end and becomes what firmware intends it to be - a second path that works. That is strictly better than making the first path fail less often.
Notes
Subnet.spec.dhcp.mode: lease). Proxy mode needs checking too:config/bootd/README.mdalready records that dnsmasq's proxyDHCP engine only answersPXEClient, so the same client gets nothing there.dhcp-matchon option 93 already handles the TFTP side.spec.online, wait for the machine to stop, set it again.