VM Fails to Boot After bootc Upgrade

[repires]

Issue Summary

After performing a system upgrade, the Dakota VM fails to boot into the OS. The VM was previously functional and booting correctly from both the ISO and the installed vdisk. Post-upgrade, the boot process stalls or fails to reach the OS login/GUI.

sudo bootc status

  Staged image: ghcr.io/projectbluefin/dakota:latest
        Digest: sha256:a0b53a56ce627a8df3c4d81942c1ae471b9099c9514f7eb9786ead346bd29c24 (amd64)
        Verity: 4cfb1209daef19e7b521d0780bc920c1dad569960a2688e89b067641f0b36d9d92398fcdbde328e80e87a112045df34e9fe24109f5d274e22071a64daf2797b0
     Timestamp: 2026-05-17T13:44:31Z

● Booted image: ghcr.io/projectbluefin/dakota:latest
        Digest: sha256:d314abc2c1b643db0f412b778700dd1325f83b43781be02da90fbbf52497bf6e (amd64)
        Verity: 32db235d8c08f95a28e188acb4bf81a91070b3ef520d74cc642c125dc63c2bd6fd8009d9596dfba0724bf49e0e042abe2c457735df06e2dc76fd079085390aa5
     Timestamp: 2026-04-20T03:37:39Z

---

Environment Information

• Host OS: Unraid 7.3.0 (Kernel 6.18.29)
• Libvirt Version: 12.2.0
• QEMU Version: 10.2.2
• Firmware: OVMF (pure EFI)
• Machine Type: pc-q35-9.2

---

CPU Info Summary

• Model: 2x Intel(R) Xeon(R) CPU E5-2667 (Sandy Bridge-EP)
• Cores/Threads: 12 Cores / 24 Threads total
• Microarchitecture Level: x86-64-v2 (+ AVX)
• v3 Support: ❌ Missing (No avx2, bmi1, bmi2, fma, or movbe)

---

Storage & Persistence Context

• VDisk Path: ZFS dataset (/mnt/user/domains/Dakota/vdisk1.img) via Unraid User Shares
• NVRAM Path: /var/lib/libvirt/qemu/nvram/ (stored on Unraid RAM disk)

---

Comparison Logs

Log: Installed and Working VM (Pre-Upgrade / Initial State)

1 -device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.2","addr":"0x0"}' \
2 -blockdev
  '{"driver":"file","filename":"/mnt/user/domains/Dakota/vdisk1.img","node-name":"libvirt-3-storage","read-only":false,"discard":"unmap","cache":{"direct":false,"no-flush":false}}' \
3 -device
  '{"driver":"virtio-blk-pci","bus":"pci.3","addr":"0x0","drive":"libvirt-3-storage","id":"virtio-disk2","bootindex":2,"write-cache":"on","serial":"vdisk1"}' \
4 -blockdev '{"driver":"file","filename":"/mnt/disk1/isos/dakota-live-latest-tunadownload.iso","node-name":"libvirt-2-storage","read-only":true}' \
5 -device '{"driver":"ide-cd","bus":"ide.0","drive":"libvirt-2-storage","id":"sata0-0-0","bootindex":1}' \
6 -device '{"driver":"ide-cd","bus":"ide.1","id":"sata0-0-1"}' \
7 ...
8 2026-05-18 07:11:39.918+0000: Domain id=10 is tainted: high-privileges
9 char device redirected to /dev/pts/6 (label charserial0)

Log: VM Does Not Boot After Upgrade (Failure State)

1 2026-05-18 08:11:59.072+0000: starting up libvirt version: 12.2.0, qemu version: 10.2.2, kernel: 6.18.29-Unraid, hostname: Matrix
2 LC_ALL=C \
3 PATH=/bin:/sbin:/usr/bin:/usr/sbin \
4 HOME=/var/lib/libvirt/qemu/domain-12-Dakota \
5 ...
6 -blockdev
   '{"driver":"file","filename":"/mnt/user/domains/Dakota/vdisk1.img","node-name":"libvirt-3-storage","read-only":false,"discard":"unmap","cache":{"direct":false,"no-flush":false}}' \
7 -device
   '{"driver":"virtio-blk-pci","bus":"pci.3","addr":"0x0","drive":"libvirt-3-storage","id":"virtio-disk2","bootindex":2,"write-cache":"on","serial":"vdisk1"}' \
8 -device '{"driver":"ide-cd","bus":"ide.0","id":"sata0-0-0","bootindex":1}' \
9 -device '{"driver":"ide-cd","bus":"ide.1","id":"sata0-0-1"}' \
10 ...
11 2026-05-18 08:11:59.072+0000: Domain id=12 is tainted: high-privileges
12 char device redirected to /dev/pts/5 (label charserial0)

---

Key Difference Observed

In the working state, the ISO CD-ROM device includes an attached backing file:

-drive":"libvirt-2-storage"

and the corresponding block device definition:

-blockdev '{"driver":"file","filename":"/mnt/disk1/isos/dakota-live-latest-tunadownload.iso"

In the failure state, the IDE CD device is still present, but the ISO backing file definition is missing entirely.

This may indicate:

• lost EFI boot entry persistence,
• invalid or reset OVMF NVRAM state,
• fallback bootloader failure,
• or dependency on the attached ISO for bootloader discovery after the upgrade.

---

Reproduction Steps

1. Boot the Dakota VM using the latest available ISO.
2. Execute a bootc upgrade within the guest OS.
3. Reboot the VM.
4. Observe that the VM no longer reaches the OS login/GUI despite the VM XML and virtual hardware remaining unchanged.

Additional Notes

• The VM previously booted successfully from both:

  • the installer/live ISO, and
  • the installed vdisk.

• The issue only appeared after the upgrade/reboot cycle.

• The NVRAM file resides on Unraid's RAM-backed filesystem, which may be relevant if EFI boot entries are not persisting across VM lifecycle events.

vm-installed-does-not-boot.log
vm-installed-working.log