ARM64 instance fails to cleanup properly on KVM-based ARM hosts

[rohityadavcloud]

problem

When expunging KVM arm64 instances, sometimes it fails to remove the domain on the host or at least the nvram/fd is not removed.

versions

ACS 4.20.0, on RPi5/ARM64 hosts

The steps to reproduce the bug

1. Create a arm64 instance
2. The instance shows up as running, for example i-2-961-VM in the example below:

root@armpi:/home/rohit# virsh list --all
 Id   Name         State
----------------------------
 1    i-2-938-VM   running
 2    i-2-961-VM   running

3. Next destroy and expunge the instance from CloudStack. The instance is expunged from CloudStack with its qcow2 disk gone, however on the KVM host the instance can sometimes be in shut off state:

root@armpi:/home/rohit# virsh list --all
 Id   Name         State
-----------------------------
 1    i-2-938-VM   running
 -    i-2-961-VM   shut off

4. On manual undefining the following error is seen sometimes:

root@armpi:/home/rohit# virsh undefine i-2-961-VM
error: Failed to undefine domain 'i-2-961-VM'
error: Requested operation is not valid: cannot undefine domain with nvram

Workaround: edit/remove the nvram section in the domain and manually delete the nvram file.

5. The nvram file needs to be always manually cleaned up on the arm64 host:

root@armpi:/home/rohit# ls -lahi /var/lib/libvirt/qemu/nvram/
total 193M
278646 drwxr-xr-x  2 root root 4.0K Apr 30 12:06 .
275497 drwxr-x--- 10 root root 4.0K Apr 30 12:06 ..
275244 -rw-------  1 root root  64M Apr 28 18:06 i-2-938-VM_VARS.fd
278831 -rw-------  1 root root  64M Feb 12 22:15 i-2-939-VM_VARS.fd
275251 -rw-------  1 root root  64M Apr 30 12:06 i-2-961-VM_VARS.fd
root@armpi:/home/rohit# rm /var/lib/libvirt/qemu/nvram/i-2-961-VM_VARS.fd

What to do about it?

ARM64 destroy/expunge should properly cleanup all additional files and configs (such as the nvram fd).