Palo Alto PAN VM container: Unable to connect to qemu monitor (port 4000)

[arturas-dot]

After creating docker image I have issues running the container - it crashes with the following log:

2025-03-15 12:35:14,498: launch         DEBUG Environment variables: environ({'HOME': '/root', 'HOSTNAME': 'cbc7435abd5b', 'PATH': '/.venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'VIRTUAL_ENV': '/.venv', 'LC_CTYPE': 'C.UTF-8'})
2025-03-15 12:35:14,498: vrnetlab       DEBUG class: PAN_vm, disk_image: /PA-VM-KVM-10.2.5.qcow2, overlay: /PA-VM-KVM-10.2.5-overlay.qcow2
2025-03-15 12:35:14,498: vrnetlab       DEBUG Creating overlay disk image
2025-03-15 12:35:14,507: vrnetlab       DEBUG Starting vrnetlab PAN
2025-03-15 12:35:14,508: vrnetlab       DEBUG VMs: [<__main__.PAN_vm object at 0x7f78471000b0>]
2025-03-15 12:35:14,508: vrnetlab       DEBUG VM not started; starting!
2025-03-15 12:35:14,508: vrnetlab       INFO ----------------START ENVIRONMENT VARIABLES-----------------
2025-03-15 12:35:14,508: vrnetlab       INFO HOME: /root
2025-03-15 12:35:14,508: vrnetlab       INFO HOSTNAME: cbc7435abd5b
2025-03-15 12:35:14,508: vrnetlab       INFO LC_CTYPE: C.UTF-8
2025-03-15 12:35:14,508: vrnetlab       INFO PATH: /.venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2025-03-15 12:35:14,508: vrnetlab       INFO VIRTUAL_ENV: /.venv
2025-03-15 12:35:14,508: vrnetlab       INFO -----------------END ENVIRONMENT VARIABLES------------------
2025-03-15 12:35:14,508: vrnetlab       INFO Launching PAN_vm with 2,sockets=1,cores=1 SMP/VCPU and 6144 M of RAM
2025-03-15 12:35:14,508: vrnetlab       INFO Scrapli: Disabled
2025-03-15 12:35:14,508: vrnetlab       INFO Transparent mgmt interface: Disabled
2025-03-15 12:35:14,508: vrnetlab       DEBUG number of provisioned data plane interfaces is 0
2025-03-15 12:35:14,508: vrnetlab       DEBUG qemu cmd: qemu-system-x86_64 -enable-kvm -display none -machine pc -monitor tcp:0.0.0.0:4000,server,nowait -serial telnet:0.0.0.0:5000,server,nowait -m 6144 -cpu host,level=9 -smp 2,sockets=1,cores=1 -drive if=ide,file=/PA-VM-KVM-10.2.5-overlay.qcow2 -uuid aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa -device pci-bridge,chassis_nr=1,id=pci.1 -device virtio-net-pci,netdev=p00,mac=0C:00:e3:51:15:00 -netdev user,id=p00,net=10.0.0.0/24,host=10.0.0.2,dns=10.0.0.3,dhcpstart=10.0.0.15,hostfwd=tcp:0.0.0.0:22-10.0.0.15:22,hostfwd=udp:0.0.0.0:161-10.0.0.15:161,hostfwd=tcp:0.0.0.0:80-10.0.0.15:80,hostfwd=tcp:0.0.0.0:443-10.0.0.15:443,hostfwd=tcp:0.0.0.0:830-10.0.0.15:830,hostfwd=tcp:0.0.0.0:6030-10.0.0.15:6030,hostfwd=tcp:0.0.0.0:8080-10.0.0.15:8080,hostfwd=tcp:0.0.0.0:9339-10.0.0.15:9339,hostfwd=tcp:0.0.0.0:32767-10.0.0.15:32767,hostfwd=tcp:0.0.0.0:57400-10.0.0.15:57400,tftp=/tftpboot
2025-03-15 12:35:14,626: vrnetlab       INFO STDOUT:
2025-03-15 12:35:14,626: vrnetlab       INFO STDERR: qemu-system-x86_64: error: failed to set MSR 0x345 to 0x2000
qemu-system-x86_64: ../../target/i386/kvm/kvm.c:3183: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.

2025-03-15 12:35:14,633: vrnetlab       ERROR Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 1)
2025-03-15 12:35:15,633: vrnetlab       ERROR Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 2)
2025-03-15 12:35:16,634: vrnetlab       ERROR Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 3)
2025-03-15 12:35:17,634: vrnetlab       ERROR Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 4)
2025-03-15 12:35:18,635: vrnetlab       ERROR Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 5)
2025-03-15 12:35:19,635: vrnetlab       ERROR Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 6)
2025-03-15 12:35:20,636: vrnetlab       ERROR Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 7)
2025-03-15 12:35:21,636: vrnetlab       ERROR Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 8)
2025-03-15 12:35:22,637: vrnetlab       ERROR Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 9)

I am using containerlab version 0.66.0, QEMU emulator version 7.2.15 and the latest Master branch,

However if I switch to the commit 'git checkout --detach 78e7127' and re-launching the image, I am not getting errors related to qemu, the container just loads in unhealthy state and after typing './launch.py --username admin --password Admin@123 --hostname palo01 --connection-mode tc --trace' I am getting - "launch INFO Disk image was not found"

[sskras]

2025-03-15 12:35:14,626: vrnetlab       INFO STDERR: qemu-system-x86_64: error: failed to set MSR 0x345 to 0x2000
qemu-system-x86_64: ../../target/i386/kvm/kvm.c:3183: kvm_buf_set_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.

Might be a bug in dockerized qemu as discussed here 3y ago:
https://pagure.io/releng/failed-composes/issue/2548#comment-735475

Two workarounds were considered at the time:

• rebuilding qemu 5.2.0 with the commit ea39f9b reverted.
• disabling the PMU

[t0m3kz]

As mentioned by @sskras - the easiest way is to edit:
https://github.com/hellt/vrnetlab/blob/master/pan/docker/launch.py#L52
and add:

username, password, disk_image=disk_image, ram=6144, cpu=“host,level=9,pmu=off”, smp=“2,sockets=1,cores=1”

Then re-create an image and should be good.

[hellt]

maybe then we should just add it by default to the panos?

[sskras]

According to [RFC PATCH 00/41] KVM: x86/pmu: Introduce passthrough vPM, it might be a good idea:

This emulated vPMU has these drawbacks:

1. Poor performance, guest PMU MSR accessing has VM-exit and some has
   expensive host perf API call. Once guest PMU is multiplexing its
   counters, KVM will waste majority of time re-creating/starting/releasing
   KVM perf events, then the guest perf performance is dropped dramatically.

Still it would surprise me much if QEMU wasn't fixed in Debian Bookworm in the 3y long run. (If that's the PAN image base)

OTOH, as a newbie I wonder how did that went through unnoticed for year, especially through #317 and #318.

Could it be that this part of the large commit from #198 introduced a regression in July last year, by chance?

PS. Is this image rarely tested / used? No offence :)

[arturas-dot]

After applying:

username, password, disk_image=disk_image, ram=6144, cpu="host,level=9,pmu=off", smp="2,sockets=1,cores=1"

everything started to work ! Thank you !

[crytectobi]

2025-07-09 17:46:09,019: vrnetlab   	INFO Launching PAN_vm with 2,sockets=1,cores=1 SMP/VCPU and 6144 M of RAM
2025-07-09 17:46:09,019: vrnetlab   	INFO Scrapli: Disabled
2025-07-09 17:46:09,019: vrnetlab   	INFO Transparent mgmt interface: Disabled
2025-07-09 17:46:09,019: vrnetlab   	DEBUG number of provisioned data plane interfaces is 0

    INFO |                      node | 2025-07-09 17:46:09,020: vrnetlab   	DEBUG qemu cmd: qemu-system-x86_64 -display none -machine pc -monitor tcp:0.0.0.0:4000,server,nowait -serial telnet:0.0.0.0:5000,server,nowait -m 6144 -cpu host,level=9,pmu=off -smp 2,sockets=1,cores=1 -drive if=ide,file=/PA-VM-KVM-10.2.10-overlay.qcow2 -uuid aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa -device pci-bridge,chassis_nr=1,id=pci.1 -device virtio-net-pci,netdev=p00,mac=0C:00:3d:69:f0:00 -netdev user,id=p00,net=10.0.0.0/24,host=10.0.0.2,dns=10.0.0.3,dhcpstart=10.0.0.15,hostfwd=tcp:0.0.0.0:22-10.0.0.15:22,hostfwd=udp:0.0.0.0:161-10.0.0.15:161,hostfwd=tcp:0.0.0.0:80-10.0.0.15:80,hostfwd=tcp:0.0.0.0:443-10.0.0.15:443,hostfwd=tcp:0.0.0.0:830-10.0.0.15:830,hostfwd=tcp:0.0.0.0:6030-10.0.0.15:6030,hostfwd=tcp:0.0.0.0:8080-10.0.0.15:8080,hostfwd=tcp:0.0.0.0:9339-10.0.0.15:9339,hostfwd=tcp:0.0.0.0:32767-10.0.0.15:32767,hostfwd=tcp:0.0.0.0:50051-10.0.0.15:50051,hostfwd=tcp:0.0.0.0:57400-10.0.0.15:57400,tftp=/tftpboot

    INFO |                      node | 2025-07-09 17:46:09,049: vrnetlab   	INFO STDOUT: 
2025-07-09 17:46:09,049: vrnetlab   	INFO STDERR: qemu-system-x86_64: CPU model 'host' requires KVM or HVF


    INFO |                      node | 2025-07-09 17:46:09,057: vrnetlab   	ERROR Unable to connect to qemu monitor (port 4000), retrying in a second (attempt 1)

I have the same issue sadly, I applied the settings mentioned above, but still no luck. Anything else you did to get this running?