[Bug] vm migration from esxi

[crcro]

Hey there! 👋 Thanks for taking the time to report this bug — we appreciate it!

Please keep in mind: PegaProx is developed and maintained entirely by volunteers in our free time. We do our best, but we can't work magic 🪄 — please be patient, we'll get to your issue as soon as we can.

Want to help keep PegaProx alive?

• ⭐ Star the project — it's free and helps a lot!
• 💖 Become a Sponsor — helps us dedicate more time to development
• 🤝 Contributing code or docs is always welcome too!

---

Describe the bug

retesting vm migration between esxi and proxmox, using latest update,
tested method

• live-mirror

Steps to Reproduce

1. connect esxi host
2. select a vm to be migrated
3. leave everything as it was detected
4. start migration task

Expected behavior

Start/finish the migration

Environment

• PegaProx Version: Beta 0.9.10.3
• Installation Method: Manual
• OS: Ubuntu 24.04
• Browser: Firefox
• Behind Reverse Proxy? No
• UI? Corporate

Logs

[10:15:00] libssh cannot connect - trying SSHFS-backed boot (OpenSSH)...
[10:15:01] SSHFS disk 0: /tmp/v2p-0f73ccec/NCAERODC01PRJ01/NCAERODC01PRJ01-flat.vmdk (50.0 GB)
[10:15:02] SSHFS QEMU test: OK
[10:15:02] Setting AppArmor complain mode for FUSE access...
[10:15:03] Starting Proxmox VM (sshfs backend + cache=writeback)...
[10:15:11] VM 109 STARTED via sshfs - DOWNTIME ENDS
[10:15:11] (Running on sshfs-backed storage with writeback cache)
[10:15:13] Transfer: bs=64MB, compress=gzip, nc=yes
[10:15:15] Background copy isolation: cgroup=v2p-copy-109, io.weight=10
[10:15:15] Starting live storage migration (drive-mirror)...
[10:15:15] Disk 0: sshfs-disk0 → /mnt/pve/tgt01-nfs-hdd/images/109/vm-109-disk-0.raw (50.0 GB)
[10:15:18] drive-mirror response: qm> </pve/tgt01-nfs-hdd/images/109/vm-109-disk-0.raw raw
[10:15:27] drive-mirror: job not visible in block-jobs after 4s
[10:15:27] block-jobs output: No active jobs
[10:15:28] Disk 0: drive-mirror failed to start
[10:15:28] drive-mirror failed - using qemu-img convert -U (VM stays running)
[10:15:28] === BACKGROUND COPY: SSHFS → local storage ===
[10:15:28] (VM keeps running on SSHFS while disks copy to local storage)
[10:15:28] Disk 0: /tmp/v2p-0f73ccec/NCAERODC01PRJ01/NCAERODC01PRJ01-flat.vmdk → /mnt/pve/tgt01-nfs-hdd/images/109/vm-109-disk-0.raw (50.0 GB)
[10:15:32] Background copy started (dd: raw → raw)

--- nothing happens past this point ---

Screenshots

Checklist

• I have searched existing issues to make sure this is not a duplicate
• I am using the latest version of PegaProx

[mkellermann97]

Hey @crcro — two commits landed on Testing covering both halves of your trace:

3f1d02d — drive-mirror + dd-background-copy logging (your specific path):

• The drive-mirror response in the task log was truncated to 150 chars (which is why your log showed qm> </pve/tgt01-nfs-hdd/images/109/vm-109-disk-0.raw raw half-cut). Truncation raised to 500 chars + the exact qm monitor command is now logged before the call, so the diagnostic loop has ground truth.
• "Job visible in block-jobs" wait extended from 4s to 10s with a third poll at +6s. Larger sshfs-backed mirrors on slow targets can take longer than 4s to register.
• On the dd-background-copy that ran after your drive-mirror fell back: pgrep order was wrong (matched the bash wrapper script's PID first, which had near-zero write_bytes). Now pgrep dd/qemu-img directly, fall back to the wrapper as a last resort.
• Method-2 progress regex was qemu-img-only ((N/100%)). For dd's status=progress output it never matched → polling loop was silent for the whole copy. Added a dd-format regex ((\d+) bytes copied) — keeps qemu-img matching for vmdk-descriptor sources.

3bdb0a2 — _ssh_pipe_transfer + _monitor_disk_write (the broader pattern):

• All three offline-VM transfer methods (HTTPS / SSH dd pipe / SSHFS dd) discarded stderr from _pve_node_exec. Same anti-pattern as [Bug] #411 fixed elsewhere. Now captured, and on rc != 0 or under-90%-complete the last ~6 lines of stderr are logged inline so auth/missing-binary/network errors show up in the task log instead of leaving "SSH: only 0 bytes" with no clue why.
• _monitor_disk_write was polling stat on vol_path but only updating internal task.progress — never task.log(). So the log feed went silent during the entire transfer. Now logs on each 10% threshold crossing (disk0: 30% (3.5/10.0 GB, ~80 MB/s)) plus a heartbeat every 30s between thresholds.
• Block-device targets (/dev/pve/vm-X-disk-0) where stat -c %s returns the device size instead of bytes-written: detected and degraded to lvs --noheadings -o data_percent for LVM thin volumes; heartbeats-only when even that's not applicable.
• "monitor alive, 0 bytes written so far (Ns elapsed)" every 60s if the writer hasn't produced output — distinguishes "stuck but PegaProx is still polling" from "everything died silently".

Lab end-to-end verified the broader pattern: my lab migration that hung on Saturday + earlier this morning at "SSH dd pipe (1.0 GB)..." now runs to completion (1 GB in 494s at 2.2 MB/s, network-limited) with visible progress every 10% / 30s heartbeats. The drive-mirror-specific half (your exact path) is logic-verified but not end-to-end-replayed — our lab takes the offline-VM _ssh_pipe_transfer path instead.

When the next release cuts: please retest and drop the new log here. With the diagnostic improvements, "where it stops" will be visible in the trace — whether it's the drive-mirror command rejection, the qemu-img convert hang, or something downstream of either.

Donations at https://opencollective.com/pegaprox if multi-layer visibility patches earn their keep.

— Marcus

[mkellermann97]

@crcro — v0.9.11 bundles both halves (3f1d02d + 3bdb0a2). Drive-mirror logging is actionable now, dd-progress detection holds on QEMU 11 / PVE 9.2. Closing.

— Marcus

[crcro]

now the process completes, but still the created vm does not have any disk attached, but on the storage there are the .raw and .qcow2 disk images:

qcow2 - is faulty (vm cannot boot from it)
raw - is faulty also (but vm can boot from it)

[mkellermann97]

Reopening — fix nailed the dd-progress + drive-mirror logging but you're hitting a different mile-marker now: disk-copy completes, image files exist, but the target VM config never gets the scsi0: / virtio0: ref so PVE renders the VM empty next to two orphaned blobs. Could you attach a support bundle from this run plus the output of qm config <vmid> on the target node? That'll show whether the disk-attach call is missing entirely, or firing with the wrong storage:volid format.

If the project pulls its weight on these tricky cross-hypervisor paths: opencollective.com/pegaprox.

— MK

[crcro]

qm config 108:

root@pve11:~# qm config 108
agent: enabled=1
bios: seabios
cores: 2
cpu: host
hotplug: disk,network
machine: pc
memory: 8192
meta: creation-qemu=11.0.0,ctime=1779712033
name: NCAERODC01PRJ01
net0: vmxnet3=00:50:56:bb:c5:c2,bridge=OVSbr0
ostype: l26
scsihw: lsi
smbios1: uuid=fdec83b0-4595-4814-882e-fb7f62c2aa10
sockets: 1
vga: vmware
vmgenid: 1489513f-33a5-441c-b4fa-1f4b83c0d840

pegaprox_support_20260525_125259.zip

[mkellermann97]

Bundle digested — confirms zero disks attached + the migration stalled mid-flight (audit log has vmware.migration.started 12:27 with no follow-up event). The pegaprox.log in your bundle is empty: file isn't at canonical paths AND the journalctl fallback hit a permission wall (your pegaprox service user isn't in the systemd-journal group).

Two ways forward to see the v2p trace:

• sudo usermod -aG systemd-journal pegaprox && sudo systemctl restart pegaprox, retry one migration, regenerate the bundle.
• Or set PEGAPROX_FILE_LOG_LEVEL=DEBUG + point logs at a writable path (env vars shipped in v0.9.10.3).

Working hypothesis: the attach loop hits a silent fail between qemu-img convert and qm set --scsi0 <volid>. I'll know which side once the log lands. Donations keep cross-hypervisor debug hours alive: opencollective.com/pegaprox.

— MK

[crcro]

attached support bundle after:

sudo usermod -aG systemd-journal pegaprox && sudo systemctl restart pegaprox

and new migration started

pegaprox_support_20260526_091150.zip

[mkellermann97]

Bundle journal-group fix worked — got 1000 lines this time. Unfortunately the log window is only ~10 min (09:02–09:11), and your migration audit event is at 06:01 — pre-systemctl-restart. The restart wiped the journal-cursor so the migration trace got cut off.

Could you trigger one more migration with PEGAPROX_LOG_LEVEL=DEBUG set in the service env, then generate the bundle WITHIN ~30 min of the migration starting? That captures the v2p flow at the DEBUG level which has the qemu-img + qm-set lines we need.

Project funding keeps the cross-hypervisor debug going: opencollective.com/pegaprox.

— MK

[crcro]

pegaprox_support_20260526_114158.zip