--disable-microvm + virtiofsd + hotpluggable memory == BOOM

[marcosps]

I was about to check something related to memory hotplug, and by it I changed virtme-ng a little:

  1199⋮    │        qemuargs.extend(["-m", args.memory])
      ⋮1199│        qemuargs.extend(["-m", f"{args.memory},slots=4,maxmem=4G"])

And executed vng:

vng --disable-microvm --user root -m 1G --verbose

Once he VM booted, I entered in the QEMU monitor mode (Ctrl-C), and executed the following commands:

object_add memory-backend-ram,id=mem1,size=1G
device_add pc-dimm,id=dimm1,memdev=mem1

This immediately triggered the following error:

thread 'vring_worker' panicked at /home/abuild/rpmbuild/BUILD/virtiofsd-1.13.2-build/virtiofsd-1.13.2/src/vhost_user.rs:271:18:
called `Result::unwrap()` on an `Err` value: QueueWriter(FindMemoryRegion)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

The issue is gone if I use force-9p. Have you seem something similar to it? I'll try to contact someone at SUSE to check if this is expected or not. I opened this bug, because someone may face the same problem, and I just wanted to advice to use --force-9p in this case.

[arighi]

Hey @marcosps can you try adding -o cache=always to the virtiofsd command?

diff --git a/virtme/commands/run.py b/virtme/commands/run.py
index ba4eede..bb0780c 100644
--- a/virtme/commands/run.py
+++ b/virtme/commands/run.py
@@ -746,7 +746,7 @@ class VirtioFS:
             stderr = ""
         os.system(
             f"{virtiofsd_path} --syslog --no-announce-submounts "
-            + f"--socket-path {self.sock} --shared-dir {path} --sandbox none {stderr} &"
+            + f"--socket-path {self.sock} --shared-dir {path} --sandbox none {stderr} -o cache=always &"
         )
         max_attempts = 5
         check_duration = 0.1

In this way filesystem data should always go through the page cache instead of being mapped directly into guest memory, so virtiofsd doesn’t need to rely on the vhost-user memory table for hotplugged regions, which should avoid the FindMemoryRegion panic you’re seeing.

[mhartmay]

...

This immediately triggered the following error:

thread 'vring_worker' panicked at /home/abuild/rpmbuild/BUILD/virtiofsd-1.13.2-build/virtiofsd-1.13.2/src/vhost_user.rs:271:18:
called `Result::unwrap()` on an `Err` value: QueueWriter(FindMemoryRegion)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

That's a virtiofsd bug, no?

[marcosps]

Hey @marcosps can you try adding -o cache=always to the virtiofsd command?

diff --git a/virtme/commands/run.py b/virtme/commands/run.py
index ba4eede..bb0780c 100644
--- a/virtme/commands/run.py
+++ b/virtme/commands/run.py
@@ -746,7 +746,7 @@ class VirtioFS:
             stderr = ""
         os.system(
             f"{virtiofsd_path} --syslog --no-announce-submounts "
-            + f"--socket-path {self.sock} --shared-dir {path} --sandbox none {stderr} &"
+            + f"--socket-path {self.sock} --shared-dir {path} --sandbox none {stderr} -o cache=always &"
         )
         max_attempts = 5
         check_duration = 0.1

In this way filesystem data should always go through the page cache instead of being mapped directly into guest memory, so virtiofsd doesn’t need to rely on the vhost-user memory table for hotplugged regions, which should avoid the FindMemoryRegion panic you’re seeing.

Still the same results:

(qemu) [   21.812186] Fallback order for Node 0: 0 
[   21.812197] Built 1 zonelists, mobility grouping on.  Total pages: 209254
[   21.812702] Policy zone: Normal

thread 'vring_worker' panicked at /home/abuild/rpmbuild/BUILD/virtiofsd-1.13.2-build/virtiofsd-1.13.2/src/vhost_user.rs:271:18:
called `Result::unwrap()` on an `Err` value: QueueWriter(FindMemoryRegion)
qemu-system-x86_64: Unexpected end-of-file before all data were read

It's something that it's urgent for me, I just opened a bug here because I found this issue. Otherwise we can leave this open until it's fixed on upstream or one of us have some time to dig in :)

[marcosps]

...

This immediately triggered the following error:

thread 'vring_worker' panicked at /home/abuild/rpmbuild/BUILD/virtiofsd-1.13.2-build/virtiofsd-1.13.2/src/vhost_user.rs:271:18:
called `Result::unwrap()` on an `Err` value: QueueWriter(FindMemoryRegion)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

That's a virtiofsd bug, no?

It seems so, but my idea was to track the issue here, so we can workaround the issue, or maybe somebody has some idea how to fix it.