DHCP "Network is down"

[bjackman]

If I boot with unshare -r vng --verbose --run ~/kbuild/scratch/arch/x86/boot/bzImage --user root --root ./image, I see this:

root@virtme-ng:/# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
root@virtme-ng:/# ls -R /sys/bus/virtio/drivers/virtio_net/
/sys/bus/virtio/drivers/virtio_net/:
bind  uevent  unbind

My Kconfig comes from vng --kconfig, it has VIRTIO_NET enabled:

❯❯  grep VIRTIO_NET ~/kbuild/scratch/.config
CONFIG_VIRTIO_NET=y

I'm booting a minimal rootfs generated by mkosi using this config, you can download it from here.

When I add --net user to my vng command the boot gets stuck:

❯❯  unshare -r vng --verbose --run ~/kbuild/scratch/arch/x86/boot/bzImage --user root --root ~/src/limmat-kerunshare -r vng --verbose --run ~/kbuild/scratch/arch/x86/boot/bzImage --user root --root ~/src/limmat-kernel/mkosi-rootfs/image/ --net user   
...
[    1.029623] mount (99) used greatest stack depth: 13208 bytes left
[    1.052375] mkdir (100) used greatest stack depth: 12936 bytes left
mount: /etc/fstab: mount point does not exist.
       dmesg(1) may have more information after failed mount system call.
cp: cannot stat '/etc/hosts': No such file or directory
[    1.630627] tsc: Refined TSC clocksource calibration: 2112.006 MHz
[    1.630802] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x1e717e40680, max_idle_ns: 440795297614 ns
[    1.631275] clocksource: Switched to clocksource tsc
mount: /etc/hosts: mount point does not exist.
       dmesg(1) may have more information after failed mount system call.
[    1.675288] virtme-init: basic initialization done
[    1.696850] virtme-init: starting udevd
[    1.698624] virtme-init: running systemd-tmpfiles
Starting systemd-udevd version 252.36-1~deb12u1
[    1.784525] virtme-init: triggering udev coldplug
/usr/lib/tmpfiles.d/journal-nocow.conf:26: Failed to resolve specifier: uninitialized /etc/ detected, skipping.
All rules containing unresolvable specifiers will be skipped.
[    1.916709] virtme-init: waiting for udev to settle
[    2.754394] virtme-init: udev is done
[    2.917244] virtme-init: Setting hostname to virtme-ng...
[    2.993944] ip (219) used greatest stack depth: 12360 bytes left
udhcpc: started, v1.35.0
udhcpc: broadcasting discover
udhcpc: sendto: Network is down
udhcpc: read error: Network is down, reopening socket
udhcpc: read error: Network is down, reopening socket
[etc]

Running virtme-ng from 1e7b05c.

[bjackman]

Hm, this seems to be specific to the shell init script - if I run make from the virtme-ng tree and explicitly use the vng script from the source tree (instead of pipx), the network works. Does that suggest this isn't an issue with the QEMU cmdline?

[matttbe]

Hello,

Hm, this seems to be specific to the shell init script - if I run make from the virtme-ng tree and explicitly use the vng script from the source tree (instead of pipx), the network works.

That's strange, the two versions should call busybox udhcpc the same way. virtme-ng-init will retry if it cannot access /sys/bus/virtio/drivers/virtio_net, but according to the logs, udhcpc was launched so it should not be that. Could you eventually add set -x after if cat /proc/cmdline |grep -q -E '(^| )virtme.dhcp($| )'; then in virtme-init and at the beginning of virtme-udhcpc-script, then relaunch vng. It might be good to compare the version with the new init program.

[    1.630627] tsc: Refined TSC clocksource calibration: 2112.006 MHz
[    1.630802] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x1e717e40680, max_idle_ns: 440795297614 ns
[    1.631275] clocksource: Switched to clocksource tsc

It looks like KVM is not used, is that right? Out of curiosity, is it intentional? Is it because you are using unshare and QEMu cannot access /dev/kvm? Or another permission issue?

[arighi]

I think running vng with unshare -r just opens a bunch of problems and it'd be really difficult to catch/fix all of them...

[arighi]

I'm also wondering if there's a different order of operation between virtme-init and virtme-ng-init that might trigger this issue (something between systemd-tmpfs and udev maybe?).

As a random quick test, can you try to add a sleep 1 after starting systemd-tmpfiles to make sure it completes before starting udev (in the bash virtme-init)? This is kind of a random test, but I think I remember we had a similar issue in the past if we were not synchronizing systemd-tmpfiles and udev...

[bjackman]

It looks like KVM is not used, is that right? Out of curiosity, is it intentional? Is it because you are using unshare and QEMu cannot access /dev/kvm? Or another permission issue?

Oh this was just because my machine seems to have the virtualization extensions disabled in the BIOS (no idea how that happened) and I'm travelling so I can't fix it. I have just reproduced locally on my laptop which does have /dev/kvm.

I think running vng with unshare -r just opens a bunch of problems and it'd be really difficult to catch/fix all of them...

Well, the alternative is that the user needs sudo to use virtme-ng with --root, no (they would need to write a rootfs that's owned by root into their host filesystem)? That's an unfortunate degradation IMO. To get away from unshare -r right now I think my only option would be to actually write my rootfs to my disk with root ownership.

Having said that, it does work if I boot using my host rootfs:

~/src/virtme-ng/vng --verbose --run ~/kbuild/scratch/arch/x86/boot/bzImage --user root --network user --no-virtme-ng-init

This is awkward because I can either run using my host rootfs and root userns (--network works), or I can run using a specific guest rootfs plus unshare -r (--network doesn't work). But I can't decouple those things to verify whether the unshare -r is related to the issue or not:

• unshare -r vng --verbose --run ~/kbuild/scratch/arch/x86/boot/bzImage --user root --network user does eventually boot, the networking doesn't work but I think it's a separate issue (not pictured: loads of other "Read-only file system" errors):

  fchownat() of /tmp/.ICE-unix failed: Read-only file system                                                                                                                                      17:36:02 [10/12339]
  fchownat() of /tmp/.XIM-unix failed: Read-only file system                                                                                                                                                         
  fchownat() of /tmp/.font-unix failed: Read-only file system                                                                                                                                                        
  fchownat() of /var/lib/colord failed: Read-only file system                                                                                                                                                        
  Failed to opendir() '/proc/self/fd/3': No such file or directory                                                                                                                                                   
  ACL operation on "/var/log/journal" failed: No such file or directory                                                                                                                                              
  Failed to reopen '/var/log/journal': Bad file descriptor                                                                                                                                                           
  fchownat() of /var/log/journal failed: Read-only file system                                                                                                                                                       
  ACL operation on "/var/log/journal/6415daa159594d8db50034f8638f7659" failed: No such file or directory                                                                                                             
  Failed to reopen '/var/log/journal/6415daa159594d8db50034f8638f7659': Bad file descriptor                                                                                                                          
  fchownat() of /var/log/journal/6415daa159594d8db50034f8638f7659 failed: Read-only file system                                                                                                                      
  ACL operation on "/run/log/journal" failed: No such file or directory
  fchownat() of /run/log/journal failed: Read-only file system
  ACL operation on "/var/log/journal/6415daa159594d8db50034f8638f7659/system.journal" failed: No such file or directory
  fchownat() of /var/log/journal/6415daa159594d8db50034f8638f7659/system.journal failed: Read-only file system
  [   19.466824] virtme-init: waiting for udev to settle
  [   19.659511] virtme-init: udev is done
  mount: /sys/fs/cgroup: must be superuser to use mount.
         dmesg(1) may have more information after failed mount system call.
  mount: /dev/pts: must be superuser to use mount.
         dmesg(1) may have more information after failed mount system call.
  mount: /dev/shm: must be superuser to use mount.
         dmesg(1) may have more information after failed mount system call.
  [   20.254159] virtme-init: Setting hostname to virtme-ng...
  [   20.367051] ip (200) used greatest stack depth: 12920 bytes left
  mktemp: failed to create file via template '/run/tmp/tmp.XXXXXXXXXX': No such file or directory
  /usr/lib/python3/dist-packages/virtme/guest/virtme-init: line 237: $tmpfile: ambiguous redirect
  /usr/lib/python3/dist-packages/virtme/guest/virtme-init: line 238: $tmpfile: ambiguous redirect
  chmod: cannot access '': No such file or directory
  mount: /etc/sudoers: must be superuser to use mount. 
         dmesg(1) may have more information after failed mount system call.
            _      _                                     
     __   _(_)_ __| |_ _ __ ___   ___       _ __   __ _ 
     \ \ / / |  __| __|  _   _ \ / _ \_____|  _ \ / _  |
      \ V /| | |  | |_| | | | | |  __/_____| | | | (_| |
       \_/ |_|_|   \__|_| |_| |_|\___|     |_| |_|\__  |
                                                  |___/ 
     kernel version: 6.14.0-rc6+ x86_64
     (CTRL+d to exit)
  
  mkdir: cannot create directory '/run/user/0': Read-only file system
  fatal library error, lookup self
  bash: /root/.bashrc: Permission denied
  root@virtme-ng:/home/jackmanb/src/mmtests-run/rootfs# curl google.com                                                                                                                                              
  curl: (6) Could not resolve host: google.com                                                                                                                                                                       
  root@virtme-ng:/home/jackmanb/src/mmtests-run/rootfs# ip a                                                                                                                                                         
  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000                                                                                                                        
      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00                                                                                                                                                          
      inet 127.0.0.1/8 scope host lo                                                                                                                                                                                 
         valid_lft forever preferred_lft forever                                                           
  2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
  

• vng --verbose --run ~/kbuild/scratch/arch/x86/boot/bzImage --user root --network user --root ./image doesn't boot for the reasons I mentioned above:

  mount: /run: must be superuser to use mount.
         dmesg(1) may have more information after failed mount system call.
  [    1.156638] mount (100) used greatest stack depth: 13208 bytes left
  mkdir: cannot create directory '/run/virtme': Read-only file system
  [    1.189610] mkdir (101) used greatest stack depth: 12936 bytes left
  mount: /run/virtme/guesttools: mount point does not exist.
         dmesg(1) may have more information after failed mount system call.
  /bin/sh: 1: exec: /run/virtme/guesttools/virtme-init: not found
  [    1.249219] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00
  

As a random quick test, can you try to add a sleep 1 after starting systemd-tmpfiles

~/src/virtme-ng (main|✚1) <3> 
❯❯  git rev-parse HEAD
3e7a6db3f28795a86b8b3b94f4df94109703bbb7

~/src/virtme-ng (main|✚1) <3> 
❯❯  git diff
diff --git i/virtme/guest/virtme-init w/virtme/guest/virtme-init
index 5137e67..7a28b95 100755
--- i/virtme/guest/virtme-init
+++ w/virtme/guest/virtme-init
@@ -152,6 +152,9 @@ if command -v systemd-tmpfiles &> /dev/null; then
     systemd-tmpfiles --create --boot --exclude-prefix="/dev" --exclude-prefix="/root"
 fi
 
+log 'eepy slepy'
+sleep 5
+
 # Make dbus work (if tmpfiles wasn't there or didn't create the directory).
 install -d /run/dbus

I still reproduce the issue with the above plus unshare -r ~/src/virtme-ng/vng --verbose --run ~/kbuild/scratch/arch/x86/boot/bzImage --user root --root ~/src/limmat-kernel/mkosi-rootfs/image/ --network user --no-virtme-ng-init