Allow setting network configuration via network-scripts

[pjgeorg]

Not sure this is indeed a feature request / issue /..., but probably more like a request for comments how one could handle my current usecase:

Instead of specifying network options on the cmdline I add network-scripts to initrd. I mainly use this setup to boot up clusters of diskless nodes. I'm still running CentOS 8, however I'm currently evaluating possible updates.

The only network related option I specify on the command line is: bootdev=ib0
For CentOS 8 I added the following patches to make it work:

--- usr/lib/dracut/modules.d/35network-manager/nm-lib.sh
+++ usr/lib/dracut/modules.d/35network-manager/nm-lib.sh
@@ -6,6 +6,7 @@
 {
     rm -f /run/NetworkManager/system-connections/*
     /usr/libexec/nm-initrd-generator -- $(getcmdline)
+    rm -f /run/NetworkManager/system-connections/*
 
     if getargbool 0 rd.neednet; then
         for i in /usr/lib/NetworkManager/system-connections/* \
--- usr/lib/dracut/modules.d/35network-manager/nm-run.sh
+++ usr/lib/dracut/modules.d/35network-manager/nm-run.sh
@@ -17,11 +17,14 @@
   break
 done
 
+BOOTDEV=$(getarg bootdev=)
+
 for _i in /sys/class/net/*
 do
     state=/run/NetworkManager/devices/$(cat $_i/ifindex)
     grep -q connection-uuid= $state 2>/dev/null || continue
     ifname=${_i##*/}
+    [ -n "$BOOTDEV" ] && [ "$BOOTDEV" = "$ifname" ] || continue
     sed -n 's/root-path/new_root_path/p;s/next-server/new_next_server/p' <$state >/tmp/dhclient.$ifname.dhcpopts
     source_hook initqueue/online $ifname
     /sbin/netroot $ifname

The first patch is required as nm-initrd-generator creates a default_connection.nmconnection, which is for an interface already handled by bundled network-scripts. Afaik with a more up to date NetworkManager this file is now called $bootdev.nmconnection. Afaik this is due to the following upstream commit and me specifying bootdev= on the command line: https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/25a2b6e14ff525be9838f4b0d90680e49b145f2b
However, issue is still the same: This interface is already taken care of by an network-script. I want nm-initrd-generator to simply create no new connections. Hence one might probably just not run it at all.

The second part of the patch (together with specifying bootdev= on the command line) ensures that only this particular interface is online to then mount the root directory using NFS over this interface with no possible interference of any other interface.

I'm not a 100% sure yet how this interacts with this upstream commit: https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/f114e16fddf854d72a8b08318359569cd83245e3

I do know bootdev= was not meant for this particular use case, but it seemed suitable to me to "misuse" it for this purpose.

With the patches mentioned above this setup currently works as I need it to. However I wanted to ask whether this is a good approach, I'm simply missing some feature of dracut / NetworkManager that allows me to do this without any patches, or this approach (using network-scripts only) is simply not supported upstream?

Thanks for any comments / help!

[haraldh]

CC @dustymabe

[dustymabe]

what is your full kernel command line? My understanding is that if there are no networking arguments (i.e. ip=) then nm-initrd-generator won't write anything to /run/NetworkManager/system-connections/*.

[pjgeorg]

Here's the full command line:

$ cat /proc/cmdline 
initrd=initramfs.img root=nfs4:10.100.10.0:/rootfs/qpz:rdma ro net.ifnames=0 biosdevname=0 rd.bootif=0 rd.net.dhcp.retry=10 bootdev=ib0

Luckily one can easily test what nm-initrd-generator produces depending on the command line without actually booting the system.
Running on CentOS 8 Stream with NetworkManager-1.32.0-0.1.el8.x86_64:

$ /usr/libexec/nm-initrd-generator -s initrd=initramfs.img root=nfs4:10.100.10.0:/rootfs/qpz:rdma ro net.ifnames=0 biosdevname=0 rd.bootif=0 rd.net.dhcp.retry=10 bootdev=ib0

*** Connection 'ib0' ***

[connection]
id=ib0
uuid=8109931b-6024-4ec8-b8d9-4225db9e6447
type=infiniband
autoconnect-retries=1
interface-name=ib0
multi-connect=1
permissions=

[infiniband]
transport-mode=datagram

[ipv4]
dhcp-timeout=900
dns-search=
method=auto

[ipv6]
addr-gen-mode=eui64
dhcp-timeout=900
dns-search=
method=auto

[proxy]

Removing bootdev=ib0 from the command line:

$ /usr/libexec/nm-initrd-generator -s initrd=initramfs.img root=nfs4:10.100.10.0:/rootfs/qpz:rdma ro net.ifnames=0 biosdevname=0 rd.bootif=0 rd.net.dhcp.retry=10

So indeed adding bootdev=ib0 to the command line arguments causes this issue. However, just like when one specifies multiple ip= arguments on the command line for various connections, I also need a way to specify the boot device in case of multiple connection configurations being added to the initramfs.
Creating a connection when bootdev=ib0 but no ip= is specified for this interface seems to be the exptected behavior, e.g. see NetoworkManager commit https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/25a2b6e14ff525be9838f4b0d90680e49b145f2b for more details on this.
After going through this commit's message once again, I think this is actually at least partially a bug in NetworkManager or this setup is simply not supported by NetworkManager. As far as I understand it when bootdev=ib0 is specified on the command line nm-initrd-generator should only create a new connection if it does not exist yet. However afaik it only detects connections that have been specifed on the command line, e.g. using ip=. It does not check for connections that have been added to the initramfs via config files.

Hence for my desired setup to work there are two options in my opinion:

1. Change nm-initrd-generator to also check for configurations in /usr/lib/NetworkManager/system-connections, /etc/NetworkManager/system-connections and /etc/sysconfig/network-scripts/ifcfg-*. Maybe allow command line arguments to overwrite existing connections.

2. Add logic to nm-lib.sh after running nm-initrd-generator to delete all created connection for devices that have already been configured in /usr/lib/NetworkManager/system-connections, /etc/NetworkManager/system-connections or /etc/sysconfig/network-scripts/ifcfg-*. My simple solution shown in my first post assumes that all connections created by nm-initrd-generator have already been configured. Of course this solution does not work in case user wants to add some configs to initramfs but configure other interfaces via command line arguments. However implementing this work-around for the general use case seems to be non-trivial or even impossible.

Btw. there is an obvious mistake in the diff I posted previously:
This line

+    [ -n "$BOOTDEV" ] && [ "$BOOTDEV" = "$ifname" ] || continue

should be replaced by

+   if  [ -n "$BOOTDEV" ]; then [ "$BOOTDEV" = "$ifname" ] || continue; fi

[dustymabe]

I also need a way to specify the boot device in case of multiple connection configurations being added to the initramfs.

Is it an option to just add configuration to the initramfs for the device you need and not the others? Then all you'd need is something like rd.neednet=1 on the kernel command line. The other interfaces shouldn't get brought up in the initramfs in that case.

[pjgeorg]

Is it an option to just add configuration to the initramfs for the device you need and not the others?

I changed my own dracut module, that pulls in the ifcfg-* files, to now only pull in those that have set NETBOOT=yes. This indeed makes sense in any case. Thanks for the suggestion! @dustymabe

However, setting rd.neednet=1 causes nm-initrd-generator to create a default connection:

$ /usr/libexec/nm-initrd-generator -s rd.neednet=1

*** Connection 'default_connection' ***

[connection]
id=Wired Connection
uuid=7791a1d0-c967-4fd6-8f36-0b7908da2268
type=ethernet
autoconnect-retries=1
multi-connect=3
permissions=
wait-device-timeout=60000

[ethernet]
mac-address-blacklist=

[ipv4]
dhcp-timeout=90
dns-search=
method=auto

[ipv6]
addr-gen-mode=eui64
dhcp-timeout=90
dns-search=
method=auto

[proxy]

While not having to specify bootdev=ib0 anymore (and checking for it to be online), I still have to delete /run/NetworkManager/system-connections/default_connection.nmconnection after running nm-initrd-generator to avoid having two connections set up for the same network interface. So the diff is now down to adding one line to nm-lib.sh.

As we have now been able to track this down to a particular behavior of nm-initrd-generator should I close this issue and open one at the NetworkManager project or do you think it is better to keep this discussion open? I assume developers involved in nm-initrd-generator and dracut probably overlap anyway.

[dustymabe]

Is it an option to just add configuration to the initramfs for the device you need and not the others?

I changed my own dracut module, that pulls in the ifcfg-* files, to now only pull in those that have set NETBOOT=yes. This indeed makes sense in any case. Thanks for the suggestion! @dustymabe

No Problem.

However, setting rd.neednet=1 causes nm-initrd-generator to create a default connection:

$ /usr/libexec/nm-initrd-generator -s rd.neednet=1

*** Connection 'default_connection' ***

[connection]
id=Wired Connection
uuid=7791a1d0-c967-4fd6-8f36-0b7908da2268
type=ethernet
autoconnect-retries=1
multi-connect=3
permissions=
wait-device-timeout=60000

[ethernet]
mac-address-blacklist=

[ipv4]
dhcp-timeout=90
dns-search=
method=auto

[ipv6]
addr-gen-mode=eui64
dhcp-timeout=90
dns-search=
method=auto

[proxy]

I see. Yes, you are right.

While not having to specify bootdev=ib0 anymore (and checking for it to be online), I still have to delete /run/NetworkManager/system-connections/default_connection.nmconnection after running nm-initrd-generator to avoid having two connections set up for the same network interface. So the diff is now down to adding one line to nm-lib.sh.

It's hacky but you could just set rd.neednet=0 and echo '[ -f /tmp/nm.done ]' > "$hookdir"/initqueue/finished/nm.sh from your own module. Of course over time this will cease to work as it's mostly an implementation detail. For example there were recent changes to run NM via systemd and not via initqueue.

As we have now been able to track this down to a particular behavior of nm-initrd-generator should I close this issue and open one at the NetworkManager project or do you think it is better to keep this discussion open? I assume developers involved in nm-initrd-generator and dracut probably overlap anyway.

I'm not really sure what we'd be asking nm-initrd-generator to do differently so I don't know if I can answer your question just yet.