[NixOS] Handling multi-layer and parent symlinks for `/etc/resolv.conf`

[KiruyaMomochi]

The pull request #318 adds support for escaping resolv.conf symlinks.
However, I've encountered some issues in more complex environments like NixOS, where symlinks can have multiple layers and even parent folders can be symlinked.

Multi-layer symbolic link

NixOS often generates files under /etc from a config file and places them in /nix/store, symlinking them back to /etc.

This can result in multi-layer symlinks like:

/etc/resolv.conf
  -> /nix/store/random-hash/etc/resolv.conf
  -> /run/systemd/resolve/stub-resolv.conf

In my humble opinion, one of the solution could be mounting the realpath to the first unresolvable layer of symlink.

Parent folder is also symbolic link

The real case is not that simple.
The file /etc/resolv.conf is actually a symlink to /etc/static/resolv.conf.
It is /etc/static that being symlinked to /nix/store/:

/etc/static
  -> /nix/store/random-hash/etc/

/etc/resolv.conf
  -> /etc/static/resolv.conf
 (-> /nix/store/random-hash/etc/resolv.conf)
  -> /run/systemd/resolve/stub-resolv.conf

This makes it difficult to resolve the actual path using readlink or realpath.

Possible Solutions and Workarounds

1. Check readability after pivot_root:
   At the very least, I think we may add a check after pivot_root to warn users if /etc/resolv.conf isn't readable.
   This could provide insights into DNS resolution issues.
2. Bind mounting all layers and parent paths:
   We may need to find paths to do bind mount from every layers of the symlink, and also each parent paths.
   This could potentially ensure resolvability after switching root.
3. Use a different syscall that can mount over a symlink.
   It will be easier if we can bind mount to /etc/resolv.conf directly. During searching I have found MS_NOSYMFOLLOW in mount(2), but that does not work for me. I have also found https://serverfault.com/a/1126837/979197, which suggests using AT_SYMLINK_NOFOLLOW.

As a alternative, it's also possible to patch slirp4ns in Nix package registry to additionly bind-mount /nix into the sandbox.

[AkihiroSuda]

Possible Solutions and Workarounds

4. Just copy the resolv.conf file, instead of mounting it

cc @giuseppe

[AkihiroSuda]

@KiruyaMomochi Do you plan to submit a PR?

[ktoulliou]

Same problem with Ubuntu 22.04.5 LTS with a full NetworkManager+systemd-resolve configuration.

As I'm not root, I can't modify this symbolic link /etc/resolv.conf -> /run/systemd/resolve/resolv.conf (as explained in many open/closed tickets).

In my docker rootless container, I have a 10.0.2.3 name server in /etc/resolv.conf (internet works with IP but not with DNS).
Also my docker pull doesn't work (error dns).

Do you have a workaround without using root rights?

[wrr]

I found an alternative solution for /etc/resolv.conf being a link to a location that is not accessible in the container; perhaps it could be useful to someone. You can use overlayfs to achieve this:

• Create a directory (say /tmp/myetc) with a single file in it: resolv.conf.
• Mount overlayfs at a location that will become /etc in the container (say /tmp/newroot/etc. Set the lowerdir mount option to be lowerdir=/tmp/myetc:/etc. If you mount read-only, you don't need to set upperdir and workdir options. This causes the content of the /tmp/myetc/ directory to take priority over the original content of /etc, so /etc/resolv.conf is no longer a link, but your file, while all other files from the original /etc remain accessible.

Here is a sample mount call:

#include <sys/mount.h>
#include <stdio.h>
#include <errno.h>
#include <string.h>

int mount_overlay() {
    const char *opts = "lowerdir=/tmp/myetc:/etc";
    
    if (mount("overlay", "/tmp/newroot/etc", "overlay", 
              MS_NOEXEC | MS_NOSUID | MS_RDONLY, opts) == -1) {
        fprintf(stderr, "mount failed: %s\n", strerror(errno));
        return -1;
    }
    
    return 0;
}

Or an equivalent command:

mount -t overlay -o lowerdir=/tmp/myetc:/etc,noexec,nosuid,ro overlay /tmp/newroot/etc

With user namespaces, overlayfs mounts are allowed in a container for non-root users.

As an added bonus, this allows you to easily replace other files in /etc by placing them in the myetc dir.