Add a path existence check, and if not, create the directory

[t0hsumi]

Contributes to solving the following issue: #1136

The cause of the problem described in the issue is that the source directory is not correctly created when using fuse-overlayfs.

[t0hsumi]

It solves the error described in this issue, but there is no binding between "a" and "tmp tmp." Also, after exiting the Bash shell, the program redirected unknown output to "output/tmp tmp."

Below is my log with a debug flag.

$ mkdir "a" "tmp tmp"
$ sudo mount --bind "a" "tmp tmp"
$ containerexec bash --debug
2025-03-09 21:30:13 - DEBUG - This is containerexec 3.29-dev.
2025-03-09 21:30:13 - INFO - Starting command bash
2025-03-09 21:30:13 - DEBUG - Available Cgroups: {}
2025-03-09 21:30:13 - DEBUG - Starting process.
2025-03-09 21:30:13 - DEBUG - Parent: child process of RunExecutor with PID 24036 started.
2025-03-09 21:30:13 - DEBUG - Child: child process of RunExecutor with PID 24036 started
2025-03-09 21:30:13 - DEBUG - Failed to make b'/tmp/BenchExec_run_knqnlkb1/mount/home/benchexec' a bind mount: [Errno 2] mount(b'/tmp/BenchExec_run_knqnlkb1/mount/home/benchexec', b'/tmp/BenchExec_run_knqnlkb1/mount/home/benchexec', None, 4096, None) failed: No such file or directory
2025-03-09 21:30:13 - DEBUG - Using fuse-overlayfs because of mount on '/'
2025-03-09 21:30:13 - DEBUG - /usr/bin/fuse-overlayfs version: 1.13
2025-03-09 21:30:13 - DEBUG - Creating overlay mount with /usr/bin/fuse-overlayfs: target=b'/tmp/BenchExec_run_knqnlkb1/overlayfs/fuse_mount', lower=b'/', upper=b'/tmp/BenchExec_run_knqnlkb1/temp', work=b'/tmp/BenchExec_run_knqnlkb1/overlayfs/fuse_work'
2025-03-09 21:30:13 - DEBUG - Mounting '/' as overlay
2025-03-09 21:30:13 - DEBUG - [Errno 16] umount(b'/tmp/BenchExec_run_knqnlkb1/mount/') failed: Device or resource busy
2025-03-09 21:30:13 - DEBUG - Mounting '/dev' as read-only
2025-03-09 21:30:13 - DEBUG - Mounting '/dev/pts' as read-only
2025-03-09 21:30:13 - DEBUG - Mounting '/dev/shm' as read-only
2025-03-09 21:30:13 - DEBUG - Mounting '/dev/mqueue' as read-only
2025-03-09 21:30:13 - DEBUG - Mounting '/dev/hugepages' as read-only
2025-03-09 21:30:13 - DEBUG - Mounting '/run' as hidden
2025-03-09 21:30:13 - DEBUG - Mounting '/sys' as read-only
2025-03-09 21:30:13 - DEBUG - Mounting '/sys/kernel/security' as read-only
2025-03-09 21:30:13 - DEBUG - Mounting '/sys/fs/cgroup' as read-only
2025-03-09 21:30:13 - DEBUG - Mounting '/sys/fs/pstore' as read-only
2025-03-09 21:30:13 - DEBUG - Mounting '/sys/firmware/efi/efivars' as read-only
2025-03-09 21:30:13 - DEBUG - Mounting '/sys/fs/bpf' as read-only
2025-03-09 21:30:13 - DEBUG - Mounting '/sys/kernel/debug' as read-only
2025-03-09 21:30:13 - DEBUG - Mounting '/sys/kernel/tracing' as read-only
2025-03-09 21:30:13 - DEBUG - Mounting '/sys/fs/fuse/connections' as read-only
2025-03-09 21:30:13 - DEBUG - Mounting '/sys/kernel/config' as read-only
2025-03-09 21:30:13 - DEBUG - Mounting '/proc' as read-only
2025-03-09 21:30:13 - DEBUG - Mounting '/snap/bare/5' as overlay
2025-03-09 21:30:13 - DEBUG - Mounting '/snap/core20/2434' as overlay
2025-03-09 21:30:13 - DEBUG - Mounting '/snap/core20/2496' as overlay
2025-03-09 21:30:13 - DEBUG - Mounting '/snap/core22/1722' as overlay
2025-03-09 21:30:13 - DEBUG - Mounting '/snap/core22/1748' as overlay
2025-03-09 21:30:13 - DEBUG - Mounting '/snap/firefox/5437' as overlay
2025-03-09 21:30:13 - DEBUG - Mounting '/snap/firefox/5836' as overlay
2025-03-09 21:30:13 - DEBUG - Mounting '/snap/firmware-updater/167' as overlay
2025-03-09 21:30:13 - DEBUG - Mounting '/snap/gnome-42-2204/176' as overlay
2025-03-09 21:30:13 - DEBUG - Mounting '/snap/gtk-common-themes/1535' as overlay
2025-03-09 21:30:13 - DEBUG - Mounting '/snap/snap-store/1113' as overlay
2025-03-09 21:30:13 - DEBUG - Mounting '/snap/snap-store/1216' as overlay
2025-03-09 21:30:13 - DEBUG - Mounting '/snap/snapd-desktop-integration/247' as overlay
2025-03-09 21:30:13 - DEBUG - Mounting '/snap/snapd/23258' as overlay
2025-03-09 21:30:13 - DEBUG - Mounting '/snap/snapd/23771' as overlay
2025-03-09 21:30:13 - DEBUG - Mounting '/snap/snapd-desktop-integration/253' as overlay
2025-03-09 21:30:13 - DEBUG - Mounting '/snap/thunderbird/682' as overlay
2025-03-09 21:30:13 - DEBUG - Mounting '/var/snap/firefox/common/host-hunspell' as overlay
2025-03-09 21:30:13 - DEBUG - Cannot use overlay mode for /boot/efi because it has file system vfat. Using read-only mode instead. You can override this by specifying a different directory mode.
2025-03-09 21:30:13 - DEBUG - Mounting '/boot/efi' as read-only
2025-03-09 21:30:13 - DEBUG - Mounting '/var/lib/lxcfs' as read-only
2025-03-09 21:30:13 - DEBUG - Mounting '/home/t0hsumi/prog/benchexec-dev/sample/tmp tmp' as overlay
2025-03-09 21:30:13 - DEBUG - Mounting '/tmp' as hidden
2025-03-09 21:30:13 - DEBUG - Mounting '/run' as hidden
2025-03-09 21:30:13 - DEBUG - Parent: executing bash in grand child with PID 24045 via child with PID 24036.
2025-03-09 21:30:13 - DEBUG - Waiting for signals
2025-03-09 21:30:13 - DEBUG - Waiting for signals
$ ls
 a  'tmp tmp'
$ ls -al
total 8
drwxrwxr-x 4 benchexec benchexec   60  3月  9 21:30  .
drwxrwxr-x 5 benchexec benchexec 4096  3月  9 21:29  ..
drwxrwxr-x 2 benchexec benchexec 4096  3月  9 21:29  a
drwxrwxr-x 2 benchexec benchexec   80  3月  9 21:30 'tmp tmp'
$ echo "hello" > a/greeting
$ ls tmp\ tmp/
$ ls a
greeting
benchexec@benchexec:/home/t0hsumi/prog/benchexec-dev/sample$ exit
exit
2025-03-09 21:30:55 - DEBUG - Child: process bash terminated with exit code 0.
2025-03-09 21:30:55 - DEBUG - Transferring output file /proc/24036/root/tmp/BenchExec_run_knqnlkb1/home/t0hsumi/prog/benchexec-dev/sample/tmp tmp/.wh..wh..opq to output.files/tmp tmp/.wh..wh..opq
2025-03-09 21:30:55 - DEBUG - Transferring output file /proc/24036/root/tmp/BenchExec_run_knqnlkb1/home/t0hsumi/prog/benchexec-dev/sample/a/greeting to output.files/a/greeting
2025-03-09 21:30:55 - DEBUG - 2 output files matched the patterns and were transferred.
2025-03-09 21:30:55 - DEBUG - Waiting for process bash with pid 24036
2025-03-09 21:30:55 - DEBUG - Parent: child process of RunExecutor with PID 24036 terminated with return value 0.
2025-03-09 21:30:55 - DEBUG - Process terminated, exit code 0.
2025-03-09 21:30:55 - DEBUG - Cleaning up temporary directory.
$ ls -r output.files/*
'output.files/tmp tmp':

output.files/a:
greeting

Let me reconsider, and if you have any advice, please let me know.

[PhilippWendler]

Thanks for this contribution! I can confirm what you see so far.

The fact that there is no "binding" between a and tmp tmp in the container is not surprising. Once we write to tmp tmp in the container, the writes will be handled by the overlayfs, and will never be passed through downwards to the bind mount. I think there is little that we can do about it, except by re-creating all bind mounts inside the container. But this seems too difficult and not really necessary.

The fact that after the run an empty output.files/tmp tmp exists is not perfect, but also not important.

What is a problem, though, is the following:

$ mkdir "a" "b"
$ sudo mount --bind "a" "b"
$ touch a/foo
$ ls a b
a:
foo

b:
foo
$ containerexec bash
2025-03-11 08:58:58 - INFO - Starting command bash
benchexec@benchexec:/home/wendler/git/benchexec$ ls a b
a:
foo

b:
benchexec@benchexec:/home/wendler/git/benchexec$ 

Inside the container b starts as an empty directory, but it should have the same content as outside the container (as long as nothing has been written inside the container).

Any idea what is going here? Would you like to investigate this?

[t0hsumi]

Thanks for the quick response!

I understand what you want containerexec to do.
I'm not sure what's going on at the moment, so I'll investigate and report back within 24 hours.

[t0hsumi]

Here's what I've investigated so far:

The OSError discussed in this issue is raised in make_bind_mount(fuse_mount_path, mount_path) with the arguments fuse_mount_path="/tmp/BenchExec_run_xxxxxxxx/fuse_mount/path/to/b", mount_path="/tmp/BenchExec_run_xxxxxxxx/mount/path/to/b", because fuse_mount_path does not exist. However, at the beginning of the loop (see this line), it does exist. When the program executes make_bind_mount(fuse_mount_path, mount_path) with the arguments fuse_mount_path='/tmp/BenchExec_run_xxxxxxxx/overlayfs/fuse_mount/', mount_path='/tmp/BenchExec_run_xxxxxxxx/mount/', it disappears because there is no path/to/b under fuse_mount_path.

This issue might be resolved by modifying setup_fuse_overlay(temp_base, work_base) to construct the bind mount from the bottom up. However, in the definition of setup_fuse_overlay(temp_base, work_base), lowerdir is specified as b"/", and I'm unsure how to extract only the bind mount from /proc/self/mounts. This link might be helpful, but I'm not sure if it's the correct approach.

[PhilippWendler]

Here's what I've investigated so far:

Thanks!

The OSError discussed in this issue is raised in make_bind_mount(fuse_mount_path, mount_path) with the arguments fuse_mount_path="/tmp/BenchExec_run_xxxxxxxx/fuse_mount/path/to/b", mount_path="/tmp/BenchExec_run_xxxxxxxx/mount/path/to/b", because fuse_mount_path does not exist. However, at the beginning of the loop (see this line), it does exist. When the program executes make_bind_mount(fuse_mount_path, mount_path) with the arguments fuse_mount_path='/tmp/BenchExec_run_xxxxxxxx/overlayfs/fuse_mount/', mount_path='/tmp/BenchExec_run_xxxxxxxx/mount/', it disappears because there is no path/to/b under fuse_mount_path.

Hm, I don't understand why /tmp/BenchExec_run_xxxxxxxx/fuse_mount/path/to/b should be missing. /tmp/BenchExec_run_xxxxxxxx/fuse_mount should be an overlay the complete file-system hierarchy of the system, and everything directory on the host should be in it. After all, other mounts are also visible in it, aren't they?

This issue might be resolved by modifying setup_fuse_overlay(temp_base, work_base) to construct the bind mount from the bottom up.

Sorry, I am not really sure what you mean with that?

[t0hsumi]

After all, other mounts are also visible in it, aren't they?

Yeah, they are visible.

I don't know why, but fuse-overlayfs doesn't properly recognize bind-mounted directories.
The kernel's overlayfs does, but the contents of the bind target directory aren't reflected in the overlayfs-mounted directory.

I made containers/fuse-overlayfs#437, and I'm going to investigate fuse-overlayfs repository further.

Here's a simple test code:

$ mount --version
mount from util-linux 2.39.3 (libmount 2.39.3: selinux, smack, btrfs, verity, namespaces, idmapping, statx, assert, debug)
$ fuse-overlayfs --version
fuse-overlayfs: version 1.13-dev
FUSE library version 3.14.0
using FUSE kernel interface version 7.31
fusermount3 version: 3.14.0
$ mkdir lowerdir/{a,b}
$ sudo mount --bind lowerdir/a lowerdir/b
$ touch lowerdir/a/foo
$ ls lowerdir/{a,b}
lowerdir/a:
foo

lowerdir/b:
foo
$ mkdir updir workdir fuse_mount
$ fuse-overlayfs -o lowerdir=lowerdir,upperdir=updir,workdir=workdir fuse_mount/
$ ls lowerdir/ fuse_mount/
fuse_mount/:
a

lowerdir/:
a  b
$ mkdir mount
$ sudo mount -t overlay overlay -olowerdir=lowerdir,upperdir=updir/,workdir=workdir/ mount
$ ls lowerdir/ mount
lowerdir/:
a  b

mount:
a  b
$ ls mount/{a,b}
mount/a:
foo

mount/b:

Sorry, I am not really sure what you mean with that?

Please disregard it. Initially, I assumed that the issue could be resolved by mounting path/to/b/content first, followed by mounting path/to/b. However, this assumption was entirely incorrect.

[PhilippWendler]

After all, other mounts are also visible in it, aren't they?

Yeah, they are visible.

I don't know why, but fuse-overlayfs doesn't properly recognize bind-mounted directories.

Wow, I am surprised.
Could you try what happens if you use lowerdir/b as the the lowerdir of a fuse-overlay instance? Does it then show b/foo correctly?
Then we could at least create a fresh fuse-overlayfs instance for every bind mount.

The kernel's overlayfs does, but the contents of the bind target directory aren't reflected in the overlayfs-mounted directory.

Yes, with kernel overlayfs one has to replicate every single mount point in the hierarchy. That is something that we do already in BenchExec.

I made containers/fuse-overlayfs#437, and I'm going to investigate fuse-overlayfs repository further.

Thanks a lot, I think this is a very well written issue and the best way forward is to wait whether we get a reaction.

[t0hsumi]

Could you try what happens if you use lowerdir/b as the the lowerdir of a fuse-overlay instance? Does it then show b/foo correctly?

Yeah, it does!

Then we could at least create a fresh fuse-overlayfs instance for every bind mount.

What about directories that are not part of any bind mount? I tried two cases, but neither worked properly:

1. Mounting the parent directory first, then mounting the bind-mounted directory: Permission denied.

   $ fuse-overlayfs -o lowerdir=lowerdir/,upperdir=updir,workdir=workdir fuse_mount/
   $ mkdir fuse_mount/b
   $ fuse-overlayfs -o lowerdir=lowerdir/b,upperdir=updir,workdir=workdir fuse_mount/b
   fusermount3: failed to access mountpoint /home/t0hsumi/prog/benchexec-dev/fuse_mount/b: Permission denied
   fuse-overlayfs: cannot mount: Operation not permitted
   

2. Mounting the bind-mounted directory first, then mounting the parent directory: fuse_mount is unexpectedly updated.

   $ fuse-overlayfs -o lowerdir=lowerdir/b,upperdir=updir,workdir=workdir fuse_mount/b
   $ ls fuse_mount/b
   foo
   $ fuse-overlayfs -o lowerdir=lowerdir,upperdir=updir,workdir=workdir fuse_mount
   $ ls fuse_mount/
   a
   

Do you have any suggestions on how to resolve this issue?

[PhilippWendler]

Then we could at least create a fresh fuse-overlayfs instance for every bind mount.

What about directories that are not part of any bind mount? I tried two cases, but neither worked properly:

1. Mounting the parent directory first, then mounting the bind-mounted directory: Permission denied.

   $ fuse-overlayfs -o lowerdir=lowerdir/,upperdir=updir,workdir=workdir fuse_mount/
   $ mkdir fuse_mount/b
   $ fuse-overlayfs -o lowerdir=lowerdir/b,upperdir=updir,workdir=workdir fuse_mount/b
   fusermount3: failed to access mountpoint /home/t0hsumi/prog/benchexec-dev/fuse_mount/b: Permission denied
   fuse-overlayfs: cannot mount: Operation not permitted
   

Hm. Here you would need to use distinct upperdirs and workdirs for each fuse-overlayfs instance (as upperdir we would want to use updir/b in the second case, as workdir some fresh directory). But I fear that this is unrelated to the mount failure and that it would still not work.

Then yet another possibility might be to use some fresh directory fuse_mount_b as target of the second call, and afterwards bind mount that to fuse_mount/b.

2. Mounting the bind-mounted directory first, then mounting the parent directory: fuse_mount is unexpectedly updated.

   $ fuse-overlayfs -o lowerdir=lowerdir/b,upperdir=updir,workdir=workdir fuse_mount/b
   $ ls fuse_mount/b
   foo
   $ fuse-overlayfs -o lowerdir=lowerdir,upperdir=updir,workdir=workdir fuse_mount
   $ ls fuse_mount/
   a
   

Yes, this is how all mounts work. If you mount something on fuse_mount, it hides everything what was there before.

[t0hsumi]

I tried several approaches and here's what I found so far.

fuse-overlayfs with allow_other + mount --bind works fine

Here's a successful setup:

$ mkdir -p lowerdir/{a,b}
$ sudo mount --bind lowerdir/a lowerdir/b
$ touch lowerdir/a/foo
$ ls lowerdir/{a,b}
lowerdir/a:
foo

lowerdir/b:
foo
$ mkdir updir workdir fuse_mount
$ fuse-overlayfs -o lowerdir=lowerdir,upperdir=updir,workdir=workdir,allow_other fuse_mount
$ ls fuse_mount/
a
$ mkdir fuse_mount/b
$ sudo mount --bind lowerdir/b fuse_mount/b
$ ls fuse_mount/b
foo

If we proceed with this method, the users will need to modify the contents of /etc/fuse.conf for setup (man 8 fuse).

Below are the unsuccessful attempts.

fuse-overlayfs with allow_other + fuse-overlayfs with allow_other didn't work.

$ fuse-overlayfs -o lowerdir=lowerdir,upperdir=updir,workdir=workdir,allow_other fuse_mount
$ ls fuse_mount/
a
$ mkdir fuse_mount/b workdir_b updir/b
$ fuse-overlayfs -o lowerdir=lowerdir/b,upperdir=updir/b,workdir=workdir_b,allow_other fuse_mount/b
$ ls fuse_mount/b

Hm. Here you would need to use distinct upperdirs and workdirs for each fuse-overlayfs instance (as upperdir we would want to use updir/b in the second case, as workdir some fresh directory). But I fear that this is unrelated to the mount failure and that it would still not work.

I also tried fuse-overlayfs (without allow_other) + fuse-overlayfs (without allow_other) and that didn't work neither.

Then yet another possibility might be to use some fresh directory fuse_mount_b as target of the second call, and afterwards bind mount that to fuse_mount/b.

I attempted this, but it did not work (I'm not sure if I did what was intended).

$ mkdir lowerdir/{a,b}
$ sudo mount --bind lowerdir/a lowerdir/b
$ touch lowerdir/a/foo
$ mkdir updir workdir fuse_mount
$ fuse-overlayfs -o lowerdir=lowerdir,upperdir=updir,workdir=workdir fuse_mount
$ mkdir updir/b workdir_b fuse_mount_b
$ fuse-overlayfs -o lowerdir=lowerdir/b,upperdir=updir/b,workdir=workdir_b fuse_mount_b
$ ls fuse_mount
a
$ mkdir fuse_mount/b
$ sudo mount fuse_mount_b fuse_mount/b
mount: fuse_mount/b: cannot mount /home/t0hsumi/prog/benchexec-dev/fuse_mount_b read-only.
       dmesg(1) may have more information after failed mount system call.

Adding allow_other to the second overlay-fs command resulted in the same as above.

Adding allow_other to the first overlay-fs command resulted in:

mount: /home/t0hsumi/prog/benchexec-dev/fuse_mount/b: /home/t0hsumi/prog/benchexec-dev/fuse_mount_b is not a block device, and stat(2) fails?.
       dmesg(1) may have more information after failed mount system call.

Adding allow_other to the first and second overlay-fs commands resulted in:

mount: /home/t0hsumi/prog/benchexec-dev/fuse_mount/b: /home/t0hsumi/prog/benchexec-dev/fuse_mount_b is not a block device.
       dmesg(1) may have more information after failed mount system call.

[t0hsumi]

Yes, this is how all mounts work. If you mount something on fuse_mount, it hides everything what was there before.

Sorry, this is exactly what I don't want the mount to do.

[PhilippWendler]

I tried several approaches and here's what I found so far.

fuse-overlayfs with allow_other + mount --bind works fine

Here's a successful setup:

$ mkdir -p lowerdir/{a,b}
$ sudo mount --bind lowerdir/a lowerdir/b
$ touch lowerdir/a/foo
$ ls lowerdir/{a,b}
lowerdir/a:
foo

lowerdir/b:
foo
$ mkdir updir workdir fuse_mount
$ fuse-overlayfs -o lowerdir=lowerdir,upperdir=updir,workdir=workdir,allow_other fuse_mount
$ ls fuse_mount/
a
$ mkdir fuse_mount/b
$ sudo mount --bind lowerdir/b fuse_mount/b
$ ls fuse_mount/b
foo

But this directly mounts lowerdir/b to fuse_mount/b, which means that write accesses to fuse_mount/b will go to the underlying file system and not be captured by the overlayfs.

If we proceed with this method, the users will need to modify the contents of /etc/fuse.conf for setup (man 8 fuse).

Hm, that would be unfortunate, because it requires root permission at least for installation, and we are happy that most of BenchExec works without that. Also, it seems there is no way to grant this to only a particular user, and system administrators might not always want to give this permission to everyone.

But I think I found a solution (more below).

Below are the unsuccessful attempts.

fuse-overlayfs with allow_other + fuse-overlayfs with allow_other didn't work.

$ fuse-overlayfs -o lowerdir=lowerdir,upperdir=updir,workdir=workdir,allow_other fuse_mount
$ ls fuse_mount/
a
$ mkdir fuse_mount/b workdir_b updir/b
$ fuse-overlayfs -o lowerdir=lowerdir/b,upperdir=updir/b,workdir=workdir_b,allow_other fuse_mount/b
$ ls fuse_mount/b

This is also something that I do not understand. If the mount to fuse_mount/b succeeds, why should it show something empty and not the content of lowerdir/b afterwards? 😕 It even works as expected (updir/b/bar being created) if I do touch fuse_mount/b/bar...

Then yet another possibility might be to use some fresh directory fuse_mount_b as target of the second call, and afterwards bind mount that to fuse_mount/b.

I attempted this, but it did not work (I'm not sure if I did what was intended).

$ mkdir lowerdir/{a,b}
$ sudo mount --bind lowerdir/a lowerdir/b
$ touch lowerdir/a/foo
$ mkdir updir workdir fuse_mount
$ fuse-overlayfs -o lowerdir=lowerdir,upperdir=updir,workdir=workdir fuse_mount
$ mkdir updir/b workdir_b fuse_mount_b
$ fuse-overlayfs -o lowerdir=lowerdir/b,upperdir=updir/b,workdir=workdir_b fuse_mount_b
$ ls fuse_mount
a
$ mkdir fuse_mount/b
$ sudo mount fuse_mount_b fuse_mount/b
mount: fuse_mount/b: cannot mount /home/t0hsumi/prog/benchexec-dev/fuse_mount_b read-only.
       dmesg(1) may have more information after failed mount system call.

I meant this, but with --bind in the last mount call.

If I try it as you wrote it down and without allow_other, it also does not work. I didn't try with allow_other, but I managed to this series of mounts working inside a container.

My understanding is as follows: fuse-overlayfs sees a risk in allowing other users to access the mount point. This is why several operations are forbidden, basically whenever another user is involved. If we want to mount something over an existing fuse mount, this is done by another user (either explicitly root in case of sudo mount, or implicitly by the kernel in case of fuse-overlayfs over fuse-overlayfs).

But if we are in a container, all of the container setup is done by the root user of the container. This means if we first do a fuse-overlayfs mount and then a mount --bind mount over it, both are performed by the same user, and they are allowed.

To try this out we need to enter an interactive container that works the same as BenchExec's container but where we are root. (If we just start containerexec bash, we are not root and cannot use the mount command. If we start containerexec --root bash, we are root but BenchExec has removed all capabilities from us, so we also cannot use the mount command.) One way is to take BenchExec, remove the drop_capabilities() calls in the source code, and then use containerexec --root --read-only-dir / --full-access-dir . bash. Another way to create such a container is unshare -mUp --fork --mount-proc --map-root-user. Once we are in this container, we can perform this sequence:

# mkdir lowerdir lowerdir/{a,b} updir workdir fuse_mount updir/b workdir_b fuse_mount_b
# mount --bind lowerdir/a lowerdir/b
# touch lowerdir/a/foo
# fuse-overlayfs -o lowerdir=lowerdir,upperdir=updir,workdir=workdir fuse_mount
# ls fuse_mount/*
fuse_mount/a:
foo

fuse_mount/b:
foo
# fuse-overlayfs -o lowerdir=lowerdir/b,upperdir=updir/b,workdir=workdir_b fuse_mount_b
# mount --bind fuse_mount_b fuse_mount/b
# ls fuse_mount/*
fuse_mount/a:
foo

fuse_mount/b:
foo

But I tried further, and now I am even more confused. Because if I try the fuse-overlayfs mount in the container, I even manage to get it working just fine across the bind mount (cf. first ls call), no need even for a second fuse-overlayfs mount. So why does not work in BenchExec then?

I hope it is somewhat clear what I wrote? Can you reproduce this?

I think one thing that we can learn is that there are subtle differences between usage of fuse-overlayfs outside of a container as regular user, and inside a container as container root. And we only care about the latter, so we need to try out stuff in the container case. And if my last example works, then there should be in principle a way to make it work in BenchExec as well, because this case is exactly like we want it.

[t0hsumi]

But this directly mounts lowerdir/b to fuse_mount/b, which means that write accesses to fuse_mount/b will go to the underlying file system and not be captured by the overlayfs.

Sorry, I completely forgot the original intention of creating the container.

I hope it is somewhat clear what I wrote? Can you reproduce this?

Thanks for the detailed explanation. I understand what you’re confused about.
While trying to reproduce the issue, I found that if updir/b exists before executing the first fuse-overlayfs command, then fuse_mount/b/foo is present (even when I am not inside the container):

$ mkdir lowerdir lowerdir/{a,b} updir updir/b workdir fuse_mount
$ touch lowerdir/a/foo
$ sudo mount --bind lowerdir/a lowerdir/b
$ fuse-overlayfs -o lowerdir=lowerdir,upperdir=updir,workdir=workdir fuse_mount
$ ls fuse_mount/*
fuse_mount/a:
foo

fuse_mount/b:
foo

On the other hand, if updir/b does not exist before executing the first fuse-overlayfs command, then fuse_mount/b/foo is not present (even inside the container).

My guess is as follows: If updir/b does not exist, fuse-overlayfs refers to lowerdir/b but treats it as bodiless and does not handle it as expected. If updir/b exists, fuse-overlayfs recognizes b as a directory and considers its contents from both updir/b/* (empty) and lowerdir/b/* (where foo exists).

[PhilippWendler]

Thanks for the detailed explanation. I understand what you’re confused about. While trying to reproduce the issue, I found that if updir/b exists before executing the first fuse-overlayfs command, then fuse_mount/b/foo is present (even when I am not inside the container):

On the other hand, if updir/b does not exist before executing the first fuse-overlayfs command, then fuse_mount/b/foo is not present (even inside the container).

My guess is as follows: If updir/b does not exist, fuse-overlayfs refers to lowerdir/b but treats it as bodiless and does not handle it as expected. If updir/b exists, fuse-overlayfs recognizes b as a directory and considers its contents from both updir/b/* (empty) and lowerdir/b/* (where foo exists).

Oh yes, of course. 🤦

It would be interesting to test whether this works as we need, i.e., are writes then sent to updir/b, and is the content of lowerdir/b/ and updir/b visible as expected?

And does it also work if we create updir/b after starting fuse-overlayfs, or only if we create it before?

Because if all of this works, we could get a solution if we just create the necessary updir/... directories in BenchExec, right? (I hope I am not still confused and overlooking something.)

Would you like to test this? Your help in this has been invaluable already!

[t0hsumi]

Of course, I'd like to! I'll reply in detail within 24 hours.

[t0hsumi]

It would be interesting to test whether this works as we need, i.e., are writes then sent to updir/b, and is the content of lowerdir/b/ and updir/b visible as expected?

If I write to fuse_mount/b/foo, the writes are sent to updir/b. Similarly, if I write to fuse_mount/a/foo, the writes are also sent to updir/b. Otherwise, everything works as expected (i.e., modifications made under fuse_mount are reflected in updir but not in lowerdir).

$ mkdir lowerdir lowerdir/{a,b} updir updir/b workdir fuse_mount
$ touch lowerdir/a/foo
$ sudo mount --bind lowerdir/a lowerdir/b
$ fuse-overlayfs -o lowerdir=lowerdir,upperdir=updir,workdir=workdir fuse_mount
$ ls -R updir/
updir/:
b

updir/b:
$ echo "foo b" > fuse_mount/b/foo
$ cat updir/b/foo
foo b
$ echo "foo a" > fuse_mount/a/foo
$ cat updir/b/foo
foo a
$ ls -R updir/
updir/:
b

updir/b:
foo

And does it also work if we create updir/b after starting fuse-overlayfs, or only if we create it before?

If we create updir/b after starting fuse-overlayfs, fuse_mount/b does not exist, which I believe is not the intended behavior. At the very least, the content in lowerdir/b is invisible in fuse_mount.

[PhilippWendler]

It would be interesting to test whether this works as we need, i.e., are writes then sent to updir/b, and is the content of lowerdir/b/ and updir/b visible as expected?

If I write to fuse_mount/b/foo, the writes are sent to updir/b. Similarly, if I write to fuse_mount/a/foo, the writes are also sent to updir/b.

Wow, also unexpected. Maybe fuse-overlayfs sees that lowerdir/a/foo and lowerdir/b/foo have the same inode and then reuse updir/b/foo or?

But I guess this would be ok for us.

So I think we have something now that could work, and it would be worth to implement it for being able to try it out?

If we create updir/b after starting fuse-overlayfs, fuse_mount/b does not exist, which I believe is not the intended behavior. At the very least, the content in lowerdir/b is invisible in fuse_mount.

Indeed.

[t0hsumi]

Wow, also unexpected. Maybe fuse-overlayfs sees that lowerdir/a/foo and lowerdir/b/foo have the same inode and then reuse updir/b/foo or?

Yeah, it seems. lowerdir/a/foo, lowerdir/b/foo, fuse_mount/a/foo and fuse_mount/b/foo have the same inode, but updir/b/foo doesn't.

So I think we have something now that could work, and it would be worth to implement it for being able to try it out?

I agree! I just created another PR.

[PhilippWendler]

Replaced by #1156.