Nested Singularity: `failed to add as session directory`

[deliciouslytyped]

> singularity --version
singularity-ce version 4.1.1

xposted from https://stackoverflow.com/questions/67451148/failed-to-add-as-session-directory-path-is-not-an-absolute-path-in-singularit/78883567#78883567

For me this appears to have happened because I was trying to invoke Singularity in Singularity, where somehow my outer UID/GID was leaking in through wherever Singularity is looking it up from, and presumably was not finding an entry corresponding to the user in /etc/passwd; specifically the error message indicates that something that looks like it should be a directory - is empty (note the two consecutive spaces in add as).

My invocation:

Singularity> singularity shell --fakeroot --userns something.sif
WARNING: Could not lookup the current user's information: user: unknown userid 12345
INFO:    Mounting image with FUSE.
WARNING: squashfuse mount failed, falling back to extraction: use of image "/root/something.sif" as overlay requires fusermount to be installed: exec: "fusermount": executable file not found in $PATH
INFO:    Converting SIF file to temporary sandbox...
INFO:    Cleaning up image...
FATAL:   container creation failed: failed to add  as session directory: path . is not an absolute path

Once I added something like sing:x:12345:12345:Ubuntu:/home/sing:/bin/bash despite not actually using this for anything inside the container, it succeeds:

Singularity> singularity shell --fakeroot --userns something.sif
INFO:    Mounting image with FUSE.
WARNING: squashfuse mount failed, falling back to extraction: use of image "/root/something.sif" as overlay requires fusermount to be installed: exec: "fusermount": executable file not found in $PATH
INFO:    Converting SIF file to temporary sandbox...
Singularity> 

Speculation:

I searched the --debug output for "direc" (as in "directory"), and ran across the following, which is what prompted me to try passwd, knowing from observation that Singularity normally appears to mount the users home directory somewhere;

...
DEBUG   [U=0,P=3034357]    addHomeMount()                Skipping home directory mount by user request.
...
DEBUG   [U=0,P=3034878]    prepareAutofs()               Could not keep file descriptor for home directory /root: no mount point
DEBUG   [U=0,P=3034878]    prepareAutofs()               Could not keep file descriptor for current working directory /root: no mount point
..

Perpaps stuff like #1822 is related.

[dtrudg]

Thanks for the information. @cyanezstange is currently spending some time to catalogue issue with nested container execution of Singularity (though concentrating on Singularity-in-Docker). We'll look into this further when we are able.

[deliciouslytyped]

Off topic,

@cyanezstange is there anywhere you are doing this? I tend to struggle with nested container stuff rather often, so I may be able to add some points. On the other hand, it's not hard to be more of an expert at this than me, so hopefully everything is covered already. ;)

Just today I had to have it pointed out to me that the reason docker-in-singularity isn't working is probably because singularity should be setting pr_no_new_privs per https://docs.sylabs.io/guides/4.0/user-guide/security.html#runtime-user-privilege-model

[dtrudg]

@deliciouslytyped - there isn't anywhere right now, I'm afraid. This is an area that hasn't been well documented, and isn't well tested in our test suite. It's an aim for the SingularityCE 4.3 release to address this. That means it'll be worked on through to Nov / Dec.

@cyanezstange will be creating issues to track problems he finds, and once the initial work is done we'll also try and get some documentation in place, via PRs to sylabs/singularity-userdocs

It's on the roadmap here...

#3224