"permission denied" error when exec-ing non-exectuable is not friendly

[marc-rigter]

Version of Singularity
3.7.3
I also get the same behaviour with version 3.9.

Describe the bug
When I try to execute the container using
"sudo singularity exec --bind /home/marcrigter/github/mopo:/home/mopo mopo.sif /home/mopo/scripts/test.sh"
I get an error "StageTwo() permission denied" and no other debug information.

Here is the output when using the -d flag:

DEBUG [U=0,P=109209] persistentPreRun() Singularity version: 3.7.3
DEBUG [U=0,P=109209] persistentPreRun() Parsing configuration file /usr/local/etc/singularity/singularity.conf
DEBUG [U=0,P=109209] handleConfDir() /root/.singularity already exists. Not creating.
DEBUG [U=0,P=109209] execStarter() Saving umask 0022 for propagation into container
DEBUG [U=0,P=109209] execStarter() Checking for encrypted system partition
DEBUG [U=0,P=109209] Init() Image format detection
DEBUG [U=0,P=109209] Init() Check for sandbox image format
DEBUG [U=0,P=109209] Init() sandbox format initializer returned: not a directory image
DEBUG [U=0,P=109209] Init() Check for sif image format
DEBUG [U=0,P=109209] Init() sif image format detected
DEBUG [U=0,P=109209] SetContainerEnv() Forwarding LANGUAGE environment variable
DEBUG [U=0,P=109209] SetContainerEnv() Forwarding LANG environment variable
DEBUG [U=0,P=109209] SetContainerEnv() Forwarding LS_COLORS environment variable
DEBUG [U=0,P=109209] SetContainerEnv() Forwarding TERM environment variable
DEBUG [U=0,P=109209] SetContainerEnv() Forwarding MAIL environment variable
DEBUG [U=0,P=109209] SetContainerEnv() Forwarding LOGNAME environment variable
DEBUG [U=0,P=109209] SetContainerEnv() Forwarding USER environment variable
DEBUG [U=0,P=109209] SetContainerEnv() Forwarding SHELL environment variable
DEBUG [U=0,P=109209] SetContainerEnv() Forwarding SUDO_COMMAND environment variable
DEBUG [U=0,P=109209] SetContainerEnv() Forwarding SUDO_USER environment variable
DEBUG [U=0,P=109209] SetContainerEnv() Forwarding SUDO_UID environment variable
DEBUG [U=0,P=109209] SetContainerEnv() Forwarding SUDO_GID environment variable
DEBUG [U=0,P=109209] SetContainerEnv() Forwarding USER_PATH environment variable
VERBOSE [U=0,P=109209] SetContainerEnv() Setting HOME=/root
VERBOSE [U=0,P=109209] SetContainerEnv() Setting PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBUG [U=0,P=109209] init() Use starter binary /usr/local/libexec/singularity/bin/starter
VERBOSE [U=0,P=109209] print() Set messagelevel to: 5
VERBOSE [U=0,P=109209] init() Starter initialization
DEBUG [U=0,P=109209] load_overlay_module() Trying to load overlay kernel module
DEBUG [U=0,P=109209] load_overlay_module() Overlay seems supported by the kernel
VERBOSE [U=0,P=109209] is_suid() Check if we are running as setuid
DEBUG [U=0,P=109209] read_engine_config() Read engine configuration
DEBUG [U=0,P=109209] init() Wait completion of stage1
DEBUG [U=0,P=109220] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=0,P=109220] init() Spawn stage 1
DEBUG [U=0,P=109220] startup() singularity runtime engine selected
VERBOSE [U=0,P=109220] startup() Execute stage 1
DEBUG [U=0,P=109220] StageOne() Entering stage 1
DEBUG [U=0,P=109220] prepareRootCaps() Root full capabilities
DEBUG [U=0,P=109220] prepareAutofs() Found "/proc/sys/fs/binfmt_misc" as autofs mount point
DEBUG [U=0,P=109220] prepareAutofs() Could not keep file descriptor for user bind path /home/mrigter/github/mopo: no mount point
DEBUG [U=0,P=109220] prepareAutofs() Could not keep file descriptor for bind path /etc/localtime: no mount point
DEBUG [U=0,P=109220] prepareAutofs() Could not keep file descriptor for bind path /etc/hosts: no mount point
DEBUG [U=0,P=109220] prepareAutofs() Could not keep file descriptor for home directory /root: no mount point
DEBUG [U=0,P=109220] prepareAutofs() Could not keep file descriptor for current working directory /home/mrigter/github/mopo/singularity: no mount point
DEBUG [U=0,P=109220] Init() Image format detection
DEBUG [U=0,P=109220] Init() Check for sandbox image format
DEBUG [U=0,P=109220] Init() sandbox format initializer returned: not a directory image
DEBUG [U=0,P=109220] Init() Check for sif image format
DEBUG [U=0,P=109220] Init() sif image format detected
DEBUG [U=0,P=109220] setSessionLayer() Overlay seems supported and allowed by kernel
DEBUG [U=0,P=109220] setSessionLayer() Attempting to use overlayfs (enable overlay = try)
VERBOSE [U=0,P=109209] wait_child() stage 1 exited with status 0
DEBUG [U=0,P=109209] cleanup_fd() Close file descriptor 4
DEBUG [U=0,P=109209] cleanup_fd() Close file descriptor 5
DEBUG [U=0,P=109209] cleanup_fd() Close file descriptor 6
DEBUG [U=0,P=109209] init() Set child signal mask
DEBUG [U=0,P=109209] init() Create socketpair for master communication channel
DEBUG [U=0,P=109209] init() Create RPC socketpair for communication between stage 2 and RPC server
VERBOSE [U=0,P=109209] init() Spawn master process
DEBUG [U=0,P=109226] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=0,P=109226] create_namespace() Create mount namespace
VERBOSE [U=0,P=109209] enter_namespace() Entering in mount namespace
DEBUG [U=0,P=109209] enter_namespace() Opening namespace file ns/mnt
VERBOSE [U=0,P=109226] create_namespace() Create mount namespace
VERBOSE [U=0,P=109232] init() Spawn RPC server
DEBUG [U=0,P=109209] startup() singularity runtime engine selected
VERBOSE [U=0,P=109209] startup() Execute master process
DEBUG [U=0,P=109232] startup() singularity runtime engine selected
VERBOSE [U=0,P=109232] startup() Serve RPC requests
DEBUG [U=0,P=109209] setupSessionLayout() Using Layer system: overlay
DEBUG [U=0,P=109209] setupOverlayLayout() Creating overlay SESSIONDIR layout
DEBUG [U=0,P=109209] addRootfsMount() Mount rootfs in read-only mode
DEBUG [U=0,P=109209] addRootfsMount() Image type is 4096
DEBUG [U=0,P=109209] addRootfsMount() Mounting block [squashfs] image: /home/mrigter/github/mopo/singularity/mopo.sif
DEBUG [U=0,P=109209] addKernelMount() Checking configuration file for 'mount proc'
DEBUG [U=0,P=109209] addKernelMount() Adding proc to mount list
VERBOSE [U=0,P=109209] addKernelMount() Default mount: /proc:/proc
DEBUG [U=0,P=109209] addKernelMount() Checking configuration file for 'mount sys'
DEBUG [U=0,P=109209] addKernelMount() Adding sysfs to mount list
VERBOSE [U=0,P=109209] addKernelMount() Default mount: /sys:/sys
DEBUG [U=0,P=109209] addDevMount() Checking configuration file for 'mount dev'
DEBUG [U=0,P=109209] addDevMount() Adding dev to mount list
VERBOSE [U=0,P=109209] addDevMount() Default mount: /dev:/dev
DEBUG [U=0,P=109209] addHostMount() Not mounting host file systems per configuration
VERBOSE [U=0,P=109209] addBindsMount() Found 'bind path' = /etc/localtime, /etc/localtime
VERBOSE [U=0,P=109209] addBindsMount() Found 'bind path' = /etc/hosts, /etc/hosts
DEBUG [U=0,P=109209] addHomeStagingDir() Staging home directory (/root) at /usr/local/var/singularity/mnt/session/root
DEBUG [U=0,P=109209] addHomeMount() Adding home directory mount [/usr/local/var/singularity/mnt/session/root:/root] to list using layer: overlay
DEBUG [U=0,P=109209] addUserbindsMount() Adding /home/mrigter/github/mopo to mount list
DEBUG [U=0,P=109209] addTmpMount() Checking for 'mount tmp' in configuration file
VERBOSE [U=0,P=109209] addTmpMount() Default mount: /tmp:/tmp
VERBOSE [U=0,P=109209] addTmpMount() Default mount: /var/tmp:/var/tmp
DEBUG [U=0,P=109209] addScratchMount() Not mounting scratch directory: Not requested
DEBUG [U=0,P=109209] addLibsMount() Checking for 'user bind control' in configuration file
DEBUG [U=0,P=109209] addFilesMount() Checking for 'user bind control' in configuration file
DEBUG [U=0,P=109209] addResolvConfMount() Adding /etc/resolv.conf to mount list
VERBOSE [U=0,P=109209] addResolvConfMount() Default mount: /etc/resolv.conf:/etc/resolv.conf
DEBUG [U=0,P=109209] addHostnameMount() Skipping hostname mount, not virtualizing UTS namespace on user request
DEBUG [U=0,P=109209] create() Mount all
DEBUG [U=0,P=109209] mountGeneric() Mounting tmpfs to /usr/local/var/singularity/mnt/session
DEBUG [U=0,P=109209] mountImage() Mounting loop device /dev/loop8 to /usr/local/var/singularity/mnt/session/rootfs of type squashfs
DEBUG [U=0,P=109209] mountGeneric() Mounting overlay to /usr/local/var/singularity/mnt/session/final
DEBUG [U=0,P=109209] mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final
DEBUG [U=0,P=109209] setPropagationMount() Set RPC mount propagation flag to SLAVE
VERBOSE [U=0,P=109209] addIdentityMount() Not updating passwd/group files, running as root!
DEBUG [U=0,P=109209] mountGeneric() Mounting /dev to /usr/local/var/singularity/mnt/session/final/dev
DEBUG [U=0,P=109209] mountGeneric() Mounting /etc/localtime to /usr/local/var/singularity/mnt/session/final/usr/share/zoneinfo/Etc/UTC
DEBUG [U=0,P=109209] mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final/usr/share/zoneinfo/Etc/UTC
DEBUG [U=0,P=109209] mountGeneric() Mounting /etc/hosts to /usr/local/var/singularity/mnt/session/final/etc/hosts
DEBUG [U=0,P=109209] mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final/etc/hosts
DEBUG [U=0,P=109209] mountGeneric() Mounting /proc to /usr/local/var/singularity/mnt/session/final/proc
DEBUG [U=0,P=109209] mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final/proc
DEBUG [U=0,P=109209] mountGeneric() Mounting sysfs to /usr/local/var/singularity/mnt/session/final/sys
DEBUG [U=0,P=109209] mountGeneric() Mounting /root to /usr/local/var/singularity/mnt/session/root
DEBUG [U=0,P=109209] mountGeneric() Remounting /usr/local/var/singularity/mnt/session/root
DEBUG [U=0,P=109209] mountGeneric() Mounting /usr/local/var/singularity/mnt/session/root to /usr/local/var/singularity/mnt/session/final/root
DEBUG [U=0,P=109209] mountGeneric() Mounting /tmp to /usr/local/var/singularity/mnt/session/final/tmp
DEBUG [U=0,P=109209] mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final/tmp
DEBUG [U=0,P=109209] mountGeneric() Mounting /var/tmp to /usr/local/var/singularity/mnt/session/final/var/tmp
DEBUG [U=0,P=109209] mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final/var/tmp
DEBUG [U=0,P=109209] mountGeneric() Mounting /usr/local/var/singularity/mnt/session/etc/resolv.conf to /usr/local/var/singularity/mnt/session/final/etc/resolv.conf
DEBUG [U=0,P=109209] mountGeneric() Mounting /home/mrigter/github/mopo to /usr/local/var/singularity/mnt/session/final/home/mopo
DEBUG [U=0,P=109209] mountGeneric() Remounting /usr/local/var/singularity/mnt/session/final/home/mopo
DEBUG [U=0,P=109209] addCwdMount() Using /home/mrigter/github/mopo/singularity as current working directory
VERBOSE [U=0,P=109209] addCwdMount() Not mounting CWD, /home/mrigter/github/mopo/singularity doesn't exist within container
VERBOSE [U=0,P=109209] addCwdMount() Not mounting CWD, while getting /home/mrigter/github/mopo/singularity information: stat /usr/local/var/singularity/mnt/session/final/home/mrigter/github/mopo/singularity: no such file or directory
DEBUG [U=0,P=109209] create() Chroot into /usr/local/var/singularity/mnt/session/final
DEBUG [U=0,P=109232] Chroot() Hold reference to host / directory
DEBUG [U=0,P=109232] Chroot() Called pivot_root on /usr/local/var/singularity/mnt/session/final
DEBUG [U=0,P=109232] Chroot() Change current directory to host / directory
DEBUG [U=0,P=109232] Chroot() Apply slave mount propagation for host / directory
DEBUG [U=0,P=109232] Chroot() Called unmount(/, syscall.MNT_DETACH)
DEBUG [U=0,P=109232] Chroot() Changing directory to / to avoid getpwd issues
DEBUG [U=0,P=109209] create() Chdir into / to avoid errors
VERBOSE [U=0,P=109226] wait_child() rpc server exited with status 0
DEBUG [U=0,P=109226] init() Set container privileges
DEBUG [U=0,P=109226] apply_privileges() Effective capabilities: 0x0000003fffffffff
DEBUG [U=0,P=109226] apply_privileges() Permitted capabilities: 0x0000003fffffffff
DEBUG [U=0,P=109226] apply_privileges() Bounding capabilities: 0x0000003fffffffff
DEBUG [U=0,P=109226] apply_privileges() Inheritable capabilities: 0x0000003fffffffff
DEBUG [U=0,P=109226] apply_privileges() Ambient capabilities: 0x0000003fffffffff
DEBUG [U=0,P=109226] apply_privileges() Set user ID to 0
DEBUG [U=0,P=109226] set_parent_death_signal() Set parent death signal to 9
DEBUG [U=0,P=109226] startup() singularity runtime engine selected
VERBOSE [U=0,P=109226] startup() Execute stage 2
DEBUG [U=0,P=109226] StageTwo() Entering stage 2
DEBUG [U=0,P=109226] StartProcess() Setting umask in container to 0022
DEBUG [U=0,P=109226] sylogBuiltin() Sourcing /.singularity.d/env/01-base.sh
DEBUG [U=0,P=109226] sylogBuiltin() Sourcing /.singularity.d/env/10-docker2singularity.sh
DEBUG [U=0,P=109226] sylogBuiltin() Sourcing /.singularity.d/env/90-environment.sh
DEBUG [U=0,P=109226] sylogBuiltin() Sourcing /.singularity.d/env/91-environment.sh
DEBUG [U=0,P=109226] sylogBuiltin() Sourcing /.singularity.d/env/94-appsbase.sh
DEBUG [U=0,P=109226] sylogBuiltin() Sourcing /.singularity.d/env/95-apps.sh
DEBUG [U=0,P=109226] sylogBuiltin() Sourcing /.singularity.d/env/99-base.sh
DEBUG [U=0,P=109226] sylogBuiltin() Sourcing /.singularity.d/env/99-runtimevars.sh
DEBUG [U=0,P=109226] sylogBuiltin() Running action command exec
FATAL [U=0,P=109226] StageTwo() permission denied
DEBUG [U=0,P=109209] startContainer() stage 2 process reported an error, waiting status
DEBUG [U=0,P=109209] Master() Child exited with exit status 255

To Reproduce
Build container using the following definition file and execute using singularity exec as above:

Bootstrap: docker
From: continuumio/miniconda3
Stage: devel

%setup

%files
builder.py /home/
requirements.txt /home/
gpu-env.yml /home/
setup.py /home/

%environment
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/.mujoco/mujoco210/bin/
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/nvidia/lib64
export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libGLEW.so
export MJLIB_PATH=/home/.mujoco/mujoco-2.1.1/lib/libmujoco.so.2.1.1
export MUJOCO_PY_MUJOCO_PATH=/home/.mujoco/mujoco210

%post
apt-get update -q
&& DEBIAN_FRONTEND=noninteractive apt-get install -y
build-essential
wget
curl
git
libgl1-mesa-dev
libgl1-mesa-glx
libglew-dev
libopenmpi-dev
libosmesa6-dev
software-properties-common
net-tools
vim
xpra
nano
xserver-xorg-dev

#Install Mujoco Binaries. 2.1.1 for dm_control and 2.1.0 for mujoco-py
mkdir /home/.mujoco
wget https://github.com/deepmind/mujoco/releases/download/2.1.1/mujoco-2.1.1-linux-x86_64.tar.gz -O mujoco.tar.gz
tar -xf mujoco.tar.gz -C /home/.mujoco
rm mujoco.tar.gz

wget https://mujoco.org/download/mujoco210-linux-x86_64.tar.gz -O mujoco.tar.gz
tar -xf mujoco.tar.gz -C /home/.mujoco
rm mujoco.tar.gz

# create environment
/opt/conda/bin/conda env create -f /home/gpu-env.yml
echo ". /opt/conda/etc/profile.d/conda.sh" >> $SINGULARITY_ENVIRONMENT
echo "conda activate mopo" >> $SINGULARITY_ENVIRONMENT
. /opt/conda/etc/profile.d/conda.sh
conda activate mopo

#Create the mujoco environment variables
export MJLIB_PATH=/home/.mujoco/mujoco-2.1.1/lib/libmujoco.so.2.1.1
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/.mujoco/mujoco-2.1.1/bin/
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/nvidia/lib64
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/.mujoco/mujoco210/bin
export MUJOCO_PY_MUJOCO_PATH=/home/.mujoco/mujoco210
export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libGLEW.so

conda install -c conda-forge wandb
pip uninstall typing
pip install -r /home/requirements.txt

# install D4RL
git clone https://github.com/rail-berkeley/d4rl.git
rm d4rl/setup.py
cp /home/setup.py d4rl/
cd d4rl
pip install -e .
cd ..

#Install PatchELF for mujoco-py
curl -o /usr/local/bin/patchelf https://s3-us-west-2.amazonaws.com/openai-sci-artifacts/manual-builds/patchelf_0.9_amd64.elf
chmod +x /usr/local/bin/patchelf

# install libgcrypt11
wget http://archive.ubuntu.com/ubuntu/pool/main/g/glibc/multiarch-support_2.27-3ubuntu1.2_amd64.deb
dpkg -i multiarch-support_2.27-3ubuntu1.2_amd64.deb
wget https://launchpad.net/~ubuntu-security-proposed/+archive/ubuntu/ppa/+build/7110687/+files/libgcrypt11_1.5.4-2ubuntu1.1_amd64.deb
dpkg -i libgcrypt11_1.5.4-2ubuntu1.1_amd64.deb

rm libgcrypt11_1.5.4-2ubuntu1.1_amd64.deb
rm multiarch-support_2.27-3ubuntu1.2_amd64.deb

rm /opt/conda/envs/mopo/lib/python3.7/site-packages/mujoco_py/builder.py
cp /home/builder.py /opt/conda/envs/mopo/lib/python3.7/site-packages/mujoco_py/

python3 -c 'import mujoco_py'

%labels
Author Marc Rigter
Version v0.0.1

%help
singularity run --nv -B .:/home/bind --network-args "portmap=8888:8888/tcp" mujoco.sif
This container is a a demo for use with PyTorch and Cuda. Please use the flag --nv
I'd suggest using the bind command to bind in the source code for building and
using the bind command during executing to save data externally to the contaienr.
The container also contains Jupyterlab for development

Expected behavior
Expect the container to execute.

OS / Linux Distribution
Ubuntu 20.04.3 LTS (Focal Fossa)

Installation Method
Following these instructions: https://sylabs.io/guides/3.7/user-guide/quick_start.html

Additional context
I've tried on two different machines using Singularity 3.7.3 and 3.9 and have the same issue in both cases.

[marc-rigter]

I've found that the issue for me was that the script that I was trying to run was not executable. Fixed by running chmod +x.

So not an issue with singularity, but the error message I was getting is not very informative.

[dtrudg]

Agreed we should likely check the if the target of exec is executable and have a nicer error if not.

[dtrudg]

Spent some time looking into it this morning, and am afraid this is not at all trivial so it'll be deferred. The error isn't particularly nice for users, but it is essentially the same as if you try run something non-exectuable in the host shell, so I'm going to consider a friendlier message an enhancement, not a bug:

11:37 AM $ /etc/hosts
bash: /etc/hosts: Permission denied

11:37 AM $ singularity exec ubuntu_latest.sif /etc/hosts
FATAL:   permission denied

The reason it's difficult to get something more friendly out is as follows....

It's easy to check if an explicit exec /etc/hosts is an executable file and report an error . However, once we start doing our own checks on the target of an exec we need to consider other cases. Consider singularity exec echo. This should generally run /bin/echo in the container. But it gets complicated due to the fact that shells implement their own version of echo, and if you run echo in a shell you will get that, not /bin/echo.

We use a shell 'action script' to drive the execution of processes in the container. For the singularity exec action this script runs exec "$@" - so the shell is going to try to resolve the executable that is passed as an argument. As far as the shell is concerned, permission denied is the correct error if you try to execute something to which you do not have +x permission.

It's actually quite difficult to find out exactly what a shell will exec... using only commands available in the shell. which is not a shell command, and has portability issues so cannot be used. POSIX sh has type but doesn't support the flags that bash does which allow you to find the path to a binary that is overridden by a shell built-in.

Our action script is run using an embedded shell interpreter, mvdan.cc/sh. This does have the type built-in that takes a -p flag. It'll then look on $PATH for an executable, and report the path to it if it exists. In theory some complex conditionals could be used around this, and checking paths. Howver, the type -p with mvdan.cc/sh doesn't match bash... and that project generally moves toward more bash over time so I'm concerned any complex conditionals we write might be fragile.

As a consequence of all the above, we'll add a note to the documentation about the reason for permission denied, but will defer enhancing the error message until it's clear there is time to do so, and in a future-proof manner. Apologies.