Singularity support

[jooh]

Just wanted to share my notes on steps I have taken to get the workshop image to kind of work on the CBU Singularity cluster. I don't know how popular Singularity is at other HPCs (now that there's Podman for user-level container running I don't really see the point of Singularity personally), but if there's demand maybe it would be good to support it?

TLDR: for this to work fully we probably need the workshop materials to live in a folder where everyone has rw, since Singularity ignores the specified neuro user in the Dockerfile and runs the container as a new user (with the same user name as in the calling HPC server).

1. Pull docker image with singularity pull docker://miykael/workshop_pybrain. At most HPCs you will have to ask IT to do this for you.
2. Create a script called notebook.sh and place it in a scripts subfolder (here I've used the same dir where I pulled the image. This script will run inside the container, and takes care of getting conda installed and the notebook server started up. The script can look a little like this

#!/bin/bash
# singularity exec script for Michael Notter's pybrain workshop
source /opt/miniconda-latest/etc/profile.d/conda.sh
conda activate neuro
jupyter notebook --no-browser --ip='*' --notebook-dir=/home/neuro/

3. Now to run you might put the following in another script workshop.sh that executes natively. This way users can just run ./workshop.sh to get their notebook up and running with bind mounts to /output and all. Note that the paths below are specific to the CBU imaging system:

outdir=/imaging/"$USER"/pybrain_output
mkdir -p $outdir
singularity exec -C \
    --bind /imaging/local/software/singularity_images/pybrain_workshop/scripts:/scripts \
    --bind $outdir:/output:rw \
    /imaging/local/software/singularity_images/pybrain_workshop/workshop_pybrain.simg \
    bash /scripts/notebook.sh

Notice a few adaptations to local conditions: we don't use the standard 'pybrain' password because IT would have our heads, we set --ip='*' so that users can start notebook servers on the HPC and access them in their native computer web browsers (everything is behind a VPN so this is not as terrible for security as you might think), and we don't pin the port since we will have multiple users on the same server.

The main outstanding issue is caused by Singularity's strange practice of running the container as the current user on the calling HPC server, which means that the /home/neuro folders where everything is stored are read-only access. So it's not possible to save notebooks, and this also seems to break test_notebooks.py, since pytest can't create caches: PytestCacheWarning: could notcreate cache path /home/neuro/workshop/.pytest_cache/v/cache/lastfailed. I think it also breaks nipype fairly completely.

For singularity to work it would be better if the workshop contents live in a new root folder /workshop, to which all users have read/write access.

[miykael]

Great input @jooh! I'm almost finished with my notebooks and will then move over to this issue. As I'm not familiar with singularity, if I just get all notebooks into /workshop that should do the trick? Data can stay in /data? Also, does /workshop need a specific user role? As long as full read and write access, it should work I assume?

[jooh]

In principle you don't need to move anything around at all, as long as you chmod a+rw all the workshop-related folders. So maybe that's the easiest thing to try first, and I can let you know what happens with singularity.

Do you have any non standard config in the neuro user's home directory? I can probably copy over dot files as needed pretty easily with my wrapper script.

[miykael]

@jooh, everything should now be ready. With respect to the workshop and with respect to singularity. I didn't move things into /workshop but rather into /home/neuro/workshop and changed the permission as suggested by you to a+rw.

With respect to the 2nd question. No, as much as I know, I don't have any non-standard config in the neuro's home directory. Only slightly special thing I can see comes from neurodocker and is mentioned in the Dockerfile.

[jooh]

I think you left off the -R flag on the chmod, see #10

[jooh]

Thanks for looking into this. Even with all the permissions set it looks like writing to the container isn't working in Singularity. From a bit of Googling I wonder if this is by design. It seems Singularity is very restrictive with write access compared to Docker.

Anyway, I have no more time to look at this before the workshop, so let's press on with Docker only. If this comes up again in a future workshop we can maybe revisit. I think the solution would have to be to copy all the workshop materials to the bind-mount 'output' directory, and then proceed with running everything from there.