How to set up Neurodesk and HPC on a cluster

[bkossows]

Hi everyone,

I’ve been asked by our cluster administrator to submit a request for a solution where JupyterHub could function as a terminal for multiple users, while also allowing unused resources to be utilized for heavy computational tasks (e.g., parallel processing with recon-all).

The problem is, I’m not entirely sure how to describe such a setup or what tools/configurations could make this possible. Essentially, we’re looking for a way to:

• Let users work interactively through JupyterHub. Most users do not need more than 4CPUs to run fsl commands and check the results.
• Use idle resources for demanding jobs when available.

If anyone has experience with similar setups or can suggest how to approach this, I’d really appreciate your advice. Any tips on integrating this with a job scheduler like SLURM or Kubernetes would also be super helpful.

Kind regards,
Bartosz Kossowski

[stebo85]

Dear @bkossows

There are two solutions that come to my mind:

1. Run a standard SLURM cluster and install OpenOndemand (https://openondemand.org/). OpenOndemand will provide you with Jupyter and full desktop support for interactive computing. Users can then submit longer running jobs to the underlying slurm cluster. Then install Neurodesk into this setup. This is the setup most universities have and what we run as well - if you want I can show you this setup in a zoom call: https://outlook.office.com/bookwithme/user/[email protected]?anonymous&ep=pcard

2. A Kubernetes cluster: This is something we are currently developing, but it still needs a lot of Kubernetes expertise to run. We run Jupyterlab with Neurodesk on Kubernetes and we have a notebook submission system that's powered by Kubeflow under the hood. Pipeline creation is managed by Elyra.

[bkossows]

Thank you for your response. I will pass it on to the decision-makers, and I would gladly take the opportunity to meet if they are interested.

[bkossows]

Dear Steffen,
Thank you for your recommendation. So far, we have implemented a Jupyter server using Kubernetes and the Neurodesk Dockerfile (link
) as the main image. We are very happy to start using the platform; however, I have noticed a few potential bottlenecks:

• Loading new containers often takes quite a while. It seems that the CVMFS cache is only 5 GB by default and user-specific ($HOME/cvmfs_cache). Could this be the reason? How would you recommend configuring the CVMFS cache for multiple users on the Jupyter server? Is it possible to set up a shared cache, or would it be better to run a local repository instead?
• We are also planning to start working on a SLURM cluster with Open OnDemand. I think our IT expert still needs to get a better understanding of the setup. Would it be possible to schedule a demo session with You and @p-andrzejewski-fj?

Kind regards
Bartosz Kossowski

[stebo85]

Dear @bkossows,

great to hear you got it working.

Regarding CVMFS there are a few options we recommend:

1. you could run a local squid proxy that all your connections go through and that caches every tool
2. you could run your own stratum 1 server
3. you could run CVMFS as a kubernetes pod and have a shared cache between all user-pods (this is the setup we use for our Kubernetes deployments. I can also share the manifests)

Regarding OpenOndemand: Yes, I can show you how we deploy that. I am currently moving between continents, so a meeting in November would be ideal. Would that work?

[bkossows]

Thanks for the quick reply. As for now please send us the manifests for the third option. We will try to optimize the current system before we move to the OpenOnDemand.
Cheers

[stebo85]

Dear @bkossows,

Here is how you could achieve a shared cvmfs setup between pods:

# #========================================#
# # CVMFS
# #========================================#
 
helm install cvmfs-csi oci://registry.cern.ch/kubernetes/charts/cvmfs-csi --version 2.5.0 -n jupyterhub
 
# Create CVMFS pvc
kubectl create -n jupyterhub -f templates/volume-storageclass-pvc.yaml
 
# Apply Neurodesk CVMFS ConfigMaps
kubectl create -f templates/cvmfs-csi-config-d.yaml --dry-run=client -o yaml | kubectl apply -f -
kubectl create -f templates/cvmfs-csi-default-local.yaml --dry-run=client -o yaml | kubectl apply -f -

kubectl -n jupyterhub get pods -o wide
kubectl delete pod -n jupyterhub cvmfs-csi-controllerplugin-*
kubectl delete pod -n jupyterhub cvmfs-csi-nodeplugin-*
kubectl -n jupyterhub get pods -o wide
 
# echo
# echo "Remember to ..."
# # echo "kubectl edit configmap cvmfs-csi-config-d -n jupyterhub"
# # echo "kubectl edit configmap cvmfs-csi-default-local -n jupyterhub"
# # echo "And add the details in ../cloud/cvmfs-csi-config-d.yaml and cvmfs-csi-default-local.yaml"
# echo "delete the cvmfs nodeplugin pod to apply changes"
# echo "To test/troubeshoot ..."
# echo "Shell into the automount container in the cvmfs-cvmfs-csi-nodeplugin-*** pod"
# echo "mount -t cvmfs neurodesk.ardc.edu.au /mnt -v"
# echo "After testing ... umount /mnt "

volume-storageclass-pvc.yaml
cvmfs-csi-config-d.yaml
cvmfs-csi-default-local.yaml

I hope these instructions still work - they are from an older version of our setup before we moved to argo as a deployment

[stebo85]

@bkossows - just wondering where you are at with this? Would you still like to meet and chat about this?