Merge pull request #1371 from NVIDIA/dholt/retire-singularity-wrapper

Retire the Singularity wrapper

Author: michael-balint

config.example/group_vars/all.yml

@@ -199,19 +199,6 @@ enroot_environ_config_files_dgx:
     content: |
       OMPI_MCA_btl_tcp_if_exclude=lo,docker0,ib0,ib1,ib2,ib3
 
-# Singularity configuration
-# Playbook: singularity, slurm-cluster
-# Singularity target version
-# set an alternate singularity version here;
-# see roles/singularity_wrapper/defaults/main.yml for default
-# singularity_version:
-singularity_conf_path: "/etc/singularity/singularity.conf"
-bind_paths: []
-  # example:
-  #- /mnt/shared
-golang_install_dir: '/opt/go/{{ golang_version }}'
-golang_gopath: /opt/go/packages
-
 ################################################################################
 # AUTH                                                                         #
 ################################################################################

config.example/group_vars/slurm-cluster.yml

@@ -66,7 +66,6 @@ dns_late: false
 slurm_configure_etc_hosts: yes
 slurm_cluster_install_cuda: yes
 slurm_cluster_install_nvidia_driver: yes
-slurm_cluster_install_singularity: no
 
 ################################################################################
 # NFS                                                                          #
@@ -164,7 +163,10 @@ openmpi_configure: "./configure --prefix={{ openmpi_install_prefix }} --disable-
 # Open OnDemand                                                                #
 ################################################################################
 install_open_ondemand: no
-# OOD Linux-host adapter requires `slurm_cluster_install_singularity` to be true
+# The OOD Linux-host adapter runs interactive apps inside an Apptainer/Singularity
+# container. DeepOps no longer installs Singularity (the wrapper role is retired;
+# Enroot/Pyxis is the supported Slurm runtime), so install Apptainer/Singularity on
+# the login nodes yourself before enabling this adapter.
 ood_install_linuxhost_adapter: no
 
 servername: '{{ ansible_fqdn }}'

config.example/host_vars/gpu01

@@ -2,5 +2,5 @@
 # Example of a host-specific variable file
 # These variables would only be used for a host named "gpu01"
 
-# Don't install Singularity on this host
-#slurm_cluster_install_singularity: false
+# Don't install the CUDA toolkit on this host
+#slurm_cluster_install_cuda: false

docs/container/docker-login.md

@@ -43,11 +43,11 @@ The [Kubernetes documentation](https://kubernetes.io/docs/tasks/configure-pod-co
 
 ## Slurm jobs using private registries
 
-The process for using private registries is different depending on whether you are using Singularity or Enroot as your container runtime.
+The process for using private registries depends on your container runtime. Enroot/Pyxis is the runtime DeepOps installs and validates for Slurm; the Singularity/Apptainer notes below apply only if your site provides that runtime separately.
 
 ### Singularity
 
-[Singularity](https://sylabs.io/singularity/) gets container pull credentials using environment variables:
+DeepOps no longer installs Singularity (the wrapper role is retired; Singularity lives on upstream as [Apptainer](https://apptainer.org/), and Enroot/Pyxis is the supported Slurm runtime). If your site provides Apptainer/Singularity, it gets container pull credentials using environment variables:
 
 ```bash
 export SINGULARITY_DOCKER_USERNAME=<username>

docs/deepops/architecture.md

@@ -109,7 +109,6 @@ Roles run as part of the various playbooks. Includes default values and tasks as
 │   ├── roce_backend
 │   ├── rsyslog_client
 │   ├── rsyslog_server
-│   ├── singularity_wrapper
 │   ├── slurm
 │   ├── spack
 │   └── standalone-container-registry

docs/slurm-cluster/README.md

@@ -14,7 +14,7 @@ Instructions for deploying a GPU cluster with Slurm
   - [Configuring shared filesystems](#configuring-shared-filesystems)
   - [Installing tools and applications](#installing-tools-and-applications)
   - [Installing Open on Demand](#installing-open-on-demand)
-  - [Pyxis, Enroot, and Singularity](#pyxis-enroot-and-singularity)
+  - [Pyxis and Enroot](#pyxis-and-enroot)
   - [Large deployments](#large-deployments)
 
 ## Requirements
@@ -236,9 +236,11 @@ See the documentation on [software modules](./software-modules.md) for informati
 
 [Open OnDemand](https://openondemand.org/) can be installed by setting the `install_open_ondemand` variable to yes before running the `slurm-cluster.yml` playbook.
 
-## Pyxis, Enroot, and Singularity
+## Pyxis and Enroot
 
-[Pyxis](https://github.com/NVIDIA/pyxis) and [Enroot](https://github.com/NVIDIA/enroot) are installed by default and can be disabled by setting `slurm_install_enroot` and `slurm_install_pyxis` to no. Singularity can be installed by setting the `slurm_cluster_install_singularity` variable to yes before running the `slurm-cluster.yml` playbook.
+[Pyxis](https://github.com/NVIDIA/pyxis) and [Enroot](https://github.com/NVIDIA/enroot) are installed by default and can be disabled by setting `slurm_install_enroot` and `slurm_install_pyxis` to no. They are the supported, release-validated container runtime for Slurm in DeepOps.
+
+The DeepOps Singularity wrapper role has been retired. Singularity lives on upstream as [Apptainer](https://apptainer.org/); sites that still want it can install Apptainer/Singularity separately.
 
 ## Large deployments
 

docs/slurm-cluster/slurm-single-node.md

@@ -11,7 +11,7 @@ Single Node Slurm Deployment Guide
     - [Allocating GPUs](#allocating-gpus)
     - [Running Containers](#running-containers)
       - [Rootless Docker](#rootless-docker)
-    - [Enroot and Singularity](#enroot-and-singularity)
+    - [Enroot and Pyxis](#enroot-and-pyxis)
 
 ## Introduction
 
@@ -92,7 +92,7 @@ The general requirements and procedure for Slurm setup via deepops is documented
 
    b. Configuring `"config/group_vars/slurm-cluster.yml"`
 
-   Typically users cannot ssh as local users directly to compute nodes in a Slurm cluster without a Slurm reservation. However, since in this deployment a compute node also functions as a login node we need to add users to slurm configuration for the ability to ssh as local users. Again, this is needed when a compute node also functions as a login node. Additionally, set the singularity install option (which is no by default). Singularity can be used to run containers and it will be used to set up rootless options as well. Do not set a default NFS with single node deployment.
+   Typically users cannot ssh as local users directly to compute nodes in a Slurm cluster without a Slurm reservation. However, since in this deployment a compute node also functions as a login node we need to add users to slurm configuration for the ability to ssh as local users. Again, this is needed when a compute node also functions as a login node. Do not set a default NFS with single node deployment.
 
    ```bash
    vi config/group_vars/slurm-cluster.yml
@@ -101,7 +101,6 @@ The general requirements and procedure for Slurm setup via deepops is documented
    ```yaml
    slurm_enable_nfs_server: false
    slurm_enable_nfs_client_nodes: false
-   slurm_cluster_install_singularity: yes
 
    slurm_login_on_compute: true
 
@@ -331,8 +330,8 @@ attach to the tmux session in the adopted ssh session.
 
 ### Running Containers
 
-DeepOps enables running containers with several containerization platforms:
-docker, singularity, and enroot with pyxis.
+DeepOps enables running containers on Slurm with rootless Docker and with
+Enroot/Pyxis, which is installed by default.
 
 #### Rootless Docker
 
@@ -441,7 +440,7 @@ cat: /slurm/slurmdbd.conf: Permission denied
 Rootless docker supports building containers and many standard docker features
 with a few limitations. These limitations can make it challenging to run
 multi node Slurm jobs, therefore for multi node jobs on Slurm the recommended
-approach is via enroot or singularity.
+approach is via Enroot/Pyxis.
 
 A user can explicitly stop the rootless docker daemon with “stop_rootless_docker.sh”
 script, or just exit the Slurm session. Upon exit from a slurm session the
@@ -458,11 +457,11 @@ login-session:$
 These scripts “start_rootless_docker.sh” and “stop_rootless_docker.sh” appear
 on a user's path upon loading the rootless docker module.
 
-### Enroot and Singularity
+### Enroot and Pyxis
 
-Singularity and enroot could also be deployed via DeepOps. These would be
-useful for multi-node jobs if running on more than one DGX system.
-Enroot with pyxis can be tested by running:
+Enroot with Pyxis is installed by DeepOps by default and is the supported
+container runtime for multi-node jobs across more than one DGX system.
+It can be tested by running:
 
 The examples below use `registry.example.com/hpc/nccl-tests:latest` as a placeholder for a site-maintained NCCL tests image.
 
@@ -504,25 +503,10 @@ will typically be inferred from the environment. Calling mpirun approach could
 be useful, because certain binding options are not available to srun directly,
 but can be set via mpirun.
 
-Singularity could be used in a similar fashion to enroot. Don’t forget the
-“--nv” option for GPUs.
-
-```bash
-login-session:srun --mpi=pmi2 --ntasks=2 --gpus-per-task=1 \
-  singularity exec --nv docker://registry.example.com/hpc/nccl-tests:latest \
-    all_reduce_perf -b 1M -e 4G -f 2 -g 1
-```
-
-Similarly to invoke mpirun with singularity run script (same script as was used
-with enroot):
-
-```bash
-login-session:srun --ntasks=2 --gpus-per-task=1 \
-  singularity exec --nv docker://registry.example.com/hpc/nccl-tests:latest \
-    ${PWD}/test_allreduce.sh
-```
-
-Refer to singularity documentation for further details. Building containers with
-singularity is permitted to non-privileged users via the “--fakeroot” option.
-Enroot and singularity excel at running containerized multi node jobs, which is
-somewhat difficult and less convenient to do using docker on Slurm.
+The DeepOps Singularity wrapper role has been retired, so DeepOps no longer
+installs Singularity. Singularity lives on upstream as
+[Apptainer](https://apptainer.org/); if your site installs Apptainer/Singularity
+separately, it can run the same containers with `singularity exec --nv` (remember
+the “--nv” option for GPUs). Enroot/Pyxis is the supported path and excels at
+running containerized multi node jobs, which is somewhat difficult and less
+convenient to do using docker on Slurm.

docs/slurm-cluster/slurm-usage.md

@@ -128,13 +128,14 @@ scancel JOBID
 To run a deep learning job with multiple processes, use MPI:
 
 ```bash
-srun -p PARTITION --pty /bin/bash
-singularity pull docker://nvcr.io/nvidia/tensorflow:19.05-py3
-singularity run docker://nvcr.io/nvidia/tensorflow:19.05-py3
-cd /opt/tensorflow/nvidia-examples/cnn/
-mpiexec --allow-run-as-root -np 2 python resnet.py --layers=50 --batch_size=32 --precision=fp16 --num_iter=50
+srun -p PARTITION --gpus-per-task=1 \
+  --container-image=nvcr.io/nvidia/tensorflow:19.05-py3 \
+  bash -c "cd /opt/tensorflow/nvidia-examples/cnn/ && \
+    mpiexec --allow-run-as-root -np 2 python resnet.py --layers=50 --batch_size=32 --precision=fp16 --num_iter=50"
 ```
 
+The `--container-image` flag runs the job inside the image using Pyxis/Enroot, which DeepOps installs by default.
+
 ## Additional Resources
 
 - [SchedMD Slurm Quickstart Guide](https://slurm.schedmd.com/quickstart.html)

playbooks/container/singularity.yml

@@ -137,10 +137,6 @@
     hostlist: "{{ rsyslog_client_group | default('slurm-node') }}"
   when: slurm_enable_rsyslog_client|default(true)
 
-- name: Install Singularity
-  import_playbook: container/singularity.yml
-  when: slurm_cluster_install_singularity|default(true)
-
 - name: Install Open OnDemand
   import_playbook: slurm-cluster/open-ondemand.yml
   when: install_open_ondemand