|
| 1 | +--- |
| 2 | +title: "How to run cyCONDOR as container" |
| 3 | +output: rmarkdown::html_vignette |
| 4 | +vignette: > |
| 5 | + %\VignetteIndexEntry{Batch correction} |
| 6 | + %\VignetteEncoding{UTF-8} |
| 7 | + %\VignetteEngine{knitr::rmarkdown} |
| 8 | +editor_options: |
| 9 | + markdown: |
| 10 | + wrap: 72 |
| 11 | +--- |
| 12 | + |
| 13 | +The easiest way to get started uning `cyCONDOR` is by deploying the `Docker` container |
| 14 | +we provide with each release of the package. This container is based on the `Bioconductor` |
| 15 | +base image and start an `RStudio` server session in your browser. |
| 16 | + |
| 17 | +In this vignette we will cover a few option to deploy our `Docker` container in your computing infrastructure. If your specific situation is not cevered do not hesitate to contact us. |
| 18 | + |
| 19 | +First we will show how to install `Docker Desktop` on any Windows or Mac machine, we will then cover some basic set |
| 20 | +up for the configuration of `Docker` in a remote server. |
| 21 | + |
| 22 | +Last we will show how to deploy our `Docker` container using `Singularity` as for safery reasons many research |
| 23 | +institutions are not allowing `Docker` containers in their IT infrastructure. |
| 24 | + |
| 25 | +# Run cyCONDOR with Docker Desktop (PC/Mac) |
| 26 | +Install Docker or another tool to deploy Docker containers (e.g., Singularity). |
| 27 | + |
| 28 | +*Note*: depending on the operating system follow the instructions at https://www.docker.com/, on Windows also the Windows Subsystem for Linux needs to be installed: |
| 29 | + |
| 30 | +1. Set up the Windows Subsystem for Linux 2 (WSL2) and any of the available Linux distributions: |
| 31 | + - Install WSL2 first following the instructions reported here: https://learn.microsoft.com/en-us/windows/wsl/install |
| 32 | + - Install Ubuntu or another Linux distribution as reported here: https://ubuntu.com/tutorials/install-ubuntu-on-wsl2-on-windows-10#1-overview |
| 33 | + |
| 34 | +2. Download and install the latest version of Docker Desktop. |
| 35 | + |
| 36 | +3. After starting the software, verify the installation of Docker Desktop. |
| 37 | + - Open the Ubuntu terminal. |
| 38 | + - Run: |
| 39 | + |
| 40 | +```{r} |
| 41 | +docker info |
| 42 | +``` |
| 43 | + |
| 44 | +*Note*: If Docker is installed and running correctly this will output some basic information on the system, if no output is produced the installation was not successful. |
| 45 | + |
| 46 | +Since the cause of this malfunction could be extremely diverse, it is advisable to consult the troubleshooting section of Docker desktop (https://docs.docker.com/desktop/troubleshoot/overview/). |
| 47 | + |
| 48 | +You can now start a `Docker` container for cyCONDOR analysis. |
| 49 | + |
| 50 | +First download the latest version of the `Docker` image |
| 51 | + |
| 52 | +``` |
| 53 | +docker pull lorenzobonaguro/cycondor:v015 |
| 54 | +``` |
| 55 | + |
| 56 | +Now start the container with the following command (everything in `[]` need to be edited by the user). |
| 57 | + |
| 58 | +``` |
| 59 | +docker run -dp [YOUR PORT]:8787 ∖ # define the port to use |
| 60 | +-e USER=[USER] -e PASSWORD=[PW] ∖ # username and password, can be defined by the user |
| 61 | +--name cycondor_analysis ∖ # name of the container |
| 62 | +-v [LOCAL DIRECTORY PATH]:/data/ ∖ # directory to mount |
| 63 | +lorenzobonaguro/cycondor:015 #name of the docker image |
| 64 | +``` |
| 65 | + |
| 66 | +You can now access your RStudio session with any browser at the address: |
| 67 | + |
| 68 | +``` |
| 69 | +http://localhost:[YOUR PORT]/ |
| 70 | +``` |
| 71 | + |
| 72 | +Enter now the selected `[USER]` and `[PW]` and you will have access to `Rstudio`. |
| 73 | + |
| 74 | +To locate your data click on the `...` symbol above the `Files` tab (see picture below) and type `/data` to finally press enter. |
| 75 | + |
| 76 | +<img src="../man/figures/open_data.png" alt="drawing"> |
| 77 | + |
| 78 | +# Run cyCONDOR with Docker on a remote server |
| 79 | + |
| 80 | +Please follow the guideline of your institution/organization on how to access remote servers and how to manage `Docker` containers. |
| 81 | + |
| 82 | +Here we give a general introduction on how to `ssh` to a remote server. |
| 83 | + |
| 84 | +First log into the remote server by typing in the terminal: |
| 85 | + |
| 86 | +``` |
| 87 | +ssh [ServerAddress] -l [username] |
| 88 | +``` |
| 89 | + |
| 90 | +Both `Mac OS Terminal` and `Windows PowerShell` can use natively the `ssh` protocol. |
| 91 | + |
| 92 | +You can now start `cyCONDOR` `Docker` container as previously |
| 93 | + |
| 94 | +``` |
| 95 | +docker run -dp [YOUR PORT]:8787 ∖ # define the port to use |
| 96 | +-e USER=[USER] -e PASSWORD=[PW] ∖ # username and password |
| 97 | +--name cycondor_analysis ∖ # name of the container |
| 98 | +-v [LOCAL DIRECTORY PATH]:/data/ ∖ # directory to mount |
| 99 | +lorenzobonaguro/cycondor:015 #name of the docker image |
| 100 | +``` |
| 101 | + |
| 102 | +The RServer session will now be available at the following address: |
| 103 | + |
| 104 | +``` |
| 105 | +http://[ServerAddress]:[YOURPORT]/ |
| 106 | +``` |
| 107 | + |
| 108 | +*Note*: This vignette is not intended as an exhaustive tutorial on how to use Docker, we encourage the reader to explore Docker functionalities on https://www.docker.com. |
| 109 | + |
| 110 | +# Stop, restart and delete a Docker container |
| 111 | + |
| 112 | +Once you are done with your work, either on your local machine or on a remote server you can stop your `Docker` container. |
| 113 | + |
| 114 | +Stoped container can be started quickly without loss of temporary data or permanently removed. |
| 115 | + |
| 116 | +To stop your `Docker` contaienr you first need to identify the `CONTAINER ID` of your session, this can be done by typing in the terminal |
| 117 | + |
| 118 | +``` |
| 119 | +docker container ls |
| 120 | +``` |
| 121 | + |
| 122 | +This command will generate a list of the running containers, you need to copy the `CONTAINER ID` of the one matching with your `--name` (e.g. cycondor_analysis) |
| 123 | + |
| 124 | +The container can now be stopped |
| 125 | + |
| 126 | +``` |
| 127 | +docker container stop [CONTAINER ID] |
| 128 | +``` |
| 129 | + |
| 130 | +To now restart the container simply run |
| 131 | + |
| 132 | +``` |
| 133 | +docker contaienr start [CONTAINER ID] |
| 134 | +``` |
| 135 | + |
| 136 | +Or alternatively to permanently remove the contaier |
| 137 | + |
| 138 | +``` |
| 139 | +docker container rm [CONTAINER ID] |
| 140 | +``` |
| 141 | + |
| 142 | +*Note*: Keep in mind that this operation will only remove the container, the docker image will still be available to your system. If you want to remove also the docker image |
| 143 | +type `docker image rm [IMAGE NAME]`. |
| 144 | + |
| 145 | +# Run cyCONDOR with Singularity on a remote server |
| 146 | +Running cyCONDOR `Docker` container with `Singularity` as runtime is possible, first you need to download cyCONDOR `Docker` image and convert it to a `.sif` file: |
| 147 | + |
| 148 | +``` |
| 149 | +singularity pull docker://lorenzobonaguro/cycondor:v015 |
| 150 | +``` |
| 151 | + |
| 152 | +This command will save the docker image in the current working directory. |
| 153 | + |
| 154 | +You can now generate a script to start the singularity container in the same directory named for example `start_singularity.sh`. |
| 155 | + |
| 156 | +Also here everything in `[]` need to be edited by the user. |
| 157 | + |
| 158 | +``` |
| 159 | +#!/bin/bash |
| 160 | +
|
| 161 | +#create temporary file variable |
| 162 | +export WORKSPACE=[folder path to store tmp files] |
| 163 | +
|
| 164 | +#create temporary folders |
| 165 | +[ -d $WORKSPACE ] || mkdir -p $WORKSPACE |
| 166 | +cd $WORKSPACE |
| 167 | +mkdir run tmp-rstudio-server var-lib-rstudio-server |
| 168 | +
|
| 169 | +#create a database config file |
| 170 | +printf 'provider=sqlite\ndirectory=/var/lib/rstudio-server\n' > database.conf |
| 171 | +
|
| 172 | +#create a secure cookie key file |
| 173 | +cd tmp-rstudio-server |
| 174 | +uuidgen > secure-cookie-key |
| 175 | +
|
| 176 | +#prepare to load the container |
| 177 | +
|
| 178 | +PASSWORD='[UserSelectedPassword]' singularity exec \ |
| 179 | +--bind $WORKSPACE/run:/run,\ |
| 180 | +$WORKSPACE/var-lib-rstudio-server:/var/lib/rstudio-server,\ |
| 181 | +$WORKSPACE/tmp-rstudio-server:/tmp/rstudio-server,\ |
| 182 | +$WORKSPACE/database.conf:/etc/rstudio/database.conf,\ |
| 183 | +[LOCAL DIRECTORY PATH]:/data/ \ |
| 184 | +[PathToCyCONDORSifFile] rserver \ |
| 185 | +--www-address=$(hostname -i) --www-port=[YourPort] \ |
| 186 | +--auth-none=0 --auth-pam-helper-path=pam-helper --server-data-dir=/var/run/rstudio-server --server-user=[YourUsername] & |
| 187 | +``` |
| 188 | + |
| 189 | +To start the container you need now to execute the `sh` script |
| 190 | + |
| 191 | +``` |
| 192 | +sh ./start_singularity.sh |
| 193 | +``` |
| 194 | + |
| 195 | +Similarly to `Docker` containers the RStudio session will be available at the address: |
| 196 | + |
| 197 | +``` |
| 198 | +http://[ServerAddress]:[YOURPORT]/ |
| 199 | +``` |
| 200 | + |
| 201 | +# Stop a Singularity container |
| 202 | + |
| 203 | +To stop a `Singularity` container you first need to identify the `[PID]` associated to the `Rserver` session. You can easily do it by typing in the terminal |
| 204 | + |
| 205 | +``` |
| 206 | +top -u [YourUsername] |
| 207 | +``` |
| 208 | + |
| 209 | +You can now stop the container by typing |
| 210 | + |
| 211 | +``` |
| 212 | +kill [PID] |
| 213 | +``` |
| 214 | + |
| 215 | +# Run cyCONDOR on a HPC cluster as SLURM Job |
| 216 | +For really big dataset it is possible to run `Singularity` containers as `SLURM` jobs on HPC computer clusters. |
| 217 | + |
| 218 | +To set up your enviroment follow this detailed guide from the `Rocker` project: https://rocker-project.org/use/singularity.html |
| 219 | + |
| 220 | +*Note*: Mare sure your institution allows interactive session via SLURM, keep in mind this approach is reserving an entire node do your work and might interfere wiith other users. |
0 commit comments