Skip to content

Commit 89c4c6a

Browse files
mvdocyuerout
mvdoc
and
yuerout
authored
ENH Add Dockerfiles (gallantlab#39)
* adding dockerfiles * removing allow root * changing tag * fixed user in container * permission * ENH use neurodocker to generate both cpu and gpu dockerfiles * MNT remove old dockerfile --------- Co-authored-by: Yuerou Tang <[email protected]>
1 parent 245d0f9 commit 89c4c6a

File tree

5 files changed

+606
-0
lines changed

5 files changed

+606
-0
lines changed

‎docker/README.md‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
# Using the Dockerfiles
2+
3+
Although the easiest way to run the tutorials is to use Google Colab, we also provide Dockerfiles for those who prefer to run the tutorials locally.
4+
5+
We provide two versions of the Dockerfile: one for CPU and one for GPU. You should use the CPU version if you do not have a compatible Nvidia GPU or if you prefer to run the tutorials on a CPU. The GPU version is recommended for those with a compatible Nvidia GPU, as it will significantly speed up the fitting of voxelwise encoding models.
6+
7+
## CPU Version
8+
The CPU version of the Dockerfile is designed to run on any machine with a CPU. It does not require any special hardware or drivers, making it suitable for a wide range of environments. Note, however, that the CPU version will be significantly slower when fitting voxelwise encoding models compared to the GPU version.
9+
10+
### Prerequisites
11+
- Install Docker on your machine. You can find instructions for your operating system [here](https://docs.docker.com/get-docker/).
12+
13+
### Build the Docker image
14+
To build the CPU version, run the following command from the current `voxelwise_tutorials/docker` directory:
15+
16+
```bash
17+
docker build --tag voxelwise_tutorials --file cpu.Dockerfile .
18+
```
19+
20+
This will create a Docker image named `voxelwise_tutorials` based on the `cpu.Dockerfile`.
21+
22+
### Run the Docker container
23+
To run the CPU version, use the following command:
24+
25+
```bash
26+
docker run --rm -it --name voxelwise_tutorials --publish 8888:8888 voxelwise_tutorials jupyter-lab --ip 0.0.0.0
27+
```
28+
29+
This command will start a Docker container from the `voxelwise_tutorials` image, mapping port 8888 on your host machine to port 8888 in the container. The `--rm` flag ensures that the container is removed after it is stopped, and the `-it` flag allows you to interact with the container. Note that all data that will be downloaded during the tutorial will be stored in the container, and it will be removed when you stop the container.
30+
31+
You should see output similar to the following:
32+
33+
```
34+
To access the server, open this file in a browser:
35+
file:///home/voxelwise/.local/share/jupyter/runtime/jpserver-7-open.html
36+
Or copy and paste one of these URLs:
37+
http://f4cb3fce5844:8888/lab?token=73d9628b0e8839023e3409945f06b9ddbdedde95fe630e00
38+
http://127.0.0.1:8888/lab?token=73d9628b0e8839023e3409945f06b9ddbdedde95fe630e00
39+
```
40+
The URL will contain a token that you can use to access the Jupyter Lab interface. Open your web browser and navigate to `http://127.0.0.1:8888/lab?token=<YOUR_TOKEN>`. This will open the Jupyter Lab interface, where you can start working with the tutorials.
41+
42+
## GPU Version
43+
The GPU version of the Dockerfile is designed to take advantage of Nvidia GPUs for faster computation. This version is recommended if you have a compatible Nvidia GPU and the necessary drivers installed. The GPU version will significantly speed up the fitting of voxelwise encoding models.
44+
45+
## Prerequisites
46+
- Install Docker on your machine. You can find instructions for your operating system [here](https://docs.docker.com/get-docker/).
47+
- Install the Nvidia Container Toolkit. You can find instructions for your operating system [here](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html).
48+
49+
50+
## Create a Dockerfile for your version of CUDA
51+
52+
You will need to create a Dockerfile for your version of CUDA. We provide a bash script that will do this for you. The bash script uses [neurodocker](https://www.repronim.org/neurodocker/) to quickly create a Dockerfile. The bash script is called `generate_dockerfile_gpu.sh` and can be found in this directory. To use the script, run the following command:
53+
54+
```bash
55+
bash generate_dockerfile_gpu.sh CUDA_VERSION [<OS_VERSION>]
56+
```
57+
58+
Replace `CUDA_VERSION` with the version of CUDA you have installed (e.g., `10.2`). The optional `<OS_VERSION>` argument allows you to specify the version of Ubuntu you want to use (e.g., `22.04`). If you do not specify an OS version, the script will default to `18.04`.
59+
60+
For example, to create a Dockerfile with CUDA 10.2 and Ubuntu 18.04, run the following command:
61+
62+
```bash
63+
bash generate_dockerfile_gpu.sh 10.2 18.04
64+
```
65+
This will create a Dockerfile named `gpu-cu102.Dockerfile` in the current directory. You can then use this Dockerfile to build the GPU version of the tutorials.
66+
67+
> [!IMPORTANT]
68+
> Make sure to use the correct version of CUDA that matches your Nvidia driver. Also note that not all CUDA versions are available in the latest Ubuntu images, so make sure to select both the appropriate CUDA and Ubuntu versions.
69+
70+
## Build the Docker image
71+
To build the GPU version, run the following command, replacing `gpu-cu102.Dockerfile` with the name of the Dockerfile you created in the previous step:
72+
73+
```bash
74+
docker build --tag voxelwise_tutorials --file gpu-cu102.Dockerfile .
75+
```
76+
77+
This will create a Docker image named `voxelwise_tutorials` based on the your GPU Dockerfile.
78+
79+
## Run the Docker container
80+
To run the GPU version, use the following command:
81+
82+
```bash
83+
docker run --rm -it --gpus all --name voxelwise_tutorials --publish 8888:8888 voxelwise_tutorials jupyter-lab --ip 0.0.0.0
84+
```
85+
86+
Then you can follow the same steps as in the CPU version to access the Jupyter Lab interface.

‎docker/cpu.Dockerfile‎

Lines changed: 223 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,223 @@
1+
# Generated by Neurodocker and Reproenv.
2+
3+
FROM ubuntu:20.04
4+
ENV LANG="en_US.UTF-8" \
5+
LC_ALL="en_US.UTF-8" \
6+
ND_ENTRYPOINT="/neurodocker/startup.sh"
7+
RUN export ND_ENTRYPOINT="/neurodocker/startup.sh" \
8+
&& apt-get update -qq \
9+
&& apt-get install -y -q --no-install-recommends \
10+
apt-utils \
11+
bzip2 \
12+
ca-certificates \
13+
curl \
14+
locales \
15+
unzip \
16+
&& rm -rf /var/lib/apt/lists/* \
17+
&& sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen \
18+
&& dpkg-reconfigure --frontend=noninteractive locales \
19+
&& update-locale LANG="en_US.UTF-8" \
20+
&& chmod 777 /opt && chmod a+s /opt \
21+
&& mkdir -p /neurodocker \
22+
&& if [ ! -f "$ND_ENTRYPOINT" ]; then \
23+
echo '#!/usr/bin/env bash' >> "$ND_ENTRYPOINT" \
24+
&& echo 'set -e' >> "$ND_ENTRYPOINT" \
25+
&& echo 'export USER="${USER:=`whoami`}"' >> "$ND_ENTRYPOINT" \
26+
&& echo 'if [ -n "$1" ]; then "$@"; else /usr/bin/env bash; fi' >> "$ND_ENTRYPOINT"; \
27+
fi \
28+
&& chmod -R 777 /neurodocker && chmod a+s /neurodocker
29+
ENV DEBIAN_FRONTEND="noninteractive"
30+
RUN chmod 777 /tmp
31+
RUN apt-get update -qq \
32+
&& apt-get install -y -q --no-install-recommends \
33+
build-essential \
34+
ca-certificates \
35+
git \
36+
netbase \
37+
&& rm -rf /var/lib/apt/lists/*
38+
ENV CONDA_DIR="/opt/miniconda-py311_24.4.0-0" \
39+
PATH="/opt/miniconda-py311_24.4.0-0/bin:$PATH"
40+
RUN apt-get update -qq \
41+
&& apt-get install -y -q --no-install-recommends \
42+
bzip2 \
43+
ca-certificates \
44+
curl \
45+
&& rm -rf /var/lib/apt/lists/* \
46+
# Install dependencies.
47+
&& export PATH="/opt/miniconda-py311_24.4.0-0/bin:$PATH" \
48+
&& echo "Downloading Miniconda installer ..." \
49+
&& conda_installer="/tmp/miniconda.sh" \
50+
&& curl -fsSL -o "$conda_installer" https://repo.continuum.io/miniconda/Miniconda3-py311_24.4.0-0-Linux-x86_64.sh \
51+
&& bash "$conda_installer" -b -p /opt/miniconda-py311_24.4.0-0 \
52+
&& rm -f "$conda_installer" \
53+
# Prefer packages in conda-forge
54+
&& conda config --system --prepend channels conda-forge \
55+
# Packages in lower-priority channels not considered if a package with the same
56+
# name exists in a higher priority channel. Can dramatically speed up installations.
57+
# Conda recommends this as a default
58+
# https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-channels.html
59+
&& conda config --set channel_priority strict \
60+
&& conda config --system --set auto_update_conda false \
61+
&& conda config --system --set show_channel_urls true \
62+
# Enable `conda activate`
63+
&& conda init bash \
64+
&& conda install -y --name base \
65+
"gxx_linux-64" \
66+
"notebook" \
67+
"jupyterlab" \
68+
"numpy" \
69+
"git-annex" \
70+
"ipywidgets" \
71+
# Clean up
72+
&& sync && conda clean --all --yes && sync \
73+
&& rm -rf ~/.cache/pip/*
74+
RUN chmod 777 /opt/miniconda-py311_24.4.0-0/share
75+
RUN python -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
76+
RUN test "$(getent passwd voxelwise)" \
77+
|| useradd --no-user-group --create-home --shell /bin/bash voxelwise
78+
USER voxelwise
79+
WORKDIR /home/voxelwise
80+
RUN git clone https://github.com/gallantlab/voxelwise_tutorials.git --depth 1
81+
RUN python -m pip install voxelwise_tutorials
82+
RUN git config --global user.email '[email protected]'
83+
RUN git config --global user.name 'Your Name'
84+
WORKDIR /home/voxelwise/voxelwise_tutorials/tutorials/notebooks/shortclips
85+
ENTRYPOINT ["/neurodocker/startup.sh"]
86+
87+
# Save specification to JSON.
88+
USER root
89+
RUN printf '{ \
90+
"pkg_manager": "apt", \
91+
"existing_users": [ \
92+
"root" \
93+
], \
94+
"instructions": [ \
95+
{ \
96+
"name": "from_", \
97+
"kwds": { \
98+
"base_image": "ubuntu:20.04" \
99+
} \
100+
}, \
101+
{ \
102+
"name": "env", \
103+
"kwds": { \
104+
"LANG": "en_US.UTF-8", \
105+
"LC_ALL": "en_US.UTF-8", \
106+
"ND_ENTRYPOINT": "/neurodocker/startup.sh" \
107+
} \
108+
}, \
109+
{ \
110+
"name": "run", \
111+
"kwds": { \
112+
"command": "export ND_ENTRYPOINT=\\"/neurodocker/startup.sh\\"\\napt-get update -qq\\napt-get install -y -q --no-install-recommends \\\\\\n apt-utils \\\\\\n bzip2 \\\\\\n ca-certificates \\\\\\n curl \\\\\\n locales \\\\\\n unzip\\nrm -rf /var/lib/apt/lists/*\\nsed -i -e '"'"'s/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/'"'"' /etc/locale.gen\\ndpkg-reconfigure --frontend=noninteractive locales\\nupdate-locale LANG=\\"en_US.UTF-8\\"\\nchmod 777 /opt && chmod a+s /opt\\nmkdir -p /neurodocker\\nif [ ! -f \\"$ND_ENTRYPOINT\\" ]; then\\n echo '"'"'#!/usr/bin/env bash'"'"' >> \\"$ND_ENTRYPOINT\\"\\n echo '"'"'set -e'"'"' >> \\"$ND_ENTRYPOINT\\"\\n echo '"'"'export USER=\\"${USER:=`whoami`}\\"'"'"' >> \\"$ND_ENTRYPOINT\\"\\n echo '"'"'if [ -n \\"$1\\" ]; then \\"$@\\"; else /usr/bin/env bash; fi'"'"' >> \\"$ND_ENTRYPOINT\\";\\nfi\\nchmod -R 777 /neurodocker && chmod a+s /neurodocker" \
113+
} \
114+
}, \
115+
{ \
116+
"name": "env", \
117+
"kwds": { \
118+
"DEBIAN_FRONTEND": "noninteractive" \
119+
} \
120+
}, \
121+
{ \
122+
"name": "run", \
123+
"kwds": { \
124+
"command": "chmod 777 /tmp" \
125+
} \
126+
}, \
127+
{ \
128+
"name": "install", \
129+
"kwds": { \
130+
"pkgs": [ \
131+
"build-essential", \
132+
"git", \
133+
"ca-certificates", \
134+
"netbase" \
135+
], \
136+
"opts": null \
137+
} \
138+
}, \
139+
{ \
140+
"name": "run", \
141+
"kwds": { \
142+
"command": "apt-get update -qq \\\\\\n && apt-get install -y -q --no-install-recommends \\\\\\n build-essential \\\\\\n ca-certificates \\\\\\n git \\\\\\n netbase \\\\\\n && rm -rf /var/lib/apt/lists/*" \
143+
} \
144+
}, \
145+
{ \
146+
"name": "env", \
147+
"kwds": { \
148+
"CONDA_DIR": "/opt/miniconda-py311_24.4.0-0", \
149+
"PATH": "/opt/miniconda-py311_24.4.0-0/bin:$PATH" \
150+
} \
151+
}, \
152+
{ \
153+
"name": "run", \
154+
"kwds": { \
155+
"command": "apt-get update -qq\\napt-get install -y -q --no-install-recommends \\\\\\n bzip2 \\\\\\n ca-certificates \\\\\\n curl\\nrm -rf /var/lib/apt/lists/*\\n# Install dependencies.\\nexport PATH=\\"/opt/miniconda-py311_24.4.0-0/bin:$PATH\\"\\necho \\"Downloading Miniconda installer ...\\"\\nconda_installer=\\"/tmp/miniconda.sh\\"\\ncurl -fsSL -o \\"$conda_installer\\" https://repo.continuum.io/miniconda/Miniconda3-py311_24.4.0-0-Linux-x86_64.sh\\nbash \\"$conda_installer\\" -b -p /opt/miniconda-py311_24.4.0-0\\nrm -f \\"$conda_installer\\"\\n# Prefer packages in conda-forge\\nconda config --system --prepend channels conda-forge\\n# Packages in lower-priority channels not considered if a package with the same\\n# name exists in a higher priority channel. Can dramatically speed up installations.\\n# Conda recommends this as a default\\n# https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-channels.html\\nconda config --set channel_priority strict\\nconda config --system --set auto_update_conda false\\nconda config --system --set show_channel_urls true\\n# Enable `conda activate`\\nconda init bash\\nconda install -y --name base \\\\\\n \\"gxx_linux-64\\" \\\\\\n \\"notebook\\" \\\\\\n \\"jupyterlab\\" \\\\\\n \\"numpy\\" \\\\\\n \\"git-annex\\" \\\\\\n \\"ipywidgets\\"\\n# Clean up\\nsync && conda clean --all --yes && sync\\nrm -rf ~/.cache/pip/*" \
156+
} \
157+
}, \
158+
{ \
159+
"name": "run", \
160+
"kwds": { \
161+
"command": "chmod 777 /opt/miniconda-py311_24.4.0-0/share" \
162+
} \
163+
}, \
164+
{ \
165+
"name": "run", \
166+
"kwds": { \
167+
"command": "python -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu" \
168+
} \
169+
}, \
170+
{ \
171+
"name": "user", \
172+
"kwds": { \
173+
"user": "voxelwise" \
174+
} \
175+
}, \
176+
{ \
177+
"name": "workdir", \
178+
"kwds": { \
179+
"path": "/home/voxelwise" \
180+
} \
181+
}, \
182+
{ \
183+
"name": "run", \
184+
"kwds": { \
185+
"command": "git clone https://github.com/gallantlab/voxelwise_tutorials.git --depth 1" \
186+
} \
187+
}, \
188+
{ \
189+
"name": "run", \
190+
"kwds": { \
191+
"command": "python -m pip install voxelwise_tutorials" \
192+
} \
193+
}, \
194+
{ \
195+
"name": "run", \
196+
"kwds": { \
197+
"command": "git config --global user.email '"'"'[email protected]'"'"'" \
198+
} \
199+
}, \
200+
{ \
201+
"name": "run", \
202+
"kwds": { \
203+
"command": "git config --global user.name '"'"'Your Name'"'"'" \
204+
} \
205+
}, \
206+
{ \
207+
"name": "workdir", \
208+
"kwds": { \
209+
"path": "/home/voxelwise/voxelwise_tutorials/tutorials/notebooks/shortclips" \
210+
} \
211+
}, \
212+
{ \
213+
"name": "entrypoint", \
214+
"kwds": { \
215+
"args": [ \
216+
"/neurodocker/startup.sh" \
217+
] \
218+
} \
219+
} \
220+
] \
221+
}' > /.reproenv.json
222+
USER voxelwise
223+
# End saving to specification to JSON.

‎docker/generate_dockerfile_cpu.sh‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
#!/bin/bash
2+
docker run --rm repronim/neurodocker:latest generate docker \
3+
--pkg-manager apt \
4+
--base-image ubuntu:20.04 \
5+
--env "DEBIAN_FRONTEND=noninteractive" \
6+
--run "chmod 777 /tmp" \
7+
--install build-essential git ca-certificates netbase\
8+
--miniconda \
9+
version="py311_24.4.0-0" \
10+
conda_install="gxx_linux-64 notebook jupyterlab numpy git-annex ipywidgets" \
11+
--run "chmod 777 /opt/miniconda-py311_24.4.0-0/share" \
12+
--run "python -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu" \
13+
--user voxelwise \
14+
--workdir /home/voxelwise \
15+
--run "git clone https://github.com/gallantlab/voxelwise_tutorials.git --depth 1" \
16+
--run "python -m pip install voxelwise_tutorials" \
17+
--run "git config --global user.email '[email protected]'" \
18+
--run "git config --global user.name 'Your Name'" \
19+
--workdir /home/voxelwise/voxelwise_tutorials/tutorials/notebooks/shortclips \
20+
> cpu.Dockerfile

‎docker/generate_dockerfile_gpu.sh‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
#!/bin/bash
2+
# Check if an argument is passed. If not, explain how to use it.
3+
# First argument is the CUDA version.
4+
# Second optional argument is the ubuntu version.
5+
if [ $# -eq 0 ]; then
6+
echo "Usage: $0 <CUDA_VERSION> [<OS_VERSION>]"
7+
echo "Example: $0 10.2 18.04"
8+
echo "If OS_VERSION is not provided, it defaults to 18.04."
9+
exit 1
10+
fi
11+
CUDA_VERSION="$1"
12+
# Check if the second argument is provided, if not set it to 18.04
13+
if [ $# -eq 2 ]; then
14+
OS_VERSION="$2"
15+
else
16+
OS_VERSION="18.04"
17+
fi
18+
# Check if the CUDA version is valid
19+
if [[ ! "$CUDA_VERSION" =~ ^[0-9]+\.[0-9]+$ ]]; then
20+
echo "Invalid CUDA version format. Please use the format X.Y"
21+
exit 1
22+
fi
23+
24+
BASE_IMAGE="nvcr.io/nvidia/cuda:${CUDA_VERSION}-base-ubuntu${OS_VERSION}"
25+
echo "Using CUDA version: $CUDA_VERSION"
26+
echo "Using OS version: ubuntu-$OS_VERSION"
27+
echo "Using base image: $BASE_IMAGE"
28+
29+
CUDA_VERSION_SHORT=$(echo "${CUDA_VERSION}" | cut -d'.' -f1-2 | tr -d '.')
30+
echo "Using CUDA version short: $CUDA_VERSION_SHORT"
31+
32+
CONDA_VERSION="py310_25.1.1-2"
33+
echo "Using conda version: $CONDA_VERSION"
34+
35+
# Generate the Dockerfile using Neurodocker
36+
docker run --rm repronim/neurodocker:latest generate docker \
37+
--pkg-manager apt \
38+
--base-image $BASE_IMAGE \
39+
--env "DEBIAN_FRONTEND=noninteractive" \
40+
--run "chmod 777 /tmp" \
41+
--install build-essential git ca-certificates netbase\
42+
--miniconda \
43+
version="$CONDA_VERSION" \
44+
conda_install="gxx_linux-64 notebook jupyterlab numpy<2 git-annex ipywidgets" \
45+
--run "chmod 777 /opt/miniconda-$CONDA_VERSION/share" \
46+
--run "python -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu"$CUDA_VERSION_SHORT \
47+
--user voxelwise \
48+
--workdir /home/voxelwise \
49+
--run "git clone https://github.com/gallantlab/voxelwise_tutorials.git --depth 1" \
50+
--run "python -m pip install voxelwise_tutorials" \
51+
--run "git config --global user.email '[email protected]'" \
52+
--run "git config --global user.name 'Your Name'" \
53+
--workdir /home/voxelwise/voxelwise_tutorials/tutorials/notebooks/shortclips \
54+
> gpu-cu"$CUDA_VERSION_SHORT".Dockerfile

0 commit comments

Comments
 (0)