Skip to content

Commit feb51ad

Browse files
committed
add instructions back to readme
1 parent cc25714 commit feb51ad

File tree

3 files changed

+300
-134
lines changed

3 files changed

+300
-134
lines changed

INSTRUCTIONS.md

Lines changed: 0 additions & 127 deletions
This file was deleted.

README.Rmd

Lines changed: 132 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,11 +2,11 @@
22
output: github_document
33
---
44
<!-- DO NOT EDIT. CREATED BY README.RMD. Knit that. -->
5-
# NMFS Open Science Docker Stack
5+
# <a name="top">NMFS Open Science Docker Stack
66

77
### Beta release June 1, 2024.
88

9-
These are a collection of container images that provide standardized environments for Python and R with Jupyter Lab, RStudio and VS Code IDEs. The images are built off the [Rocker](https://rocker-project.org/images/devcontainer/images.html), [Pangeo](https://github.com/pangeo-data/pangeo-docker-images) and [Jupyter](https://jupyter-docker-stacks.readthedocs.io/en/latest/) base images. This repo holds the stable Docker stack for specific pipelines used in Fisheries. The images are designed to work out-of-box and identically in Jupyter Hubs, Codespaces, Binder, etc.Read the Design section below on what the NMFS Open Sci Docker Stack does. For use instructions, see [INSTRUCTIONS.md](https://nmfs-opensci/container-images/INSTRUCTIONS.md).
9+
These are a collection of container images that provide standardized environments for Python and R with Jupyter Lab, RStudio and VS Code IDEs. The images are built off the [Rocker](https://rocker-project.org/images/devcontainer/images.html), [Pangeo](https://github.com/pangeo-data/pangeo-docker-images) and [Jupyter](https://jupyter-docker-stacks.readthedocs.io/en/latest/) base images. This repo holds the stable Docker stack for specific pipelines used in Fisheries. The images are designed to work out-of-box and identically in Jupyter Hubs, Codespaces, Binder, etc.Read the Design section below on what the NMFS Open Sci Docker Stack does. For use, see [Instructions](#instructions) and [Link to files](#files).
1010

1111
## Stable set of images
1212

@@ -86,4 +86,133 @@ All code used in the images is under open licenses. Some is copy-left which mean
8686

8787
## Disclaimer
8888

89-
This repository is a scientific product and is not official communication of the National Oceanic and Atmospheric Administration, or the United States Department of Commerce. All NOAA GitHub project content is provided on an ‘as is’ basis and the user assumes responsibility for its use. Any claims against the Department of Commerce or Department of Commerce bureaus stemming from the use of this GitHub project will be governed by all applicable Federal law. Any reference to specific commercial products, processes, or services by service mark, trademark, manufacturer, or otherwise, does not constitute or imply their endorsement, recommendation or favoring by the Department of Commerce. The Department of Commerce seal and logo, or the seal and logo of a DOC bureau, shall not be used in any manner to imply endorsement of any commercial product or activity by DOC or the United States Government.
89+
This repository is a scientific product and is not official communication of the National Oceanic and Atmospheric Administration, or the United States Department of Commerce. All NOAA GitHub project content is provided on an ‘as is’ basis and the user assumes responsibility for its use. Any claims against the Department of Commerce or Department of Commerce bureaus stemming from the use of this GitHub project will be governed by all applicable Federal law. Any reference to specific commercial products, processes, or services by service mark, trademark, manufacturer, or otherwise, does not constitute or imply their endorsement, recommendation or favoring by the Department of Commerce. The Department of Commerce seal and logo, or the seal and logo of a DOC bureau, shall not be used in any manner to imply endorsement of any commercial product or activity by DOC or the United States Government.
90+
91+
# <a name="instructions">Instructions for using the images
92+
[back to readme](#top)
93+
94+
There are many ways to use Docker images. Here are common ones. Scroll to the bottom for instructions on linking your container to file systems (so you can get and store files).
95+
96+
## To run images in a JupyterHub with 'bring your image'
97+
98+
If your JupyterHub has this option:
99+
100+
- Click on the 'Bring your own image' radio button at bottom
101+
- Paste in url to your image (or any other image)
102+
- You will find the urls in the right nav bar under 'Packages'
103+
- Example `ghcr.io/nmfs-opensci/jupyter-base-notebook:latest`
104+
105+
## Run with a JupyterHub
106+
107+
Should work out of the box. Put the url to the image whereever you would use images.
108+
109+
## Run with docker
110+
111+
You can run the images on a Virtual Machine or your computer if you have Docker or Podman installed.
112+
113+
```
114+
docker run -p 8888:8888 ghcr.io/nmfs-opensci/container-images/py-rocket-base:latest
115+
```
116+
117+
On a Mac M2+ with Rosetta emulation turned on in the Docker Desktop settings.
118+
119+
```
120+
docker run --platform linux/amd64 -p 8888:8888 ghcr.io/nmfs-opensci/container-images/py-rocket-base:latest
121+
```
122+
123+
In the terminal look for something like and put that in a browser.
124+
125+
```
126+
http://127.0.0.1:8888/lab?token=6d45c7d88aba92a815647c
127+
```
128+
129+
**Running geospatial R Docker images and working with netCDF files**
130+
131+
GDAL netCDF driver needs some extra flags added to the `docker run` for GDAL to work correctly when run inside a Docker container. This doesn't affect Python as much since `xarray` works with netCDF via different drivers, but the `terra` netCDF functions use GDAL drivers under the hood to open netCDF files. You'll get error saying it can't find files. We ran into trouble when accessing cloud-hosted netCDF files. Perhaps it works ok if you download the files.
132+
133+
Add this to the call:
134+
```
135+
--cap-add SYS_PTRACE --security-opt seccomp=unconfined
136+
```
137+
so you call will look like:
138+
```
139+
docker run -p 8888:8888 --cap-add SYS_PTRACE --security-opt seccomp=unconfined ghcr.io/nmfs-opensci/container-images/py-rocket-geospatial:latest
140+
```
141+
Note we had trouble getting this to work on an Mac with Apple chips. You can test if it is going to work by running this Python code and seeing if `you can see if `DCAP_VIRTUALIO` is listed:
142+
```
143+
from osgeo import gdal
144+
nc = gdal.GetDriverByName("netCDF")
145+
nc.GetMetadata().keys()
146+
```
147+
148+
## Run with Binder
149+
150+
Create a file called `Dockerfile` and put in the base of your GitHub repository or in a folder called `binder` or `.binder`. Into that file put the following line (replacing the image url to match your desired image).
151+
```
152+
FROM ghcr.io/nmfs-opensci/container-images/py-rocket-geospatial:latest
153+
```
154+
155+
Then go to <https://mybinder.org> and paste in the url to your GitHub repo or alternatively go to the following url directly:
156+
```
157+
https://mybinder.org/v2/gh/<username or org>/<reponame>
158+
```
159+
160+
## With Codespaces
161+
162+
See the folders in the `.devcontainer` folder and create a `.devcontainer/devcontainer.json` file in your own repo by copying one of `devcontainer.json` file. They all use the same template with just the top lines changed. Note that the folder `.devcontainer/codespace` is also required. If you change the line that starts up Jupyter Lab (at the bottom of the devcontainer.json file, do not use port 8888 or else RStudio will not launch.
163+
164+
The Codespaces code is based on: <https://github.com/MichaelAkridge-NOAA/Open-Science-Codespaces>
165+
166+
## GitPod -- like Codespaces
167+
168+
Work in progress. Approach is similar to Codespaces.
169+
170+
## Run on Google Colab
171+
172+
TBD. This seems harder. See this [issue](https://github.com/nmfs-opensci/container-images/issues/14)
173+
174+
# <a name="files"> Getting access to files
175+
[back to readme](#top)
176+
177+
The container gives you a computing environment, but by design, it is a container and not connected to the file system in whatever is running the container. So you will need to get your files in/out of the container and have a way to save your work.
178+
179+
## Upload/Download files
180+
181+
Under the Files menu in Jupyter Lab or the Files tab in RStudio, you can upload and download files.
182+
183+
## Use a Git repository
184+
185+
Jupyter Lab and RStudio have Git GUIs. Use those or the command line to clone repos and push changes back to the repos.
186+
187+
```
188+
cd ~
189+
git clone <url to the repo>
190+
```
191+
192+
## Connect to a bucket
193+
194+
If you are working with large data sets, you do not want to move these into your container (slow, slow). You will want to create a bucket (like an S3 bucket) and connect to that. This is like having a external drive in the cloud.
195+
196+
Instructions to come.
197+
198+
## Mount a file system
199+
200+
You can mount a local file system and read/write directly from that. Here "local" means the machine that is running the container. "local" might be a virtual machine, a server or your computer.
201+
202+
**On a JupyterHub**: The managers of the hub most likely have created persistent memory for you. If not, use Git, upload/download, or use buckets.
203+
204+
**On your computer**: you'll add a flag to the `docker run` command to mount your local file system to the Docker container.
205+
206+
When you use `--volume` to bind-mount a file or directory, make sure it does not exist on the Docker container. So do not bind a directory like `\usr` which would destroy the container (nothing bad; it just won't work). Use something like `\home\jovyan\mydir`. `--volume` creates the endpoint for you and it is always created as a directory.
207+
208+
In this example, `mydir` needs to exist in the directory where you are running `docker run`. If you get errors, try `ls` to make sure the directory is there.
209+
```
210+
docker run --platform linux/amd64 -p 8888:8888 --volume ./myproject_files:/home/jovyan/mydir ghcr.io/nmfs-opensci/container-images/py-rocket-base:latest
211+
```
212+
as you work in `mydir` in the container, those changes will appear in your computer's `myproject_files` directory. It is as if you are working on your own computer, but you are using the development environment of the docker file.
213+
214+
Mac users with Apple chips, add `--platform linux/amd64`:
215+
```
216+
docker run --platform linux/amd64 -p 8888:8888 --volume ./myproject_files:/home/jovyan/mydir ghcr.io/nmfs-opensci/container-images/py-rocket-base:latest
217+
```
218+

0 commit comments

Comments
 (0)