Rewrite documentation

Author: 3XX0

README.md

@@ -1,256 +1,54 @@
 # ENROOT
 
-A set of scripts and utilities to run container images as unprivileged "chroot" (or what some people refer to as HPC containers).
+A simple, yet powerful tool to turn traditional container/OS images into unprivileged sandboxes.
 
-Example:
-```bash
-enroot import docker://alpine
-enroot create alpine.sqsh
-enroot start alpine
-```
-
-## Prerequisites
-
-Kernel settings:
-```bash
-# Make sure your kernel is fairly recent (~ 3.10) and supports what's required:
-grep -E '(CONFIG_NAMESPACES|CONFIG_USER_NS|CONFIG_OVERLAY_FS|CONFIG_SECCOMP_FILTER)=' /boot/config-$(uname -r)
-# For running old containers (e.g. Centos 6) on new kernels (4.8+)
-grep 'CONFIG_X86_VSYSCALL_EMULATION'/boot/config-$(uname -r) && grep 'vsyscall=emulate' /proc/cmdline
-
-# Configure namespace limits appropriately if necessary
-sudo tee -a /etc/sysctl.d/10-namespace.conf <<< "user.max_user_namespaces = 65536"
-sudo tee -a /etc/sysctl.d/10-namespace.conf <<< "user.max_mnt_namespaces = 65536"
-
-# Debian distributions
-sudo tee -a /etc/sysctl.d/10-namespace.conf <<< "kernel.unprivileged_userns_clone = 1"
-
-# RHEL distributions
-sudo grubby --args="namespace.unpriv_enable=1 user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"
-
-sudo reboot
-```
-
-Dependencies:
-```bash
-# Debian distributions (build, required, optional)
-sudo apt install -y gcc make libcap2-bin libbsd-dev
-sudo apt install -y curl tar pigz jq squashfs-tools parallel
-sudo apt install -y pv nvidia-container-cli
-
-# RHEL distributions (build, required, optional)
-sudo yum install -y epel-release
-sudo yum install -y gcc make libcap libbsd-devel
-sudo yum install -y curl tar pigz jq squashfs-tools parallel
-sudo yum install -y pv nvidia-container-cli
-````
-
-## Installation
-
-```bash
-git submodule update --init
-
-sudo make install
-# In order to allow unprivileged users to import images
-sudo make setcap
-```
-
-Environment settings:
-
-| Environment | Default | Description |
-| ------ | ------ | ------ |
-| `ENROOT_LIBEXEC_PATH` | `/usr/local/libexec/enroot` | Path to sources and utilities |
-| `ENROOT_SYSCONF_PATH` | `/usr/local/etc/enroot` | Path to system configuration files |
-| `ENROOT_CONFIG_PATH` | `${XDG_CONFIG_HOME}/enroot` | Path to user configuration files |
-| `ENROOT_CACHE_PATH` | `${XDG_CACHE_HOME}/enroot` | Path to user image/credentials cache |
-| `ENROOT_DATA_PATH` | `${XDG_DATA_HOME}/enroot` | Path to user container storage |
-| `ENROOT_RUNTIME_PATH` | `${XDG_RUNTIME_DIR}/enroot` | Path to the runtime working directory |
-
-## Usage
-```
-Usage: enroot COMMAND [ARG...]
-
- Commands:
-    version
-    import [--output|-o IMAGE] URI
-    export [--output|-o IMAGE] NAME
-    create [--name|-n NAME] IMAGE
-    list [--fancy|-f]
-    remove [--force|-f] NAME...
-    start [--root|-r] [--rw|-w] [--conf|-c CONFIG] NAME [COMMAND] [ARG...]
-    bundle [--all|-a] [--output|-o BUNDLE] [--checksum|-c] [--target|-t TARGET] [--desc|-d TEXT] IMAGE
-```
-
-## Commands
-
-### version
-Show the version of Enroot.
-
-### import
-Import and convert a Docker container image to an [Enroot image](#image-format) where the URI is of the form  
-`docker://[<user>@][<registry>#]<image>[:<tag>]`. Digests will be cached under `${ENROOT_CACHE_PATH}`.
-
-Credentials can be configured by writing to the file `${ENROOT_CONFIG_PATH}/.credentials` following the netrc file format. For example:
-```
-# NVIDIA GPU Cloud
-machine authn.nvidia.com login $oauthtoken password <TOKEN>
-# Docker Hub
-machine auth.docker.io login <LOGIN> password <PASSWORD>
-```
+Enroot can be thought of as an enhanced unprivileged `chroot(1)`. It uses the same underlying technologies as containers but removes much of the isolation they inherently provide while preserving filesystem separation.
 
-Environment settings:
+This approach is generally preferred in high-performance environments or virtualized environments where portability and reproducibility is important, but extra isolation is not warranted.
 
-| Environment | Default | Description |
-| ------ | ------ | ------ |
-| `ENROOT_GZIP_PROG` | `pigz` _or_ `gzip` | Gzip program used to uncompress digest layers |
-| `ENROOT_SQUASH_OPTS` | `-comp lzo -noD` | Options passed to `mksquashfs` to produce the image |
+Enroot is also similar to other tools like `proot(1)` or `fakeroot(1)` but instead relies on more recent features from the Linux kernel (i.e. user and mount namespaces), and provides facilities to import well known container image formats (e.g. [Docker](https://www.docker.com/)).
 
-### export
-Export a container root filesystem found under `${ENROOT_DATA_PATH}` to a container image.  
-The resulting artifact can then be unpacked using the [create](#create) command.
+Usage example:
 
-Environment settings:
-
-| Environment | Default | Description |
-| ------ | ------ | ------ |
-| `ENROOT_SQUASH_OPTS` | `-comp lzo -noD` | Options passed to `mksquashfs` to produce the image |
-
-### create
-Take a container image and unpack its root filesystem under `${ENROOT_DATA_PATH}` with the given name (optionally).  
-The resulting artifact can then be started using the [start](#start) command.
-
-### list
-List all the containers along with their size on disk (optionally).
-
-### remove
-Remove a container, deleting its root filesystem from disk.
-
-### start
-Start a previously [created](#create) container by executing its command script (or entrypoint), refer to [Image format (/etc/rc)](#image-format).  
-By default the root filesystem of the container is made read-only unless the `--rw` option has been provided.  
-The `--root` option can also be provided in order to remap your current user to be root inside the container.
-
-Additionally, a configuration script can be provided with `--conf` to perform specific actions before the container starts.  
-This script is a standard bash script called before any configuration happens with the command and arguments passed as input parameters.
-One or more of the following functions can be defined:
-
-| Function | Description |
-| ------ | ------ |
-| `environ()` | Outputs [environment configuration](#environment-configuration-files) |
-| `mounts()` | Outputs [mount configuration](#mount-configuration-files) |
-| `hooks()` | A specific instance of [pre-start hook scripts](#pre-start-hook-scripts) |
-
-Here is an example of such configuration:
-
-```bash
-environ() {
-    # Keep all the environment from the host
-    env
-}
-
-mounts() {
-    # Mount the X11 unix-domain socket
-    echo "/tmp/.X11-unix /tmp/.X11-unix none x-create=dir,bind"
-    # Mount the current working directory to /mnt
-    echo "${PWD} /mnt none bind"
-}
-
-hooks() {
-    # Set the DISPLAY environment variable if not set
-    [ -z "${DISPLAY-}" ] && echo "DISPLAY=:0.0" >> ${ENROOT_ENVIRON}
-    # Record the date when the container was last started
-    date > ${ENROOT_ROOTFS}/last_started
-}
-```
-
-Environment settings:
-
-| Environment | Default | Description |
-| ------ | ------ | ------ |
-| `ENROOT_LOGIN_SHELL` | `/bin/sh` | Login shell used to run the container initialization (i.e. `/init`)|
-| `ENROOT_ROOTFS_RW` | | Equivalent to `--rw` if set |
-| `ENROOT_REMAP_ROOT` | | Equivalent to `--root` if set |
-
-### bundle
-Create a self-extracting bundle from a container image which can be used to start a container with no external dependencies (on most Linux distributions).  
-The resulting bundle takes the same arguments as the [start](#start) command with the addition of `--info` which displays the bundle information, and
-`--keep` which keeps the container filesystem extracted to the target directory after exiting. If `--keep` was not provided, `${TMPDIR}` is used for extraction.
-
-By default, only system-wide configuration is copied to the bundle unless `--all` is specified, in which case user-specified configuration is copied as well.  
-The target directory used to keep the container filesystem can be defined using the `--target` option and defaults to `${PWD}/<bundle>`.  
-Additionally, a checksum can be generated and a description provided with `--checksum` and `--desc` respectively.
-
-Example:
-```bash
-enroot import docker://alpine
-enroot bundle -t '${HOME}/.local/share/enroot/foobar' alpine.sqsh
-./alpine.run --keep --rw cp /etc/os-release /release
-enroot start foobar cat /release
+```sh
+# Import and start an Ubuntu image from DockerHub
+$ enroot import docker://ubuntu
+$ enroot create ubuntu.sqsh
+$ enroot start ubuntu
 ```
 
-Environment settings:
-
-| Environment | Default | Description |
-| ------ | ------ | ------ |
-| `ENROOT_BUNDLE_ALL` | | Equivalent to `--all` if set |
-| `ENROOT_BUNDLE_SUM` | | Equivalent to `--checksum` if set |
-
-## Image format
-Enroot images are standard squashfs images with the following configuration files influencing runtime behaviors.
-
-| File | Description |
-| ------ | ------ |
-| `/etc/rc` | Command script of the container (entrypoint) |
-| `/etc/fstab` | Mount configuration of the container |
-| `/etc/environment` | Environment of the container |
+## Key Concepts
 
-These files follow the same format as the standard Linux/Unix ones (see _fstab(5)_, _rc(8)_, _pam_env(8)_) with the exceptions listed below.
-
-`/etc/rc`:
-  - The command and arguments of the [start](#start) command are passed as input parameters.
-
-
-`/etc/fstab`:
-  - Adds two additional mount options, `x-create=dir` or `x-create=file` to create an empty directory or file before performing the mount.
-  - The target mountpoint is relative to the container rootfs.
-  - References to environment variables from the host of the form `${ENVVAR}` will be substituted
-
-```
-# Example mounting your home directory from the host
-${HOME} ${HOME} none x-create=dir,bind
-```
+* Adheres to the [KISS principle](https://en.wikipedia.org/wiki/KISS_principle) and [Unix philosophy](https://en.wikipedia.org/wiki/Unix_philosophy)
+* Standalone (no daemon)
+* Fully unprivileged and multi-user capable (no setuid binary, cgroup inheritance, per-user configuration/container store...)
+* Easy to use (simple image format, scriptable, root remapping...)
+* Little to no isolation (no performance overhead, simplifies HPC deployements)
+* Entirely composable and extensible (system-wide and user-specific configurations)
+* Fast Docker image import (3x to 5x speedup on large images)
+* Built-in GPU support with [libnvidia-container](https://github.com/nvidia/libnvidia-container)
+* Facilitate collaboration and development workflows (bundles, in-memory containers...)
 
-`/etc/environment`:
-  - References to environment variables from the host of the form `${ENVVAR}` will be substituted
+## Documentation
 
- ```bash
- # Example preserving the DISPLAY environment variable from the host
- DISPLAY=${DISPLAY}
- ```
+1. [Requirements](doc/requirements.md)
+1. [Installation](doc/installation.md)
+1. [Image format](doc/image-format.md)
+1. [Configuration](doc/configuration.md)
+1. [Standard Hooks](doc/standard-hooks.md)
+1. [Usage](doc/usage.md)
 
-## Configuration
 
-Common configurations can be applied to all containers by leveraging the following directories under `${ENROOT_SYSCONF_PATH}` (system-wide) and/or `${ENROOT_CONFIG_PATH}` (user-specific).
+## Copyright and License
 
-| Directory | Description |
-| ------ | ------ |
-| `environ.d` | Environment configuration files |
-| `mounts.d` | Mount configuration files |
-| `hooks.d` | Pre-start hook scripts |
+This project is released under the [BSD 3-clause license](https://github.com/NVIDIA/enroot/blob/master/LICENSE).
 
-### Environment configuration files
-Environment files have the `.env` extension and follow the same format as described in [Image format (/etc/environment)](#image-format)
+## Issues and Contributing
 
-### Mount configuration files
-Mount files have the `.fstab` extension and follow the same format as described in [Image format (/etc/fstab)](#image-format)
+* Please let us know by [filing a new issue](https://github.com/NVIDIA/enroot/issues/new)
+* You can contribute by opening a [pull request](https://help.github.com/articles/using-pull-requests/)
 
-### Pre-start hook scripts
-Pre-start hooks are standard bash scripts with the `.sh` extension.  
-They run with full capabilities before the container has switched to its final root.  
-Scripts are started with the host environment as well as the following environment variables:
+## Reporting Security Issues
 
-| Environment | Description |
-| ------ | ------ |
-| `ENROOT_PID` | PID of the container |
-| `ENROOT_ROOTFS` | Path to the container rootfs |
-| `ENROOT_ENVIRON` | Path to the container environment file to be read at startup |
+When reporting a security issue, do not create an issue or file a pull request.  
+Instead, disclose the issue responsibly by sending an email to `psirt<at>nvidia.com`.

conf/enroot.conf.in

@@ -21,7 +21,7 @@
 # Remap the current user to root inside containers by default.
 #ENROOT_REMAP_ROOT          no
 
-# Maximum time in seconds to wait for connections estabishment.
+# Maximum time in seconds to wait for connections establishment.
 #ENROOT_CONNECT_TIMEOUT     30
 
 # Use HTTP for outgoing requests instead of HTTPS (UNSECURE!).

doc/cmd/bundle.md

@@ -0,0 +1,62 @@
+# Usage
+```
+Usage: enroot bundle [options] [--] IMAGE
+
+Create a self-extracting bundle from a container image.
+
+ Options:
+   -a, --all            Include user configuration files in the bundle
+   -c, --checksum       Generate an embedded checksum
+   -d, --desc TEXT      Provide a description of the bundle
+   -o, --output BUNDLE  Name of the output bundle file (defaults to "IMAGE.run")
+   -t, --target DIR     Target directory used by --keep (defaults to "$PWD/BUNDLE")
+```
+```
+Usage: enroot-check_1.0.0.run [options] [--] [COMMAND] [ARG...]
+
+ Options:
+   -i, --info           Display the information about this bundle
+   -k, --keep           Keep the bundle extracted in the target directory
+   -q, --quiet          Supress the progress bar output
+   -v, --verify         Verify that the host configuration is compatible with the bundle
+   -x, --extract        Extract the bundle in the target directory and exit (implies --keep)
+
+   -c, --conf CONFIG    Specify a configuration script to run before the container starts
+   -e, --env KEY[=VAL]  Export an environment variable inside the container
+   -m, --mount FSTAB    Perform a mount from the host inside the container (colon-separated)
+   -r, --root           Ask to be remapped to root inside the container
+   -w, --rw             Make the container root filesystem writable
+```
+
+# Description
+
+Create a self-extracting bundle from a container image which can be used to start a container on most Linux distributions with no external dependencies.  
+The resulting bundle takes the same arguments as the [start](start.md) command.
+
+The target directory used to keep the container filesystem can be defined using the `--target` option and defaults to `$PWD/<bundle>`.  
+By default when generating a bundle, only system-wide configuration is copied to the bundle unless `--all` is specified, in which case user-specified configuration is copied as well.  
+
+Before executing a bundle, `--verify` can be used to check whether or not the host meets the necessary requirements.
+
+If `--keep` is not provided at launch, `$ENROOT_TEMP_PATH` will be used for extraction.
+
+# Configuration
+
+| Environment | Default | Description |
+| ------ | ------ | ------ |
+| `ENROOT_BUNDLE_ALL` | `no` | Include user-specific configuration inside bundles (same as `--all`) |
+| `ENROOT_BUNDLE_CHECKSUM` | `no` | Generate an embedded checksum inside bundles (same as `--checksum`) |
+
+# Example
+
+```sh
+# Import Ubuntu from DockerHub and generate a bundle from it
+$ enroot import docker://ubuntu
+$ enroot bundle --target '${HOME}/.local/share/enroot/hello-world' ubuntu.sqsh
+
+# Execute the bundle by writing a message at the root of its filesystem and keep it extracted
+$ ./ubuntu.run --keep --rw tee /message <<< "Hello World"
+
+# Display the message inside the bundle root filesystem
+$ enroot start hello-world cat /message
+```

doc/cmd/create.md

@@ -0,0 +1,23 @@
+# Usage
+
+```
+Usage: enroot create [options] [--] IMAGE
+
+Create a container root filesystem from a container image.
+
+ Options:
+   -n, --name  Name of the container (defaults to "IMAGE")
+   ```
+   
+# Description
+
+Take a container image and unpack its root filesystem under `$ENROOT_DATA_PATH/`.  
+The resulting root filesystem can be started with the [start](start.md) command or removed with the [remove](remove.md) command.
+
+# Example
+
+```sh
+# Import Ubuntu 18.04 from DockerHub and create a container out of it
+$ enroot import docker://ubuntu:18.04
+$ enroot create --name ubuntu ubuntu+18.04.sqsh
+```