Merge pull request #59 from Datahel/podman

default local development to Podman
fix Prometheus

Author: jussiarpalahti

README.md

@@ -36,57 +36,100 @@ Checklist for creating the repository:
 
 ## Working with the template:
 
-The template assumes working within a Docker container. Non-container use is possible but not recommended.
+The template assumes working within containers. Non-container development is possible but not recommended. 
 
 ### Option 1: Codespaces
 
 Repositories generated from the template are development-ready with [GitHub Codespaces](https://docs.github.com/en/codespaces/overview).
 
-Codespaces builds a container according to `.devcontainer/devcontainer.json` and attaches to it in `vsc` mode (see 'Running the container' below).
+Codespaces builds a container according to settings in `.devcontainer/devcontainer.json` and attaches to it in `vsc` mode (see 'Running the container' below).
 
 Just launch your repository in a Codespace and start working.
 
 ### Option 2: Local
 
-Clone the repository, and run the container (see 'Running the container' below).
 
-## Running the container
+Begin by selecting your tools. Here are two examples for local development: Podman + JupyterLab (recommended), and Docker + VSC (*Visual Studio Code*).
 
-The template container has three modes: two for development `vsc` (default) and `jupyterlab`, and `api` for running the API. The mode is given to Docker as an environment variable `MODE`.
+The template container has three modes: two for development `vsc` (default) and `jupyterlab`, and `api` for running the API. The mode is given an environment variable `MODE` when running the container.
 
-### Visual Studio Code (VSC):
+*Podman + JupyterLab*
 
-Install VSC and Dev Containers Extension. When you open your repository in VSC, it automatically detects `.devcontainer/devcontianer.json` and suggests opening the repository in a container. Alternatively, press `cmd/ctrl shift P`, type `Remote-Container: Rebuild and Reopen in Container` and press enter. VSC builds the container and attaches to it. 
+Podman is an open source container runtime tool, that can also operate Docker containers, and shares most of the syntax with Docker. While it is possible to configure VSC Dev Container or VSC Remote Container to use Podman or other open source container framework (see instructions with [minikube](https://benmatselby.dev/post/vscode-dev-containers-minikube/) or [Podman desktop](https://developers.redhat.com/articles/2023/02/14/remote-container-development-vs-code-and-podman#)), this requires some effort compared to Docker with Docker Desktop. In addition, third party Docker extensions may not work. Therefore, it is recommended to develop locally with Jupyterlab, which comes installed within the template container.
 
-### JupyterLab:
+```bash
+# On MacOS
 
-The template installs JupyterLab within the container. To work in JupyterLab, run the container with `MODE=jupyterlab docker-compose up`. The container starts JupyterLab. To access jupyterlab, copy the address from the terminal to your browser. 
+# Install podman, docker and hyperkit
+# These are the only requirements for your system - 
+# everything else, including python and it's packages 
+# are installed within the container!
 
-### Working offline with JupyterLab:
+$ brew install hyperkit
+$ brew install docker docker-compose
+$ brew install podman podman-compose
+$ brew install podman-desktop
+    
+# Init podman machine and allocate resources
+# NOTE: edit the allocation according to your needs.
+# You may well run out of memory with this default setup.
+$ podman machine init --cpus=2 --memory=4096
 
-The repository allows developing and running ML models completely offline.
+# Start podman machine
+$ podman machine start
 
-Steps to offline install:
+# Build container
+$ cd your_repo
+$ podman-compose build
 
-0. Clone your repo on a network connected device or use codespaces.
-1. Add required python packages to `requirements/requirements.in`. Try to include all packages you might require, because adding them later without internet access is rather difficult.
-2. Build the image, and within the `requirements` folder run the script `./update_requirements.sh`. 
-3. Rebuild the image.
-4. Pull the image and transfer it to the offline device. The offline device must have Docker installed. 
-5. Start container in the `jupyterlab` mode (see above).
+# Start dev container with JupyterLab
+$ MODE=jupyterlab podman-compose up
 
-The API works offline, too, but requires network access for external clients.
+# To stop the dev container
+# (in another terminal window or tab)
+$ cd your_repo
+$ podman-compose down
+
+# To stop podman
+$ podman machine stop
+```
+
+Check out [Podman docs](https://docs.podman.io/en/latest/index.html) for more information and instructions. 
+
+*Docker Desktop + Visual Studio Code*
+
+[Docker Desktop](https://www.docker.com/) is an open core tool for managing Docker containers. Please note that most non-personal use of Docker Desktop now requires paid license subscription.
+
+To use the template with docker desktop, install and start docker desktop according to the instructions. VSC Extensions work with Docker Desctop with default settings.
+
+Visual Studio Code is an IDE that supports container development. Install VSC and Dev Containers Extension. When you open your repository in VSC, it automatically detects `.devcontainer/devcontianer.json` and suggests opening the repository in a container. Alternatively, press `cmd/ctrl shift P`, type `Remote-Container: Rebuild and Reopen in Container` and press enter. VSC builds the container and attaches to it. 
 
 ### Running the API:
 
-To start the API as the container entrypoint, run `MODE=api docker-compose up`. This loads the latest trained model from model store, starts the API and leaves the container running. The API requires an existing model store and a stored model to function. 
+To start the API as the container entrypoint, run 
+`MODE=api podman-compose up` or `MODE=api docker-compose up` This loads the latest trained model from model store, starts the API and leaves the container running. The API requires an existing model store and a stored model to function. 
 
 > NOTE: To launch container in `api` mode, you must first train a model and save it to model store on a persistent volume or mapping, i.e. change the `local_data` volume type in compose and rebuild the container. To avoid accidentaly leaking sensitive data, model stores are by default saved to `tmpfs` storage that is removed every time the container is stopped. The `api` mode will not work without changing this setup.
 
 To develop interactively with the API running, you may start the API from within your VSC / jupyterlab terminal by running `uvicorn main:app --reload --reload-include *.pickle --host 0.0.0.0` within the API folder of the container. This does not require changing the volume types.
 
 To specify model store and model version to load, use environment variables as specified in `api/app_base.py`. The default option loads latest model from pickle store.
 
+### Working offline:
+
+The repository allows developing and running ML models completely offline.
+
+Steps to offline install:
+
+0. Clone your repo on a network connected device or use codespaces.
+1. Add required python packages to `requirements/requirements.in`. Try to include all packages you might require, because adding them later without internet access is rather difficult.
+2. Build the image, and within the `requirements` folder run the script `./update_requirements.sh`. 
+3. Rebuild the image.
+4. Pull the image and transfer it to the offline device with Podman requirements installed.
+5. Start container in the `jupyterlab` mode with your choice of container tools.
+
+The API works offline, too, but requires network access for external clients.
+
 ## Examples
 
 The `examples/` folder contains simplified single-notebook examples on how to create, train, evaluate and deploy a ML model to model store. There are two notebooks due to two alternatives for the model store. You can try out the API by:
@@ -97,6 +140,8 @@ The `examples/` folder contains simplified single-notebook examples on how to cr
 3. Automatically generated online API documentation is available at: http://127.0.0.1:8000/docs
 4. Real time metrics are available at: http://127.0.0.1:8000/metrics and at the built-in example Prometheus server at http://127.0.0.1:9090
 
+> NOTE: if working in Codespaces, right-click the localhost links from README to access the port-forwarded endpoints!
+
 The following API request calls for the example model to make a prediction on two new data points:
 
     curl -X 'POST' \
@@ -131,7 +176,7 @@ To update `requirements.txt`, run script `./update_requirements.sh` INSIDE the `
 Install new requirements with `pip install -r requirements/requirements.txt` or by rebuilding the container.
 
 The template installs python 3.10 (Ubuntu default).
-To specify another python version, edit the dockerfile to set up a virtual environment. The template is tested with python 3.10, but is expected to work with 3.8 and newer. However, other versions may require additional configuration.
+To specify another python version, edit Dockerfile to set up a virtual environment. The template is tested with python 3.10, but is expected to work with 3.8 and newer. However, other versions may require additional configuration.
 
 ## ML Pipeline
 
@@ -238,18 +283,28 @@ The template comes with an 'ethical issue template' found in `.github/ISSUE_TEMP
 
 The template was developed and tested with:
 
- - GitHub Codespaces
+    GitHub Codespaces
+
+and MacBook Pro M1 & macOS Ventura 13.2.1 with:
+
+    Docker desktop 4.15.0 (93002)
+    VSC 1.74.0
+    VSC Dev Containers Extension v0.245.2
 
-and MacBook Pro M1 & macOS Ventura 13.0 with:
+and MacBook Pro M1 & macOS Ventura 13.2.1 with:
 
- - Docker desktop 4.15.0 (93002)
- - VSC 1.74.0
- - VSC Dev Containers Extension v0.245.2
+    podman-desktop 1.1.0
+    hyperkit 0.20210107
+    podman 4.5.1
+    podman-compose 1.0.6
+    docker 24.0.2
+    docker-compose 2.18.1
 
 Additional configuration may be required for other systems.
 
 ## Known issues
 
  - nbdev_clean git hook may remove 'parameters' tag from notebook cells, even though it should be an allowed key as it is listed in settings.ini. The tag may need to be re-added manually to allow notebook parameterization with papermill.
- - nbdev documentation related functions may not work out-of-box with arm64 machines such as M1 macbooks because the container installs amd64 version of Quarto. You can bypass this by setting `platform` option for docker. However, this makes container build SUPER slow and is thus not a default setting.
+ - nbdev documentation related functions may not work out-of-box with arm64 machines such as M1 macbooks because the container installs amd64 version of Quarto. You can bypass this by setting `platform` option for Docker. However, this makes container build SUPER slow and is thus not a default setting.
  - Source code (git) and model instance version may mismatch if code changes are not committed before updating a model to the model store.
+ - Is podman not responding? See [this discussion](https://github.com/containers/podman/issues/16054) for possible solutions.

compose.yml

@@ -40,6 +40,7 @@ services:
         ./entrypoint.sh
   prometheus-local:
     image: prom/prometheus
+    user: "65534:65534"
     container_name: mlops-prometheus-local
     volumes:
       # prometheus configuration
@@ -48,6 +49,8 @@ services:
       # Select the data and aggregations (summary statistics) from api/metrics
       # to be monitored carefully and handle the time series with care
       # in order to avoid leaking sensitive information!
-      - ./monitoring/prometheus:/prometheus
+      # With this example configuration, metrics time series are stored
+      # in /prometheus volume:
+      - /prometheus
     ports:
       - 9090:9090

entrypoint.sh

@@ -17,5 +17,5 @@ then
     jupyter-lab --allow-root --ip 0.0.0.0 --port 8888
 
 else
-    echo "unknown mode: "$MODE", use 'api', 'vsc' or leave empty (defaults to 'vsc')"
+    echo "unknown mode: "$MODE", use 'api', 'vsc', 'jupyterlab' or leave empty (defaults to 'vsc')"
 fi

monitoring/prometheus.yml

@@ -2,11 +2,11 @@ global:
   scrape_interval:     15s
 
 scrape_configs:
-    - job_name: 'mlops-fastapi-api'
+  - job_name: 'mlops-api-metrics'
     basic_auth:
         # WARNING: this is not secure way to handle secrets!
         username: 'stanleyjobson'
         password: 'swordfish'
 
     static_configs:
-      - targets: ['mlops-api:8000']
+      - targets: ['mlops:8000']