update README

Author: Nuutti Kytö

README.md

@@ -16,15 +16,15 @@
 Create ETL & machine learning pipelines, model store, API, monitoring & logging - all in a single container, with minimum setup required! 
 
 ![mlops process chart & architecture](mlops_template.png)
-*ML Ops template example generic architecture & process chart.*
+*ML Ops template generic structure and process flowchart*
 
-The generic ML Ops process and structure of the template is described in the figure above. The core structure of the template is built on three elements: a notebook-based ML pipeline (`ml_pipe/`), a model store with two alternatives (`model_store/`), and and API with logging and monitoring properties (`api/`).
+The generic ML Ops process and structure of the template is described in the figure above. The core structure of the template is built on three elements: a notebook-based ML pipeline (`ml_pipe/`), a model store with two alternatives (`model_store/`), and and API with generic logging and monitoring functions (`api/`).
 
-In addition the repository contains `requirements/` folder for managing requirements, `Dockerfile` and `compose.yml` for building and running the container, and `examples/` showing two simple examples on how to create a ML model and save it into a model store. The api template is created so that it works with the examples as-is, but may need to be adjusted for your model.
+In addition the repository contains `requirements/` folder for managing requirements, `Dockerfile` and `compose.yml` for building and running the container, and `examples/` showing two simple examples on how to create a ML model and save it into a model store. The api template is created so that it works with the examples as-is, but is built modular and easy to adopt for your use case.
 
 ## Creating a new repo from this template
 
-For your project, create new repo from this template ([see GitHub instructions](https://docs.github.com/en/repositories/creating-and-managing-repositories/creating-a-repository-from-a-template)). Commit history will be cleaned and the authors won't have access to your new repository (although sharing your work is encouraged).
+For your ML project, create new repo from this template ([see GitHub instructions](https://docs.github.com/en/repositories/creating-and-managing-repositories/creating-a-repository-from-a-template)). Commit history will be cleaned and the authors won't have access to your new repository (although sharing your work is encouraged).
 
 > NOTE: Updates to the template cannot be automatically synchronized to child projects!
 
@@ -73,19 +73,19 @@ Steps to offline install:
 2. Build the image, and within the `requirements` folder run the script `./update_requirements.sh`. 
 3. Rebuild the image.
 4. Pull the image and transfer it to the offline device. The offline device must have Docker installed. 
-5. Start container in the jupyterlab mode (see above).
+5. Start container in the `jupyterlab` mode (see above).
 
 The API works offline, too, but requires network access for external clients.
 
 ### Running the API:
 
 To start the API as the container entrypoint, run `MODE=api docker-compose up`. This loads the latest trained model from model store, starts the API and leaves the container running. The API requires an existing model store and a stored model to function. 
 
-> NOTE: To launch container in `api` mode, you must first train a model and save it to model store on a persistent volume or mapping, i.e. change the `local_data` volume type in compose and rebuild the container. To avoid accidentaly leaking sensitive data, model stores are by default saved to `tmpfs` storage that is removed every time the container is stopped.
+> NOTE: To launch container in `api` mode, you must first train a model and save it to model store on a persistent volume or mapping, i.e. change the `local_data` volume type in compose and rebuild the container. To avoid accidentaly leaking sensitive data, model stores are by default saved to `tmpfs` storage that is removed every time the container is stopped. The `api` mode will not work without changing this setup.
 
-To develop interactively with the API running, you may start the API from within your VSC / jupyterlab terminal by running `uvicorn main:app --reload --reload-include *.pickle --host 0.0.0.0` within the API folder of the container.
+To develop interactively with the API running, you may start the API from within your VSC / jupyterlab terminal by running `uvicorn main:app --reload --reload-include *.pickle --host 0.0.0.0` within the API folder of the container. This does not require changing the volume types.
 
-To specify model store and model version to load, use environment variables as specified in `app_base.py`. The default option loads latest model from pickle store.
+To specify model store and model version to load, use environment variables as specified in `api/app_base.py`. The default option loads latest model from pickle store.
 
 ## Examples
 
@@ -124,14 +124,14 @@ The following API request calls for the example model to make a prediction on tw
 
 The template uses [pip-tools](https://pypi.org/project/pip-tools/) for calculating dependencies.
 Requirements are listed in `*.in` files inside the `requirements/` folder. They are separated in four files:
-prefix `api` is for API, `dev` is for developer tools, `model_store` is for model store and basic `requirements.in` for everything else. You should only need to edit `requirements.in`, unless you wish to edit the core tools the template was built on.
+prefix `api` is for API, `dev` is for developer tools, `model_store` is for model store and basic `requirements.in` for everything else. You should only need to edit `requirements.in`, unless you wish to deviate or add on to the core tools the template was built on.
 
 To update `requirements.txt`, run script `./update_requirements.sh` INSIDE the `requirements/` folder. Do not edit `requirements.txt` manually or with `pip freeze`.
 
 Install new requirements with `pip install -r requirements/requirements.txt` or by rebuilding the container.
 
 The template installs python 3.10 (Ubuntu default).
-To specify another python version, edit the dockerfile to set up a virtual environment. 
+To specify another python version, edit the dockerfile to set up a virtual environment. The template is tested with python 3.10, but is expected to work with 3.8 and newer. However, other versions may require additional configuration.
 
 ## ML Pipeline
 
@@ -152,7 +152,7 @@ The template is planned for taking use of the following two nbdev functions:
 
 Other features may be useful but are not guaranteed to work out-of-box with the template. See list of commands with `nbdev_help`.
 
-Check out [nbdev tutorial](https://nbdev.fast.ai/tutorials/tutorial.html) for more info on the tool. 
+Check out [nbdev tutorial](https://nbdev.fast.ai/tutorials/tutorial.html) for more info on the tool. You can get most out of nbdev following [fastcore](https://fastcore.fast.ai) principles, which we do yet fully utilize, but is a practice of coding we are headed towards.
 
 [Papermill](https://papermill.readthedocs.io/en/latest/) allows running notebooks from python, parameterized. 
 
@@ -165,24 +165,26 @@ First is a simple pickle store, that stores ML model, input and output schemas a
 
 > NOTE: Because default model versioning depends on git head, commit all changes before running the model workflow, to be able to match model version to source code!
 
-Other option is `mlflow` model store. We do not yet take full advantage of mlflows capabilities, but decided to add the option for further exploration and promising features of the tool. The mlflow model store creates a passive mlflow model store that the user can configure for what is stored in there in addition to the model. You can easily store results, graphs, notebooks, data, metrics. In addition, the mlflow model store can be combined with the mlflow server and UI to compare different model runs in a convenient way. The mlflow store has built in support for most common ml model types, and allows lot of configuration. However, if you are not already familiar with the tool, starting with the pickle store is recommended for simplicity.
+Other option is `mlflow` model store. We do not yet take full advantage of all [mlflow](https://www.mlflow.org/docs/latest/index.html) capabilities, but decided to add the option for further exploration and promising features of the tool. The mlflow model store creates a passive mlflow storage. The user can configure for what is stored in there in addition to the model. You can easily store results, graphs, notebooks, data, metrics. In addition, the mlflow model store can be combined with the mlflow server and UI to compare different model runs in a convenient way. The mlflow store has built in support for most common ml model types, and allows lot of configuration. However, if you are not already familiar with the tool, starting with the pickle store is recommended for simplicity.
 
+Both model stores are by default created in the `local_data/` folder. To ensure persistance of your model store (no matter which option you choose), you should take backups. You can either back-up this folder or volume (according to your configuration) or use [git-svn](https://git-scm.com/docs/git-svn) and optionally [git-lfs](https://git-lfs.com) to version control your model store. You should also include `ml_pipe/` worklflow created notebook copies to the backup.
 
 ## Testing
 
-For the ML pipe you can write assertion tests in the ml_pipe notebooks. However, a passing notebook run is already a great starting point for a test. Use of automatic acceptance tests before deploying trained models to model store is encouraged.
+For the ML pipe you can write assertion tests in the ml_pipe notebooks. However, a passing notebook run is already a great starting point for a test. You should separate code and ML tests: code tests should be run with small sample or generated data and intended to ensure that your code works as code, whereas ML tests are to ensure the quality of data and performance of your model. 
+Use of automatic acceptance tests, including tests for bias and other ethical aspects is encouraged. Models that do not pass these tests should not be saved to model store - versioned copies of the notebooks and results that are automatically generated are enough to keep track of these experiments.
 
 ### Locust load testing for API
 
-API load testing is recommended. [Locust](https://docs.locust.io/en/stable/) is a recommended tool for this. 
+API load testing, is recommended. [Locust](https://docs.locust.io/en/stable/) is a recommended tool for this. 
 
 There is an example script in `api/misc/`.
 
-For the example load test, start the api and run:
+For the example load test, start the api with the example model and run:
 
     locust -f locustfile.py -H http://127.0.0.1:8000
 
-Then open http://127.0.0.1:8000 in browser
+Then open http://127.0.0.1:8000 in browser to view the results.
 
 Change log_mode in main.py to test how logging method affects to troughput.
 
@@ -199,7 +201,7 @@ For the API, there are now two different ways to log structured data. Standard l
 
 The `compose.yml` file and `monitoring/` folder contain a simple example configuration for monitoring with [Prometheus](https://prometheus.io/docs/introduction/overview/). The container will launch a local Prometheus instance by default. 
 
-Generic functions for monitoring ML models are presented in `api/metrics/` and demonstrated in the api template. Instead of Prometheus histograms, most of the ML metrics are calculated from a fixed-size FIFO queues. This is because the primary function of the ML metrics is to detect drift in data, models and performance, and thus must be comparable throughout the monitoring timeframe. Generic health metrics from [prometheus-client](https://github.com/prometheus/client_python) are also used by default.
+Generic functions for monitoring ML models are presented in `api/metrics/` and demonstrated in the api template. Instead of Prometheus histograms, most of the ML metrics are calculated from a fixed-size FIFO queues (see `DriftMonitor`  class in `api/metrics/prometheus_metrics`). This is because the primary function of the ML metrics is to detect drift in data, models and performance, and thus must be comparable throughout the monitoring timeframe. Generic health metrics from [prometheus-client](https://github.com/prometheus/client_python) are also used by default.
 
 Adjust both metrics and monitoring for your needs. 
 For a centralized view over multiple algorithms, it is recommended to scrape the local Prometheus instances instead of the API directly. This way you can still view the local time series in case of network issues.
@@ -215,12 +217,12 @@ The template is especially designed for working with personal and sensitive data
 Here is a couple of things that we've considered:
 
 - Ports & network: the template is set up for development. The api and monitoring endpoints are set up for localhost. Check out the configuration before exposing any endpoints to networks.
-- Data: Avoid making copies of data. If possible, load data straight from the source at runtime. If you must copy data locally, store it under `ml_pipe/data/` - this folder is ignored by git. However, it is included in a volume. Begin development with anonymized or generated data. Utilize [tabular_anonymizer](https://github.com/Datahel/tabular-anonymizer) and  [presidio-text-anonymizer](https://github.com/Datahel/presidio-text-anonymizer).
-- Data generated and collected by API is stored under `localdata/` by default. This folder is ignored by git. It is also set up as a `tmpfs` storage in `config.yml` - this means that the contents of the folder only exist in runtime memory of the container and are cleared when the container stops. You may want to change this to a volume or a bind mount - but evaluate the effects on data protection before doing so.
+- Data: Avoid making copies of data. If feasible, load data straight from the source at runtime. If you must copy data locally, store it under `ml_pipe/data/` - this folder is ignored by git. However, it is included in the container volume. Begin development with anonymized or generated data. Utilize [tabular_anonymizer](https://github.com/Datahel/tabular-anonymizer) and  [presidio-text-anonymizer](https://github.com/Datahel/presidio-text-anonymizer).
+- Data generated and collected by API is stored under `local_data/` by default. This folder is ignored by git. It is also set up as a `tmpfs` storage in `config.yml` - this means that the contents of the folder only exist in runtime memory of the container and are cleared when the container stops. You may want to change this to a volume or a bind mount - but evaluate the effects on data protection before doing so.
 - Protect API endpoints with authentication. Currently the template comes with examples on basic http authentication defined in `api/security/`.
 - Use proper security handling for setting and storing passwords, keys, database access tokens etc.
 - You can run API and dev as two separate instances of the same image, i.e. run API so that it does not have direct access to training data.
-- If you want to share source code publicly, but you process or sensitive data, manually recreate a new, public repository based on your private one. This repository must not 'touch' the real data or secrets to avoid contamination (e.g. accidentaly including sensitive data to a commit) - it is just copy of the code. This is laboursome, but a way to avoid accidentally storing and leaking data through git.
+- If you want to share source code publicly, but you process or sensitive data, manually recreate a new, public repository based on your private one. This repository must not come in contact with the real data or secrets to avoid contamination (e.g. accidentaly including sensitive data to a commit) - it is just copy of the code. This is laboursome, but you only have to update this public repo for major updates in the source code - not for every commit.
 - Consider if opening your source code will risk data protection or allow malicious or unintended use of the model. Source code cannot always be shared openly.
 
 ## Ethical aspects
@@ -250,4 +252,4 @@ Additional configuration may be required for other systems.
 
  - nbdev_clean git hook may remove 'parameters' tag from notebook cells, even though it should be an allowed key as it is listed in settings.ini. The tag may need to be re-added manually to allow notebook parameterization with papermill.
  - nbdev documentation related functions may not work out-of-box with arm64 machines such as M1 macbooks because the container installs amd64 version of Quarto. You can bypass this by setting `platform` option for docker. However, this makes container build SUPER slow and is thus not a default setting.
- - Source code and model instance version may mismatch if code changes are not committed before updating a model to the model store.
+ - Source code (git) and model instance version may mismatch if code changes are not committed before updating a model to the model store.