Merge pull request #72 from cseptesting/68-improve-report-module

Updated documentation with tutorials using docker and parallel execution

Author: pabloitu

docs/conf.py

@@ -70,7 +70,6 @@
 html_theme = "sphinx_rtd_theme"
 html_static_path = ["_static"]
 html_theme_options = {
-    "display_version": True,
     "prev_next_buttons_location": "both",
     "sticky_navigation": True,
     "collapse_navigation": True,

docs/guide/model_config.rst

@@ -159,9 +159,6 @@ Here you can find a comprehensive list of parameters used to configure models
    * - **forecast_unit** (required)
      - All
      - Specifies the time unit for the forecast. Use **years** for time-independent models and **days** for time-dependent models.
-   * - **store_db** (optional)
-     - All
-     - If the model consists on only files, this is a boolean (true/false) specifying whether to store the forecast in a database (HDF5).
    * - **flavours** (optional)
      - All
      - A set of parameter variations to generate multiple model variants (e.g., different settings for the same model).

docs/index.rst

@@ -64,6 +64,8 @@ Quickstart
 |                                                  | - :ref:`case_f`                     |
 | |api| **API Reference**                          | - :ref:`case_g`                     |
 |                                                  | - :ref:`case_h`                     |
+|                                                  | - :ref:`case_i`                     |
+|                                                  | - :ref:`case_j`                     |
 +--------------------------------------------------+-------------------------------------+
 
 What is floatCSEP
@@ -158,6 +160,8 @@ Collaborators
    tutorials/case_f.rst
    tutorials/case_g.rst
    tutorials/case_h.rst
+   tutorials/case_i.rst
+   tutorials/case_j.rst
 
 
 .. toctree::

docs/intro/installation.rst

@@ -105,6 +105,26 @@ Having installed the binary dependencies of **pyCSEP** (see `Installing pyCSEP <
     Or downloaded manually from the `latest release  <https://github.com/cseptesting/floatcsep/releases>`_.
 
 
+.. _docker-install:
+
+Docker (Model Containerization & Parallel Run)
+----------------------------------------------
+
+Some tutorials and experiments **containerize models with Docker**, and the engine can use Docker
+for **parallel** execution. To use these features, please install Docker and (on Linux) perform the
+post-installation steps:
+
+- Docker install guide: https://docs.docker.com/engine/install/
+- Linux post-installation (non-root usage and add your user to the `docker` group):
+  https://docs.docker.com/engine/install/linux-postinstall/
+
+.. tip::
+   After installing, verify Docker works:
+
+   .. code-block:: console
+
+      $ docker run --rm hello-world
+
 
 For Developers
 --------------

docs/tutorials/case_e.rst

@@ -3,7 +3,7 @@
 E - A Time-Independent Experiment
 =================================
 
-This example shows how to run a realistic testing experiment (based on :ref:`Schorlemmer et al. 2010<References>`) while summarizing the concepts from the previous tutorials.
+This example shows how to run a realistic testing experiment (based on :ref:`Schorlemmer et al. 2010<case_e_references>`) while summarizing the concepts from the previous tutorials.
 
 .. currentmodule:: floatcsep
 
@@ -139,6 +139,8 @@ Plot command
     and re-run with the ``plot`` command. A forecast figure will re-appear in ``results/{window}/forecasts`` with a different colormap. Additional forecast and catalog plotting options can be found in the :func:`csep.utils.plots.plot_spatial_dataset` and :func:`csep.utils.plots.plot_catalog` ``pycsep`` functions.
 
 
+.. _case_e_references:
+
 References
 ----------
 

docs/tutorials/case_f.rst

@@ -49,7 +49,7 @@ The source files can be found in the ``tutorials/case_e`` folder or in  `the Git
 Model
 -----
 
-The time-dependency of a model is manifested here by the provision of different forecasts, i.e., statistical descriptions of seismicity, for different time-windows. In this example, the forecasts were created from an external model https://github.com/lmizrahi/etas (:ref:`Mizrahi et al. 2021<References>`_), with which the experiment has no interface. This means that we use **only the forecast files** and no source code. We leave the handling of a model source code for subsequent tutorials.
+The time-dependency of a model is manifested here by the provision of different forecasts, i.e., statistical descriptions of seismicity, for different time-windows. In this example, the forecasts were created from an external `ETAS model <https://github.com/lmizrahi/etas>`_ (:ref:`Mizrahi et al. 2021 <case_f_references>`), with which the experiment has no interface. This means that we use **only the forecast files** and no source code. We leave the handling of a model source code for subsequent tutorials.
 
 
 
@@ -120,6 +120,8 @@ Running the experiment
     This will automatically set all the calculation paths (testing catalogs, evaluation results, figures) and will create a summarized report in ``results/report.md``.
 
 
+.. _case_f_references:
+
 References
 ----------
 

docs/tutorials/case_g.rst

@@ -14,7 +14,11 @@ Here, we set up a time-dependent model from its **source code** for an experimen
         $ floatcsep run config.yml
 
     After the calculation is complete, the results will be summarized in ``results/report.md``.
+    For troubleshooting, run the experiment with:
 
+    .. code-block:: console
+
+        $ floatcsep run config.yml --debug
 
 .. currentmodule:: floatcsep
 

docs/tutorials/case_h.rst

@@ -14,6 +14,11 @@ Here, we run an experiment that accesses, containerizes and executes multiple **
         $ floatcsep run config.yml
 
     After the calculation is complete, the results will be summarized in ``results/report.md``.
+    For troubleshooting, run the experiment with:
+
+    .. code-block:: console
+
+        $ floatcsep run config.yml --debug
 
 .. currentmodule:: floatcsep
 

docs/tutorials/case_i.rst

@@ -0,0 +1,213 @@
+.. _case_i:
+
+I — Containerizing a Model with Docker
+======================================
+
+**Goal.** Show how to containerize a forecasting model with **Docker** and run it inside
+the floatCSEP engine, producing catalog forecasts and evaluating them with an N-test.
+This case uses simple mock models as examples.
+
+.. warning::
+
+   **Docker is required** to containerize models and (optionally) to run in parallel.
+   Please install Docker and complete the Linux post-installation steps if applicable.
+   See :ref:`docker-install` in the Installation guide.
+
+.. admonition:: **TL; DR**
+
+    In a terminal, navigate to ``floatcsep/tutorials/case_i`` and type:
+
+    .. code-block:: console
+
+        $ floatcsep run config.yml
+
+    After the calculation is complete, the results will be summarized in ``results/report.md``.
+    For troubleshooting, run the experiment with:
+
+    .. code-block:: console
+
+        $ floatcsep run config.yml --debug
+
+.. currentmodule:: floatcsep
+
+.. contents:: Contents
+    :local:
+
+
+
+Experiment layout
+-----------------
+
+The experiment input files are:
+
+::
+
+    case_i
+        └── pymock
+            └── pymock  # src code
+                | ...
+            ├── Dockerfile  # Docker image build instructions
+            └── setup.cfg  # Python build configuration
+        └── pymock_slow  # Same as pymock, but slower (to test cpu and DAG usage)
+            | ...
+        ├── catalog.csep
+        ├── config.yml
+        ├── tests.yml
+        └── models.yml
+
+
+
+Configuration
+-------------
+
+``config.yml``
+^^^^^^^^^^^^^^
+
+.. code-block:: yaml
+
+   name: case_i
+
+   time_config:
+     start_date: 2012-5-23T00:00:00
+     end_date:   2012-8-23T00:00:00
+     horizon:    7days
+     exp_class:  td
+
+   region_config:
+     region:    italy_csep_region
+     mag_min:   3.5
+     mag_max:   8.0
+     mag_bin:   0.5
+     depth_min: 0
+     depth_max: 70
+
+   run_mode:     parallel
+   force_rerun:  True
+   catalog:      catalog.csv
+   model_config: models.yml
+   test_config:  tests.yml
+
+   postprocess:
+     plot_forecasts: false
+
+**Notes**
+
+- ``exp_class: td`` declares a time-dependent experiment with rolling windows of length ``horizon``.
+- ``run_mode: parallel`` enables parallel solving (if resources allow).
+- ``force_rerun: True`` forces regeneration of forecasts even if files exist.
+
+
+``models.yml``
+^^^^^^^^^^^^^^
+
+.. code-block:: yaml
+
+   - pymock:
+       path: pymock
+       func: pymock
+       func_kwargs:
+         n_sims: 1000
+         mag_min: 3.5
+       build: docker
+
+   - pymock_slow:
+       path: pymock_slow
+       func: pymock
+       prefix: pymock
+       func_kwargs:
+         n_sims: 1000
+         mag_min: 3.5
+       build: docker
+
+**Notes**
+
+- ``build: docker`` tells floatCSEP to **build a Docker image** for the model located at ``path``.
+- ``func`` is the entry-point used **inside** the container to run the forecasts (defined in ``setup.cfg``)
+- ``func_kwargs`` are passed to ``model/input/args`` **inside** the container. They are stored and can be visualized in ``results/{time_window/input/{model}``.
+- You can add the options ``force_build`` to rebuild the Docker image.
+
+
+How containerization works here
+-------------------------------
+
+- For each model block marked ``build: docker``, floatCSEP:
+
+  1. Builds a Docker image from the model directory at ``path`` (expects a valid Dockerfile and the model’s code/config there).
+  2. Runs the container, mounting standard I/O folders used by the model (e.g., an ``input/`` and a model ``forecasts/`` directory), and executes your model’s entry-point (``func``).
+  3. Collects the produced forecast files and proceeds with evaluations.
+
+
+**Checklist to Dockerize your own model**
+
+- Add a **Dockerfile** in your model folder (``path``).
+- Ensure your entry-point can be invoked by the engine (e.g., a Python module or script that maps to ``func`` and accepts ``func_kwargs``).
+- The model should be able to read input data (catalog and args) from ``{model_basedir}/input``
+- Write outputs to the expected location (e.g., a ``{model_basedir}/forecasts/`` directory).
+- (Optional): Avoid writing to root-owned paths inside the container; keep everything under the mounted work basedir.
+- Test your image locally with a dry run:
+
+
+from the ``{model_basedir}``
+
+  .. code-block:: console
+
+     $ docker build \
+    --build-arg USERNAME=$USER \
+    --build-arg USER_UID=$(id -u) \
+    --build-arg USER_GID=$(id -g) \
+    -t {model_name} .
+
+     $ docker run --rm --volume $PWD:/usr/src/pymock:rw model_pymock python run.py input/args.txt
+
+
+What happens under the hood
+---------------------------
+
+1. Controller parses rolling time windows from ``start_date`` to ``end_date`` with a 7-day horizon.
+2. For each Dockerized model (``pymock``, ``pymock_slow``), the controller builds an image and launches
+   containers with the appropriate inputs/arguments.
+3. Forecasts are written to each model’s ``forecasts/`` directory (or central output).
+4. The **Catalog N-test** runs on the collected forecasts and generates diagnostic plots.
+
+
+Outputs
+-------
+
+You should find:
+
+- Weekly gridded forecasts in each model’s ``forecasts/`` folder.
+- N-test results (tables/JSON) in ``results/{time_window}/evaluations``.
+- Markdown and PDF reports summarizing the experiment results in ``results/report.md``.
+
+
+Running the experiment
+----------------------
+
+From the ``tutorials/case_i`` folder, run:
+
+.. code-block:: console
+
+   $ floatcsep run config.yml
+
+This will build the Docker images (if needed), run the containers for each time window and model,
+perform the evaluations, and create a summarized report in ``results/report.md``.
+
+
+Troubleshooting
+---------------
+
+- Run in debug mode with ``floatcsep run config.yml --debug``, or add ``--log`` to output a log file ``results/log``
+- **Docker not found / permission denied**:
+  - Ensure Docker is installed and your user can run containers (Linux: add to ``docker`` group).
+  - See :ref:`docker-install`.
+- **Model image fails to build**:
+  - Check a clean installation (without Docker) and running using its entry-point.
+  - Check the Dockerfile, base image, and that all runtime deps are installed in the image.
+  - Try building manually with ``docker build`` for clearer error messages.
+  - Adapt one of the provided Dockerfiles in the model folders.
+- **No forecasts produced**:
+  - Confirm your model writes to the expected ``forecasts/`` directory.
+  - Verify that the code actually reads from ``input/args.txt`` (or ``.yml`` or ``.json``).
+- **Plots not generated**:
+  - Inspect logs under ``results/`` for tracebacks.
+