ewoks install: fresh env by default and support pip+venv, uv, poetry, pixi and conda

Author: woutdenolf

CHANGELOG.md

@@ -9,13 +9,32 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- `ewoks install`: create a new python environment for the workflow (named after the workflow, in
+  the directory the package manager creates environments in) instead of installing in the current
+  environment. New arguments: `--env-root`, `--env-name`, `--clean`, `--in-place`,
+  `--python-version` and `--with-ewoks` (add `ewoks` itself when the requirements do not contain
+  it, so the workflow can be executed in the environment).
+- `ewoks execute`: add `--env` argument to execute a workflow in another python environment, for
+  example one created by `ewoks install`.
 - `ewoks install`: add `--package-manager-name` and `--package-manager-command` arguments.
 - `ewoks convert`: add `--package-manager-name` and `--package-manager-command` arguments.
 - Add `ewoks lint` CLI command to check if a workflow is conform to the Ewoks specification.
+- Support the `uv`, `conda`, `pixi` and `poetry` package managers to generate and install workflow
+  requirements. Every package manager stores the files it needs to reproduce the environment
+  (`uv.lock`, `environment.yml`, ...) as well as the list of installed python distributions, from
+  which any other package manager generates its own files to reproduce the environment.
+- Documentation: tutorial "Install and execute a workflow" and how-to guide "Reproduce a workflow
+  environment".
+
+### Changed
+
+- The `pip` package manager is named `pip-venv`: it creates environments with `venv` and installs
+  in them with `pip`.
 
 ### Removed
 
 - `ewoks install`: remove `-p/--python` argument.
+- Support for the `pipenv` package manager.
 
 ## [7.0.0rc1] - 2026-08-02
 

doc/howtoguides.rst

@@ -15,6 +15,7 @@ Getting started
     howtoguides/inspect_workflows
     howtoguides/gui
     howtoguides/python
+    howtoguides/requirements
     howtoguides/engines
 
 Automation

doc/howtoguides/requirements.rst

@@ -0,0 +1,121 @@
+.. _requirements:
+
+Reproduce a workflow environment
+================================
+
+A :term:`workflow` stores the python environment it was created in as the ``requirements``
+field of its ``graph`` field. ``ewoks install`` recreates that environment and
+``ewoks execute --env`` runs the :term:`workflow` in it. See
+:ref:`this tutorial <install_tutorial>` for a step-by-step example.
+
+What is stored
+--------------
+
+``ewoks convert`` and ``ewoks execute -o convert_destination=...`` store
+
+* ``python`` and ``system``: the python interpreter and the operating system.
+* ``distributions``: every installed python package with its version and, when it was not
+  installed from the python package index, the git commit or the archive it came from. Any
+  :term:`package manager` can recreate the environment from this list.
+* ``manager``: the :term:`package manager` that generated the requirements, with the content
+  of the files it needs to recreate the environment: ``requirements.txt`` for pip-venv,
+  ``pyproject.toml`` and ``uv.lock`` for uv, ``pyproject.toml`` and ``poetry.lock`` for
+  poetry, ``environment.yml`` for conda, ``pixi.toml`` and ``pixi.lock`` for pixi.
+
+Use ``--exclude-requirements`` to store nothing.
+
+.. note::
+
+    A :term:`workflow` that stores a list of requirements instead of the structure above is
+    still supported: the list is parsed as a ``requirements.txt`` file.
+
+Select a package manager
+------------------------
+
+``ewoks convert`` uses the :term:`package manager` of the current python environment. It is
+detected from the environment variables and files that package managers leave behind, and
+from the tool that installed most of the packages. Use ``--package-manager-name`` to select
+one explicitly
+
+.. code-block:: bash
+
+    ewoks convert demo demo.json --test --package-manager-name uv
+
+``ewoks install`` uses the :term:`package manager` that generated the requirements, unless it
+is not installed on the machine. It installs the stored files of that :term:`package manager`.
+When the requirements do not contain them, or installing them fails, it generates its own
+files from the ``distributions`` list and installs those instead. Both installations are
+confirmed separately unless ``--yes`` is provided
+
+.. code-block:: bash
+
+    # requirements generated by conda, environment created by uv
+    ewoks install demo.json --yes --package-manager-name uv
+
+``--package-manager-command`` provides the command that invokes the tool, for example when it
+is not on the ``PATH`` or when a faster implementation should be used
+
+.. code-block:: bash
+
+    ewoks install demo.json --yes --package-manager-name conda \
+        --package-manager-command micromamba
+
+Choose the environment
+----------------------
+
+.. code-block:: bash
+
+    # a root directory of your choice instead of the one of the package manager
+    ewoks install demo.json --yes --env-root /tmp/envs
+
+    # a name of your choice instead of the workflow identifier
+    ewoks install demo.json --yes --env-name demo-env
+
+    # remove the environment when it already exists
+    ewoks install demo.json --yes --clean
+
+    # another python version than the one stored in the requirements
+    ewoks install demo.json --yes --python-version 3.12
+
+    # add ewoks itself when the requirements do not contain it
+    ewoks install demo.json --yes --with-ewoks
+
+    # install in the current python environment instead of creating one
+    ewoks install demo.json --yes --in-place
+
+The default name is the identifier of the :term:`workflow`, which is the ``id`` field of its
+``graph`` field. A :term:`workflow` without an identifier gets a name derived from its
+content.
+
+Without ``--env-root`` the :term:`package manager` decides where the environment goes: conda
+creates it in its own environment directory (``conda config --show envs_dirs``), so
+``conda activate <name>`` finds it, and poetry in the directory it creates project
+environments in (``poetry config virtualenvs.path``). venv, uv and pixi create an environment
+wherever they are told to, so for those ewoks uses ``~/.ewoks/envs``.
+
+An environment that already exists is installed in, which adds the requirements to what is
+already there. Use ``--clean`` to remove it first. Only a directory that contains a python
+environment is removed.
+
+The environment of a :term:`workflow` is a directory. For pip-venv and conda it is a python
+environment, for uv and poetry it is a project with the environment in ``.venv`` and for pixi
+it is a workspace with the environment in ``.pixi/envs/default``. ``ewoks execute --env`` takes
+that directory, not the python interpreter inside it.
+
+Limitations
+-----------
+
+* ``--in-place`` is only supported by pip-venv and uv. The other package managers only install
+  in an environment they created themselves.
+* ``--python-version`` is a request: uv, conda and pixi can provide any python version, while
+  pip-venv and poetry can only use the version of the python interpreter that creates the
+  environment. A warning is emitted when the version cannot be provided.
+* uv, poetry and pixi resolve the ``distributions`` into a lock file, which requires access to
+  the package index. When that fails, a warning is emitted and the requirements are stored
+  without files: the environment is then recreated from the ``distributions`` list.
+* A lock file is only read by a :term:`package manager` that understands its format version, so
+  reproducing an environment can require a version of the tool that is at least as recent as
+  the one that generated the requirements.
+* poetry 1.8 or later is required. Older versions are reported as not available.
+* A package installed from a local directory cannot be recreated elsewhere. This is reported as
+  a warning when the requirements are generated.

doc/reference/cli.rst

@@ -18,9 +18,27 @@ ewoks install
 
     If no ``requirements`` field exist, **ewoks install** will try to extract requirements from the :term:`tasks <Task>` in the :term:`workflows <Workflow>` before installing them.
 
-    Unless ``--yes`` is provided, **ewoks install** will ask for confirmation before installing the packages.
+    **ewoks install** installs the files of the :term:`package manager` that generated the ``requirements``.
+    When the ``requirements`` do not contain those files, or installing them fails, the Python ``distributions`` they contain are installed instead.
 
-    By default, packages are installed in the current Python environment: if **ewoks install** is run in a virtual environment, the packages will be installed in this virtual environment.
+    Unless ``--yes`` is provided, **ewoks install** will ask for confirmation before each installation and before removing an existing environment.
+
+    By default, packages are installed in a new Python environment named after the workflow identifier, which is the ``id`` field of its ``graph`` field.
+    The environment is created where the :term:`package manager` creates named environments (``~/.ewoks/envs`` for package managers that do not have such a directory).
+    Use ``--env-root`` for another directory to create the environment in and ``--env-name`` for another name than the workflow identifier.
+    An environment that already exists is installed in, unless ``--clean`` is provided to remove it first.
+    The :term:`workflow` can be executed in that environment when it contains **ewoks** itself, which is the case when the ``requirements`` contain it.
+    Use ``--with-ewoks`` to add **ewoks** when the ``requirements`` do not contain it, without changing any of the versions they do contain:
+
+    .. code-block:: bash
+
+        ewoks install myworkflow.json --yes --with-ewoks
+        ewoks execute --env ~/.ewoks/envs/myworkflow myworkflow.json
+
+    Use ``--in-place`` to install in the current Python environment instead. This is not supported by all package managers.
+
+    The package manager that generated the ``requirements`` is used to reproduce the environment when it is available.
+    Provide ``--package-manager-name`` (and optionally ``--package-manager-command``) to use another one: the ``requirements`` always contain the list of installed Python distributions as a fallback.
 
 ewoks convert
 -------------

doc/reference/glossary.rst

@@ -28,5 +28,8 @@ Glossary
     Execution engine
         An execution engine is the underlying software used to execute the :term:`workflow`. :term:`Ewoks` supports multiple execution engines: pypushflow, orange, dask and the ewoks internal excution engine.
 
+    Package manager
+        A package manager creates python environments and installs packages in them. :term:`Ewoks` uses package managers to store the environment in which a :term:`workflow` was created and to recreate it: pip with venv, uv, poetry, conda and pixi.
+
     blissdata
         `Blissdata <https://bliss.gitlab-pages.esrf.fr/blissdata>`_ is an API for accessing data from BLISS in memory.

doc/tutorials.rst

@@ -8,6 +8,7 @@ Tutorials
     tutorials/hello_world
     tutorials/create_workflow
     tutorials/execute
+    tutorials/install
     tutorials/infrastructure
     tutorials/external
     tutorials/develop

doc/tutorials/execute.rst

@@ -13,6 +13,17 @@ The :ref:`python interface <execute_python>` for executing :term:`workflows <Wor
 
 The equivalent exists for the :ref:`command-line <cli>`.
 
+A :term:`workflow` can also be executed in a python environment created from the
+:term:`workflow` itself
+
+.. code-block:: bash
+
+    ewoks install /path/to/graph.json --yes
+    ewoks execute --env ~/.ewoks/envs/mygraph /path/to/graph.json
+
+See :ref:`this tutorial <install_tutorial>` and
+:ref:`this how-to guide <requirements>`.
+
 The :code:`engine=None` argument selects the default :term:`execution engine <Execution engine>`. Documentation on different :term:`execution engines <Execution engine>`:
 
 * `ewoksppf <https://ewoksppf.readthedocs.io/>`_ : execute cyclic :term:`workflows <Workflow>`

doc/tutorials/getting_started.rst

@@ -87,6 +87,16 @@ To inspect or modify a workflow, convert it to a JSON file:
 
    ewoks convert demo demo.json --test
 
+The JSON file also stores the python environment the :term:`workflow` needs, so it can be
+recreated later:
+
+.. code-block:: bash
+
+   ewoks install demo.json --yes
+   ewoks execute --env ~/.ewoks/envs/demo demo.json --outputs=all
+
+See the `Install and execute a workflow tutorial <./install.html>`_.
+
 Learn More
 ==========
 

doc/tutorials/install.rst

@@ -0,0 +1,105 @@
+.. _install_tutorial:
+
+Install and execute a workflow
+==============================
+
+A :term:`workflow` needs the python packages of its :term:`tasks <Task>` to be installed.
+:term:`Ewoks` can store the python environment in which a :term:`workflow` was created inside
+the :term:`workflow` itself and recreate that environment later, on another machine or at
+another time.
+
+This tutorial only needs the ``ewoks`` python package
+
+.. code-block:: bash
+
+    pip install ewoks
+
+Store the requirements
+----------------------
+
+``ewoks convert`` saves the packages installed in the current python environment as the
+``requirements`` of the destination :term:`workflow`
+
+.. code-block:: bash
+
+    ewoks convert demo demo.json --test
+
+The ``requirements`` field of the ``graph`` field looks like this
+
+.. code-block:: json
+
+    {
+        "python": {"version": "3.12.11", "implementation": "CPython", "...": "..."},
+        "system": {"system": "Linux", "machine": "x86_64", "...": "..."},
+        "distributions": [
+            {"name": "ewokscore", "version": "5.1.0", "installer": "pip"},
+            {"name": "networkx", "version": "3.4.2", "installer": "pip"}
+        ],
+        "manager": {
+            "name": "pip-venv",
+            "version": "25.0.1",
+            "files": {"requirements.txt": "ewokscore==5.1.0\nnetworkx==3.4.2\n"}
+        }
+    }
+
+* ``distributions`` are the installed python packages. Any :term:`package manager` can
+  recreate the environment from this list.
+* ``manager`` is the :term:`package manager` that generated the requirements together with
+  the files it needs to recreate the environment exactly, for example a lock file.
+
+Create the environment
+----------------------
+
+``ewoks install`` creates a python environment for the :term:`workflow`
+
+.. code-block:: bash
+
+    ewoks install demo.json --yes
+
+It prints the python interpreter of the environment it created, the command to execute the
+:term:`workflow` in it and the command to remove it again
+
+.. code-block:: text
+
+    Installed requirements for demo.json
+      Python : ~/.ewoks/envs/demo/bin/python
+      Execute: ewoks execute --env ~/.ewoks/envs/demo demo.json
+      Remove : rm -rf ~/.ewoks/envs/demo
+
+Without ``--yes`` you are asked to confirm after the packages have been listed. The
+environment is named after the :term:`workflow` and created where the :term:`package manager`
+creates named environments. Use ``--env-root`` to create it in a directory of your choice,
+``--env-name`` to give it another name and ``--clean`` to remove it when it already exists.
+
+Execute the workflow
+--------------------
+
+``--env`` executes the :term:`workflow` with the python interpreter of that environment
+instead of the current one
+
+.. code-block:: bash
+
+    ewoks execute --env ~/.ewoks/envs/demo demo.json --outputs=all
+
+This works because the requirements contain ``ewoks`` itself: it was installed in the
+environment in which the :term:`workflow` was converted. When they do not, add
+``--with-ewoks`` to ``ewoks install`` to install it without changing any of the versions
+coming from the :term:`workflow`.
+
+Remove the environment
+----------------------
+
+The environment is a normal directory, so removing it is enough
+
+.. code-block:: bash
+
+    rm -rf ~/.ewoks/envs/demo
+
+This is the command that ``ewoks install`` prints when it creates the environment.
+
+.. note::
+
+    ``ewoks install`` uses the :term:`package manager` that generated the requirements when it
+    is available and falls back to another one otherwise. See
+    :ref:`this how-to guide <requirements>` to select a :term:`package manager`, to install
+    in the current python environment or to choose a python version.

pyproject.toml

@@ -46,7 +46,8 @@ test = [
     "ipykernel",
     "importlib_resources",
     "pyqt5",
-    "pytest-venv",
+    "uv",
+    "poetry >=1.8",
     "defusedxml",
 ]
 dev = [