ewoks install: fresh env by default and support pip+venv, uv, poetry, pixi and conda

Author: woutdenolf

.github/workflows/test.yml

@@ -32,6 +32,7 @@ jobs:
         ${{ matrix.extra-pytest-warnings }}
       enable-coverage: ${{ matrix.enable-coverage }}
       jupyter-platform-dirs: "1"
+      package-managers: ${{ matrix.package-managers }}
       codecov-flags: "unit"
     strategy:
       fail-fast: false
@@ -41,42 +42,50 @@ jobs:
             python-version: "3.8"
             enable-coverage: "false"
             extra-pytest-warnings: ""
+            package-managers: "uv poetry pixi conda"
 
           - os: ubuntu-latest
             python-version: "3.9"
             enable-coverage: "false"
             # https://github.com/networkx/networkx/issues/7372
             extra-pytest-warnings: "-W ignore::RuntimeWarning:networkx.utils.backends"
+            package-managers: "uv poetry pixi conda"
 
           - os: ubuntu-latest
             python-version: "3.10"
             enable-coverage: "false"
             extra-pytest-warnings: ""
+            package-managers: "uv poetry pixi conda"
 
           - os: ubuntu-latest
             python-version: "3.11"
             enable-coverage: "false"
             extra-pytest-warnings: ""
+            package-managers: "uv poetry pixi conda"
 
           - os: ubuntu-latest
             python-version: "3.12"
             enable-coverage: "true"
             extra-pytest-warnings: ""
+            package-managers: "uv poetry pixi conda"
 
           - os: ubuntu-latest
             python-version: "3.13"
             enable-coverage: "false"
             extra-pytest-warnings: ""
+            package-managers: "uv poetry pixi conda"
 
           - os: ubuntu-latest
             python-version: "3.14"
             enable-coverage: "false"
             extra-pytest-warnings: ""
+            package-managers: "uv poetry pixi conda"
 
           - os: windows-latest
             python-version: "3.9"
             enable-coverage: "false"
             extra-pytest-warnings: ""
+            package-managers: "uv poetry pixi conda"
 
   # Run linter / checks
   checks:

CHANGELOG.md

@@ -9,13 +9,34 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- `ewoks install`: create a new python environment for the workflow (named after the workflow, in
+  the directory the package manager creates environments in) instead of installing in the current
+  environment. New arguments: `--env-root`, `--env-name`, `--clean`, `--in-place`,
+  `--python-version` and `--with-ewoks` (add `ewoks` itself when the requirements do not contain
+  it, so the workflow can be executed in the environment).
+- `ewoks execute`: add `--env` argument to execute a workflow in another python environment, for
+  example one created by `ewoks install`.
 - `ewoks install`: add `--package-manager-name` and `--package-manager-command` arguments.
 - `ewoks convert`: add `--package-manager-name` and `--package-manager-command` arguments.
+- `ewoks convert`: print the package manager that generated the requirements, or that the
+  requirements are not saved.
 - Add `ewoks lint` CLI command to check if a workflow is conform to the Ewoks specification.
+- Support the `uv`, `conda`, `pixi` and `poetry` package managers to generate and install workflow
+  requirements. Every package manager stores the files it needs to reproduce the environment
+  (`uv.lock`, `environment.yml`, ...) as well as the list of installed python distributions, from
+  which any other package manager generates its own files to reproduce the environment.
+- Documentation: tutorial "Install and execute a workflow" with an end-to-end walk-through for each
+  package manager, and how-to guide "Reproduce a workflow environment".
+
+### Changed
+
+- The `pip` package manager is named `pip-venv`: it creates environments with `venv` and installs
+  in them with `pip`.
 
 ### Removed
 
 - `ewoks install`: remove `-p/--python` argument.
+- Support for the `pipenv` package manager.
 
 ## [7.0.0rc1] - 2026-08-02
 

doc/conf.py

@@ -24,6 +24,7 @@
     "nbsphinx",
     "nbsphinx_link",
     "sphinx_copybutton",
+    "sphinx_tabs.tabs",
 ]
 templates_path = ["_templates"]
 exclude_patterns = ["build", "**.ipynb_checkpoints"]

doc/howtoguides.rst

@@ -15,6 +15,7 @@ Getting started
     howtoguides/inspect_workflows
     howtoguides/gui
     howtoguides/python
+    howtoguides/requirements
     howtoguides/engines
 
 Automation

doc/howtoguides/requirements.rst

@@ -0,0 +1,122 @@
+.. _requirements:
+
+Reproduce a workflow environment
+================================
+
+A :term:`workflow` stores the python environment it was created in as the ``requirements``
+field of its ``graph`` field. ``ewoks install`` recreates that environment and
+``ewoks execute --env`` runs the :term:`workflow` in it. See
+:ref:`this tutorial <install_tutorial>` for a step-by-step example.
+
+What is stored
+--------------
+
+``ewoks convert`` and ``ewoks execute -o convert_destination=...`` store
+
+* ``python`` and ``system``: the python interpreter and the operating system.
+* ``distributions``: every installed python package with its version and, when it was not
+  installed from the python package index, the git commit or the archive it came from. Any
+  :term:`package manager` can recreate the environment from this list.
+* ``manager``: the :term:`package manager` that generated the requirements, with the content
+  of the files it needs to recreate the environment: ``requirements.txt`` for pip-venv,
+  ``pyproject.toml`` and ``uv.lock`` for uv, ``pyproject.toml`` and ``poetry.lock`` for
+  poetry, ``environment.yml`` for conda, ``pixi.toml`` and ``pixi.lock`` for pixi.
+
+Use ``--exclude-requirements`` to store nothing.
+
+.. note::
+
+    A :term:`workflow` that stores a list of requirements instead of the structure above is
+    still supported: the list is parsed as a ``requirements.txt`` file.
+
+Select a package manager
+------------------------
+
+``ewoks convert`` uses the :term:`package manager` of the current python environment. It is
+detected from the environment variables and files that package managers leave behind, and
+from the tool that installed most of the packages. Use ``--package-manager-name`` to select
+one explicitly
+
+.. code-block:: bash
+
+    ewoks convert demo demo.json --test --package-manager-name uv
+
+``ewoks install`` uses the :term:`package manager` that generated the requirements, unless it
+is not installed on the machine. It installs the stored files of that :term:`package manager`.
+When the requirements do not contain them, or installing them fails, it generates its own
+files from the ``distributions`` list and installs those instead. Both installations are
+confirmed separately unless ``--yes`` is provided
+
+.. code-block:: bash
+
+    # requirements generated by conda, environment created by uv
+    ewoks install demo.json --yes --package-manager-name uv
+
+``--package-manager-command`` provides the command that invokes the tool, for example when it
+is not on the ``PATH`` or when a faster implementation should be used
+
+.. code-block:: bash
+
+    ewoks install demo.json --yes --package-manager-name conda \
+        --package-manager-command micromamba
+
+Choose the environment
+----------------------
+
+.. code-block:: bash
+
+    # a root directory of your choice instead of the one of the package manager
+    ewoks install demo.json --yes --env-root /tmp/envs
+
+    # a name of your choice instead of the workflow identifier
+    ewoks install demo.json --yes --env-name demo-env
+
+    # remove the environment when it already exists
+    ewoks install demo.json --yes --clean
+
+    # another python version than the one stored in the requirements
+    ewoks install demo.json --yes --python-version 3.12
+
+    # add ewoks itself when the requirements do not contain it
+    ewoks install demo.json --yes --with-ewoks
+
+    # install in the current python environment instead of creating one
+    ewoks install demo.json --yes --in-place
+
+The default name is the identifier of the :term:`workflow`, which is the ``id`` field of its
+``graph`` field. A :term:`workflow` without an identifier gets a name derived from its
+content.
+
+Without ``--env-root`` the :term:`package manager` decides where the environment goes: conda
+creates it in its own environment directory (``conda config --show envs_dirs``), so
+``conda activate <name>`` finds it, and poetry in the directory it creates project
+environments in (``poetry config virtualenvs.path``). venv, uv and pixi create an environment
+wherever they are told to, so for those ewoks uses ``~/.ewoks/envs``.
+
+An environment that already exists is installed in, which adds the requirements to what is
+already there. Use ``--clean`` to remove it first. Only a directory that contains a python
+environment is removed.
+
+The environment of a :term:`workflow` is a directory. For pip-venv and conda it is a python
+environment, for uv and poetry it is a project with the environment in ``.venv`` and for pixi
+it is a workspace with the environment in ``.pixi/envs/default``. ``ewoks execute --env`` takes
+that directory, not the python interpreter inside it.
+
+Limitations
+-----------
+
+* ``--in-place`` is only supported by pip-venv and uv. The other package managers only install
+  in an environment they created themselves.
+* ``--python-version`` is a request: uv can provide any python version, conda and pixi provide
+  the patch versions built by their channel, and pip-venv and poetry can only use the version of
+  the python interpreter that creates the environment. A warning is emitted when the version
+  cannot be provided.
+* uv, poetry and pixi resolve the ``distributions`` into a lock file, which requires access to
+  the package index. When that fails, a warning is emitted and the requirements are stored
+  without files: the environment is then recreated from the ``distributions`` list.
+* A lock file is only read by a :term:`package manager` that understands its format version, so
+  reproducing an environment can require a version of the tool that is at least as recent as
+  the one that generated the requirements.
+* poetry 1.8 or later is required. Older versions are reported as not available.
+* A package installed from a local directory cannot be recreated elsewhere. This is reported as
+  a warning when the requirements are generated.

doc/reference/cli.rst

@@ -18,9 +18,27 @@ ewoks install
 
     If no ``requirements`` field exist, **ewoks install** will try to extract requirements from the :term:`tasks <Task>` in the :term:`workflows <Workflow>` before installing them.
 
-    Unless ``--yes`` is provided, **ewoks install** will ask for confirmation before installing the packages.
+    **ewoks install** installs the files of the :term:`package manager` that generated the ``requirements``.
+    When the ``requirements`` do not contain those files, or installing them fails, the Python ``distributions`` they contain are installed instead.
 
-    By default, packages are installed in the current Python environment: if **ewoks install** is run in a virtual environment, the packages will be installed in this virtual environment.
+    Unless ``--yes`` is provided, **ewoks install** will ask for confirmation before each installation and before removing an existing environment.
+
+    By default, packages are installed in a new Python environment named after the workflow identifier, which is the ``id`` field of its ``graph`` field.
+    The environment is created where the :term:`package manager` creates named environments (``~/.ewoks/envs`` for package managers that do not have such a directory).
+    Use ``--env-root`` for another directory to create the environment in and ``--env-name`` for another name than the workflow identifier.
+    An environment that already exists is installed in, unless ``--clean`` is provided to remove it first.
+    The :term:`workflow` can be executed in that environment when it contains **ewoks** itself, which is the case when the ``requirements`` contain it.
+    Use ``--with-ewoks`` to add **ewoks** when the ``requirements`` do not contain it, without changing any of the versions they do contain:
+
+    .. code-block:: bash
+
+        ewoks install myworkflow.json --yes --with-ewoks
+        ewoks execute --env ~/.ewoks/envs/myworkflow myworkflow.json
+
+    Use ``--in-place`` to install in the current Python environment instead. This is not supported by all package managers.
+
+    The package manager that generated the ``requirements`` is used to reproduce the environment when it is available.
+    Provide ``--package-manager-name`` (and optionally ``--package-manager-command``) to use another one: the ``requirements`` always contain the list of installed Python distributions as a fallback.
 
 ewoks convert
 -------------

doc/reference/glossary.rst

@@ -28,5 +28,8 @@ Glossary
     Execution engine
         An execution engine is the underlying software used to execute the :term:`workflow`. :term:`Ewoks` supports multiple execution engines: pypushflow, orange, dask and the ewoks internal excution engine.
 
+    Package manager
+        A package manager creates python environments and installs packages in them. :term:`Ewoks` uses package managers to store the environment in which a :term:`workflow` was created and to recreate it: pip with venv, uv, poetry, conda and pixi.
+
     blissdata
         `Blissdata <https://bliss.gitlab-pages.esrf.fr/blissdata>`_ is an API for accessing data from BLISS in memory.

doc/tutorials.rst

@@ -8,6 +8,7 @@ Tutorials
     tutorials/hello_world
     tutorials/create_workflow
     tutorials/execute
+    tutorials/install
     tutorials/infrastructure
     tutorials/external
     tutorials/develop

doc/tutorials/execute.rst

@@ -13,6 +13,17 @@ The :ref:`python interface <execute_python>` for executing :term:`workflows <Wor
 
 The equivalent exists for the :ref:`command-line <cli>`.
 
+A :term:`workflow` can also be executed in a python environment created from the
+:term:`workflow` itself
+
+.. code-block:: bash
+
+    ewoks install /path/to/graph.json --yes
+    ewoks execute --env ~/.ewoks/envs/mygraph /path/to/graph.json
+
+See :ref:`this tutorial <install_tutorial>` and
+:ref:`this how-to guide <requirements>`.
+
 The :code:`engine=None` argument selects the default :term:`execution engine <Execution engine>`. Documentation on different :term:`execution engines <Execution engine>`:
 
 * `ewoksppf <https://ewoksppf.readthedocs.io/>`_ : execute cyclic :term:`workflows <Workflow>`

doc/tutorials/getting_started.rst

@@ -87,6 +87,16 @@ To inspect or modify a workflow, convert it to a JSON file:
 
    ewoks convert demo demo.json --test
 
+The JSON file also stores the python environment the :term:`workflow` needs, so it can be
+recreated later:
+
+.. code-block:: bash
+
+   ewoks install demo.json --yes
+   ewoks execute --env ~/.ewoks/envs/demo demo.json --outputs=all
+
+See the `Install and execute a workflow tutorial <./install.html>`_.
+
 Learn More
 ==========