Migrating the extension dependencies to pip-tools

[Thibault-Pelletier]

Following recent PRs in the repo and the recent addition to Slicer packaging, the dependency handling for SlicerNUNet should be moved to pip-tools.

Overview

Pip tools migration implies:

• A requirements.in / pyproject.toml file compiling the bare requirements necessary for the extension including Python and platform specific requirements
• A constraints.txt file listing requirements that shouldn't be touched in Slicer environment (in the long run the contraints.txt file would be handled by Slicer)

Pip tools would then make it possible to generate requirements lock file per supported python version and supported OS.
This way of specifying the requirements should be compatible with the new Slicer packaging infrastructure.

Example of content

Example requirements.in content:

nnunetv2>=2.7.0; platform_system == "Darwin"
nnunetv2>=2.3.0; platform_system != "Darwin"
acvl-utils==0.2.0; python_version < "3.12"
dynamic_network_architectures!=0.4

Example constraints.txt content:

SimpleITK==9999
torch==9999
requests==9999
dynamic_network_architectures!=0.4

Lock generation

Lock generation per platform would imply generating the locks on the CI when generating a new version.

The lock generation can be done / tested on the Github CI first. In the future, lock generation may make more sense on Kitware's CI when packaging the wheel and after running the unit tests to make sure the locked versions are actually working in Slicer's environment.

On Github lock generation can be done in a job that will:

• Checkout the latest stable Slicer version
• Run pip-compile for the given Slicer to generate the lock file
• Install the extension
• Run the self tests of the extension
• Commit the updated lock file provided the self tests are working

As Slicer ships with given python versions, requirements locks should probably mapped to requirements-py<version>-<os>.lock tags.

Enforcing version

Pip tools comes with the pip-sync method followed by args that force the environment to be synced to the required package version.
Running pip-sync requirements-py<version>-<os>.lock --pip-args "--no-deps" would force the environment to be updated to locked versions.

This differs from pip_ensure which does a soft check based on the requirements.in only. For the NNUNet package, it may be necessary to use a hard sync as compared to the soft pip_ensure given the package's compatibility flakiness.

@lassoan @ebrahimebrahim @pieper happy to get your thoughts on this!

[Thibault-Pelletier]

Related to #15

[ebrahimebrahim]

Just now learning about pip-tools

I am wondering, if we adopt uv in Slicer, would that remove the need for a pip-tools approach? We'd have the uv.lock, which sounds like it serves a similar purpose but I am not sure

There's a possibility of moving to uv for python package installation in slicer, possibly extracting the approach from Slicer/Slicer#8181

[Thibault-Pelletier]

From my understanding, and from David's PR, UV seems to be a more global package manager approach which requires deeper integration in the 3D Slicer python packaging environment.

It can handle creating new virtual environments, handles caching and more.

So I think that this can be a long run solution for 3D Slicer, but I'm not sure the rest of David's PR makes a lot of sens at the granularity of the extensions.

To support compatibility in both trame-slicer and Slicer, I'm thinking that python extensions should be maintained as python packages directly. This would mean having a pyproject.toml per extension, source files to be packaged and installed in wheels, PyPI publishing (if applicable).

Installing the extension in Slicer would then be done using a pip install command (or UV install) while resolving 3D Slicer's constraints.

Here pip-tools is more of a short term solution to handling dependencies and lock at Slicer's level. Its fully pip installable and can be called using python -m module calls and is compatible with pyproject.toml inputs to generate a requirements lock file (also compatible with UV).

So I think it's a low risk, low effort, possibly high return investment compatible with long term Slicer maintenance.

[lassoan]

Pip tools would then make it possible to generate requirements lock file per supported python version and supported OS.

The working combination of package versions also depend on what features of what Slicer extensions you use. A custom Slicer app developer can create requirements.in file(s) because he has all this information. However, an extension developer would not have enough information for this, because he does not know which features of his extension would be used and what other features of other extensions are needed and what their Python dependencies are.

Another issue with pip-tools is that pip-sync may uninstall all the python packages that the user installed interactively in the Python console, which would be really annoying when doing prototyping.

SimpleITK==9999
torch==9999
requests==9999

To me it seems that this does not skip packages but makes dependency resolution fail and then the custom Slicer app developer has to update the requirements.in file manually, considering all the extensions and their features that he wants to use in Slicer. This may work for him, but not for extension developers (they don't know about what else the end user needs) and not for end users (who are not developers and may not know anything about Python).

Selective pip install in Slicer is a fully automatic solution that allows removal and overriding of any dependency. Of course, it would be nicer if we could rely on an existing tool, but unfortunately pip-tools does not have a SimpleITK==skip option.

To support compatibility in both trame-slicer and Slicer, I'm thinking that python extensions should be maintained as python packages directly.

I agree. In the long term, all Slicer extensions (not just purely-Python-scripted but those that have C++ components as well) should be pip-installable Python packages. That would allow composing an application using a Python package manager - the same way as all Python applications.

[Thibault-Pelletier]

So after playing with UV a few minutes, I take back my earlier comment.

UV supports lock generation, excluded dependencies and can generate pip compatible requirements file.
UV installed in Slicer can resolve dependencies for a given pyproject.toml file, but this will be limited to the Python version shipped in Slicer. This will generate errors when trying to support multi python versions in the pyproject.toml config

For example acvl_utils pinning cannot be resolved using Slicer 5.10 + UV to generate a lock file for python 3.9 :

requires-python = ">=3.9"
dependencies = [
    "nnunetv2; sys_platform != 'darwin'",
    "nnunetv2 >= 2.7.0; sys_platform == 'darwin'",
    "acvl_utils == 0.2.0; python_version < '3.12'",
    "dynamic_network_architectures == 0.2.0; python_version < '3.12'",
    "imagecodecs <= 2026.3.6; sys_platform == 'darwin'",
]

[tool.uv]
exclude-dependencies = [
    "torch",
    "SimpleITK",
    "requests"
]

I still have to play a bit more with the uv sync --inexact option to check how it works (for instance when colliding with other installs).

On a side note, UV is indeed blazing fast, esp. with its parallel dependencies downloads.

[ebrahimebrahim]

UV installed in Slicer can resolve dependencies for a given pyproject.toml file, but this will be limited to the Python version shipped in Slicer. This will generate errors when trying to support multi python versions in the pyproject.toml config

I wonder why you are running into this limitation. Normally uv lock resolves dependencies across all python versions included in requires-python

[Thibault-Pelletier]

That's because usually uv runs on the system and not in the Slicer environment.
When trying to resolve for Python 3.9 it somehow finds a python I have on my system (not in the Path) which may add to the error as well.

So the question becomes what generates the lock vs what synchronizes the lock.

What I can test is using a system UV to generate the lock and check what happens once loading from Slicer.
What I would like to test as well is generating UV locks for more than the SlicerNNUNet, syncing in Slicer and seeing how it explodes.
As this takes a bit of exploration, I'm not sure when I'll get to it though 🤔 ...