Update to recent Python versions and packaging structure#141
Update to recent Python versions and packaging structure#141jacobtomlinson merged 19 commits intodask:mainfrom
Conversation
|
@jacobtomlinson Just a ping if there are any plans on updating dask-mpi? I wanted to use it in my project, but unfortunately other dependencies require python versions >3.12 (strangely, uv and pixi have no problems installing it, but pip, which i have to use, fails solving the dependencies) Before I spend effort downgrading, maybe this pull will eventually get merged and resolve my issues. Thanks! |
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
jacobtomlinson
changed the title
| auto-update-conda: false | ||
| python-version: ${{ matrix.python }} | ||
| environment-file: ${{ env.ENV_FILE }} | ||
| use-only-tar-bz2: true # IMPORTANT: This needs to be set for caching to work properly! |
There was a problem hiding this comment.
This is not needed as of conda-incubator/setup-miniconda@v3
dask_mpi/_version.py
Outdated
There was a problem hiding this comment.
This file now gets generated dynamically by setuptools-scm (via hatch-vcs) during build/install.
| worker_options, | ||
| name, | ||
| ): | ||
| from mpi4py import MPI |
There was a problem hiding this comment.
Importing this at import time was causing a race condition.
| if worker_task is not None: | ||
| await worker_task |
There was a problem hiding this comment.
Worker processes can hang if we fail to await the worker task in Python >= 3.11 so we are awaiting it explicitly during shutdown.
OpenMPI 5.x (PRRTE runtime) removed --oversubscribe and the old OMPI_MCA_rmaps_base_oversubscribe MCA parameter. The correct way to enable oversubscription is now via PRRTE_MCA_rmaps_default_mapping_policy. Additionally, OpenMPI 5.x's default CPU binding policy can hang in virtualized CI environments (e.g. GitHub Actions Azure VMs) where hwloc misdetects the CPU topology. Disabling binding via PRTE_MCA_rmaps_default_binding_policy=none fixes this. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
| # OpenMPI 5.x removed --oversubscribe; use PRRTE's mapping policy instead | ||
| PRTE_MCA_rmaps_default_mapping_policy: ':OVERSUBSCRIBE' | ||
| # Disable CPU binding to avoid hangs on virtualized CI hardware (OpenMPI 5.x) |
There was a problem hiding this comment.
These options were updated in openmpi 5
|
This is now ready for review. @kmpaul if you have some time I would much appreciate it! |
|
In the interest of getting a release out and the low maintenance capacity we currently have in Dask I'm going to merge this tomorrow if there are no further comments. |
2 participants
setup.pytopyproject.tomlsetuptoolstohatchlingversioneertohatch-vcs(most of the deleted lines are related to removing this vendored dependency)