Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/build-docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ jobs:
- name: Install Python dependencies
run: |
python -m pip install --upgrade pip
python -m pip install pathos pygraphviz neuron cloud-volume k3d scikit-image open3d
python -m pip install joblib pathos pygraphviz neuron cloud-volume k3d scikit-image open3d
python -m pip install -e .[test-notebook,all,docs,flybrains,cloud-volume]
- name: Download test data
run: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/test-tutorials.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ jobs:
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install pathos pygraphviz neuron cloud-volume k3d scikit-image open3d
python -m pip install joblib pathos pygraphviz neuron cloud-volume k3d scikit-image open3d
python -m pip install -e .[test-notebook,all]
- name: Download test data
run: |
Expand Down
34 changes: 29 additions & 5 deletions docs/examples/6_misc/tutorial_misc_00_multiprocess.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,16 +5,24 @@
This notebook will show you how to use parallel processing with `navis`.

By default, most {{ navis }} functions use only a single thread/process (although some third-party functions
used under the hood might). Distributing expensive computations across multiple cores can speed things up considerable.
used under the hood might use more). Distributing expensive computations across multiple cores can speed things
up considerable.

Many {{ navis }} functions natively support parallel processing. This notebook will illustrate various ways
to use parallelism. Before we get start: {{ navis }} uses `pathos` for multiprocessing - if you installed
{{ navis }} with `pip install navis[all]` you should be all set. If not, you can install `pathos` separately:
to use parallelism. Before we get started: by default, {{ navis }} uses `joblib` as backend for multiprocessing.
However, you can also use `pathos` as an alternative. If you installed {{ navis }} with `pip install navis[all]`
you should be all set to use either. If not, you can install the packages separately:

```shell
pip install joblib tqdm-joblib -U
pip install pathos -U
```

!!! tip
Parallel processing incurs overhead: we have to spawn additional processes and move data between the main
& the worker processes. If you have fast single-core performance and/or small tasks, that overhead might outweigh
the benefits of parallelism. See also additional notes at the bottom of this tutorial.

## Running {{ navis }} functions in parallel

Since version `0.6.0` many {{ navis }} functions accept a `parallel=True` and an (optional) `n_cores` parameter:
Expand All @@ -36,7 +44,7 @@ def time_func(func, *args, **kwargs):
# %%
# !!! important
# This documentation is built on Github Actions where the number of cores can be as low as 2. The speedup on
# your machine should be more pronounced than what you see below. That said: parallel processing has some
# your machine should be more pronounced than the times you see below. That said: parallel processing has some
# overhead and for small tasks the overhead can be larger than the speed-up.

# %%
Expand Down Expand Up @@ -87,6 +95,17 @@ def time_func(func, *args, **kwargs):
# the number of available CPU cores. However, doing so will likely over-subscribe your CPU and end up
# slowing things down.

# %%
# Additional parameters for controlling parallelism:
# - `backend`: either "auto" (default), "joblib", or "pathos". This determines which parallel processing
# backend to use. "auto" will pick "joblib" if available, otherwise "pathos". Note: `joblib` is compatible
# with Dask to run on clusters. See the joblib documentation for details on how to set that up and then
# use `backend="joblib:dask"` in {{ navis }}.
# - `chunksize`: either "auto" (default) or an integer. This determines the number of neurons
# that will be processed per worker in each batch. "auto" will pick a chunksize that tries to balance
# load across workers while minimizing overhead. You can also set a fixed chunksize
# (e.g. `chunksize=10`).

# %%
# ## Parallelizing generic functions
#
Expand Down Expand Up @@ -114,5 +133,10 @@ def my_func(x):
nl.apply, my_func, parallel=True
)

"""
## A note on free-threading


With version 3.13, Python introduced a free-threading build where the Global Interpreter Lock (GIL) is removed.
In theory, this would allow true multi-threading in Python and make parallel processing with threads much more efficient.
However, as of late 2025, many of the libraries used by {{ navis }} (e.g. `igraph`) do not yet support free-threading.
"""
16 changes: 14 additions & 2 deletions docs/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,10 +123,22 @@ directly, are listed below:

---

#### `joblib`: [joblib](https://joblib.readthedocs.io)

Joblib is a multiprocessing library. {{ navis }} uses it to parallelize functions
across lists of neurons. Please note that you will also need to install `tqdm-joblib`
for progress bars to work with joblib.

``` shell
pip install joblib tqdm-joblib
```

---

#### `pathos`: [pathos](https://github.com/uqfoundation/pathos)

Pathos is a multiprocessing library. {{ navis }} uses it to parallelize functions
across lists of neurons.
Pathos is a multiprocessing library. {{ navis }} uses it as an alternative backend
to parallelize functions across lists of neurons.

``` shell
pip install pathos
Expand Down
Loading
Loading