Skip to content

Use joblib as default parallel backend#190

Open
schlegelp wants to merge 2 commits into
masterfrom
joblib_backend
Open

Use joblib as default parallel backend#190
schlegelp wants to merge 2 commits into
masterfrom
joblib_backend

Conversation

@schlegelp

@schlegelp schlegelp commented Nov 10, 2025

Copy link
Copy Markdown
Collaborator

With this PR we switch the default backend from pathos to joblib. pathos is still supported via the new backend parameter but joblibhas a few advantages:

  1. In my hands it seeems considerable faster for small tasks (possibly due to lower overhead/faster serialization)
  2. It temporarily keeps processes alive in the background to avoid having to spawn them again (this is a bit hit & miss)
  3. Automatically determines batch sizes
  4. Avoids oversubscription by automatically limiting the number of inner threads
  5. Compatible with Dask (see also Hackathon: dask backend #111)

The plan is to let this incubate a little before merging.

TODO:

  • explore using joblib for NBLAST
  • make joblib a required dependency (currently optional)
  • incorporate tqdm-joblib (it's only ~40 lines of code)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant