Serialization error when trying to use n_workers > 1 in create_nwbs

[gshvarts]

I'm getting the following errors when trying to run create_nwbs with n_workers > 1:

Making NWB for animal == 'bobrick' and date == 20231206
Object <function create_nwbs.<locals>.pass_func at 0x7f30e2e9d6c0> cannot be deterministically hashed. This likely indicates that the object cannot be serialized deterministically.
Making NWB for animal == 'bobrick' and date == 20231207
[/home/gl-willow/mambaforge/envs/trodes_to_nwb_3/lib/python3.13/site-packages/distributed/node.py:187](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/mambaforge/envs/trodes_to_nwb_3/lib/python3.13/site-packages/distributed/node.py:187): UserWarning: Port 8787 is already in use.
Perhaps you already have a cluster running?
Hosting the HTTP server on port 46375 instead
  warnings.warn(
Object <function create_nwbs.<locals>.pass_func at 0x7f2ff3e73c40> cannot be deterministically hashed. This likely indicates that the object cannot be serialized deterministically.
Making NWB for animal == 'bobrick' and date == 20231208
[/home/gl-willow/mambaforge/envs/trodes_to_nwb_3/lib/python3.13/site-packages/distributed/node.py:187](https://file+.vscode-resource.vscode-cdn.net/home/gl-willow/mambaforge/envs/trodes_to_nwb_3/lib/python3.13/site-packages/distributed/node.py:187): UserWarning: Port 8787 is already in use.
Perhaps you already have a cluster running?
Hosting the HTTP server on port 37473 instead
  warnings.warn(
Object <function create_nwbs.<locals>.pass_func at 0x7f2ff1d27ba0> cannot be deterministically hashed. This likely indicates that the object cannot be serialized deterministically.

I don't remember seeing this before when trying to up the number of workers (previously I think we were running up against memory errors instead but I don't remember exactly). Any idea what's going on here?

[samuelbray32]

Sorry haven't been able to get to this. I think this is an issue because the pass_func is defined inside of create_nwbs and new versions of dask are more strict in their hashing.

If you could let me know

1. If this is still an issue
2. The version of dask in your environment

I can make a branch to try out a hopefully easy fix