concurrent.futures._base.CancelledError occurred when running grnboost2

[YimengQiao]

Hi, thanks for the great packages! I am trying to use pyscenic to build GRNs for my custom dataset on an HPC cluster. However, there raised an error that looks like "concurrent.futures._base.CancelledError: finalize-42b352fb7aa8be55b379c245a43ab99e" either I use a conda environment or singularity image.

The code I ran is that:

#!/bin/bash
#SBATCH --job-name=grn
#SBATCH -N 1 -n 8
#SBATCH -p cpu

module load singularity
singularity build aertslab-pyscenic-0.12.1.sif docker://aertslab/pyscenic:0.12.1
singularity run aertslab-pyscenic-0.12.1.sif \
    pyscenic grn \
        --num_workers 8 \
        -o ./expr_mat.adjacencies.tsv \
        ./data/panE_unfiltered.loom \
        ./data/allTFs_hg38.txt

And the error is:

2024-08-01 16:08:20,913 - pyscenic.cli.pyscenic - INFO - Loading expression matrix.

2024-08-01 16:09:35,776 - pyscenic.cli.pyscenic - INFO - Inferring regulatory networks.
2024-08-01 16:14:58,092 - distributed.client - ERROR - 
Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/distributed/utils.py", line 742, in wrapper
    return await func(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/distributed/client.py", line 1298, in _reconnect
    await self._ensure_connected(timeout=timeout)
  File "/opt/venv/lib/python3.10/site-packages/distributed/client.py", line 1328, in _ensure_connected
    comm = await connect(
  File "/opt/venv/lib/python3.10/site-packages/distributed/comm/core.py", line 291, in connect
    comm = await asyncio.wait_for(
  File "/usr/local/lib/python3.10/asyncio/tasks.py", line 432, in wait_for
    await waiter
asyncio.exceptions.CancelledError

Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/distributed/utils.py", line 742, in wrapper
    return await func(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/distributed/client.py", line 1508, in _handle_report
    await self._reconnect()
  File "/opt/venv/lib/python3.10/site-packages/distributed/utils.py", line 742, in wrapper
    return await func(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/distributed/client.py", line 1298, in _reconnect
    await self._ensure_connected(timeout=timeout)
  File "/opt/venv/lib/python3.10/site-packages/distributed/client.py", line 1328, in _ensure_connected
    comm = await connect(
  File "/opt/venv/lib/python3.10/site-packages/distributed/comm/core.py", line 291, in connect
    comm = await asyncio.wait_for(
  File "/usr/local/lib/python3.10/asyncio/tasks.py", line 432, in wait_for
    await waiter
asyncio.exceptions.CancelledError
preparing dask client
parsing input
creating dask graph
8 partitions
computing dask graph
not shutting down client, client was created externally
finished

Traceback (most recent call last):
  File "/opt/venv/bin/pyscenic", line 33, in <module>
    sys.exit(load_entry_point('pyscenic==0.12.1+0.gce41b61.dirty', 'console_scripts', 'pyscenic')())
  File "/opt/venv/lib/python3.10/site-packages/pyscenic/cli/pyscenic.py", line 713, in main
    args.func(args)
  File "/opt/venv/lib/python3.10/site-packages/pyscenic/cli/pyscenic.py", line 106, in find_adjacencies_command
    network = method(
  File "/opt/venv/lib/python3.10/site-packages/arboreto/algo.py", line 39, in grnboost2
    return diy(expression_data=expression_data, regressor_type='GBM', regressor_kwargs=SGBM_KWARGS,
  File "/opt/venv/lib/python3.10/site-packages/arboreto/algo.py", line 134, in diy
    return client \
  File "/opt/venv/lib/python3.10/site-packages/distributed/client.py", line 3337, in compute
    result = self.gather(futures)
  File "/opt/venv/lib/python3.10/site-packages/distributed/client.py", line 2291, in gather
    return self.sync(
  File "/opt/venv/lib/python3.10/site-packages/distributed/utils.py", line 339, in sync
    return sync(
  File "/opt/venv/lib/python3.10/site-packages/distributed/utils.py", line 406, in sync
    raise exc.with_traceback(tb)
  File "/opt/venv/lib/python3.10/site-packages/distributed/utils.py", line 379, in f
    result = yield future
  File "/opt/venv/lib/python3.10/site-packages/tornado/gen.py", line 762, in run
    value = future.result()
  File "/opt/venv/lib/python3.10/site-packages/distributed/client.py", line 2155, in _gather
    raise exc
concurrent.futures._base.CancelledError: finalize-42b352fb7aa8be55b379c245a43ab99e

Finally, I will get an empty .adj.tsv file.

I'd appreciate if you could help me with that problem. Thanks!