the loop started by dask-ctl.lifecycle.create_cluster conflicts with jupyterlab's loop #66
Description
I tried to launch a local cluster on jupyterlab using the python API, and received an error saying that the loop was already running:
import dask_ctl.lifecycle
import pathlib
spec = """
version: 1
module: "dask.distributed"
class: "LocalCluster"
args: []
kwargs:
n_workers: 2
threads_per_worker: 1
memory_limit: '1GB'
"""
path = pathlib.Path("spec.yaml")
path.write_text(spec)
dask_ctl.lifecycle.create_cluster(str(path))traceback
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
Cell In[1], line 17
14 path = pathlib.Path("spec.yaml")
15 path.write_text(spec)
---> 17 dask_ctl.lifecycle.create_cluster(str(path))
File .../lib/python3.12/site-packages/dask_ctl/lifecycle.py:73, in create_cluster(spec_path, local_fallback, asynchronous)
70 cluster.shutdown_on_close = False
71 return cluster
---> 73 return loop.run_sync(_create_cluster)
File .../lib/python3.12/site-packages/tornado/ioloop.py:533, in IOLoop.run_sync(self, func, timeout)
530 self.stop()
532 timeout_handle = self.add_timeout(self.time() + timeout, timeout_callback)
--> 533 self.start()
534 if timeout is not None:
535 self.remove_timeout(timeout_handle)
File .../lib/python3.12/site-packages/tornado/platform/asyncio.py:205, in BaseAsyncIOLoop.start(self)
204 def start(self) -> None:
--> 205 self.asyncio_loop.run_forever()
File .../lib/python3.12/asyncio/base_events.py:630, in BaseEventLoop.run_forever(self)
628 """Run until stop() is called."""
629 self._check_closed()
--> 630 self._check_running()
631 self._set_coroutine_origin_tracking(self._debug)
633 old_agen_hooks = sys.get_asyncgen_hooks()
File .../lib/python3.12/asyncio/base_events.py:622, in BaseEventLoop._check_running(self)
620 def _check_running(self):
621 if self.is_running():
--> 622 raise RuntimeError('This event loop is already running')
623 if events._get_running_loop() is not None:
624 raise RuntimeError(
625 'Cannot run the event loop while another loop is running')
RuntimeError: This event loop is already runningversions
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
anyio 4.6.2.post1 pyhd8ed1ab_0 conda-forge
argon2-cffi 23.1.0 pyhd8ed1ab_0 conda-forge
argon2-cffi-bindings 21.2.0 py312h66e93f0_5 conda-forge
arrow 1.3.0 pyhd8ed1ab_0 conda-forge
asttokens 2.4.1 pyhd8ed1ab_0 conda-forge
async-lru 2.0.4 pyhd8ed1ab_0 conda-forge
attrs 24.2.0 pyh71513ae_0 conda-forge
aws-c-auth 0.7.31 hcdce11a_5 conda-forge
aws-c-cal 0.7.4 hd3f4568_4 conda-forge
aws-c-common 0.9.31 hb9d3cd8_0 conda-forge
aws-c-compression 0.2.19 hf20e7d7_4 conda-forge
aws-c-event-stream 0.5.0 h72d8268_0 conda-forge
aws-c-http 0.8.10 h6bb76cc_5 conda-forge
aws-c-io 0.14.20 h389d861_2 conda-forge
aws-c-mqtt 0.10.7 had056f2_5 conda-forge
aws-c-s3 0.7.0 hc85afc5_0 conda-forge
aws-c-sdkutils 0.1.19 hf20e7d7_6 conda-forge
aws-checksums 0.1.20 hf20e7d7_3 conda-forge
aws-crt-cpp 0.29.0 h07ed512_0 conda-forge
aws-sdk-cpp 1.11.407 h9c41b47_6 conda-forge
azure-core-cpp 1.14.0 h5cfcd09_0 conda-forge
azure-identity-cpp 1.10.0 h113e628_0 conda-forge
azure-storage-blobs-cpp 12.13.0 h3cf044e_1 conda-forge
azure-storage-common-cpp 12.8.0 h736e048_1 conda-forge
azure-storage-files-datalake-cpp 12.12.0 ha633028_1 conda-forge
babel 2.14.0 pyhd8ed1ab_0 conda-forge
beautifulsoup4 4.12.3 pyha770c72_0 conda-forge
bleach 6.1.0 pyhd8ed1ab_0 conda-forge
bokeh 3.6.0 pyhd8ed1ab_0 conda-forge
brotli-python 1.1.0 py312h2ec8cdc_2 conda-forge
bzip2 1.0.8 h4bc722e_7 conda-forge
c-ares 1.34.2 heb4867d_0 conda-forge
ca-certificates 2024.8.30 hbcca054_0 conda-forge
cached-property 1.5.2 hd8ed1ab_1 conda-forge
cached_property 1.5.2 pyha770c72_1 conda-forge
certifi 2024.8.30 pyhd8ed1ab_0 conda-forge
cffi 1.17.1 py312h06ac9bb_0 conda-forge
charset-normalizer 3.4.0 pyhd8ed1ab_0 conda-forge
click 8.1.7 unix_pyh707e725_0 conda-forge
cloudpickle 3.1.0 pyhd8ed1ab_1 conda-forge
comm 0.2.2 pyhd8ed1ab_0 conda-forge
contourpy 1.3.0 py312h68727a3_2 conda-forge
cytoolz 1.0.0 py312h66e93f0_1 conda-forge
dask 2024.10.0 pyhd8ed1ab_0 conda-forge
dask-core 2024.10.0 pyhd8ed1ab_0 conda-forge
dask-ctl 2024.1.0 pyhd8ed1ab_0 conda-forge
dask-expr 1.1.16 pyhd8ed1ab_0 conda-forge
debugpy 1.8.7 py312h2ec8cdc_0 conda-forge
decorator 5.1.1 pyhd8ed1ab_0 conda-forge
defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge
distributed 2024.10.0 pyhd8ed1ab_0 conda-forge
entrypoints 0.4 pyhd8ed1ab_0 conda-forge
exceptiongroup 1.2.2 pyhd8ed1ab_0 conda-forge
executing 2.1.0 pyhd8ed1ab_0 conda-forge
fqdn 1.5.1 pyhd8ed1ab_0 conda-forge
freetype 2.12.1 h267a509_2 conda-forge
fsspec 2024.10.0 pyhff2d567_0 conda-forge
gflags 2.2.2 h5888daf_1005 conda-forge
glog 0.7.1 hbabe93e_0 conda-forge
h11 0.14.0 pyhd8ed1ab_0 conda-forge
h2 4.1.0 pyhd8ed1ab_0 conda-forge
hpack 4.0.0 pyh9f0ad1d_0 conda-forge
httpcore 1.0.6 pyhd8ed1ab_0 conda-forge
httpx 0.27.2 pyhd8ed1ab_0 conda-forge
hyperframe 6.0.1 pyhd8ed1ab_0 conda-forge
icu 75.1 he02047a_0 conda-forge
idna 3.10 pyhd8ed1ab_0 conda-forge
importlib-metadata 8.5.0 pyha770c72_0 conda-forge
importlib_metadata 8.5.0 hd8ed1ab_0 conda-forge
importlib_resources 6.4.5 pyhd8ed1ab_0 conda-forge
ipykernel 6.29.5 pyh3099207_0 conda-forge
ipython 8.29.0 pyh707e725_0 conda-forge
isoduration 20.11.0 pyhd8ed1ab_0 conda-forge
jedi 0.19.1 pyhd8ed1ab_0 conda-forge
jinja2 3.1.4 pyhd8ed1ab_0 conda-forge
json5 0.9.25 pyhd8ed1ab_0 conda-forge
jsonpointer 3.0.0 py312h7900ff3_1 conda-forge
jsonschema 4.23.0 pyhd8ed1ab_0 conda-forge
jsonschema-specifications 2024.10.1 pyhd8ed1ab_0 conda-forge
jsonschema-with-format-nongpl 4.23.0 hd8ed1ab_0 conda-forge
jupyter-lsp 2.2.5 pyhd8ed1ab_0 conda-forge
jupyter_client 8.6.3 pyhd8ed1ab_0 conda-forge
jupyter_core 5.7.2 pyh31011fe_1 conda-forge
jupyter_events 0.10.0 pyhd8ed1ab_0 conda-forge
jupyter_server 2.14.2 pyhd8ed1ab_0 conda-forge
jupyter_server_terminals 0.5.3 pyhd8ed1ab_0 conda-forge
jupyterlab 4.2.5 pyhd8ed1ab_0 conda-forge
jupyterlab_pygments 0.3.0 pyhd8ed1ab_1 conda-forge
jupyterlab_server 2.27.3 pyhd8ed1ab_0 conda-forge
keyutils 1.6.1 h166bdaf_0 conda-forge
krb5 1.21.3 h659f571_0 conda-forge
lcms2 2.16 hb7c19ff_0 conda-forge
ld_impl_linux-64 2.43 h712a8e2_2 conda-forge
lerc 4.0.0 h27087fc_0 conda-forge
libabseil 20240722.0 cxx17_h5888daf_1 conda-forge
libarrow 18.0.0 h9c5d0aa_0_cuda conda-forge
libarrow-acero 18.0.0 h530483c_0_cuda conda-forge
libarrow-dataset 18.0.0 h530483c_0_cuda conda-forge
libarrow-substrait 18.0.0 h8ffff87_0_cuda conda-forge
libblas 3.9.0 25_linux64_openblas conda-forge
libbrotlicommon 1.1.0 hb9d3cd8_2 conda-forge
libbrotlidec 1.1.0 hb9d3cd8_2 conda-forge
libbrotlienc 1.1.0 hb9d3cd8_2 conda-forge
libcblas 3.9.0 25_linux64_openblas conda-forge
libcrc32c 1.1.2 h9c3ff4c_0 conda-forge
libcurl 8.10.1 hbbe4b11_0 conda-forge
libdeflate 1.22 hb9d3cd8_0 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 hd590300_2 conda-forge
libevent 2.1.12 hf998b51_1 conda-forge
libexpat 2.6.3 h5888daf_0 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc 14.2.0 h77fa898_1 conda-forge
libgcc-ng 14.2.0 h69a702a_1 conda-forge
libgfortran 14.2.0 h69a702a_1 conda-forge
libgfortran-ng 14.2.0 h69a702a_1 conda-forge
libgfortran5 14.2.0 hd5240d6_1 conda-forge
libgomp 14.2.0 h77fa898_1 conda-forge
libgoogle-cloud 2.30.0 h438788a_0 conda-forge
libgoogle-cloud-storage 2.30.0 h0121fbd_0 conda-forge
libgrpc 1.65.5 hf5c653b_0 conda-forge
libiconv 1.17 hd590300_2 conda-forge
libjpeg-turbo 3.0.0 hd590300_1 conda-forge
liblapack 3.9.0 25_linux64_openblas conda-forge
libnghttp2 1.64.0 h161d5f1_0 conda-forge
libnsl 2.0.1 hd590300_0 conda-forge
libopenblas 0.3.28 pthreads_h94d23a6_0 conda-forge
libparquet 18.0.0 hdbc8f64_0_cuda conda-forge
libpng 1.6.44 hadc24fc_0 conda-forge
libprotobuf 5.27.5 h5b01275_2 conda-forge
libre2-11 2024.07.02 hbbce691_1 conda-forge
libsodium 1.0.20 h4ab18f5_0 conda-forge
libsqlite 3.47.0 hadc24fc_1 conda-forge
libssh2 1.11.0 h0841786_0 conda-forge
libstdcxx 14.2.0 hc0a3c3a_1 conda-forge
libstdcxx-ng 14.2.0 h4852527_1 conda-forge
libthrift 0.21.0 h0e7cc3e_0 conda-forge
libtiff 4.7.0 he137b08_1 conda-forge
libutf8proc 2.8.0 h166bdaf_0 conda-forge
libuuid 2.38.1 h0b41bf4_0 conda-forge
libwebp-base 1.4.0 hd590300_0 conda-forge
libxcb 1.17.0 h8a09558_0 conda-forge
libxcrypt 4.4.36 hd590300_1 conda-forge
libxml2 2.12.7 he7c6b58_4 conda-forge
libzlib 1.3.1 hb9d3cd8_2 conda-forge
locket 1.0.0 pyhd8ed1ab_0 conda-forge
lz4 4.3.3 py312hb3f7f12_1 conda-forge
lz4-c 1.9.4 hcb278e6_0 conda-forge
markdown-it-py 3.0.0 pyhd8ed1ab_0 conda-forge
markupsafe 3.0.2 py312h178313f_0 conda-forge
matplotlib-inline 0.1.7 pyhd8ed1ab_0 conda-forge
mdurl 0.1.2 pyhd8ed1ab_0 conda-forge
mistune 3.0.2 pyhd8ed1ab_0 conda-forge
msgpack-python 1.1.0 py312h68727a3_0 conda-forge
nbclient 0.10.0 pyhd8ed1ab_0 conda-forge
nbconvert-core 7.16.4 pyhd8ed1ab_1 conda-forge
nbformat 5.10.4 pyhd8ed1ab_0 conda-forge
ncurses 6.5 he02047a_1 conda-forge
nest-asyncio 1.6.0 pyhd8ed1ab_0 conda-forge
notebook-shim 0.2.4 pyhd8ed1ab_0 conda-forge
numpy 2.1.2 py312h58c1407_0 conda-forge
openjpeg 2.5.2 h488ebb8_0 conda-forge
openssl 3.3.2 hb9d3cd8_0 conda-forge
orc 2.0.2 h690cf93_1 conda-forge
overrides 7.7.0 pyhd8ed1ab_0 conda-forge
packaging 24.1 pyhd8ed1ab_0 conda-forge
pandas 2.2.3 py312hf9745cd_1 conda-forge
pandocfilters 1.5.0 pyhd8ed1ab_0 conda-forge
parso 0.8.4 pyhd8ed1ab_0 conda-forge
partd 1.4.2 pyhd8ed1ab_0 conda-forge
pexpect 4.9.0 pyhd8ed1ab_0 conda-forge
pickleshare 0.7.5 py_1003 conda-forge
pillow 11.0.0 py312h7b63e92_0 conda-forge
pip 24.3.1 pyh8b19718_0 conda-forge
pkgutil-resolve-name 1.3.10 pyhd8ed1ab_1 conda-forge
platformdirs 4.3.6 pyhd8ed1ab_0 conda-forge
prometheus_client 0.21.0 pyhd8ed1ab_0 conda-forge
prompt-toolkit 3.0.48 pyha770c72_0 conda-forge
psutil 6.1.0 py312h66e93f0_0 conda-forge
pthread-stubs 0.4 hb9d3cd8_1002 conda-forge
ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge
pure_eval 0.2.3 pyhd8ed1ab_0 conda-forge
pyarrow 18.0.0 py312h9cebb41_0 conda-forge
pyarrow-core 18.0.0 py312h09cf70e_0_cuda conda-forge
pycparser 2.22 pyhd8ed1ab_0 conda-forge
pygments 2.18.0 pyhd8ed1ab_0 conda-forge
pysocks 1.7.1 pyha2e5f31_6 conda-forge
python 3.12.7 hc5c86c4_0_cpython conda-forge
python-dateutil 2.9.0 pyhd8ed1ab_0 conda-forge
python-fastjsonschema 2.20.0 pyhd8ed1ab_0 conda-forge
python-json-logger 2.0.7 pyhd8ed1ab_0 conda-forge
python-tzdata 2024.2 pyhd8ed1ab_0 conda-forge
python_abi 3.12 5_cp312 conda-forge
pytz 2024.1 pyhd8ed1ab_0 conda-forge
pyyaml 6.0.2 py312h66e93f0_1 conda-forge
pyzmq 26.2.0 py312hbf22597_3 conda-forge
re2 2024.07.02 h77b4e00_1 conda-forge
readline 8.2 h8228510_1 conda-forge
referencing 0.35.1 pyhd8ed1ab_0 conda-forge
requests 2.32.3 pyhd8ed1ab_0 conda-forge
rfc3339-validator 0.1.4 pyhd8ed1ab_0 conda-forge
rfc3986-validator 0.1.1 pyh9f0ad1d_0 conda-forge
rich 13.9.3 pyhd8ed1ab_0 conda-forge
rpds-py 0.20.0 py312h12e396e_1 conda-forge
s2n 1.5.5 h3931f03_0 conda-forge
send2trash 1.8.3 pyh0d859eb_0 conda-forge
setuptools 75.1.0 pyhd8ed1ab_0 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
snappy 1.2.1 ha2e4443_0 conda-forge
sniffio 1.3.1 pyhd8ed1ab_0 conda-forge
sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge
soupsieve 2.5 pyhd8ed1ab_1 conda-forge
stack_data 0.6.2 pyhd8ed1ab_0 conda-forge
tblib 3.0.0 pyhd8ed1ab_0 conda-forge
terminado 0.18.1 pyh0d859eb_0 conda-forge
tinycss2 1.4.0 pyhd8ed1ab_0 conda-forge
tk 8.6.13 noxft_h4845f30_101 conda-forge
tomli 2.0.2 pyhd8ed1ab_0 conda-forge
toolz 1.0.0 pyhd8ed1ab_0 conda-forge
tornado 6.4.1 py312h66e93f0_1 conda-forge
traitlets 5.14.3 pyhd8ed1ab_0 conda-forge
types-python-dateutil 2.9.0.20241003 pyhff2d567_0 conda-forge
typing-extensions 4.12.2 hd8ed1ab_0 conda-forge
typing_extensions 4.12.2 pyha770c72_0 conda-forge
typing_utils 0.1.0 pyhd8ed1ab_0 conda-forge
tzdata 2024b hc8b5060_0 conda-forge
uri-template 1.3.0 pyhd8ed1ab_0 conda-forge
urllib3 2.2.3 pyhd8ed1ab_0 conda-forge
wcwidth 0.2.13 pyhd8ed1ab_0 conda-forge
webcolors 24.8.0 pyhd8ed1ab_0 conda-forge
webencodings 0.5.1 pyhd8ed1ab_2 conda-forge
websocket-client 1.8.0 pyhd8ed1ab_0 conda-forge
wheel 0.44.0 pyhd8ed1ab_0 conda-forge
xorg-libxau 1.0.11 hb9d3cd8_1 conda-forge
xorg-libxdmcp 1.1.5 hb9d3cd8_0 conda-forge
xyzservices 2024.9.0 pyhd8ed1ab_0 conda-forge
xz 5.2.6 h166bdaf_0 conda-forge
yaml 0.2.5 h7f98852_2 conda-forge
zeromq 4.3.5 h3b0a872_6 conda-forge
zict 3.0.0 pyhd8ed1ab_0 conda-forge
zipp 3.20.2 pyhd8ed1ab_0 conda-forge
zstandard 0.23.0 py312hef9b889_1 conda-forge
zstd 1.5.6 ha6fb4c9_0 conda-forge
As far as I can tell, the reason for this is that ipykernel runs a event loop in the background (to support top-level await calls), which causes the loop started tornado.ioloop.IOLoop.run_sync to error.
Getting this to work is an absolute rabbit hole, though: nesting loops is – intentionally, it seems – not supported by asyncio which is what tornado.ioloop wraps starting with tornado>=6, so this will likely need a separate thread for the new event loop. dask / distributed might have a mechanism to execute async code synchronously already, even in jupyterlab, which could be reused here?