Description
PR #5773 adds the ability for the Client
to run preload code, similar to the Worker
and Scheduler
. However, @jrbourbeau pointed out that users may be passing preload
and preload_argv
to the Client.__init__
with the intention of passing them to the worker configuration of LocalCluster
.
As a temporary measure, #5773 only adds Client
preload code that is specified in the Dask YAML configuration file. Since a similar problem with keyword arguments passthrough will happen if/when plugins are added for the Client, @jrbourbeau proposed deferring adding the preload
and preload_argv
keywords. When those are added, there will need to be a deprecation cycle to change how users pass kwargs to the worker configuration through LocalCluster
. This deprecation cycle would also remove the similar problem that will occur when/if plugins are added.
One possible solution to this is to modify the signature of LocalCluster.__init__
so that the **worker_kwargs
that are collected at the end of the constructor become a keyword argument of their own, similar to the existing scheduler_kwargs
. See
distributed/distributed/deploy/local.py
Lines 121 to 123 in 8d0df89
This should not affect the n_workers
and threads_per_worker
kwargs, which would still be passed through unmodified.