Deprecate asynchronous `SchedulerPlugin.{add|remove}_worker` hooks

[hendrikmakait]

Allowing synchronous and asynchronous SchedulerPlugin.{add|remove}_worker hooks makes it harder to reason about their execution order (related: #8486). I'm unaware of asynchronous hooks used in practice, so I propose deprecating these. Should there be any legitimate use cases, we can introduce an asynchronous after_{add|remove}_worker hook that would allow strict ordering.

[fjetter]

The reason why the ordering is hard to reason about is not because they are defined async but because we're not awaiting them individually here

distributed/distributed/scheduler.py

Lines 4390 to 4399 in 4425516

	awaitables = []
	for plugin in list(self.plugins.values()):
	try:
	result = plugin.add_worker(scheduler=self, worker=address)
	if result is not None and inspect.isawaitable(result):
	awaitables.append(result)
	except Exception as e:
	logger.exception(e)
	plugin_msgs = await asyncio.gather(*awaitables, return_exceptions=True)

and

distributed/distributed/scheduler.py

Lines 5192 to 5213 in 4425516

	awaitables = []
	for plugin in list(self.plugins.values()):
	try:
	try:
	result = plugin.remove_worker(
	scheduler=self, worker=address, stimulus_id=stimulus_id
	)
	except TypeError:
	parameters = inspect.signature(plugin.remove_worker).parameters
	if "stimulus_id" not in parameters and not any(
	p.kind is p.VAR_KEYWORD for p in parameters.values()
	):
	# Deprecated (see add_plugin)
	result = plugin.remove_worker(scheduler=self, worker=address) # type: ignore
	else:
	raise
	if inspect.isawaitable(result):
	awaitables.append(result)
	except Exception as e:
	logger.exception(e)
	plugin_msgs = await asyncio.gather(*awaitables, return_exceptions=True)

I don't see a reason to remove async add/remove_worker

[hendrikmakait]

The reason why the ordering is hard to reason about is not because they are defined async but because we're not awaiting them individually here

You're right that we could execute them in order. The one downside of that is that previously synchronous remove_worker calls would effectively become asynchronous. For instance, P2P's remove_worker hook relied (relies?) on being executed synchronously with the preceding code in Scheduler.remove_worker (#7798)

[hendrikmakait]

FWIW, we currently make this guarantee about the execution of synchronous remove_worker hooks:

distributed/distributed/diagnostics/plugin.py

Lines 190 to 193 in 774874e

	If this method is synchronous, it is immediately and synchronously executed
	without ``Scheduler.remove_worker`` ever yielding to the event loop.
	If it is asynchronous, it will be awaited after all synchronous
	``SchedulerPlugin.remove_worker`` hooks have executed.