Run dask with a matching interpreter

[cjwatson]

When Debian is in the process of migrating from one Python version to another, there's a transitional period when we run tests on each of two Python versions, although /usr/bin/python3 can only point to one at a time. distributed runs dask from the executable search path, and when it's installed as a system package rather than in a virtual environment, it's possible during this transitional period for distributed to be running under (e.g.) Python 3.13 while /usr/bin/dask has a #! line that causes it to be run under (e.g.) Python 3.12. This results in the following obscure failure mode in some tests in distributed/deploy/tests/test_subprocess.py:

______________________________ test_basic ______________________________

    @pytest.mark.skipif(WINDOWS, reason="distributed#7434")
    @gen_test()
    async def test_basic():
        async with SubprocessCluster(
            asynchronous=True,
            dashboard_address=":0",
            scheduler_kwargs={"idle_timeout": "5s"},
            worker_kwargs={"death_timeout": "5s"},
        ) as cluster:
            async with Client(cluster, asynchronous=True) as client:
>               result = await client.submit(lambda x: x + 1, 10)

distributed/deploy/tests/test_subprocess.py:27:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <Future: cancelled, key: lambda-e426db6b5bc630ab037e145b5e8f631c>, raiseit = True

    async def _result(self, raiseit=True):
        await self._state.wait()
        if self.status == "error":
            exc = clean_exception(self._state.exception, self._state.traceback)
            if raiseit:
                typ, exc, tb = exc
>               raise exc.with_traceback(tb)
E               SystemError: no locals found when setting up annotations

distributed/client.py:410: SystemError

Explicitly running dask using sys.executable fixes this. It's only strictly necessary to change the invocations of dask in distributed.deploy.subprocess, but it seemed better to apply the same change to the whole codebase for consistency.

• Tests added / passed
• Passes pre-commit run --all-files

[cjwatson]

OK, distributed/shuffle/_rechunk.py:756: error: Unused "type: ignore" comment [unused-ignore] fails locally for me too but I was hoping that that was a local false positive, since I didn't touch that file. I've filed another PR for that (#8976).

[cjwatson]

I'm not familiar with conda, so I can't help with the ubuntu-latest-mindeps-memray-extra_packages failure (which also seems unrelated to my changes). I hope somebody more closely involved with this project can figure that one out. Since it passed in #8976, it may just need to be retried, but I don't think I can do that myself.

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

    27 files  ±0      27 suites  ±0   11h 33m 19s ⏱️ + 8m 2s
 4 127 tests ±0   3 996 ✅  - 1    112 💤 ±0   19 ❌ +1 
51 714 runs  ±0  49 267 ✅  - 2  2 300 💤 +1  147 ❌ +1 

For more details on these failures, see this check.

Results for commit 6d6411f. ± Comparison against base commit 1b92625.

♻️ This comment has been updated with latest results.

[jacobtomlinson]

This seems like a very niche scenario. However this seems like a reasonable change, explicit is better than implicit.

[jacobtomlinson]

I've just pulled from main which should make the linter happy.

[jacobtomlinson]

Other failures look unrelated. Merging.