SpecCluster: add option to *not* shut down the scheduler when the cluster is closed

[nocnokneo]

Add shutdown_scheduler option to SpecCluster. Defaults to True so this is backward compatible.

• Closes SpecCluster implmentation closes the scheduler unexpectedly #7025

• Closes AWS - ECS/Fargate Cluster - When providing a scheduler_address, calling cluster.close() terminates the remote scheduler via tcp dask-cloudprovider#375

• Tests added / passed

• Passes pre-commit run --all-files

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

    27 files  ± 0      27 suites  ±0   11h 43m 41s ⏱️ + 13m 48s
 4 102 tests + 2   3 989 ✅ + 6    111 💤 ±0  2 ❌  - 4 
51 430 runs  +26  49 131 ✅ +29  2 297 💤 +1  2 ❌  - 4 

For more details on these failures, see this check.

Results for commit fa66395. ± Comparison against base commit 7df4356.

♻️ This comment has been updated with latest results.

[jacobtomlinson]

This looks good to me. I think we will need to go through the various implementations of SpecCluster and set this value in cases where we also set shutdown_on_close=False.

[jacobtomlinson]

Merged in from main to resolve CI issues