Failing tests in `packages/cuml/python/cuml/cuml/tests/dask`

[TomAugspurger]

Lots of failures in https://github.com/rapidsai/dask-upstream-testing/actions/runs/14522270490/job/40745795745

I think https://github.com/rapidsai/dask-upstream-testing/actions/runs/14522270490/job/40745795745#step:10:14281 is the start of the tracebacks.

I'll triage the different failure types later.

[TomAugspurger]

dask/distributed#9050 is probably the cause of most / all of these failures.

[TomAugspurger]

We're seeing different failures now, in e.g. packages/cuml/python/cuml/cuml/tests/dask/test_dask_base.py::test_get_combined_model[True-data_size0-LinearRegression-float32]:

______ test_get_combined_model[True-data_size0-LinearRegression-float32] _______
Traceback (most recent call last):
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/cuml/dask/common/part_utils.py", line 168, in _extract_partitions
    [(first(who_has[key]), part) for key, part in key_to_part]
      ^^^^^^^^^^^^^^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/toolz/itertoolz.py", line 375, in first
    return next(iter(seq))
           ^^^^^^^^^^^^^^^
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 341, in from_call
    result: Optional[TResult] = func()
                                ^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 262, in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 182, in _multicall
    return outcome.get_result()
           ^^^^^^^^^^^^^^^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_result.py", line 100, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 177, in pytest_runtest_call
    raise e
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 169, in pytest_runtest_call
    item.runtest()
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/python.py", line 1792, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/python.py", line 194, in pytest_pyfunc_call
    result = testfunction(**testargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/packages/cuml/python/cuml/cuml/tests/dask/test_dask_base.py", line 61, in test_get_combined_model
    model.fit(X_train, y_train)
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/cuml/dask/linear_model/linear_regression.py", line 84, in fit
    models = self._fit(
             ^^^^^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/cuml/dask/common/base.py", line 397, in _fit
    data = DistributedDataHandler.create(data=data, client=self.client)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/cuml/dask/common/input_utils.py", line 112, in create
    gpu_futures = client.sync(_extract_partitions, data, client)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/utils.py", line 376, in sync
    return sync(
           ^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/utils.py", line 452, in sync
    raise error
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/utils.py", line 426, in f
    result = yield future
             ^^^^^^^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/tornado/gen.py", line 766, in run
    value = future.result()
            ^^^^^^^^^^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/tornado/gen.py", line 785, in run
    yielded = self.gen.send(value)
              ^^^^^^^^^^^^^^^^^^^^
RuntimeError: generator raised StopIteration
----------------------------- Captured stderr call -----------------------------
2025-04-22 18:41:04,847 - distributed.worker - ERROR - Compute Failed
Key:       ('Future-from-value-getitem-25ac592522d689bb03f74b7f971d96a8', 0, 0)
State:     executing
Task:  <Task ('Future-from-value-getitem-25ac592522d689bb03f74b7f971d96a8', 0, 0) _execute_subgraph(...)>
Exception: 'TypeError("\'Future\' object is not subscriptable")'
Traceback: '  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/dask/array/chunk.py", line 421, in getitem\n    result = obj[index]\n             ~~~^^^^^^^\n'

______ test_get_combined_model[True-data_size0-LinearRegression-float64] _______
Traceback (most recent call last):
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/cuml/dask/common/part_utils.py", line 168, in _extract_partitions
    [(first(who_has[key]), part) for key, part in key_to_part]
      ^^^^^^^^^^^^^^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/toolz/itertoolz.py", line 375, in first
    return next(iter(seq))
           ^^^^^^^^^^^^^^^
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 341, in from_call
    result: Optional[TResult] = func()
                                ^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 262, in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 182, in _multicall
    return outcome.get_result()
           ^^^^^^^^^^^^^^^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_result.py", line 100, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 177, in pytest_runtest_call
    raise e
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 169, in pytest_runtest_call
    item.runtest()
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/python.py", line 1792, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/python.py", line 194, in pytest_pyfunc_call
    result = testfunction(**testargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/packages/cuml/python/cuml/cuml/tests/dask/test_dask_base.py", line 61, in test_get_combined_model
    model.fit(X_train, y_train)
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/cuml/dask/linear_model/linear_regression.py", line 84, in fit
    models = self._fit(
             ^^^^^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/cuml/dask/common/base.py", line 397, in _fit
    data = DistributedDataHandler.create(data=data, client=self.client)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/cuml/dask/common/input_utils.py", line 112, in create
    gpu_futures = client.sync(_extract_partitions, data, client)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/utils.py", line 376, in sync
    return sync(
           ^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/utils.py", line 452, in sync
    raise error
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/utils.py", line 426, in f
    result = yield future
             ^^^^^^^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/tornado/gen.py", line 766, in run
    value = future.result()
            ^^^^^^^^^^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/tornado/gen.py", line 785, in run
    yielded = self.gen.send(value)
              ^^^^^^^^^^^^^^^^^^^^
RuntimeError: generator raised StopIteration
----------------------------- Captured stderr call -----------------------------
2025-04-22 18:41:04,978 - distributed.worker - ERROR - Compute Failed
Key:       ('Future-from-value-getitem-c639176a13b6ec9aa3ff97dee3d2fec9', 0, 0)
State:     executing
Task:  <Task ('Future-from-value-getitem-c639176a13b6ec9aa3ff97dee3d2fec9', 0, 0) _execute_subgraph(...)>
Exception: 'TypeError("\'Future\' object is not subscriptable")'
Traceback: '  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/dask/array/chunk.py", line 421, in getitem\n    result = obj[index]\n             ~~~^^^^^^^\n'

from https://github.com/rapidsai/dask-upstream-testing/actions/runs/14601900890/job/40961858256#step:10:1016

[TomAugspurger]

rapidsai/cuml#6576 fixes a handful.

Here are the remaining failures I'm seeing locally after that:

FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_dbscan.py::test_dbscan[int320-20-500-float32-1000.0] - AssertionError: assert <array_equal: [0 0 0 ... 0 0 0] [  0   1   2 ... 497 498 499] unit_tol=0.0001 total_tol=0.0001 with_sign=True>
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_dbscan.py::test_dbscan[int320-20-500-float32-None] - Failed: Timeout >120.0s
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_dbscan.py::test_dbscan[int320-20-500-float64-1000.0] - Failed: Timeout >120.0s
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_dbscan.py::test_dbscan[int321-20-500-float32-1000.0] - Failed: Timeout >120.0s
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_dbscan.py::test_dbscan[int640-20-500-float32-1000.0] - Failed: Timeout >120.0s
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_dbscan.py::test_dbscan[int641-20-500-float32-1000.0] - Failed: Timeout >120.0s
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_dbscan.py::test_dbscan_precomputed[int32-500-float32-1] - Failed: Timeout >120.0s
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_dbscan.py::test_dbscan_precomputed[int32-500-float64-1] - Failed: Timeout >120.0s
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_dbscan.py::test_dbscan_precomputed[int64-500-float32-1] - Failed: Timeout >120.0s
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_dbscan.py::test_dbscan_sklearn_comparison[0.05-500-noisy_moons] - Failed: Timeout >120.0s
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_dbscan.py::test_dbscan_sklearn_comparison[0.05-500-blobs] - distributed.comm.core.CommClosedError: in <TCP (closed) ConnectionPool.scatter local=tcp://127.0.0.1:43614 remote=tcp://127.0.0.1:35779>: Stream is closed
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_dbscan.py::test_dbscan_sklearn_comparison[0.05-500-no_structure] - AssertionError: Core point mismatch at #0: 2147483647 (expected -1)
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_dbscan.py::test_dbscan_sklearn_comparison[0.1-500-noisy_moons] - AssertionError: Border point not connected to cluster at #0: 1 (reference: 0)
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_doctest.py::test_docstring[LabelBinarizer] - AssertionError: 3 of 17 doctests failed for LabelBinarizer:
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_func.py::test_tree_reduce_futures[2] - TypeError: unsupported operand type(s) for +: 'int' and 'Future'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_func.py::test_tree_reduce_futures[10] - TypeError: unsupported operand type(s) for +: 'int' and 'Future'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_func.py::test_tree_reduce_futures[15] - TypeError: unsupported operand type(s) for +: 'int' and 'Future'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_func.py::test_reduce_futures[2] - TypeError: unsupported operand type(s) for +: 'int' and 'Future'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_func.py::test_reduce_futures[10] - StopIteration
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_func.py::test_reduce_futures[15] - StopIteration
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_kmeans.py::test_end_to_end[dataframe-True-None-5-10-1000.0] - assert 5 == 8
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_kmeans.py::test_end_to_end[dataframe-True-None-5-30-1000.0] - assert 5 == 8
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_kmeans.py::test_end_to_end[dataframe-False-None-5-10-1000.0] - assert 5 == 8
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_kmeans.py::test_end_to_end[array-True-None-5-10-1000.0] - assert 5 == 8
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_kneighbors_classifier.py::test_predict_and_score[dataset0-parameters0-dask_cudf] - distributed.client.FutureCancelledError: finalize-hlgfinalizecompute-2e8605037c2f41dcadaaaf9cbf58962e cancelled for reason: lost dependencies.
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_kneighbors_regressor.py::test_predict_and_score[dataset0-parameters0-dask_cudf] - distributed.client.FutureCancelledError: finalize-hlgfinalizecompute-4f3bc5da49a645af99a5b1a6bb710dd2 cancelled for reason: lost dependencies.
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_label_binarizer.py::test_basic_functions[True-labels0] - distributed.client.FutureCancelledError: finalize-hlgfinalizecompute-97c8b394881f42deb9be8347b93917ef cancelled for reason: lost dependencies.
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_label_binarizer.py::test_basic_functions[True-labels1] - Exception: AssertionError(<TaskState ('rechunk-merge-8c05fbba295671d954151b28b0a55cac', 0) processing>)
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_label_binarizer.py::test_basic_functions[False-labels0] - distributed.client.FutureCancelledError: finalize-hlgfinalizecompute-c43f895cb7a143e18b7612081c0b1836 cancelled for reason: lost dependencies.
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_logistic_regression.py::test_sparse_all_zeroes[False-False] - Failed: Timeout >120.0s
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_logistic_regression.py::test_sparse_all_zeroes[False-True] - Failed: Timeout >120.0s
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_logistic_regression.py::test_sparse_all_zeroes[True-False] - Failed: Timeout >120.0s
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_logistic_regression.py::test_sparse_one_gpu_all_zeroes[False] - Failed: Timeout >120.0s
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_logistic_regression.py::test_sparse_one_gpu_all_zeroes[True] - Failed: Timeout >120.0s
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_naive_bayes.py::test_model_multiple_chunks[float32] - RuntimeError: 1 of 1 worker jobs failed: 'Future' object has no attribute 'feature_count_'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_naive_bayes.py::test_model_multiple_chunks[float64] - RuntimeError: 1 of 1 worker jobs failed: 'Future' object has no attribute 'feature_count_'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_naive_bayes.py::test_model_multiple_chunks[int32] - RuntimeError: 1 of 1 worker jobs failed: 'Future' object has no attribute 'feature_count_'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_rf_classification_multi_class[3] - KeyError: 'tcp://127.0.0.1:42593'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_rf_regression_dask_fil[5-float32] - KeyError: 'tcp://127.0.0.1:46463'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_rf_classification_dask_array[5] - KeyError: 'tcp://127.0.0.1:42593'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_rf_regression_dask_cpu[5] - KeyError: 'tcp://127.0.0.1:42593'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_rf_classification_dask_fil_predict_proba[5] - KeyError: 'tcp://127.0.0.1:42593'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_rf_concatenation_dask[classification] - KeyError: 'tcp://127.0.0.1:46463'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_rf_concatenation_dask[regression] - KeyError: 'tcp://127.0.0.1:46463'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_single_input_regression[True] - KeyError: 'tcp://127.0.0.1:42375'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_single_input_regression[False] - KeyError: 'tcp://127.0.0.1:42375'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_rf_get_json[regression-5-1] - KeyError: 'tcp://127.0.0.1:46463'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_rf_get_json[regression-5-2] - KeyError: 'tcp://127.0.0.1:46463'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_rf_get_json[regression-5-3] - KeyError: 'tcp://127.0.0.1:46463'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_rf_get_json[regression-5-5] - KeyError: 'tcp://127.0.0.1:46463'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_rf_get_json[regression-5-10] - KeyError: 'tcp://127.0.0.1:46463'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_rf_get_json[regression-5-15] - KeyError: 'tcp://127.0.0.1:46463'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_rf_get_json[regression-5-20] - KeyError: 'tcp://127.0.0.1:46463'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_rf_get_json[regression-10-1] - KeyError: 'tcp://127.0.0.1:46463'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_rf_get_json[regression-20-1] - KeyError: 'tcp://127.0.0.1:46463'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_rf_get_json[classification-5-1] - KeyError: 'tcp://127.0.0.1:46463'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_rf_instance_count[5-1] - KeyError: 'tcp://127.0.0.1:46463'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_rf_instance_count[5-2] - KeyError: 'tcp://127.0.0.1:46463'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_rf_instance_count[5-3] - KeyError: 'tcp://127.0.0.1:46463'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_rf_instance_count[5-5] - KeyError: 'tcp://127.0.0.1:46463'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_rf_instance_count[5-10] - KeyError: 'tcp://127.0.0.1:46463'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_rf_instance_count[5-15] - KeyError: 'tcp://127.0.0.1:46463'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_rf_instance_count[5-20] - KeyError: 'tcp://127.0.0.1:46463'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_rf_instance_count[10-1] - KeyError: 'tcp://127.0.0.1:46463'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_rf_instance_count[20-1] - KeyError: 'tcp://127.0.0.1:46463'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_rf_get_combined_model_right_aftter_fit[regression] - KeyError: 'tcp://127.0.0.1:46463'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_rf_get_combined_model_right_aftter_fit[classification] - KeyError: 'tcp://127.0.0.1:46463'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_rf_get_text[True-5] - KeyError: 'tcp://127.0.0.1:46463'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_rf_get_text[True-10] - KeyError: 'tcp://127.0.0.1:46463'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_rf_get_text[True-20] - KeyError: 'tcp://127.0.0.1:46463'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_rf_get_text[False-5] - KeyError: 'tcp://127.0.0.1:46463'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_rf_broadcast[True-True-classification] - KeyError: 'tcp://127.0.0.1:46463'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_rf_broadcast[True-True-regression] - KeyError: 'tcp://127.0.0.1:46463'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_rf_broadcast[True-False-classification] - KeyError: 'tcp://127.0.0.1:46463'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_random_forest.py::test_rf_broadcast[False-True-classification] - KeyError: 'tcp://127.0.0.1:46463'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_tfidf.py::test_tfidf_transformer[True-True-True-l1-base_case] - RuntimeError: 1 of 1 worker jobs failed: 'Future' object has no attribute 'use_idf'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_tfidf.py::test_tfidf_transformer[True-True-True-l1-diag] - RuntimeError: 1 of 1 worker jobs failed: 'Future' object has no attribute 'use_idf'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_tfidf.py::test_tfidf_transformer[True-True-True-l1-empty_feature] - RuntimeError: 1 of 1 worker jobs failed: 'Future' object has no attribute 'use_idf'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_tfidf.py::test_tfidf_transformer[True-True-True-l1-123] - RuntimeError: 1 of 1 worker jobs failed: 'Future' object has no attribute 'use_idf'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_tfidf.py::test_tfidf_transformer[True-True-True-l1-empty_doc] - RuntimeError: 1 of 1 worker jobs failed: 'Future' object has no attribute 'use_idf'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_tfidf.py::test_tfidf_transformer[True-True-True-l2-base_case] - RuntimeError: 1 of 1 worker jobs failed: 'Future' object has no attribute 'use_idf'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_tfidf.py::test_tfidf_transformer[True-True-True-None-base_case] - RuntimeError: 1 of 1 worker jobs failed: 'Future' object has no attribute 'use_idf'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_tfidf.py::test_tfidf_transformer[True-False-True-l1-base_case] - RuntimeError: 1 of 1 worker jobs failed: 'Future' object has no attribute 'use_idf'
FAILED packages/cuml/python/cuml/cuml/tests/dask/test_dask_tfidf.py::test_tfidf_transformer[False-True-True-l1-base_case] - RuntimeError: 1 of 1 worker jobs failed: 'Future' object has no attribute 'use_idf'
========================================================= 84 failed, 284 passed, 1 skipped, 4691 deselected, 9 xfailed, 1 xpassed, 61 warnings in 2932.59s (0:48:52) ==========================================================

Many have a similar root cause: unnecessarily wrapping a Future in a Delayed.