Prevent threaded Dask workers in DistRDF backend

[JAGANNATHANJP]

This PR prevents unsupported threaded Dask workers in DistRDF.

Distributed RDataFrame with Dask threads may lead to crashes and does not provide advantages due to Python GIL limitations. This change validates worker configuration at backend initialization and raises a RuntimeError when threaded workers are detected.

Suggested configuration:

• processes=True
• threads_per_worker=1

[github-actions]

Test Results

    22 files      22 suites   3d 8h 32m 55s ⏱️
 3 851 tests  2 774 ✅   0 💤 1 077 ❌
69 074 runs  67 560 ✅ 436 💤 1 078 ❌

For more details on these failures, see this check.

Results for commit 9a58aa7.

♻️ This comment has been updated with latest results.

[vepadulano]

Thank you @JAGANNATHANJP ! Before proceeding with the CI, I would like to request you to add a test for this new behaviour, perhaps in roottest/python/distrdf/backends/check_backend.py

[JAGANNATHANJP]

Added a test covering the case where scheduler_info() does not provide worker information during Dask backend initialization.

[JAGANNATHANJP]

Resolved the merge conflict with the latest master and updated the branch.

[vepadulano]

Hi @JAGANNATHANJP ,

Apparently there are still some conflicts, see e.g. https://github.com/root-project/root/actions/runs/28595013336/job/85566036258?pr=22393

Auto-merging roottest/python/distrdf/backends/check_backend.py
  CONFLICT (content): Merge conflict in roottest/python/distrdf/backends/check_backend.py
  error: could not apply def6df24ab... Add tests for Dask worker validation

[vepadulano]

Dear @JAGANNATHANJP ,

The test suite is failing on all platforms with the following error

 >       with pytest.warns(
              UserWarning, match="Number of partitions 2 is greater than number of entries 1 in the dataframe"
          ):
  E       Failed: DID NOT WARN. No warnings of type (<class 'UserWarning'>,) were emitted.
  E        Emitted warnings: [].
  
  ../../../../../src/roottest/python/distrdf/backends/check_backend.py:161: Failed
  =========================== short test summary info ============================
  FAILED ../../../../../src/roottest/python/distrdf/backends/test_all.py::TestInitialization::test_initialization_method[dask] - Failed: DID NOT WARN. No warnings of type (<class 'UserWarning'>,) were emitted.
   Emitted warnings: [].
  !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
  ========================= 1 failed, 5 passed in 4.32s ==========================

Did you run the test suite locally before resubmitting your changes?

[JAGANNATHANJP]

Hi @vepadulano,

Apologies for that! The failure was caused by an accidental indentation error during the last merge conflict resolution (df.GetValue() got shifted outside the pytest.warns block).

I have fixed the indentation in commit 9a58aa7 and verified the full test suite locally inside my build environment. All 14 tests in check_backend.py (including the new Dask worker validation test and test_initialization_method) passed 100% cleanly locally.

Could you please approve running the CI checks when you have a moment? Thank you!

[JAGANNATHANJP]

Hi @vepadulano,

I investigated the recent CI failures. The failure on fedora44 (pyunittests-roottest-python-distrdf-backends-all) was caused by the Dask LocalCluster attempting to spin up a web dashboard on port 8787 during the test, which crashed on the CI runner due to port conflicts/restrictions.

I have pushed a commit to explicitly pass dashboard_address=None in check_backend.py to ensure it runs completely silently.

(Note: The other failures on Alma8/Alma9/Mac appear to be unrelated upstream C++ IO and CMake flakes).

Could you please review when the CI finishes? Thanks!