Dask only uses ~40% percentage of CPU on Windows

[carlosg-m]

Describe the issue: After multiple attempts, including setting the LocalCluster in "processes" mode to multiple combinations of n_workers and threads_per_worker, using Dask Dataframes with MapPartitions, Dask Bags and Dask Arrays with simple snippets of code. The CPU does not go over 40%, only 16 cores are working but capped below 100%.

Minimal Complete Verifiable Example:

import dask.array as da
import numpy as np
from dask.distributed import Client, LocalCluster

# Create a local Dask cluster
cluster = LocalCluster(n_workers=24, threads_per_worker=1)
client = Client(cluster)

dask_array = dask.array.random.random((1000000, 100000))

# Perform a computation on the Dask array
result = dask_array.sum()

# Compute and get the result
print(result.compute())

# Close the Dask client and cluster
client.close()
cluster.close()

Anything else we need to know?:

Environment:

• Dask version: 2025.4.1
• Python version: 3.13.3
• Operating System: Windows 11 24H2
• Install method (conda, pip, source): pip
• CPU: 13th Gen Intel(R) Core(TM) i9-13950HX, 2200 Mhz, 24 Cores, 32 Threads
• Workstation: Laptop

[carlosg-m]

I also made sure to have more tasks/partitions than workers. Still the CPU seems locked.

[jacobtomlinson]

This usually happens when each task is too small. The workers aren't fully saturated because they are all waiting on the scheduler to give them more tasks. When your tasks are small the scheduler becomes the bottleneck.

In your example above you have a small array and no specified chunking. Then you are doing a simple sum operation. I'm not surprised the workers aren't saturated. Try making your array bigger and specifyuing a chunks kwarg and make each chunk big enough that the sum actually takes some amount of computation time.

[carlosg-m]

@jacobtomlinson, thank you for the quick reply!

I tried different setups but the result is always the same.

For example this function takes approximately 14s to execute.

Now using this setup I'm going to try to use Dask to achieve full parallelism:

import numpy as np
import dask.array as da
from dask.distributed import Client, LocalCluster

def test():

    for _ in range(20):
        a = np.random.random((10000000))
        b = np.random.random((10000000))
        
    return np.correlate(a, b)

# Create a local Dask cluster
cluster = LocalCluster(n_workers=32, threads_per_worker=1, processes=True)
client = Client(cluster)

example = [dask.delayed(test)() for i in range(64)]
dask.compute(*example)

I get the correct result, but the point is that only a small percentage of my CPU resources are used:

[jacobtomlinson]

Looking at that screenshot it seems to be reporting 100% utilisation in the top right.

But only 53% utilisation in the bottom

Looking at the spec for your laptop CPU it has 8 performance cores and 16 efficiency cores. Intel performance cores support hyperthreading (2 threads per core) so 32 logical threads in total.

Task Manager is showing half of your cores pegged at 100%, and the other half not really doing anything. It's not really clear which half is performance and which is efficiency, but my guess is that Windows is restricting Python/Dask to one type of core only.

Are you using your laptop on battery, or set to some kind of eco power mode? I could imagine this would cause Windows to only use the efficiency cores.

[carlosg-m]

This actually makes a lot of sense. I've used Dask successfully in the past without any issue on Colab/Paperspace VMs (however professionally I use Spark/PySpark).
The laptop was plugged in and on high performance mode, during all tests.
Is there any guidelines for users using laptops and Windows?
Other people must have had this issue before, is there a way to bypass these Windows restrictions?

[jacobtomlinson]

I've not seen this question come up before, as you can see from PyPI Stats Windows only accounts for around 2% of Dask installs, so we get way fewer bug reports about Windows.

This Microsoft article seems to suggest running your Python process as administrator can bypass some restrictions around performance. Specifically this comment:

Thanks, this indeed works around the problem entirely. I am running many Python processes in parallel but without administrator rights they only fill up 4 logical cores instead of all 16 as with administrator rights, which makes a huge difference of course. It's actually not just on laptops, this is happening on an i5-12600k for me.

I'm not a Windows expert by any stretch, so other folks may be better placed to advise you here.

[carlosg-m]

Thank you @jacobtomlinson!
Elevating Jupyter Lab to admin privileges worked flawlessly.
Dask is using 100% of all resources, and now it takes just a fraction of the time.