Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature request: Allow separately specifying --nthreads and resource limits #63

Open
kazimuth opened this issue May 28, 2020 · 1 comment
Labels
chart/dask Related to the dask chart

Comments

@kazimuth
Copy link
Contributor

I'm using Dask to coordinate some ML jobs, which have internal parallelism. I'd like to restrict my cluster so that only one job can run on a node at a time, even though nodes have multiple CPUs. However, this is currently impossible, since the Kubernetes worker resource limit is used to customize --nthreads:
https://github.com/dask/helm-chart/blob/master/dask/templates/dask-worker-deployment.yaml#L37-L38

This also prevents giving nodes e.g. 0.5 or 1.5 cpus, because even though Kubernetes allows it Dask will get confused and crash (i.e. --nthreads fails to parse as an integer.)

This is moderately esoteric so if I should just fork the chart and use that I don't mind.

@martindurant
Copy link
Member

I don't think this is unreasonable. Could you do it with the existing behaviour as the fallback if you don't explicitly supply both limits? We would welcome a PR like that.

@consideRatio consideRatio added the chart/dask Related to the dask chart label Nov 6, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
chart/dask Related to the dask chart
Projects
None yet
Development

No branches or pull requests

3 participants