Open
Description
Some functions may require more resources than others so the ability to set the function cost is useful to avoid overloading workers.
Imagine function we have 1 worker with 4 threads and F(x)
, G(x)
while 4 instances of G(x)
could be computed easily on the worker, 2 or more instances of F(x)
will overload the worker CPU thus a possible API could be.
f.submit(x=123, compute_cost=4)
This will allow the function to fill up the 4 threads that the worker has without submitting more tasks to it before completing this function.
Of course, the API requires considerable design choices but this issue serves as a discussion starter.