Dedicated worker for job type

[jtwaleson]

I have some jobs that are very CPU intensive and I would only like to schedule one of them at the same time. I have others that are more network intensive and can run many of them concurrently.

A solution would be to add a parameter to each worker type to select which kind of jobs it will take.

You could have:

• worker_type_1: filter "CPUJob", workers: 1
• worker_type_2: filter "!CPUJob", workers: 5
• worker_type_3: filter "*", workers: 2

If there's some interest in this I could attempt a PR. We could also argue that this is too specialized and for use cases like this, an external scheduler should be used.

[kaplanelad]

@jtwaleson We currently support num_workers as a global configuration, which is not tied to specific worker types.

I agree that having num_workers configurable per worker type could be a useful feature.

Would you like to open a PR for this implementation? Alternatively, if you prefer, you can share your design approach, and we can discuss the best way to implement it.

[jtwaleson]

Ok, will start in about two weeks, busy period right now :)

[kaplanelad]

No rush, I'll do it

[kaplanelad]

Issue: ##1327

[fbartho]

Silly question: @kaplanelad -- if I wanted to push some jobs into SQS from loco.rs, and have a non-loco-based worker, is that possible now?

---

Context:

Specifically, I believe we're going to use unstructured.io (using their Open Source/self-hosted option) which uses machine learning models to extract text from specific document types. This is primarily a python stack. I was thinking about organizing it as a worker that processes jobs from a specific SQS queue.

I was thinking about shoving a small rust binary into it to interact with the queue and then call into python for the functionality. But that didn't need to be a full loco application.

Any pointers or recommendations?

[kaplanelad]

Silly question: @kaplanelad -- if I wanted to push some jobs into SQS from loco.rs, and have a non-loco-based worker, is that possible now?

we are not supporting SQS as a queues right now.