Multiple worker-pools (sketch)

[timbu]

A sketch of how multiple worker-pools could be added.

• Allows an extra kwarg worker_group on @http decorators.

    @http('GET', '/run/<job_id>, worker_group='background_tasks')
    def run_job(self, job_id):
        ...

• Combined with new config:

MAX_WORKERS_PER_GROUP:
    background_tasks: 5
    another_group: 10

• However, it can't work properly for all QueueConsumer entrypoints until we solve shared-queues/prefetch-counts problems.

Motivation
By allowing separate worker pools we could introduce simple management http entrypoints that could perform health-checks / introspections even when the rest of the service was at max workers.

Questions

• Should we start rejecting requests if we are at max workers (server too busy?) Clients could then re-try on a different server.
• Is it not worth implementing multiple worker pools until we have changed the way we use QueueConsumers?