Ideas to improve the processing of background jobs

[davidor]

These are some ideas that we have discussed in the past to improve the processing of background jobs:

• Evaluate Sidekiq as a replacement for Resque.
• Implement a way to process at least a small % of non-priority jobs even when there are priority jobs pending.
• Break up big report jobs into smaller ones.
• Improve the way we reschedule failed jobs to avoid growing queues under stress load. The problem with this is that when an incident happens, the queues start growing, and the priority one in particular grows very fast. Shoving more jobs on top of a growing queue is not a good idea, as the system clearly can't cope with the workload

[andrewdavidmackenzie]

I'd be interested to see a split of time taken:

• Listeners: required processing / time to create and enqueue jobs
• Workers: time to dequeue jobs / required processing

to see just how far we are from the crossover point where:

• Workers required processing time < Listeners enqueue time

and we could scrap the whole background processing split, queues, sidekiq/rescue, priorities and all that goes with it...