Skip to content

Pass Async::Semaphore to replace Async::Idler, in order to control the exact concurrency limit. #4

@shaokun

Description

@shaokun

Async::Job::Processor::Redis::Server#initialize(...parent: nil), by default, the parent is nil, hence it's set to Async::Idler.new.
But I want to be able to control the exact number of concurrency, so I am trying to pass Async::Semaphore to parent.

I do it in this way:

# config/initializers/async_job.rb

Rails.application.configure do
  # Create a queue for the "default" backend:
  config.async_job.define_queue "default" do
    dequeue Async::Job::Processor::Redis, parent: Async::Semaphore.new(10)
  end
end

It's passed to the parent parameter, but the Server still dequeue as many jobs as possible without taking the concurrency limit into account.
From code:

def start!
  return false if @task
  
  @task = true
  
  @parent.async(transient: true, annotation: self.class.name) do |task|
    @task = task
    
    while true
      self.dequeue(task)
    end
  ensure
    @task = nil
  end
end

It looks like that the infinite loop is within the @parent.async, hence the concurrency limit doesn't work at all.
Should it be?

while true
  @parent.async(transient: true, annotation: self.class.name) do |task| do
    self.dequeue
  end
end

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions