Skip to content

Issue: Parallel executor performance is mostly linear #326

Description

@byroot

We use capistrano to deploy on several hundred servers at once, and we noticed capistrano's performance was heavily tied to the number of servers.

To prove it I ran the following benchmark:

require 'benchmark'
require 'csv'
task :bench do
  servers = roles(:borg).to_a
  timings = {}

  if ENV['PREWARM']
    on servers do
      execute :echo, 'pong > /dev/null'
    end
  end

  [1, 10, 50, 100].each do |count|
    timings[count] = Benchmark.measure do
      on servers.pop(count) do
        execute :echo, 'pong > /dev/null'
      end
    end
  end

  res = [%w(count user system real)]
  timings.each do |count, timing|
    res << [count, timing.utime, timing.stime, timing.real]
  end
  puts res.map(&:to_csv).join
end

Here are the results: Spreadsheet

capture d ecran 2016-02-02 a 18 41 29

The first graph is with "cold" connections, meaning it's connection establishment plus the command execution. In the second graph, all the connections were pre established before the benchmark.

I'm still investigating to figure out where exactly the bottleneck (or bottlenecks) exactly is. I know the GIL is not for nothing, but capistrano / SSHKit being IO heavy, I think there is other reasons.

cc @kirs as well as @csfrancis

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions