task: detangle build-time and runtime dependencies

[yaauie]

There are a few interconnected issues with how we use bundler for both the build chain and the plugin manager. This is not an exhaustive list.

• The runtime and development dependencies of logstash-core are somewhat arbitrarily split between Gemfile.template and logstash-core/logstash-core.gemspec
  • use gemspec path: 'logstash-core' in the Gemfile.template instead of making logstash-core a runtime dependency; this ensures that the add_development_dependency declarations in logstash-core/logstash-core.gemspec are effectively development dependencies of the Gemfile.template
  • move non-plugin (a) runtime dependency declarations and (b) development declarations from Gemfile.template to the gemspec
  • NOTE: version constraints for plugins must be in Gemfile.template in order to support plugin upgrades in shipped artifacts
  • temporary pins of transitive dependencies can continue to live in Gemfile.template
• bin/logstash-plugin install --development installs not only the development dependencies of logstash core, but also of each of the plugins that are currently installed, and it does not merge the requirements so that the last one wins (even if it overrides a runtime requirement that is in the checked-in logstash-core.gemspec or Gemfile.template).
  • do we ever actually need the development dependencies of all of the plugins while developing Logstash core? Why aren't the dev dependencies we need specified as development dependencies of logstash core?
• Because we build artifacts directly from source, the dependencies that are required for building the artifacts but not required for the built artifact itself are a part of the shipped bundle (e.g., rake, octokit, childprocess, minitar, fpm).
  • I believe this can be solved by using an intermediate staging directory and installing its bundle using its plugin manager, at which point the build group of dependencies in the Gemfile.template could be made development dependencies in logstash-core/logstash-core.gemspec, which would ensure that they're never installed in the built artifact.