8.17.4 cannot load Elasticsearch input plugin

[lpeter91]

Basically what the title says. Can't tell if it's an issue with Logstash, the plugin or the "packaging", so reporting it here in hopes it gets more attention. (: Quite simple to reproduce:

POC config:

$ cat /tmp/pipeline/test.conf

input {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "test"
    user => "test"
    password => "test"
  }
}

output {
  stdout {
    codec => json
  }
}

Trying to run:

$ docker run --rm -it -v /tmp/pipeline/:/usr/share/logstash/pipeline/ docker.elastic.co/logstash/logstash:8.17.4

Using bundled JDK: /usr/share/logstash/jdk
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2025-03-30T16:42:13,687][WARN ][org.logstash.deprecation.settings.DeprecatedAlias] The setting `http.host` is a deprecated alias for `api.http.host` and will be removed in version 9. Please use `api.http.host` instead
[2025-03-30T16:42:13,697][INFO ][logstash.runner          ] Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties
[2025-03-30T16:42:13,698][WARN ][deprecation.logstash.runner] 'pipeline.buffer.type' setting is not explicitly defined.Before moving to 9.x set it to 'heap' and tune heap size upward, or set it to 'direct' to maintain existing behavior.
[2025-03-30T16:42:13,699][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.17.4", "jruby.version"=>"jruby 9.4.9.0 (3.1.4) 2024-11-04 547c6b150e OpenJDK 64-Bit Server VM 21.0.6+7-LTS on 21.0.6+7-LTS +indy +jit [x86_64-linux]"}
[2025-03-30T16:42:13,701][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dlogstash.jackson.stream-read-constraints.max-string-length=200000000, -Dlogstash.jackson.stream-read-constraints.max-number-length=10000, -Dls.cgroup.cpuacct.path.override=/, -Dls.cgroup.cpu.path.override=/, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dio.netty.allocator.maxOrder=11]
[2025-03-30T16:42:13,727][INFO ][org.logstash.jackson.StreamReadConstraintsUtil] Jackson default value override `logstash.jackson.stream-read-constraints.max-string-length` configured to `200000000`
[2025-03-30T16:42:13,727][INFO ][org.logstash.jackson.StreamReadConstraintsUtil] Jackson default value override `logstash.jackson.stream-read-constraints.max-number-length` configured to `10000`
[2025-03-30T16:42:13,734][INFO ][logstash.settings        ] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[2025-03-30T16:42:13,737][INFO ][logstash.settings        ] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[2025-03-30T16:42:13,870][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"f1aa32e7-31b9-415e-802d-5b90b281ad0e", :path=>"/usr/share/logstash/data/uuid"}
[2025-03-30T16:42:14,165][WARN ][logstash.monitoringextension.pipelineregisterhook] xpack.monitoring.enabled has not been defined, but found elasticsearch configuration. Please explicitly set `xpack.monitoring.enabled: true` in logstash.yml
[2025-03-30T16:42:14,165][WARN ][deprecation.logstash.monitoringextension.pipelineregisterhook] Internal collectors option for Logstash monitoring is deprecated and targeted for removal in the next major version.
Please configure Elastic Agent to monitor Logstash. Documentation can be found at:
https://www.elastic.co/guide/en/logstash/current/monitoring-with-elastic-agent.html
[2025-03-30T16:42:14,386][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
[2025-03-30T16:42:17,583][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"elasticsearch: Name or service not known", :exception=>Manticore::ResolutionFailure, :cause=>#<Java::JavaNet::UnknownHostException: elasticsearch: Name or service not known>}
[2025-03-30T16:42:17,585][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch: Name or service not known"}
[2025-03-30T16:42:17,590][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"elasticsearch", :exception=>Manticore::ResolutionFailure, :cause=>#<Java::JavaNet::UnknownHostException: elasticsearch>}
[2025-03-30T16:42:17,591][WARN ][logstash.licensechecker.licensereader] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch {:url=>http://elasticsearch:9200/, :error_message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2025-03-30T16:42:17,592][WARN ][logstash.licensechecker.licensereader] Attempt to fetch Elasticsearch cluster info failed. Sleeping for 0.02 {:fail_count=>1, :exception=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch"}
[2025-03-30T16:42:17,614][ERROR][logstash.licensechecker.licensereader] Unable to retrieve Elasticsearch cluster info. {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError}
[2025-03-30T16:42:17,615][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
[2025-03-30T16:42:17,631][ERROR][logstash.monitoring.internalpipelinesource] Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.
[2025-03-30T16:42:17,698][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2025-03-30T16:42:17,888][INFO ][org.reflections.Reflections] Reflections took 95 ms to scan 1 urls, producing 152 keys and 530 values
[2025-03-30T16:42:18,873][ERROR][logstash.plugins.registry] Unable to load plugin. {:type=>"input", :name=>"elasticsearch"}
[2025-03-30T16:42:18,880][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (PluginLoadingError) Couldn't find any input plugin named 'elasticsearch'. Are you sure this is correct? Trying to load the elasticsearch input plugin resulted in this error: Unable to load the requested plugin named elasticsearch of type input. The plugin is not installed.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:137)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:240)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$initialize.call(AbstractPipelineExt$INVOKER$i$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:847)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1379)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:139)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:363)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:446)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)", "org.jruby.RubyClass.newInstance(RubyClass.java:949)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:446)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:548)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:363)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:88)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:238)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:225)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:228)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:476)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:293)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:324)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:118)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:144)", "org.jruby.RubyProc.call(RubyProc.java:354)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:111)", "java.base/java.lang.Thread.run(Thread.java:1583)"], :cause=>{:exception=>Java::OrgJrubyExceptions::Exception, :message=>"(PluginLoadingError) Couldn't find any input plugin named 'elasticsearch'. Are you sure this is correct? Trying to load the elasticsearch input plugin resulted in this error: Unable to load the requested plugin named elasticsearch of type input. The plugin is not installed.", :backtrace=>["RUBY.lookup_pipeline_plugin(/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:245)", "RUBY.lookup(/usr/share/logstash/logstash-core/lib/logstash/plugin.rb:165)", "org.logstash.plugins.factory.PluginFactoryExt.plugin(org/logstash/plugins/factory/PluginFactoryExt.java:229)", "org.logstash.execution.AbstractPipelineExt.initialize(org/logstash/execution/AbstractPipelineExt.java:240)", "RUBY.initialize(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47)", "org.jruby.RubyClass.new(org/jruby/RubyClass.java:949)", "RUBY.execute(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:50)", "RUBY.converge_state(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:420)"]}}
[2025-03-30T16:42:18,892][INFO ][logstash.runner          ] Logstash shut down.
[2025-03-30T16:42:18,898][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:924) ~[jruby.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:883) ~[jruby.jar:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:90) ~[?:?]

Having an actual ES available doesn't fix this issue (this is jut a POC, but I have this issue in a staging system). I didn't test if this is actually limited to the container image or not. Also did not test if this is limited to this input or other plugins are affected too. logstash-plugin list does list the plugin as installed. 8.17.3 seems to work fine.

[lpeter91]

Update: I tested with the .tar.gz distribution, it's also affected, so it's not an issue with the container image itself.

[yaauie]

Updated versions of both the Elasticsearch Filter (v3.17.1) and Elasticsearch Input (v4.21.2) are available to resolve this issue, and can be updated in-place:

bin/logstash-plugin update logstash-filter-elasticsearch logstash-input-elasticsearch

On my console, I get the following output:

╭─{ rye@REDACTED:~/src/elastic/REDACTED/logstash-8.17.4 }
╰─○ bin/logstash-plugin update logstash-filter-elasticsearch logstash-input-elasticsearch
Using bundled JDK: /Users/rye/src/elastic/REDACTED/logstash-8.17.4/jdk.app/Contents/Home
Updating logstash-filter-elasticsearch, logstash-input-elasticsearch
Bundler attempted to update logstash-mixin-ca_trusted_fingerprint_support but its version stayed the same
Bundler attempted to update logstash-mixin-normalize_config_support but its version stayed the same
Bundler attempted to update logstash-mixin-ecs_compatibility_support but its version stayed the same
Bundler attempted to update logstash-mixin-event_support but its version stayed the same
Bundler attempted to update logstash-mixin-validator_support but its version stayed the same
Bundler attempted to update logstash-mixin-scheduler but its version stayed the same
Updated logstash-filter-elasticsearch 3.17.0 to 3.17.1
Updated logstash-input-elasticsearch 4.21.1 to 4.21.2
[success (00:00:18)]

These updated versions will also be included in the forthcoming patch releases of Logstash itself.

[robbavey]

@lpeter91 Did this resolve your issue?

[lpeter91]

@robbavey Well I did not explicitly try the plugin update method since I'm using Logstash via ECK and it's just "too much of a hassle" to add this customization just to remove it after the next release. So from my point of view this issue is resolved when the new version is released and I can actually see it working. :)

But I can take your word for it that it will work, so I'm closing the issue.