Using comments in the add_field section causes syntax error #137
Description
In Logstash 6.3.0 I've noticed the following phenomenon:
I have configuration A and configuration B. I run them with the following command:
/usr/share/logstash/bin/logstash -f /home/user/A.cf --path.data="/srv"
/usr/share/logstash/bin/logstash -f /home/user/B.cf --path.data="/srv"
Configuration A:
input {
stdin {}
}
filter {
mutate {
add_field => {
# Comment 1
"some_field_1" => "some value"
# Comment 2
"some_field_2" => "some value"
}
}
}
output {
stdout {}
}
Configuration A yields the following error:
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2019-05-31 12:58:29.124 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2019-05-31 12:58:30.131 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.3.0"}
[ERROR] 2019-05-31 12:58:31.253 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, {, } at line 14, column 13 (byte 188) after filter {\n\n mutate {\n\n add_field => {\n # Comment 1\n \"some_field_1\" => \"some value\"\n \n # Comment 2\n ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:49:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:167:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:305:in `block in converge_state'"]}
Configuration B:
input {
stdin {}
}
filter {
mutate {
add_field => {
# Comment 1
"some_field_1" => "some value"
"some_field_2" => "some value"
}
}
}
output {
stdout {}
}
Configuration B yields no errors:
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2019-05-31 12:59:49.383 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2019-05-31 12:59:50.726 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.3.0"}
[INFO ] 2019-05-31 12:59:59.327 [Converge PipelineAction::Create<main>] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[INFO ] 2019-05-31 12:59:59.594 [Converge PipelineAction::Create<main>] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x395ce82 run>"}
The stdin plugin is now waiting for input:
[INFO ] 2019-05-31 12:59:59.730 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2019-05-31 13:00:00.123 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9603}
Test
{
"@timestamp" => 2019-05-31T11:00:04.690Z,
"some_field_1" => "some value",
"some_field_2" => "some value",
"message" => "Test",
"@version" => "1",
"host" => "some.host.com"
}
The only difference between the two configurations is this part:
A:
add_field => {
# Comment 1
"some_field_1" => "some value"
# Comment 2
"some_field_2" => "some value"
}
B:
add_field => {
# Comment 1
"some_field_1" => "some value"
"some_field_2" => "some value"
}
Am I right to assume that you can only use comments at the "beginning" of an add_field section, but not in the middle? Is it a bug or a feature?
Of course the obvious solution to this issue is avoiding these kinds of comments, but I wanted to gather some plus information about this before moving onward.
Udate: We have examined the issue further and come to the conclusion that this issue is present in every plugin which has a hash as a parameter (split, translate etc.).