Multiline codec pattern not working for gelf input

[fewknow]

Please post all product and debugging questions on our forum. Your questions will reach our wider community members there, and if we confirm that there is a bug, then we can open a new issue here.

For all general issues, please provide the following details for fast resolution:

• Version: logstash 6.4.2
• Operating System:

NAME="Ubuntu"
VERSION="18.04.1 LTS (Bionic Beaver)"

• Config File :

input {
  stdin { }
  gelf {
    host => "0.0.0.0"
    port => 12201
    codec => multiline {
      pattern => "^\t"
      what => "previous"
    }
  }

  udp {
    codec => json
    port => 5001
  }

  tcp {
    port => 5000
    codec => json
  }

  beats {
    port => 5044
  }

  http {
    port => 8000
    type => "elb-healthcheck"
  }

}

filter {
  if [type] == "elb-healthcheck" {
    drop { }
  }
}


#filter {
#    multiline {
#      pattern => "^(%{TIMESTAMP_ISO8601})"
#      negate => true
#      what => "previous"
#   }
#   grok {
#      # Do multiline matching with (?m) as the above mutliline filter may add newlines to the log messages.
#      match => [ "message", "(?m)^%{TIMESTAMP_ISO8601:logtime}%{SPACE}%{LOGLEVEL:loglevel} %{SPACE}%{NUMBER:pid}%{SPACE}%{SYSLOG5424SD:threadname}%{SPACE}---%{SPACE}%{JAVACLASSSHORT:classname}%{SPACE}:%{SPACE}%{GREEDYDATA:logmessage}" ]
#   }
#}

filter {

 ## Map log levels to integers, the "level" index is an integer and blows up when it receives a string:w
 ## Log level mappings from: https://docs.python.org/2/library/logging.html
  mutate {
    gsub => [
      "level", "DEBUG", "10",
      "level", "INFO", "20",
      "level", "WARN", "30",
      "level", "ERROR", "40",
      "level", "CRITICAL", "50",
      "level", "NOTSET", "0"
    ]
 }

 ## Convert log level to an integer (after above mapping)
 mutate {
   convert => {
        "level" => "integer"
   }
 }
}
output {
  stdout { codec => rubydebug  }
}

• Sample Data:

{
      "container_id" => "4ca881d3376eba3063bbebe17b1a6ffce4485bbf885f99f7bf1087fceb49fde5",
               "tag" => "sand-multitenant-microservice",
    "container_name" => "ecs-sand-multitenant-microservice-1-sand-multitenant-microservice-feedf7aad4f5d8a4d901",
        "@timestamp" => 2018-10-18T21:29:47.957Z,
          "image_id" => "sha256:12f59bfb89234141d9b0b1e583489563c8644e63b145f0a266a9476a3ddb1f90",
     "short_message" => "",
           "command" => "java -jar multitenant-microservice.jar",
              "host" => "ip-10-90-66-117",
             "level" => 6,
          "@version" => "1",
           "version" => "1.1",
        "image_name" => "038131160342.dkr.ecr.us-east-1.amazonaws.com/multitenant-microservice:12584d13310d3ff836ff4f37842d1556ea2a98a6",
       "source_host" => "10.90.66.117",
           "created" => "2018-10-18T21:29:23.568019329Z"
}
{
      "container_id" => "4ca881d3376eba3063bbebe17b1a6ffce4485bbf885f99f7bf1087fceb49fde5",
               "tag" => "sand-multitenant-microservice",
    "container_name" => "ecs-sand-multitenant-microservice-1-sand-multitenant-microservice-feedf7aad4f5d8a4d901",
        "@timestamp" => 2018-10-18T21:29:47.957Z,
          "image_id" => "sha256:12f59bfb89234141d9b0b1e583489563c8644e63b145f0a266a9476a3ddb1f90",
           "command" => "java -jar multitenant-microservice.jar",
              "host" => "ip-10-90-66-117",
             "level" => 6,
          "@version" => "1",
           "version" => "1.1",
        "image_name" => "038131160342.dkr.ecr.us-east-1.amazonaws.com/multitenant-microservice:12584d13310d3ff836ff4f37842d1556ea2a98a6",
       "source_host" => "10.90.66.117",
           "message" => "\tat org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:155) ~[spring-boot-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]",
           "created" => "2018-10-18T21:29:23.568019329Z"
}
{
      "container_id" => "4ca881d3376eba3063bbebe17b1a6ffce4485bbf885f99f7bf1087fceb49fde5",
               "tag" => "sand-multitenant-microservice",
    "container_name" => "ecs-sand-multitenant-microservice-1-sand-multitenant-microservice-feedf7aad4f5d8a4d901",
        "@timestamp" => 2018-10-18T21:29:47.957Z,
          "image_id" => "sha256:12f59bfb89234141d9b0b1e583489563c8644e63b145f0a266a9476a3ddb1f90",
           "command" => "java -jar multitenant-microservice.jar",
              "host" => "ip-10-90-66-117",
             "level" => 6,
          "@version" => "1",
           "version" => "1.1",
        "image_name" => "038131160342.dkr.ecr.us-east-1.amazonaws.com/multitenant-microservice:12584d13310d3ff836ff4f37842d1556ea2a98a6",
       "source_host" => "10.90.66.117",
           "message" => "\tat org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:140) ~[spring-boot-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]",
           "created" => "2018-10-18T21:29:23.568019329Z"
}
{
      "container_id" => "4ca881d3376eba3063bbebe17b1a6ffce4485bbf885f99f7bf1087fceb49fde5",
               "tag" => "sand-multitenant-microservice",
    "container_name" => "ecs-sand-multitenant-microservice-1-sand-multitenant-microservice-feedf7aad4f5d8a4d901",
        "@timestamp" => 2018-10-18T21:29:47.959Z,
          "image_id" => "sha256:12f59bfb89234141d9b0b1e583489563c8644e63b145f0a266a9476a3ddb1f90",
           "command" => "java -jar multitenant-microservice.jar",
              "host" => "ip-10-90-66-117",
             "level" => 6,
          "@version" => "1",
           "version" => "1.1",
        "image_name" => "038131160342.dkr.ecr.us-east-1.amazonaws.com/multitenant-microservice:12584d13310d3ff836ff4f37842d1556ea2a98a6",
       "source_host" => "10.90.66.117",
           "message" => "\tat org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:398) [spring-boot-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]",
           "created" => "2018-10-18T21:29:23.568019329Z"
}
{
      "container_id" => "4ca881d3376eba3063bbebe17b1a6ffce4485bbf885f99f7bf1087fceb49fde5",
               "tag" => "sand-multitenant-microservice",
    "container_name" => "ecs-sand-multitenant-microservice-1-sand-multitenant-microservice-feedf7aad4f5d8a4d901",
        "@timestamp" => 2018-10-18T21:29:47.987Z,
          "image_id" => "sha256:12f59bfb89234141d9b0b1e583489563c8644e63b145f0a266a9476a3ddb1f90",
           "command" => "java -jar multitenant-microservice.jar",
              "host" => "ip-10-90-66-117",
             "level" => 6,
          "@version" => "1",
           "version" => "1.1",
        "image_name" => "038131160342.dkr.ecr.us-east-1.amazonaws.com/multitenant-microservice:12584d13310d3ff836ff4f37842d1556ea2a98a6",
       "source_host" => "10.90.66.117",
           "message" => "\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:1258) [spring-boot-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]",
           "created" => "2018-10-18T21:29:23.568019329Z"
}

Multiline is not working with gelf input.

The pattern is not being picked up.

I also see that it is setting the pattern as "^\t" fom "^\t"

[2018-10-18T16:35:21,262][TRACE][logstash.codecs.multiline] Registered multiline plugin {:type=>nil, :config=>{"pattern"=>"^\\t", "what"=>"previous", "id"=>"27e36f0b-9aad-44ff-b3de-4da18659b951", "enable_metric"=>true, "negate"=>false, "patterns_dir"=>[], "charset"=>"UTF-8", "multiline_tag"=>"multiline", "max_lines"=>500, "max_bytes"=>10485760}}
[2018-10-18T16:35:21,274][DEBUG][logstash.inputs.gelf     ] config LogStash::Inputs::Gelf/@host = "0.0.0.0"
[2018-10-18T16:35:21,275][DEBUG][logstash.inputs.gelf     ] config LogStash::Inputs::Gelf/@codec = <LogStash::Codecs::Multiline pattern=>"^\\t", what=>"previous", id=>"27e36f0b-9aad-44ff-b3de-4da18659b951", enable_metric=>true, negate=>false, charset=>"UTF-8", multiline_tag=>"multiline", max_lines=>500, max_bytes=>10485760>
[2018-10-18T16:35:21,276][DEBUG][logstash.inputs.gelf     ] config LogStash::Inputs::Gelf/@id = "600a8979b5fed5bc50d159d1b20bb74593ba55df2b4e831fd9b0f8c8f8b70c02"
[2018-10-18T16:35:21,276][DEBUG][logstash.inputs.gelf     ] config LogStash::Inputs::Gelf/@port = 12201
[2018-10-18T16:35:21,276][DEBUG][logstash.inputs.gelf     ] config LogStash::Inputs::Gelf/@enable_metric = true
[2018-10-18T16:35:21,276][DEBUG][logstash.inputs.gelf     ] config LogStash::Inputs::Gelf/@add_field = {}
[2018-10-18T16:35:21,276][DEBUG][logstash.inputs.gelf     ] config LogStash::Inputs::Gelf/@remap = true
[2018-10-18T16:35:21,276][DEBUG][logstash.inputs.gelf     ] config LogStash::Inputs::Gelf/@strip_leading_underscore = true
[2018-10-18T16:35:21,276][DEBUG][logstash.inputs.gelf     ] config LogStash::Inputs::Gelf/@use_tcp = false
[2018-10-18T16:35:21,277][DEBUG][logstash.inputs.gelf     ] config LogStash::Inputs::Gelf/@use_udp = true

Thank you for your time.

[colinsurprenant]

@fewknow Thanks for your feedback.

Please note that the gelf does not make use of codecs. In fact, the GELF format is serialized in JSON so the gelf input simply deserialize the received JSON data - there is not point in supporting codecs for that input.

Could you please share more details about what exactly you are trying to achieve and which field is part of a multi lines output that you are trying to consolidate back into a single event?

[fewknow]

@colinsurprenant I am trying to consolidate java logs coming from my docker containers. This is in aws on an ECS cluster. There is no support for log4j currently, but there is GELF support on ECS tasks. Currently I have this pumping logs to logstash and then to elasticsearch.

The issue I am facing is a single log for every line of the log over GELF. Was hoping to help out my developers by collapsing the stack trace into a more desirable log once it gets to elasticsearch. Please let me know best way to do this.

I know there is filebeat and a log4j plugin but this doesn't work well with our docker containers as they are a single PID process currenlty and would like to keep them this way. Any and all help is appreciated.

Thanks

[colinsurprenant]

I would suggest creating a feature request issue to explore the possibility of enabling codecs per your multi-line use-case but I don't see that happening in the short term unless the community pick it up.

Alternatively, since this is more a usage question I would also suggest to bring it up in the forums https://discuss.elastic.co/c/logstash where it might reach a wider audience.

[squalou]

hi @fewknow ,
by chance would you have found a way to solve your issue ?
I'm facing the exact same situation.

Since "multiline" is not available as a filter, I wondered maybe if "aggregate" filter would be of any help ... didn't try ye though.

[squalou]

(for the records, I switched from geld to firelens / fluentbit to solve this)