Skip to content

index.asciidoc

Latest commit

 

History

History
370 lines (271 loc) · 11.2 KB

index.asciidoc

File metadata and controls

370 lines (271 loc) · 11.2 KB
Raw

Elasticsearch filter plugin

Description

Search Elasticsearch for a previous log event and copy some fields from it into the current event. Below are two complete examples of how this filter might be used.

The first example uses the legacy 'query' parameter where the user is limited to an Elasticsearch query_string. Whenever logstash receives an "end" event, it uses this elasticsearch filter to find the matching "start" event based on some operation identifier. Then it copies the @timestamp field from the "start" event into a new field on the "end" event. Finally, using a combination of the "date" filter and the "ruby" filter, we calculate the time duration in hours between the two events.

if [type] == "end" {
   elasticsearch {
      hosts => ["es-server"]
      query => "type:start AND operation:%{[opid]}"
      fields => { "@timestamp" => "started" }
   }

   date {
      match => ["[started]", "ISO8601"]
      target => "[started]"
   }

   ruby {
      code => "event.set('duration_hrs', (event.get('@timestamp') - event.get('started')) / 3600)"
   }
}

The example below reproduces the above example but utilises the query_template. This query_template represents a full Elasticsearch query DSL and supports the standard Logstash field substitution syntax. The example below issues the same query as the first example but uses the template shown.

if [type] == "end" {
      elasticsearch {
         hosts => ["es-server"]
         query_template => "template.json"
         fields => { "@timestamp" => "started" }
      }

      date {
         match => ["[started]", "ISO8601"]
         target => "[started]"
      }

      ruby {
         code => "event.set('duration_hrs', (event.get('@timestamp') - event.get('started')) / 3600)"
      }
}

template.json:

{
  "size": 1,
  "sort" : [ { "@timestamp" : "desc" } ],
  "query": {
    "query_string": {
      "query": "type:start AND operation:%{[opid]}"
    }
  },
  "_source": ["@timestamp"]
}

As illustrated above, through the use of 'opid', fields from the Logstash events can be referenced within the template. The template will be populated per event prior to being used to query Elasticsearch.

Notice also that when you use query_template, the Logstash attributes result_size and sort will be ignored. They should be specified directly in the JSON template, as shown in the example above.

Authentication

Authentication to a secure Elasticsearch cluster is possible using one of the following options:

Authorization

Authorization to a secure Elasticsearch cluster requires read permission at index level and monitoring permissions at cluster level. The monitoring permission at cluster level is necessary to perform periodic connectivity checks.

Elasticsearch Filter Configuration Options

This plugin supports the following configuration options plus the [plugins-filters-elasticsearch-common-options] described later.

Setting Input type Required

aggregation_fields

hash

No

api_key

password

No

ca_file

a valid filesystem path

No

cloud_auth

password

No

cloud_id

string

No

docinfo_fields

hash

No

enable_sort

boolean

No

fields

array

No

hosts

array

No

index

string

No

password

password

No

proxy

uri

No

query

string

No

query_template

string

No

result_size

number

No

sort

string

No

ssl

boolean

No

tag_on_failure

array

No

user

string

No

Also see [plugins-filters-elasticsearch-common-options] for a list of options supported by all filter plugins.

 

aggregation_fields

  • Value type is hash

  • Default value is {}

Hash of aggregation names to copy from elasticsearch response into Logstash event fields

Example:

    filter {
      elasticsearch {
        aggregation_fields => {
          "my_agg_name" => "my_ls_field"
        }
      }
    }
api_key

  • Value type is password

  • There is no default value for this setting.

Authenticate using Elasticsearch API key. Note that this option also requires enabling the ssl option.

Format is id:api_key where id and api_key are as returned by the Elasticsearch {ref}/security-api-create-api-key.html[Create API key API].

ca_file

  • Value type is path

  • There is no default value for this setting.

SSL Certificate Authority file

cloud_auth

  • Value type is password

  • There is no default value for this setting.

Cloud authentication string ("<username>:<password>" format) is an alternative for the user/password pair.

For more info, check out the {logstash-ref}/connecting-to-cloud.html[Logstash-to-Cloud documentation].

cloud_id

  • Value type is string

  • There is no default value for this setting.

Cloud ID, from the Elastic Cloud web console. If set hosts should not be used.

For more info, check out the {logstash-ref}/connecting-to-cloud.html[Logstash-to-Cloud documentation].

docinfo_fields

  • Value type is hash

  • Default value is {}

Hash of docinfo fields to copy from old event (found via elasticsearch) into new event

Example:

    filter {
      elasticsearch {
        docinfo_fields => {
          "_id" => "document_id"
          "_index" => "document_index"
        }
      }
    }
enable_sort

  • Value type is boolean

  • Default value is true

Whether results should be sorted or not

fields

  • Value type is array

  • Default value is {}

An array of fields to copy from the old event (found via elasticsearch) into the new event, currently being processed.

In the following example, the values of @timestamp and event_id on the event found via elasticsearch are copied to the current event’s started and start_id fields, respectively:

fields => {
  "@timestamp" => "started"
  "event_id" => "start_id"
}
hosts

  • Value type is array

  • Default value is ["localhost:9200"]

List of elasticsearch hosts to use for querying.

This option can accept an environment variable containing one or more hostnames separated by whitespace. Strings separated by whitespace are treated as separate entries.

Examples:

index

  • Value type is string

  • Default value is ""

Comma-delimited list of index names to search; use _all or empty string to perform the operation on all indices. Field substitution (e.g. index-name-%{date_field}) is available

password

  • Value type is password

  • There is no default value for this setting.

Basic Auth - password

proxy

  • Value type is uri

  • There is no default value for this setting.

Set the address of a forward HTTP proxy. An empty string is treated as if proxy was not set, and is useful when using environment variables e.g. proxy ⇒ '${LS_PROXY:}'.

query

  • Value type is string

  • There is no default value for this setting.

Elasticsearch query string. More information is available in the {ref}/query-dsl-query-string-query.html#query-string-syntax[Elasticsearch query string documentation].

query_template

  • Value type is string

  • There is no default value for this setting.

File path to elasticsearch query in DSL format. More information is available in the {ref}/query-dsl.html[Elasticsearch query documentation].

result_size

  • Value type is number

  • Default value is 1

How many results to return

sort

  • Value type is string

  • Default value is "@timestamp:desc"

Comma-delimited list of <field>:<direction> pairs that define the sort order

ssl

  • Value type is boolean

  • Default value is false

SSL

tag_on_failure

  • Value type is array

  • Default value is ["_elasticsearch_lookup_failure"]

Tags the event on failure to look up previous log event information. This can be used in later analysis.

user

  • Value type is string

  • There is no default value for this setting.

Basic Auth - username

../../../../logstash/docs/include/filter.asciidoc