Key-Value Pairs Parser Plugin for Fluentd

Fluentd built-in parser_ltsv has been provided all feature of this plugin since Fluentd v1.1.0.

Overview

This is a parser plugin for Fluentd. Learn more about parser plugins here.

This plugin allows you to parse inputs that look like key-value pairs. For example, if your text logs look like

"this_field=10000  that_field=hello time=2013-01-01T12:34:00"

It is parsed as

{"this_field":10000, "that_field":"hello"}

with the event's time being 2013-01-01T12:34:00

Requirements

fluent-plugin-kv-parser	fluentd	ruby
>= 1.0.0	>= v0.14.0	>= 2.1
< 1.0.0	>= v0.12.0	>= 1.9

How to Install and Use

For Fluentd,

gem install fluent-plugin-kv-parser

For Treasure Agent,

/usr/sbin/td-agent-gem install fluent-plugin-kv-parser

Then, for parser-plugin enabled input plugins (including in_tail, in_tcp, in_udp and in_syslog, you can just write format kv

For example, using in_tcp with the following configuration:

<source>
  @type tcp
  port 24225
  tag kv_log
  <parse>
    @type kv
    time_key my_time
    types k1:integer,my_time:time
  </parse>
</source>
<match kv_log>
  @type stdout
</match>

Running

echo 'my_time=2014-12-31T00:00:00 k1=1234 k2=hello' | nc localhost 24224

gives

2014-12-31 00:00:00 +0000 kv_log: {"k1":1234,"k2":"hello"}

Parameters

parameter	description	default
kv_delimiter	The delimiter for key-value pairs. If the value starts and ends with the character '/', the separator is interpreted to be a regexp. Else, it is interpreted to be a string. Hence,kv_delimiter /a+/ splits on one or more "a"s, kv_delimiter a splits on a single "a"	/\s+/
kv_char	The string to split the key from the value	"="
time_key	The time key field among the key-value pairs to be used as the time for the event. If missing or unparsable, the current time is used	time
types	The parameter to convert the values of key-value pairs. The syntax is <key_name>:<type_name>. For example, to convert the key "k1" into integer, write types k1:integer. For the time type, one can write <key_name>:time:<time_format> to convert the string into a time object. For example, to convert the string "my_time=12/31/2014 12:00:00", use my_time:time:%m/%d/%Y %H:%M:%S. This parameter is same as the one used for in_tail and others (see under the "types" section over there)

License

Apache 2.0. Copyright Kiyoto Tamura