Reference

Table of Contents

Classes

fluentbit: Installs and configures fluentbit
fluentbit::config: configures the main fluentbit main config
fluentbit::install: Installs fluentbit package
fluentbit::repo: configures the fluentbit repo
fluentbit::repo::debian: A short summary of the purpose of this class
fluentbit::service: Manages the td-agent-bit service

Defined types

fluentbit::filter::modify: Filter to modify records using rules and conditions
fluentbit::input::forward: Inputs / listen and forward messages
fluentbit::input::syslog: Ingest logs from (r)syslog
fluentbit::input::tail: Input plugin to monitor one or several text files.
fluentbit::output::es: Plugin to output logs to a configured elasticsearch instance
fluentbit::output::http: Manage the configuration of a HTTP output plugin.

Classes

fluentbit

Installs and configures fluentbit

See also https://docs.fluentbit.io/manual/

Examples

include fluentbit

Parameters

The following parameters are available in the fluentbit class.

`manage_package_repo`

Data type: Boolean

Installs the package repositories

`service_name`

Data type: String[1]

the td-agent-bit service name

`input_plugins`

Data type: Array[Fluentbit::Plugin]

Hash of the INPUT plugins to be configured

`output_plugins`

Data type: Array[Fluentbit::Plugin]

Hash of the OUTPUT plugins to be configured

`filter_plugins`

Data type: Array[Fluentbit::Plugin]

Hash of the filter to be configured

`manage_package_repo`

Installs the package repositories

`repo_key_fingerprint`

Data type: String[1]

GPG key identifier of the repository

`repo_key_location`

Data type: Stdlib::HTTPUrl

GPG key location

`package_ensure`

Data type: String[1]

Whether to install the Fluentbit package, and what version to install. Values: 'present', 'latest', or a specific version number. Default value: 'present'.

`package_name`

Data type: String[1]

Specifies the Fluentbit package to manage. Default value: 'td-agent-bit'

`manage_service`

Data type: Boolean

Whether to manage the service at all. Default value: true

`service_enable`

Data type: Boolean

Whether to enable the fluentbit service at boot. Default value: true.

`service_ensure`

Data type: Stdlib::Ensure::Service

Whether the fluentbit service should be running. Default value: 'running'.

`service_manage`

Whether to manage the fluentbit service. Default value: true.

`service_hasstatus`

Whether the service has a functional status command. Default value: true.

`service_hasrestart`

Whether the service has a restart command. Default value: true.

`manage_config_dir`

Data type: Boolean

Whether to manage the configuration directory. When enabled, will remove all unmanaged files from the directory the configuration resides in. Default value: true

`config_file`

Data type: Stdlib::Absolutepath

Path of the daemon configuration.

`config_file_mode`

Data type: Stdlib::Filemode

File mode to apply to the daemon configuration file

`storage_path`

Data type: Optional[Stdlib::Absolutepath]

Set an optional location in the file system to store streams and chunks of data. If this parameter is not set, Input plugins can only use in-memory buffering.

`storage_sync`

Data type: Optional[Enum['normal', 'full']]

Configure the synchronization mode used to store the data into the file system. It can take the values normal or full. Default value: 'normal'

`storage_checksum`

Data type: Boolean

Enable the data integrity check when writing and reading data from the filesystem. The storage layer uses the CRC32 algorithm. Default value: false

`storage_backlog_mem_limit`

Data type: Optional[String[1]]

If storage.path is set, Fluent Bit will look for data chunks that were not delivered and are still in the storage layer, these are called backlog data. This option configure a hint of maximum value of memory to use when processing these records. Default value: 5M

`manage_plugins_file`

Data type: Boolean

Whether to manage the enabled external plugins

`plugins_file`

Data type: Stdlib::Absolutepath

A plugins configuration file allows to define paths for external plugins.

`plugins`

Data type: Array[Stdlib::Absolutepath]

List of external plugin objects to enable

`manage_streams_file`

Data type: Boolean

Whether to manage the stream processing configuration

`streams_file`

Data type: Stdlib::Absolutepath

Path for the Stream Processor configuration file.

`streams`

Data type: Array[Fluentbit::Stream]

Stream processing tasks

`manage_parsers_file`

Data type: Boolean

Whether to manage the parser definitions

`parsers_file`

Data type: Stdlib::Absolutepath

Path for a parsers configuration file. Multiple Parsers_File entries can be used.

`parsers`

Data type: Array[Fluentbit::Parser]

List of parser definitions. The default value consists of all the available definitions provided by the upstream project as of version 1.3

`flush`

Data type: Integer

Set the flush time in seconds. Everytime it timeouts, the engine will flush the records to the output plugin.

`daemon`

Data type: Boolean

Boolean value to set if Fluent Bit should run as a Daemon (background) or not. Allowed values are: yes, no, on and off.

`log_file`

Data type: Optional[Stdlib::Absolutepath]

Absolute path for an optional log file.

`log_level`

Data type: Enum['error', 'warning', 'info', 'debug', 'trace']

Set the logging verbosity level. Values are: error, info, debug and trace. Values are accumulative, e.g: if 'debug' is set, it will include error, info and debug. Note that trace mode is only available if Fluent Bit was built with the WITH_TRACE option enabled.

`http_server`

Data type: Boolean

Enable built-in HTTP Server

`http_listen`

Data type: Stdlib::IP::Address::Nosubnet

Set listening interface for HTTP Server when it's enabled

`http_port`

Data type: Stdlib::Port

Set TCP Port for the HTTP Server

`coro_stack_size`

Data type: Integer

Set the coroutines stack size in bytes. The value must be greater than the page size of the running system.

`variables`

Data type: Hash

macro definitions to use in the configuration file the will be registered using the @SET command.

`service_has_status`

Data type: Boolean

`service_has_restart`

Data type: Boolean

fluentbit::config

Includes all [input] and [output] configs. (@include) Sets global variables (@set) Configures global [service] section

fluentbit::install

Installs fluentbit package

fluentbit::repo

configures the fluentbit repo

fluentbit::repo::debian

A description of what this class does

Examples

include fluentbit::repo::debian

fluentbit::service

Manages the td-agent-bit service

Defined types

fluentbit::filter::modify

Filter to modify records using rules and conditions

Examples

fluentbit::filter::modify { 'namevar': }

Parameters

The following parameters are available in the fluentbit::filter::modify defined type.

`configfile`

Data type: String

Path to the filter configfile. Naming should be filter_*.conf to make sure it's getting included by the main config.

Default value: '/etc/td-agent-bit/filter_modify.conf'

`set`

Data type: Optional

Add a key/value pair with key KEY and value VALUE. If KEY already exists, this field is overwritten

Default value: undef

`add`

Data type: Optional

Add a key/value pair with key KEY and value VALUE if KEY does not exist

Default value: undef

`remove`

Data type: Optional

Remove a key/value pair with key KEY if it exists

Default value: undef

`remove_wildcard`

Data type: Optional

Remove all key/value pairs with key matching wildcard KEY

Default value: undef

`remove_regex`

Data type: Optional

Remove all key/value pairs with key matching regexp KEY

Default value: undef

`rename`

Data type: Optional

Rename a key/value pair with key KEY to RENAMED_KEY if KEY exists AND RENAMED_KEY does not exist

Default value: undef

`hard_rename`

Data type: Optional

Rename a key/value pair with key KEY to RENAMED_KEY if KEY exists. If RENAMED_KEY already exists, this field is overwritten

Default value: undef

`copy`

Data type: Optional

Copy a key/value pair with key KEY to COPIED_KEY if KEY exists AND COPIED_KEY does not exist

Default value: undef

`hard_copy`

Data type: Optional

Copy a key/value pair with key KEY to COPIED_KEY if KEY exists. If COPIED_KEY already exists, this field is overwritten

Default value: undef

`match`

Data type: String

Default value: '*'

fluentbit::input::forward

Forward is the protocol used by Fluent Bit and Fluentd to route messages between peers. This plugin implements the input service to listen for Forward messages.

Parameters

The following parameters are available in the fluentbit::input::forward defined type.

`configfile`

Data type: String

Path to the input configfile. Naming should be input_*.conf to make sure it's getting included by the main config.

Default value: "/etc/td-agent-bit/input_forward_${name}.conf"

`listen`

Data type: Stdlib::IP::Address::Nosubnet

Listener network interface

Default value: '0.0.0.0'

`port`

Data type: Stdlib::Port

TCP port to listen for incoming connections.

Default value: 24224

`buffer_max_size`

Data type: Optional[String]

Specify the maximum buffer memory size used to receive a Forward message. The value must be according to the Unit Size specification.

Default value: undef

`buffer_chunk_size`

Data type: Optional[String]

By default the buffer to store the incoming Forward messages, do not allocate the maximum memory allowed, instead it allocate memory when is required. The rounds of allocations are set by Buffer_Chunk_Size. The value must be according to the Unit Size specification.

Default value: undef

fluentbit::input::syslog

Setups config file for fluentbit and configures listen on rsyslog. Reloads services if something changes.

Parameters

The following parameters are available in the fluentbit::input::syslog defined type.

`mode`

Data type: String

Defines transport protocol mode: unix_udp (UDP over Unix socket), unix_tcp (TCP over Unix socket), tcp or udp

Default value: 'unix_udp'

`listen`

Data type: Optional[String]

If Mode is set to tcp, specify the network interface to bind.

Default value: $mode

`port`

Data type: Optional[String]

If Mode is set to tcp, specify the TCP port to listen for incoming connections.

Default value: $mode

`path`

Data type: Optional[String]

If Mode is set to unix_tcp or unix_udp, set the absolute path to the Unix socket file.

Default value: $mode

`parser`

Data type: String

Specify an alternative parser for the message. By default, the plugin uses the parser syslog-rfc3164. If your syslog messages have fractional seconds set this Parser value to syslog-rfc5424 instead.

Default value: 'syslog-rfc3164'

`rsyslog_config`

Data type: String

Specify the path to the rsyslog config file for fluentbit to enable listening.

Default value: '/etc/rsyslog.d/60-fluent-bit.conf'

`configfile`

Data type: String

Path to the input configfile. Naming should be input_*.conf to make sure it's getting included by the main config.

Default value: '/etc/td-agent-bit/input_syslog.conf'

fluentbit::input::tail

The tail input plugin allows to monitor one or several text files. It has a similar behavior like tail -f shell command.

Parameters

The following parameters are available in the fluentbit::input::tail defined type.

`configfile`

Data type: Stdlib::Absolutepath

Path to the input configfile. Naming should be input_*.conf to make sure it's getting included by the main config.

Default value: "/etc/td-agent-bit/input_tail_${name}.conf"

`path`

Data type: Stdlib::Absolutepath

Pattern specifying a specific log files or multiple ones through the use of common wildcards.

`routing_tag`

Data type: Optional[String[1]]

Set a tag (with regex-extract fields) that will be placed on lines read.

Default value: undef

`storage_type`

Data type: Optional[Enum['memory', 'filesystem']]

Specify the buffering mechanism to use. Values: memory, filesystem

Default value: undef

`buffer_chunk_size`

Data type: Optional[Fluentbit::Sizeunit]

Set the initial buffer size to read files data.

Default value: undef

`buffer_max_size`

Data type: Optional[Fluentbit::Sizeunit]

Set the limit of the buffer size per monitored file.

Default value: undef

`path_key`

Data type: Optional[String[1]]

Appends the name of the monitored file as part of the record. The value assigned becomes the key in the map.

Default value: undef

`exclude_path`

Data type: Array[String]

Shell patterns to exclude files matching a certain criteria.

Default value: []

`refresh_interval`

Data type: Optional[Integer[1]]

The interval of refreshing the list of watched files in seconds.

Default value: undef

`rotate_wait`

Data type: Optional[Integer[1]]

Specify the number of extra time in seconds to monitor a file once is rotated in case some pending data is flushed.

Default value: undef

`ignore_older`

Data type: Optional[Fluentbit::Timeunit]

Ignores files that have been last modified before this time.

Default value: undef

`skip_long_lines`

Data type: Boolean

When a monitored file reach it buffer capacity due to a very long line (Buffer_Max_Size), the default behavior is to stop monitoring that file. Skip_Long_Lines alters that behavior and instructs Fluent Bit to skip long lines and continue processing other lines that fits into the buffer size.

Default value: false

`db`

Data type: Optional[Stdlib::Absolutepath]

Specify the database file to keep track of monitored files and offsets.

Default value: undef

`db_sync`

Data type: Optional[Enum['Extra', 'Full', 'Normal', 'Off']]

Set a default synchronization (I/O) method. Values: Extra, Full, Normal, Off.

Default value: undef

`mem_buf_limit`

Data type: Optional[Fluentbit::Sizeunit]

Set a limit of memory that Tail plugin can use when appending data to the Engine.

Default value: undef

`key`

Data type: Optional[String[1]]

Name of the key to append any unstructured data to the output as.

Default value: undef

`tag_regex`

Data type: Optional[String[1]]

Set a regex to exctract fields from the file.

Default value: undef

`multiline`

Data type: Boolean

Try to discover multiline messages and use the proper parsers to compose the outgoing messages.

Default value: false

`multiline_flush`

Data type: Optional[Integer[1]]

Wait period time in seconds to process queued multiline messages.

Default value: undef

`parsers`

Data type: Array[String[1]]

Name of parsers to use for disecting the message. When multiline is enabled, the first array entry becomes the configuration value for Parser_Firstline, with the remaining entry being assigned to the Parser_N attributes. When multiline is disabled, only the first item is used for the Parser configuration value.

Default value: []

`docker_mode`

Data type: Boolean

Recombine split Docker log lines before passing them to any parser.

Default value: false

`docker_mode_flush`

Data type: Optional[Integer[1]]

Wait period time in seconds to flush queued unfinished split lines.

Default value: undef

fluentbit::output::es

Plugin to output logs to a configured elasticsearch instance

Examples

include fluentbit::output::es

Parameters

The following parameters are available in the fluentbit::output::es defined type.

`configfile`

Data type: String

Path to the output configfile. Naming should be output_*.conf to make sure it's getting included by the main config.

Default value: "/etc/td-agent-bit/output_es_${name}.conf"

`match`

Data type: String

Tag to route the output.

Default value: '*'

`tls`

Data type: Enum['on', 'off']

Turn TLS encrypted communication with Elasticsearch on / off.

Default value: 'off'

`host`

Data type: Stdlib::Host

IP address or hostname of the target Elasticsearch instance

Default value: '127.0.0.1'

`port`

Data type: Stdlib::Port

TCP port of the target Elasticsearch instance

Default value: 9200

`path`

Elasticsearch accepts new data on HTTP query path "/_bulk". But it is also possible to serve Elasticsearch behind a reverse proxy on a subpath. This option defines such path on the fluent-bit side. It simply adds a path prefix in the indexing HTTP POST URI.

`buffer_size`

Specify the buffer size used to read the response from the Elasticsearch HTTP service. This option is useful for debugging purposes where is required to read full responses, note that response size grows depending of the number of records inserted. To set an unlimited amount of memory set this value to False, otherwise the value must be according to the Unit Size specification.

`pipeline`

Newer versions of Elasticsearch allows to setup filters called pipelines. This option allows to define which pipeline the database should use. For performance reasons is strongly suggested to do parsing and filtering on Fluent Bit side, avoid pipelines.

`http_user`

Data type: Optional[String]

Optional username credential for Elastic X-Pack access

Default value: undef

`http_passwd`

Data type: Optional[String]

Password for user defined in HTTP_User

Default value: undef

`index`

Data type: String

Index name

Default value: 'fluentbit'

`type`

Data type: String

Type name

Default value: 'flb_type'

`logstash_format`

Enable Logstash format compatibility.

`logstash_prefix`

When Logstash_Format is enabled, the Index name is composed using a prefix and the date, e.g: If Logstash_Prefix is equals to 'mydata' your index will become 'mydata-YYYY.MM.DD'. The last string appended belongs to the date when the data is being generated.

`logstash_dateformat`

Time format (based on strftime) to generate the second part of the Index name.

`time_key`

When Logstash_Format is enabled, each record will get a new timestamp field. The Time_Key property defines the name of that field.

`time_key_format`

When Logstash_Format is enabled, this property defines the format of the timestamp.

`include_tag_key`

When enabled, it append the Tag name to the record.

`tag_key`

When Include_Tag_Key is enabled, this property defines the key name for the tag.

`generate_id`

When enabled, generate _id for outgoing records. This prevents duplicate records when retrying ES.

`replace_dots`

When enabled, replace field name dots with underscore, required by Elasticsearch 2.0-2.3.

`trace_output`

When enabled print the elasticsearch API calls to stdout (for diag only)

`current_time_index`

Use current time for index generation instead of message record

`logstash_prefix_key`

Prefix keys with this string

fluentbit::output::http

The http output plugin allows to flush your records into a HTTP endpoint. This type manages the configuration for it.

Examples

fluentbit::output::http { 'logstash': }

Parameters

The following parameters are available in the fluentbit::output::http defined type.

`configfile`

Data type: Stdlib::Absolutepath

Path to the output configfile. Naming should be output_*.conf to make sure it's getting included by the main config.

Default value: "/etc/td-agent-bit/output_http_${name}.conf"

`match`

Data type: Optional[String[1]]

Tag to route the output.

Default value: undef

`tls`

Data type: Fluentbit::TLS

TLS configuration. By default TLS is disabled.

Default value: {}

`headers`

Data type: Hash[String[1], String[1]]

Map of headers to add to requests.

Default value: {}

`retry_limit`

Data type: Variant[Undef, Boolean, Integer[1]]

Number of retries if upstream refuses the records. false will disable retries, true will cause it to retry once. All other values are passed on verbatim.

Default value: undef

`host`

Data type: Optional[Stdlib::Host]

IP address or hostname of the target HTTP Server.

Default value: undef

`port`

Data type: Optional[Stdlib::Port]

TCP port of the target HTTP Server.

Default value: undef

`http_user`

Data type: Optional[String[1]]

Basic Auth Username.

Default value: undef

`http_passwd`

Data type: Optional[String[1]]

Basic Auth Password. Requires HTTP_User to be set.

Default value: undef

`proxy`

Data type: Optional[Stdlib::HTTPUrl]

Specify an HTTP Proxy.

Default value: undef

`uri`

Data type: Optional[Stdlib::Absolutepath]

Specify an optional HTTP URI for the target web server.

Default value: undef

`format`

Data type: Optional[Enum['msgpack', 'json', 'json_stream', 'json_lines', 'gelf']]

Specify the data format to be used in the HTTP request body.

Default value: undef

`header_tag`

Data type: Optional[String[1]]

Specify an optional HTTP header field for the original message tag.

Default value: undef

`json_date_key`

Data type: Optional[String[1]]

Specify the name of the date field in output.

Default value: undef

`json_date_format`

Data type: Optional[Enum['double', 'iso8601']]

Specify the format of the date.

Default value: undef

`gelf_timestamp_key`

Data type: Optional[String[1]]

Specify the key to use for timestamp in gelf format.

Default value: undef

`gelf_host_key`

Data type: Optional[String[1]]

Specify the key to use for host in gelf format.

Default value: undef

`gelf_short_messge_key`

Data type: Optional[String[1]]

Specify the key to use for short in gelf format.

Default value: undef

`gelf_full_message_key`

Data type: Optional[String[1]]

Specify the key to use for full in gelf format.

Default value: undef

`gelf_level_key`

Data type: Optional[String[1]]

Specify the key to use for level in gelf format.

Default value: undef

License

helio/puppet-fluentbit

ErrorLooks like something went wrong!

About

Topics

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors 2

Languages

Error
Looks like something went wrong!

Packages