Skip to content

file.md

Latest commit

 

History

History
175 lines (130 loc) · 5.52 KB

file.md

File metadata and controls

175 lines (130 loc) · 5.52 KB
Raw

file

The file buffer plugin provides a persistent buffer implementation. It uses files to store buffer chunks on disk.

{% hint style='info' %} In the following examples, it assumes pre-configured chunk keys by default with <buffer>. If you want to understand buffer behavior precisely, see Buffer Section Configuration documentation. {% endhint %}

Parameters

path

type default version
string nil 0.9.0

The directory path where buffer chunks are stored. Don't share this directory path with other buffers. Be sure to specify a unique path for each buffer.

<match pattern>
  ...
  <buffer>
    @type file
    path /var/log/fluent/buf
  </buffer>
</match>

This config outputs the buffer chunk files as follows. The file name is buffer.b{chunk_id}{path_suffix}.

/var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.log
/var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.log.meta

With multiple workers, a directory is automatically created for each worker. So there is no need to specify a unique path for each worker.

<system>
  workers 2
</system>

...

<match pattern>
  ...
  <buffer>
    @type file
    path /var/log/fluent/buf
  </buffer>
</match>

This config outputs the buffer chunk files as follows. The directory worker{worker_id} is automatically created.

/var/log/fluentd/buf/worker0/buffer.b58eec11d08ca8143b40e4d303510e0bb.log
/var/log/fluentd/buf/worker0/buffer.b58eec11d08ca8143b40e4d303510e0bb.log.meta

/var/log/fluentd/buf/worker1/buffer.b5e2a5aca2bcd9818ad6718845ddc456a.log
/var/log/fluentd/buf/worker1/buffer.b5e2a5aca2bcd9818ad6718845ddc456a.log.meta

If you specify root_dir in system configuration and @id of the plugin, then you can omit this parameter.

<system>
  root_dir /var/log/fluentd
</system>

...

<match pattern>
  @id test_id
  ...
  <buffer>
    @type file
  </buffer>
</match>

This config outputs the buffer chunk files as follows. The directory {root_dir}/worker{worker_id}/{@id}/buffer is used for the path. In this case, the worker{worker_id} directory is created even for a single worker.

/var/log/fluentd/worker0/test_id/buffer/buffer.b58eec11d08ca8143b40e4d303510e0bb.log
/var/log/fluentd/worker0/test_id/buffer/buffer.b58eec11d08ca8143b40e4d303510e0bb.log.meta

Please make sure that you have enough space in the path directory. Running out of disk space is a problem frequently reported by users.

path_suffix

type default version
string .log 1.6.3

Changes the suffix of the buffer file.

# default 
/var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.log
/var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.log.meta

# with 'path_suffix .buf'
/var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.buf
/var/log/fluentd/buf/buffer.b58eec11d08ca8143b40e4d303510e0bb.buf.meta

This parameter is useful when .log is not fit for your environment. See also this issue's comment.

Tips

Customize a filename of the buffer chunk

You can customize the prefix of filename (buffer by default) by adding .* to the end of the path parameter.

<match pattern>
  ...
  <buffer>
    @type file
    path /var/log/fluent/buf/custom.*
  </buffer>
</match>

This config outputs the buffer chunk files as follows. The prefix buffer is changed to custom.

/var/log/fluentd/buf/custom.b58eec11d08ca8143b40e4d303510e0bb.log
/var/log/fluentd/buf/custom.b58eec11d08ca8143b40e4d303510e0bb.log.meta

You can also customize the entire filename by adding .*. to the path parameter.

<match pattern>
  ...
  <buffer>
    @type file
    path /var/log/fluent/buf/custom_prefix.*.custom_suffix
  </buffer>
</match>

This config outputs the buffer chunk files as follows. In this case, path_suffix parameter is not used.

/var/log/fluentd/buf/custom_prefix.b58eec11d08ca8143b40e4d303510e0bb.custom_suffix
/var/log/fluentd/buf/custom_prefix.b58eec11d08ca8143b40e4d303510e0bb.custom_suffix.meta

Limitation

{% hint style='warning' %} Caution: file buffer implementation depends on the characteristics of the local file system. Don't use file buffer on remote file systems e.g. NFS, GlusterFS, HDFS, etc. We observed major data loss by using the remote file system. {% endhint %}

{% hint style='warning' %} path is implicitly used as file buffer path if file buffer path is not explicitly configured in <buffer> section. In such a case, it must not contain [ or ] in the actually evaluated path. If such a character was included in the path, it might be trouble (ignored) after restarting of Fluentd during resuming buffer handling process. As a workaround, override path parameter in <buffer> section which does not contain such a character. {% endhint %}

If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is an open-source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.