Skip to content

tcp-and-tls.md

Latest commit

 

History

History
233 lines (173 loc) · 7.48 KB

tcp-and-tls.md

File metadata and controls

233 lines (173 loc) · 7.48 KB
Raw

TCP and TLS

The TCP and TLS output plugin lets you send records to a remote TCP server. The payload can be formatted in different ways as required.

Configuration parameters

This plugin supports the following parameters:

Key Description Default
format Specify the data format to be printed. Supported formats: msgpack, json, json_lines, json_stream. msgpack
host Target host where Fluent Bit or Fluentd are listening for Forward messages. 127.0.0.1
json_date_format Specify the format of the date. Supported formats: double, epoch, epoch_ms, iso8601, java_sql_timestamp. double
json_date_key Specify the name of the time key in the output record. To disable the time key, set the value to false. date
port TCP port of the target service. 5170
raw_message_key If set, the value of this key is sent as the raw message payload without additional formatting. none
workers The number of workers to perform flush operations for this output. 2

TLS configuration parameters

The following parameters are available to configure a secure channel connection through TLS:

Key Description Default
tls Enable or disable TLS support. off
tls.verify Force certificate validation. on
tls.debug Set TLS debug verbosity level. Allowed values: 0 (No debug), 1 (Error), 2 (State change), 3 (Informational), 4 (Verbose) 1
tls.ca_file Absolute path to CA certificate file. none
tls.crt_file Absolute path to Certificate file. none
tls.key_file Absolute path to private Key file. none
tls.key_passwd Optional password for tls.key_file file. none

Getting started

Command line

JSON format

This example specifies gathering CPU usage metrics and send them in JSON lines mode to a remote endpoint using netcat service.

fluent-bit -i cpu -o tcp://127.0.0.1:5170 -p format=json_lines -v

Run the following in a separate terminal, netcat will start listening for messages on TCP port 5170. After it connects to Fluent Bit you should see the output in JSON format:

nc -l 5170

Which returns results similar to:

...
{"date":1644834856.905985,"cpu_p":1.1875,"user_p":0.5625,"system_p":0.625,"cpu0.p_cpu":0.0,"cpu0.p_user":0.0,"cpu0.p_system":0.0,"cpu1.p_cpu":1.0,"cpu1.p_user":1.0,"cpu1.p_system":0.0,"cpu2.p_cpu":4.0,"cpu2.p_user":2.0,"cpu2.p_system":2.0,"cpu3.p_cpu":1.0,"cpu3.p_user":0.0,"cpu3.p_system":1.0,"cpu4.p_cpu":1.0,"cpu4.p_user":0.0,"cpu4.p_system":1.0,"cpu5.p_cpu":1.0,"cpu5.p_user":1.0,"cpu5.p_system":0.0,"cpu6.p_cpu":0.0,"cpu6.p_user":0.0,"cpu6.p_system":0.0,"cpu7.p_cpu":3.0,"cpu7.p_user":1.0,"cpu7.p_system":2.0,"cpu8.p_cpu":0.0,"cpu8.p_user":0.0,"cpu8.p_system":0.0,"cpu9.p_cpu":1.0,"cpu9.p_user":0.0,"cpu9.p_system":1.0,"cpu10.p_cpu":1.0,"cpu10.p_user":0.0,"cpu10.p_system":1.0,"cpu11.p_cpu":0.0,"cpu11.p_user":0.0,"cpu11.p_system":0.0,"cpu12.p_cpu":0.0,"cpu12.p_user":0.0,"cpu12.p_system":0.0,"cpu13.p_cpu":3.0,"cpu13.p_user":2.0,"cpu13.p_system":1.0,"cpu14.p_cpu":1.0,"cpu14.p_user":1.0,"cpu14.p_system":0.0,"cpu15.p_cpu":0.0,"cpu15.p_user":0.0,"cpu15.p_system":0.0}
...

msgpack format

Repeat the JSON approach but using the msgpack output format.

fluent-bit -i cpu -o tcp://127.0.0.1:5170 -p format=msgpack -v

You could send this to stdout but as it's a serialized format you would end up with strange output.

This should be handled by a msgpack receiver to unpack. The following example uses the Python msgpack library handle it:

#Python3
import socket
import msgpack

unpacker = msgpack.Unpacker(use_list=False, raw=False)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(("127.0.0.1", 5170))
s.listen(1)
connection, address = s.accept()

while True:
    data = connection.recv(1024)
    if not data:
        break
    unpacker.feed(data)
    for unpacked in unpacker:
        print(unpacked)

Run the following commands:

 pip install msgpack
 python3 test.py

Which return results like:

...
(ExtType(code=0, data=b'b\n5\xc65\x05\x14\xac'), {'cpu_p': 0.1875, 'user_p': 0.125, 'system_p': 0.0625, 'cpu0.p_cpu': 0.0, 'cpu0.p_user': 0.0, 'cpu0.p_system': 0.0, 'cpu1.p_cpu': 0.0, 'cpu1.p_user': 0.0, 'cpu1.p_system': 0.0, 'cpu2.p_cpu': 1.0, 'cpu2.p_user': 0.0, 'cpu2.p_system': 1.0, 'cpu3.p_cpu': 0.0, 'cpu3.p_user': 0.0, 'cpu3.p_system': 0.0, 'cpu4.p_cpu': 0.0, 'cpu4.p_user': 0.0, 'cpu4.p_system': 0.0, 'cpu5.p_cpu': 0.0, 'cpu5.p_user': 0.0, 'cpu5.p_system': 0.0, 'cpu6.p_cpu': 0.0, 'cpu6.p_user': 0.0, 'cpu6.p_system': 0.0, 'cpu7.p_cpu': 0.0, 'cpu7.p_user': 0.0, 'cpu7.p_system': 0.0, 'cpu8.p_cpu': 0.0, 'cpu8.p_user': 0.0, 'cpu8.p_system': 0.0, 'cpu9.p_cpu': 1.0, 'cpu9.p_user': 1.0, 'cpu9.p_system': 0.0, 'cpu10.p_cpu': 0.0, 'cpu10.p_user': 0.0, 'cpu10.p_system': 0.0, 'cpu11.p_cpu': 0.0, 'cpu11.p_user': 0.0, 'cpu11.p_system': 0.0, 'cpu12.p_cpu': 0.0, 'cpu12.p_user': 0.0, 'cpu12.p_system': 0.0, 'cpu13.p_cpu': 0.0, 'cpu13.p_user': 0.0, 'cpu13.p_system': 0.0, 'cpu14.p_cpu': 0.0, 'cpu14.p_user': 0.0, 'cpu14.p_system': 0.0, 'cpu15.p_cpu': 0.0, 'cpu15.p_user': 0.0, 'cpu15.p_system': 0.0})
...

Configuration file

The following is a basic setup to forward logs over TCP:

{% tabs %} {% tab title="fluent-bit.yaml" %}

pipeline:
  inputs:
    - name: cpu
      tag: cpu

  outputs:
    - name: tcp
      match: '*'
      host: 192.168.10.12
      port: 5170
      format: json_lines

{% endtab %} {% tab title="fluent-bit.conf" %}

[INPUT]
  Name cpu
  Tag  cpu

[OUTPUT]
  Name  tcp
  Match *
  Host  192.168.10.12
  Port  5170
  Format json_lines

{% endtab %} {% endtabs %}

Examples

Raw message key

When raw_message_key is set, the plugin sends the value of the specified key as a raw message instead of applying formatting, causing the format property to be ignored:

{% tabs %} {% tab title="fluent-bit.yaml" %}

pipeline:
  inputs:
    - name: tail
      path: /var/log/app.log
      tag: app

  outputs:
    - name: tcp
      match: '*'
      host: 127.0.0.1
      port: 5170
      raw_message_key: $message

{% endtab %} {% tab title="fluent-bit.conf" %}

[INPUT]
  Name tail
  Path /var/log/app.log
  Tag  app

[OUTPUT]
  Name           tcp
  Match          *
  Host           127.0.0.1
  Port           5170
  Raw_Message_Key $message

{% endtab %} {% endtabs %}

The $ sign is required to signal the record accessor to extract the value from the specified key - this can be a top-level or nested field:

{% tabs %} {% tab title="Top-level field" %}

Example log:

{
  "timestamp": "2026-07-05T14:32:07Z",
  "message": "<34>Jul  5 14:32:07 myhost app: Cache key S2VlcCBjYWxtIGFuZCBsb2cgb24="
}

Setting raw_message_key to $message will yield:

<34>Jul  5 14:32:07 myhost app: Cache key S2VlcCBjYWxtIGFuZCBsb2cgb24=

{% endtab %} {% tab title="Nested field" %}

Example log:

{
  "timestamp": "2026-07-05T14:32:07Z",
  "message": {
    "original": "<34>Jul  5 14:32:07 myhost app: Cache key S2VlcCBjYWxtIGFuZCBsb2cgb24="
  }
}

Setting raw_message_key to $message['original'] will yield:

<34>Jul  5 14:32:07 myhost app: Cache key S2VlcCBjYWxtIGFuZCBsb2cgb24=

{% endtab %} {% endtabs %}