Skip to content

in_exec with out_exec in one config create file descriptors leak #3627

@LukasJerabek

Description

@LukasJerabek

Describe the bug

Hello, I think I have found the core issue of my problems with fluentd configuration, could someone have look on this please?
I have set everything according to https://docs.fluentd.org/installation/before-install.
Using configuration and two scripts below I can notice that if I do these for a while:

cd /proc/<fluentd PID>/ 
find fd | wc -l

Then I can see that number of file descriptors is rising infinitely eventually resulting in memory problems and "too many open files" error because of reaching fd limit (ulimit -n). This only occures if both in_exec and out_exec are used, since if I do following swaps, the number of file descriptors stabilizes. I can do either of these and the problems will disappear.

File descriptors count is stabilized if I swap match section for this:

<match hardware.memory>
  @type file
  append true
  path {{ home }}/fluentd/test_bug2.log
  add_path_suffix false
  <buffer []>
    @type memory
    flush_mode immediate
  </buffer>
  <format>
    @type json
  </format>
</match>

or swap source and filter section for this:

<source>
  @type sample
  sample {"mem_available":"1.5","swap_total":"1.5","swap_free":"1.2","time_local":"2022-02-09T10:00:00+0100"}
  tag hardware.memory
</source>

To Reproduce

Install fluentd:
gem install fluentd --version 1.14.5 --no-doc
Create folder "fluentd" in home folder
Exchange {{ home }} or {{ user }} for your real values in every file below.
Put fluentd.conf (with configuration below), get_memory_metrics.sh, psql_memory.sh into {{ home }}/fluentd/
do chmod +x on get_memory_metrics.sh, psql_memory.sh
run {{ home }}/bin/fluentd 1.14.5 -c {{ home }}/fluentd/fluentd.conf -vv

Now start checking the file descriptors in /proc/<fluentd PID>/ with:
find fd | wc -l
Keep cheking it for some time, youll see the count of file descriptors never stops rising.

Hopefully I did not miss anything.

Expected behavior

Number of file descriptors is not rising infinitely and not causing memory leaks.

Your Environment

- Fluentd version: 1.14.5 (installed as gem)
- TD Agent version: -
- Operating system: Rocky Linux 8
- Kernel version: 4.18.0-348.12.2.el8_5.x86_64
- Ruby version: 2.5.9

Your Configuration

<system>
  process_name fluentd_monitor
</system>

<source>
  @type exec
  tag hardware.memory
  run_interval 10s
  command {{ home }}/fluentd/get_memory_metrics.sh
  <parse>
    @type json
    types mem_available:float,swap_total:float,swap_free:float
  </parse>
</source>

<filter hardware.memory>
  @type record_transformer
  enable_ruby
  <record>
    time_local ${time.strftime('%Y-%m-%dT%H:%M:%S%z')}
  </record>
</filter>

<match hardware.memory>
  @type exec
  command {{ home }}/fluentd/psql_memory.sh first_argument second_argument
  <format>
    @type csv
    force_quotes false
    delimiter |
    fields time_local,mem_available,swap_total,swap_free
  </format>
  <buffer time>
    @type file
    path {{ home }}/fluentd/hardware.memory.buff
    timekey 50s
    timekey_wait 0s
    flush_mode lazy
    total_limit_size 200MB
    retry_type periodic  # exponential backoff is broken https://github.com/fluent/fluentd/issues/3609
    retry_max_times 0  # need to go to secondary as soon as possible https://github.com/fluent/fluentd/issues/3610
  </buffer>
  <secondary>
    @type secondary_file
    directory {{ home }}/fluentd/secondary/
    basename dump.${chunk_id}
  </secondary>
</match>

Your Error Log

No errors in fluentd.log using --v

Additional context

get_memory_metrics.sh

#!/bin/bash
mem_info=$(awk '{ \
    if (/MemAvailable:/) {mem_available=$2}; \
    if (/SwapTotal:/) {swap_total=$2}; \
    if (/SwapFree:/) {swap_free=$2}; \
    if (mem_available && swap_total && swap_free) \
      {printf "%.3f %.3f %.3f", mem_available/1024/1024, swap_total/1024/1024, swap_free/1024/1024;exit} \
  }' /proc/meminfo)
read -r mem_available swap_total swap_free<<<"${mem_info}"

printf '{"mem_available":"%s","swap_total":"%s","swap_free":"%s"}' \
"$mem_available" \
"$swap_total" \
"$swap_free"

psql_memory.sh

#!/bin/sh
echo $1 $2 $3

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingmemory

    Type

    No type

    Projects

    • Status

      Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions