Description
Describe the bug
Hello, I think I have found the core issue of my problems with fluentd configuration, could someone have look on this please?
I have set everything according to https://docs.fluentd.org/installation/before-install.
Using configuration and two scripts below I can notice that if I do these for a while:
cd /proc/<fluentd PID>/
find fd | wc -l
Then I can see that number of file descriptors is rising infinitely eventually resulting in memory problems and "too many open files" error because of reaching fd limit (ulimit -n). This only occures if both in_exec and out_exec are used, since if I do following swaps, the number of file descriptors stabilizes. I can do either of these and the problems will disappear.
File descriptors count is stabilized if I swap match section for this:
<match hardware.memory>
@type file
append true
path {{ home }}/fluentd/test_bug2.log
add_path_suffix false
<buffer []>
@type memory
flush_mode immediate
</buffer>
<format>
@type json
</format>
</match>
or swap source and filter section for this:
<source>
@type sample
sample {"mem_available":"1.5","swap_total":"1.5","swap_free":"1.2","time_local":"2022-02-09T10:00:00+0100"}
tag hardware.memory
</source>
To Reproduce
Install fluentd:
gem install fluentd --version 1.14.5 --no-doc
Create folder "fluentd" in home folder
Exchange {{ home }} or {{ user }} for your real values in every file below.
Put fluentd.conf (with configuration below), get_memory_metrics.sh, psql_memory.sh into {{ home }}/fluentd/
do chmod +x on get_memory_metrics.sh, psql_memory.sh
run {{ home }}/bin/fluentd 1.14.5 -c {{ home }}/fluentd/fluentd.conf -vv
Now start checking the file descriptors in /proc/<fluentd PID>/
with:
find fd | wc -l
Keep cheking it for some time, youll see the count of file descriptors never stops rising.
Hopefully I did not miss anything.
Expected behavior
Number of file descriptors is not rising infinitely and not causing memory leaks.
Your Environment
- Fluentd version: 1.14.5 (installed as gem)
- TD Agent version: -
- Operating system: Rocky Linux 8
- Kernel version: 4.18.0-348.12.2.el8_5.x86_64
- Ruby version: 2.5.9
Your Configuration
<system>
process_name fluentd_monitor
</system>
<source>
@type exec
tag hardware.memory
run_interval 10s
command {{ home }}/fluentd/get_memory_metrics.sh
<parse>
@type json
types mem_available:float,swap_total:float,swap_free:float
</parse>
</source>
<filter hardware.memory>
@type record_transformer
enable_ruby
<record>
time_local ${time.strftime('%Y-%m-%dT%H:%M:%S%z')}
</record>
</filter>
<match hardware.memory>
@type exec
command {{ home }}/fluentd/psql_memory.sh first_argument second_argument
<format>
@type csv
force_quotes false
delimiter |
fields time_local,mem_available,swap_total,swap_free
</format>
<buffer time>
@type file
path {{ home }}/fluentd/hardware.memory.buff
timekey 50s
timekey_wait 0s
flush_mode lazy
total_limit_size 200MB
retry_type periodic # exponential backoff is broken https://github.com/fluent/fluentd/issues/3609
retry_max_times 0 # need to go to secondary as soon as possible https://github.com/fluent/fluentd/issues/3610
</buffer>
<secondary>
@type secondary_file
directory {{ home }}/fluentd/secondary/
basename dump.${chunk_id}
</secondary>
</match>
Your Error Log
No errors in fluentd.log using --v
Additional context
get_memory_metrics.sh
#!/bin/bash
mem_info=$(awk '{ \
if (/MemAvailable:/) {mem_available=$2}; \
if (/SwapTotal:/) {swap_total=$2}; \
if (/SwapFree:/) {swap_free=$2}; \
if (mem_available && swap_total && swap_free) \
{printf "%.3f %.3f %.3f", mem_available/1024/1024, swap_total/1024/1024, swap_free/1024/1024;exit} \
}' /proc/meminfo)
read -r mem_available swap_total swap_free<<<"${mem_info}"
printf '{"mem_available":"%s","swap_total":"%s","swap_free":"%s"}' \
"$mem_available" \
"$swap_total" \
"$swap_free"
psql_memory.sh
#!/bin/sh
echo $1 $2 $3
Metadata
Metadata
Assignees
Labels
Type
Projects
Status
Done