Fluentd buffer is flushed constantly duplicating the record

[yukuraga]

Describe the bug

Hello,

We are using fluentd as our central log collector component of our project in a containerized fashion. Basically fluentd collects logs from various input sources and outputs them to our postgresql database via postgres output plugin.

Recently, we came across a situation where a buffer file is flushed over and over and never gets expired resulting quite number of duplicated log records in the database (for a case, two lines of logs reached 30000+ duplicated records).

To Reproduce

N/A (System is being tested and at it's not clear which point exactly it came into this situation)

Expected behavior

Flushed buffer should be expired.

Your Environment

- Fluentd version: (Docker image) fluent/fluentd:v1.16.6-debian-1.0

Your Configuration

Buffer section of postgres output plugin:

    <buffer time>
      @type file
      path /fluentd/buffer/sinecnsp_diagnostic_event
      flush_thread_count 8
      total_limit_size 16GB
      flush_mode interval
      flush_interval 15s
      retry_max_times 8
      timekey 300
      timekey_wait 30
      timekey_use_utc true
      flush_at_shutdown true
    </buffer>

Your Error Log

No error log

Additional context

We backed up the mentioned .log file and .log.meta file and if I put them at the buffer path of different setup, issue arises on the new setup also.

For the sake of info security, we cannot share the .log file itself, if it's going to help, I can share the .log.meta file.

I shared only buffer section of postgres output plugin because our configuration is split into quite number of different configuration files and I think only this buffer section is relevant. If you need, I can share from other config files also.

[daipom]

@yukuraga Thanks for this report!
I moved this issue to Discussions since it is difficult to reproduce this issue and judge if this is a bug or not with the current information.

We need to investigate the cause of the problem.
It would be great to know the general reproduction steps of this behavior.

I will make some comments later.

[daipom]

@yukuraga
Is the problem limited to specific buffer files? (Do new such buffer files keep appearing?)

[yukuraga]

As per our configuration, buffer is organized as file buffer with timekey, so it generates new buffer files as logs continue to come.

It's like this,

When you go into docker container via docker exec -it and you go to buffer path, you run watch ls -al command at the path; you see the "problematic" .log and .log.meta files are there, as per its timekey period new buffer files are created and when the new ones expire they are deleted but this problematic pair stays. I can try to create a GIF of this scenario.

[yukuraga]

This is the binary content of .log.meta file.

[yukuraga]

After activating debug logs, I see this:

I marked the chunk id and as you can see from previous binary content share, it is exactly the buffer we are facing problem.

Does this suggest a possible corruption of buffer or buffer meta files?
How come I see error log from postgres output plugin but actually it is inserted into database?

[Watson1978]

Can you attach raw logs instead of screenshots?

[daipom]

Looks like some flush error occur.
This would be the cause.