Dead Letter Queue Chunks are deleted on Fluent Bit restart

[optimistichare9]

Bug Report

Describe the bug
The .flb chunks created to the DLQ rejected folder are deleted when Fluent Bit is restarted.

To Reproduce

• Intentionally cause chunks to fail by using bad output configs so that chunks are sent to the rejected folder.
• Restart Fluent Bit
• I can see this in the output and the files are deleted

[error] [storage backlog] could not retrieve chunk tag from rejected/iis.log_0_opensearch.0_000001FED68F1500.flb, removing it from the queue

Expected behavior
The chunks should be replayed to avoid data loss.

Your Environment

• Version used: fluent-bit-4.2.2-win64
• Configuration:

[SERVICE]
    Flush        1
    Daemon       Off
    Log_Level    trace
    Parsers_File ..\parsers.conf
    storage.path **\buffer\
    storage.keep.rejected on
    storage.rejected.path rejected

[INPUT]
    Name              tail
    Path              ***\*.log
    Tag               iis.log
    Parser            iis-parser
    Path_Key          LogFile
    storage.type  filesystem

[OUTPUT]
    Name opensearch
    Match iis.log
    Host ***.amazonaws.com
    Port 443
    AWS_Auth On
    AWS_Region ca-central-1
    Index test-iis-logs
    tls On
    Suppress_Type_Name On
    Retry_Limit 2

• Operating System and version: Windows 11 24H2

Additional context
Trying to simulate a scenario where the output becomes unavailable - I want to use the DLQ to preserve failed chunks so they can be replayed later after the output is fixed.

I have also run https://github.com/chronosphereio/calyptia-chunk-inspector and the chunks seem OK.

go run . check -file /calyptia-chunk-inspector-main/iis.log_0_opensearch.0_000001AEFDAB01C0.flb
Filename /calyptia-chunk-inspector-main/iis.log_0_opensearch.0_000001AEFDAB01C0.flb OK

[lecaros]

@optimistichare9 thanks again for reporting this.
I'm adding reduced config and log files

config

pipeline:
  inputs:
    - name: dummy
      storage.type: filesystem
      tag: test
      samples: 3
  outputs:
    - name: http
      host: localhostsss
      port: 8080
      match: '*'
    - name: stdout
      match: '*'
service:
  flush: 1
  storage.path: ./storage
  storage.keep.rejected: on
  storage.rejected.path: rejected

logs

Fluent Bit v5.0.0
* Copyright (C) 2015-2025 The Fluent Bit Authors
* Fluent Bit is a CNCF graduated project under the Fluent organization
* https://fluentbit.io

______ _                  _    ______ _ _           _____  _____           _            
|  ___| |                | |   | ___ (_) |         |  ___||  _  |         | |           
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   _|___ \ | |/' |______ __| | _____   __
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / /   \ \|  /| |______/ _` |/ _ \ \ / /
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V //\__/ /\ |_/ /     | (_| |  __/\ V / 
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/ \____(_)\___/       \__,_|\___| \_/


[2026/01/22 13:18:36.318105000] [ info] [fluent bit] version=5.0.0, commit=6a6bce1555, pid=41561
[2026/01/22 13:18:36.319842000] [ info] [storage] ver=1.1.6, type=memory+filesystem, sync=normal, checksum=off, max_chunks_up=128
[2026/01/22 13:18:36.319850000] [ info] [storage] backlog input plugin: storage_backlog.1
[2026/01/22 13:18:36.319855000] [ info] [simd    ] disabled
[2026/01/22 13:18:36.319859000] [ info] [cmetrics] version=1.0.6
[2026/01/22 13:18:36.319885000] [ info] [ctraces ] version=0.6.6
[2026/01/22 13:18:36.319990000] [ info] [input:dummy:dummy.0] initializing
[2026/01/22 13:18:36.319994000] [ info] [input:dummy:dummy.0] storage_strategy='filesystem' (memory + filesystem)
[2026/01/22 13:18:36.320055000] [ info] [input:storage_backlog:storage_backlog.1] initializing
[2026/01/22 13:18:36.320058000] [ info] [input:storage_backlog:storage_backlog.1] storage_strategy='memory' (memory only)
[2026/01/22 13:18:36.320077000] [ info] [input:storage_backlog:storage_backlog.1] queue memory limit: 95.4M
[2026/01/22 13:18:36.320197000] [ info] [output:http:http.0] worker #0 started
[2026/01/22 13:18:36.320210000] [ info] [output:http:http.0] worker #1 started
[2026/01/22 13:18:36.320259000] [ info] [output:stdout:stdout.1] worker #0 started
[2026/01/22 13:18:36.320279000] [ info] [sp] stream processor started
[2026/01/22 13:18:36.320356000] [error] [storage backlog] could not retrieve chunk tag from rejected/test_0_http.0_0x106d24da0.flb, removing it from the queue
[2026/01/22 13:18:36.320620000] [error] [storage backlog] could not retrieve chunk tag from rejected/test_0_http.0_0x106d1aca0.flb, removing it from the queue
[2026/01/22 13:18:36.320796000] [error] [storage backlog] could not retrieve chunk tag from rejected/test_0_http.0_0x823804000.flb, removing it from the queue
[2026/01/22 13:18:36.320912000] [ info] [engine] Shutdown Grace Period=5, Shutdown Input Grace Period=2
[0] test: [[1769098717.322403000, {}], {"message"=>"dummy"}]
[2026/01/22 13:18:38.374604000] [ warn] [net] getaddrinfo(host='localhostsss', err=4): Domain name not found
[2026/01/22 13:18:38.374682000] [error] [output:http:http.0] no upstream connections available to localhostsss:8080
[2026/01/22 13:18:38.374746000] [ warn] [engine] failed to flush chunk '41561-1769098717.322575000.flb', retry in 11 seconds: task_id=0, input=dummy.0 > output=http.0 (out_id=0)
^C[2026/01/22 13:18:38] [engine] caught signal (SIGINT)
[2026/01/22 13:18:38.732748000] [ info] [input] pausing dummy.0
[2026/01/22 13:18:38.732842000] [ info] [input] pausing storage_backlog.1
[2026/01/22 13:18:38.732876000] [ info] [output:http:http.0] thread worker #0 stopping...
[2026/01/22 13:18:38.732939000] [ info] [output:http:http.0] thread worker #0 stopped
[2026/01/22 13:18:38.733116000] [ info] [output:http:http.0] thread worker #1 stopping...
[2026/01/22 13:18:38.733256000] [ info] [output:http:http.0] thread worker #1 stopped
[2026/01/22 13:18:38.733377000] [ info] [output:stdout:stdout.1] thread worker #0 stopping...
[2026/01/22 13:18:38.733426000] [ info] [output:stdout:stdout.1] thread worker #0 stopped

This config should be run twice. It will only generate 3 messages/chunks per run.

After

[2026/01/22 13:18:36.320356000] [error] [storage backlog] could not retrieve chunk tag from rejected/test_0_http.0_0x106d24da0.flb, removing it from the queue

the files are deleted.

[lecaros]

There's a fix for this in #11389. Testing is appreciated :)

[optimistichare9]

There's a fix for this in #11389. Testing is appreciated :)

Thanks @lecaros ! I tested #11389 and it seems fixed to me. The .flb files are no longer deleted and I can see
[2026/01/23 11:37:58.616230507] [debug] [storage backlog] skipping DLQ stream 'rejected'