prometheus_remote_write drops metrics older than 1 hour

[asg7443]

Bug Report

Describe the bug
prometheus_remote_write output plugin drops metrics older than 1 hour

To Reproduce

• Create directories

mkdir -p /srv/logshipper/{conf,data,log}

• Create config which enables file system storage and simulates an unreachable server

---
# /srv/logshipper/conf/fluent-bit.yaml
service:
  log_level: trace
  log_file: /fluent-bit/log/main.log
  flush: 300
  storage.path: /fluent-bit/data/storage
  scheduler.base: 60
  scheduler.cap: 3600

pipeline:
  inputs:
    - name: node_exporter_metrics
      tag:  node_metrics
      metrics: "loadavg"
      scrape_interval: 60
      storage.type: filesystem

  outputs:
    - name: prometheus_remote_write
      match: node_metrics
      host: unreachable.test

      storage.total_limit_size: 100M
      retry_limit: 1000

• Run fluent-bit
  podman run --rm -d -v /srv/logshipper/conf:/fluent-bit/etc:ro -v /srv/logshipper/data:/fluent-bit/data:rw -v /srv/logshipper/log:/fluent-bit/log:rw docker.io/fluent/fluent-bit:4.2.2 -c /fluent-bit/etc/fluent-bit.yaml

• Wait for 1.5-2 hours

• See the chunk files being deleted from /srv/logshipper/data with messages in /srv/logshipper/log/main.log like these:

[2026/01/27 11:16:45.224905057] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] task_id=1 assigned to thread #1
[2026/01/27 11:16:45.224971777] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] cmetrics msgpack size: 2110
[2026/01/27 11:16:45.225003737] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] cmetric_id=0 decoded 0-422 payload_size=0
[2026/01/27 11:16:45.225012738] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] cmetric_id=1 decoded 422-844 payload_size=0
[2026/01/27 11:16:45.225024378] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] cmetric_id=2 decoded 844-1266 payload_size=0
[2026/01/27 11:16:45.225031738] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] cmetric_id=3 decoded 1266-1688 payload_size=0
[2026/01/27 11:16:45.225040218] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] cmetric_id=4 decoded 1688-2110 payload_size=0
[2026/01/27 11:16:45.225042938] [debug] [output:prometheus_remote_write:prometheus_remote_write.0] final payload size: 0
[2026/01/27 11:16:45.225058978] [debug] [out flush] cb_destroy coro_id=43
[2026/01/27 11:16:45.225092898] [ info] [engine] flush chunk '1-1769507997.474870989.flb' succeeded at retry 7: task_id=1, input=node_exporter_metrics.0 > output=prometheus_remote_write.0 (out_id=0)
[2026/01/27 11:16:45.225102658] [debug] [task] destroy task=0xfc3a1fa23960 (task_id=1)
[2026/01/27 11:16:45.225109058] [debug] [input chunk] remove chunk 1-1769507997.474870989.flb with 4096 bytes from plugin prometheus_remote_write.0, the updated fs_chunks_size is 65536 bytes

Expected behavior
Chunk files are not deleted and being kept for as long as it's configured with storage.total_limit_size and retry_limit parameters

Your Environment

• Version used: 4.2.2, official Docker image

Additional context
It makes it impossible to use Fluent Bit as a temporary buffer in the event of a prolonged server outage. Old metrics in this scenario are lost.

[asg7443]

Prometheus Remote-Write specification:

Backfill The specification does not place a limit on how old series can be pushed, however server/implementation specific constraints may exist.

Prometheus has a parameter to set such a limit, but it looks like by default it's disabled:

# Configures how old an out-of-order/out-of-bounds sample can be w.r.t. the TSDB max time.
# An out-of-order/out-of-bounds sample is ingested into the TSDB as long as the timestamp
# of the sample is >= TSDB.MaxTime-out_of_order_time_window.
#
# When out_of_order_time_window is >0, the errors out-of-order and out-of-bounds are
# combined into a single error called 'too-old'; a sample is either (a) ingestible
# into the TSDB, i.e. it is an in-order sample or an out-of-order/out-of-bounds sample
# that is within the out-of-order window, or (b) too-old, i.e. not in-order
# and before the out-of-order window.
#
# When out_of_order_time_window is greater than 0, it also affects experimental agent. It allows
# the agent's WAL to accept out-of-order samples that fall within the specified time window relative
# to the timestamp of the last appended sample for the same series.
[ out_of_order_time_window: [<duration>](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#duration) | default = 0s ]

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#tsdb

Maybe it would make sense to remove this restriction in prometheus_remote_write

#11082 is probably a related issue.