Skip to content

File exporter rotation does not handle file size limits properly when using datetime patterns #70

@marcosnav

Description

@marcosnav

The OTEL_EXPORTER_OTLP_TRACES_FILE variable allows to set a pattern to something like traces-%Y-%m-%d-%T.jsonl, resulting in traces files named as traces-2026-01-31-12-00-00.jsonl.

The problem:

With OTEL_EXPORTER_OTLP_TRACES_FILE_ROTATE_SIZE and OTEL_EXPORTER_OTLP_TRACES_FILE_FILE_SIZE set. Every time the flush interval kicks, and traces are written to disk, a new file is created with the current time and does not wait for the file to reach the configured size limit. So, as an example, if the total rotation files limit is set to 5 and the file size limit is set to 50MB, it would be expected to have an end result of files like the following example, where backup files are written until the latest file is close to hit the 50MB:

-rw-------   1 mn  staff     49999364 Jan  31 10:01 trace-2026-01-31-10:01:03.jsonl
-rw-------   1 mn  staff     49998765 Feb   1 15:56 trace-2026-02-01-15:56:46.jsonl
-rw-------   1 mn  staff     49999023 Feb   2 01:23 trace-2026-02-02-01:23:21.jsonl
-rw-------   1 mn  staff     49998994 Feb   4 23:38 trace-2026-02-04-23:38:55.jsonl
-rw-------   1 mn  staff     49998846 Feb   5 12:56 trace-2026-02-05-12:56:01.jsonl
-rw-------   2 mn  staff       568965 Feb   5 16:13 trace-2026-02-05-latest.jsonl

BUT! we are getting this, all backup files allocated with just a small amount of traces data:

-rw-------   1 mn  staff     9680 Feb  5 16:13 trace-2026-02-05-22:13:52.jsonl
-rw-------   1 mn  staff     8248 Feb  5 16:13 trace-2026-02-05-22:13:53.jsonl
-rw-------   1 mn  staff     9680 Feb  5 16:13 trace-2026-02-05-22:13:54.jsonl
-rw-------   1 mn  staff     9680 Feb  5 16:13 trace-2026-02-05-22:13:55.jsonl
-rw-------   1 mn  staff     8192 Feb  5 16:13 trace-2026-02-05-22:13:56.jsonl
-rw-------   2 mn  staff     8192 Feb  5 16:13 trace-2026-02-05-latest.jsonl

Workaround

For file exporter to respect the file size and backup count limits, I found that using a plain %N index (`traces-%N.jsonl) results in something functional like this:

-rw-------   1 mn  staff          979 Feb   5 16:13 job.stderr
-rw-------   1 mn  staff         2134 Feb   5 16:13 job.stdout
-rw-------   1 mn  staff     49999364 Jan  31 10:01 trace-0.jsonl
-rw-------   1 mn  staff     49998765 Feb   1 15:56 trace-1.jsonl
-rw-------   1 mn  staff     49999023 Feb   2 01:23 trace-2.jsonl
-rw-------   1 mn  staff     49998994 Feb   4 23:38 trace-3.jsonl
-rw-------   1 mn  staff     49998846 Feb   5 12:56 trace-4.jsonl
-rw-------   2 mn  staff       568965 Feb   5 16:13 trace-5.jsonl
-rw-------   2 mn  staff       568965 Feb   5 16:13 trace-latest.jsonl

Still, it is a bummer that we cannot use date formats for rotation backups.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions