fluentbit unable to start up intermittently

[michaelzanggl]

Bug Report

Describe the bug
fluentbit is unable to start up properly intermittently.

All we see in the logs is

Fluent Bit v3.2.8
* Copyright (C) 2015-2025 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io
______ _ _ ______ _ _ _____ _____
| ___| | | | | ___ (_) | |____ |/ __ \
| |_ | |_ _ ___ _ __ | |_ | |_/ /_| |_ __ __ / /`' / /'
| _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / \ \ / /
| | | | |_| | __/ | | | |_ | |_/ / | |_ \ V /.___/ /./ /___
\_| |_|\__,_|\___|_| |_|\__| \____/|_|\__| \_/ \____(_)_____/
[2025/03/13 00:30:42] [ info] Configuration:
[2025/03/13 00:30:42] [ info] flush time | 1.000000 seconds
[2025/03/13 00:30:42] [ info] grace | 5 seconds
[2025/03/13 00:30:42] [ info] daemon | 0
[2025/03/13 00:30:42] [ info] ___________
[2025/03/13 00:30:42] [ info] inputs:
[2025/03/13 00:30:42] [ info] tail
[2025/03/13 00:30:42] [ info] tail
[2025/03/13 00:30:42] [ info] ___________
[2025/03/13 00:30:42] [ info] filters:
[2025/03/13 00:30:42] [ info] parser.0
[2025/03/13 00:30:42] [ info] modify.1
[2025/03/13 00:30:42] [ info] parser.2
[2025/03/13 00:30:42] [ info] modify.3
[2025/03/13 00:30:42] [ info] ___________
[2025/03/13 00:30:42] [ info] outputs:
[2025/03/13 00:30:42] [ info] es.0
[2025/03/13 00:30:42] [ info] ___________
[2025/03/13 00:30:42] [ info] collectors:

It never goes beyond that step.
Occassionally, we see the following logs afterward and it starts logging to ES. But a pod restart breaks it again (intermittently)

[2025/03/13 02:03:01] [ info] [fluent bit] version=3.2.8, commit=d13e8e4ab2, pid=1
[2025/03/13 02:03:01] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2025/03/13 02:03:01] [ info] [simd ] disabled
[2025/03/13 02:03:01] [ info] [cmetrics] version=0.9.9
[2025/03/13 02:03:01] [ info] [ctraces ] version=0.6.0

To Reproduce
Unable to reproduce consistently.
Posting all relevant files here.

fluent-bit.conf:

[SERVICE]
    Flush        1
    Log_Level    info
    Daemon    off
    Parsers_File /fluent-bit/etc/parsers.conf
    Parsers_File /fluent-bit/etc/custom-parsers.conf

[INPUT]
    Name tail
    Path /var/log/nginx/access_file*
    Tag nginx.access
    DB /var/log/nginx/flb_nginx.db
    Mem_Buf_Limit 5MB
    Skip_Long_Lines On

[FILTER]
    Name parser
    Match nginx.access
    Key_Name log
    Parser nginx
    Preserve_Key On
    Reserve_Data On

[FILTER]
    Name modify
    Match nginx.access
    Add logType nginx_access

[INPUT]
    Name tail
    Path /var/log/nginx/error_file*
    Tag nginx.error
    DB /var/log/nginx/flb_nginx.db
    Mem_Buf_Limit 5MB
    Skip_Long_Lines On

[FILTER]
    Name parser
    Match nginx.error
    Key_Name log
    Parser nginx_error
    Preserve_Key On
    Reserve_Data On

[FILTER]
    Name modify
    Match nginx.error
    Add logType nginx_error

[OUTPUT]
    Name es
    Match nginx.*
    Host ${ELASTICSEARCH_HOST}
    Port 9200
    Generate_ID On
    Logstash_Format On
    Logstash_Prefix ${ELASTICSEARCH_INDEX}
    Logstash_DateFormat %Y-%m-%d
    Retry_Limit  False
    Trace_Error On
    Trace_Output On
    Write_Operation upsert

custom-parsers.conf:

[PARSER]
    Name         nginx_error
    Format       regex
    Regex        ^(?<time>[^ ]+ [^ ]+) \[(?<level>[^ ]+)\] (?<pid>[0-9]+)#(?<tid>[0-9]+): \*(?<cid>[0-9]+) (?<message>[^,]+)(, client: (?<client>[^ ]+))?(, server: (?<server>[^ ]+))?(, request: "(?<request>[^"]+)")?(, host: "(?<host>[^"]+)")?$
    Time_Key     time
    Time_Format  %d/%b/%Y:%H:%M:%S %z

relevant parts in kubernetes yaml:

containers:
        - name: nginx
          // ...
          volumeMounts:
            - name: log-volume
              mountPath: /var/log/nginx
        - name: fluentbit
          image: fluent/fluent-bit:3.2.8
          resources:
            requests:
              cpu: "100m"
              memory: "50Mi"
            limits:
              cpu: "200m"
              memory: "100Mi"
          volumeMounts:
            - name: log-volume
              mountPath: /var/log/nginx
            - name: fluentbit-config-volume
              mountPath: /fluent-bit/etc/fluent-bit.conf
              subPath: fluent-bit.conf
            - name: fluentbit-config-volume
              mountPath: /fluent-bit/etc/custom-parsers.conf
              subPath: custom-parsers.conf
      volumes:
        - name: log-volume
          emptyDir: {}
        - name: fluentbit-config-volume
          configMap:
            name: fluentbit-config-map

Your Environment

• Version used: 3.2.8
• Configuration: see above
• Environment name and version (e.g. Kubernetes? What version?): kubernetes

Additional context
We are loading fluentbit as a sidecar to a nginx container, delivering nginx logs to Elasticsearch.
We tried to see if there's any logs/errors in the container itself but could not find anything.
We also tried increasing memory/cpu without luck.