[ECS FARGATE] Frequent SIGSEGV issues

[alandreasyans]

Describe the question/issue

With the ECS Fargate (v1.4) oftelny is getting the [engine] caught signal (SIGSEGV) issue even when task has enough time for the proper shutdown(grace period 30sec).

Configuration

Fluent Bit Log Output

March 5, 2026, 19:13	[2026/03/05 15:13:06.482369276] [debug] [upstream] KA connection #138 to my_cluster-id.eu-west-2.es.amazonaws.com:443 is connected	log_router
March 5, 2026, 19:13	[2026/03/05 15:13:06] [engine] caught signal (SIGSEGV)	log_router
March 5, 2026, 19:13	[2026/03/05 15:13:06.468995880] [ info] [task] emitter/emitter_for_multiline.0 has 0 pending task(s):	log_router
March 5, 2026, 19:13	[2026/03/05 15:13:06.469030681] [ info] [task] emitter/emitter_for_rewrite_tag has 1 pending task(s):	log_router
March 5, 2026, 19:13	[2026/03/05 15:13:06.469078429] [ info] [task] task_id=0 still running on route(s): opensearch/Application	log_router
March 5, 2026, 19:13	[2026/03/05 15:13:06.469141540] [ info] [input:storage_backlog:storage_backlog.4] register emitter.6/1-1772723585.218186378.flb	log_router
March 5, 2026, 19:13	[2026/03/05 15:13:06.468654247] [ info] [engine] pending chunk count: memory=0, filesystem=1; grace_timer=1	log_router
March 5, 2026, 19:13	[2026/03/05 15:13:06.468759761] [ info] [task] tcp/tcp.0 has 0 pending task(s):	log_router
March 5, 2026, 19:13	[2026/03/05 15:13:06.468815348] [ info] [task] forward/forward.1 has 0 pending task(s):	log_router
March 5, 2026, 19:13	[2026/03/05 15:13:06.468861331] [ info] [task] forward/forward.2 has 0 pending task(s):	log_router
March 5, 2026, 19:13	[2026/03/05 15:13:06.468908097] [ info] [task] forward/forward.3 has 0 pending task(s):	log_router
March 5, 2026, 19:13	[2026/03/05 15:13:06.468941921] [ info] [task] storage_backlog/storage_backlog.4 has 0 pending task(s):	log_router
March 5, 2026, 19:13	[2026/03/05 15:13:06.464083642] [ warn] [engine] service will shutdown in max 30 seconds	log_router
March 5, 2026, 19:13	[2026/03/05 15:13:06.464160522] [debug] [engine] task 0 already scheduled to run, not re-scheduling it.	log_router
March 5, 2026, 19:13	[2026/03/05 15:13:06.464195606] [ info] [engine] pausing all inputs..	log_router
March 5, 2026, 19:13	[2026/03/05 15:13:06.464252128] [ info] [input] pausing tcp.0	log_router
March 5, 2026, 19:13	[2026/03/05 15:13:06.464314715] [ info] [input] pausing forward.1	log_router
March 5, 2026, 19:13	[2026/03/05 15:13:06.464386504] [ info] [input] pausing forward.2	log_router
March 5, 2026, 19:13	[2026/03/05 15:13:06.464434241] [debug] [input:forward:forward.3] thread pause instance	log_router
March 5, 2026, 19:13	[2026/03/05 15:13:06.464569260] [ info] [input] pausing storage_backlog.4	log_router
March 5, 2026, 19:13	[2026/03/05 15:13:06.464641587] [ info] [input] pausing emitter_for_multiline.0	log_router
March 5, 2026, 19:13	[2026/03/05 15:13:06.464690687] [ info] [input] pausing emitter_for_rewrite_tag	log_router
March 5, 2026, 19:13	[2026/03/05 15:13:06.464761710] [debug] [input:forward:forward.3] thread pause instance	log_router
March 5, 2026, 19:13	[2026/03/05 15:13:06] [engine] caught signal (SIGTERM)

Fluent Bit Version Info

March 5, 2026, 19:12	Fluent Bit v4.2 – Direct Routes Ahead	log_router
March 5, 2026, 19:12	Celebrating 10 Years of Open, Fluent Innovation!	log_router
March 5, 2026, 19:12	AWS for Fluent Bit Container Image Version 3.2.0

Cluster Details

AWS Fargate is used

Application Details

The following Fluent Bit configuration is used(s3 part removed for the better visibility)

[SERVICE]
Flush 5
Grace 30
Daemon Off
Parsers_File /parsers.conf
storage.path /var/log/flb-storage/
storage.sync normal
storage.checksum On
storage.max_chunks_up 512
storage.backlog.mem_limit 10M
storage.backlog.flush_on_shutdown On
scheduler.base 10
scheduler.cap 60
Log_Level debug

[Input]
Name forward
unix_path /var/run/fluent.sock
Mem_Buf_Limit 75MB
threaded true
storage.type memory

[FILTER]
name multiline
Match firelens
multiline.key_content log
mode partial_message

[FILTER]
Name grep
Match firelens
Regex log ^{

[FILTER]
Name parser
Match firelens
Key_Name log
Parser docker

[FILTER]
Name expect
Match firelens
key_exists LogType
action warn

[FILTER]
Name rewrite_tag
Match firelens
Rule $LogType ^(.*) $LogType false
Emitter_Name emitter_for_rewrite_tag
Emitter_Storage.type memory
Emitter_Mem_Buf_Limit 100M
[FILTER]
Name modify
Match Integration
Rename @t t
Rename @m m
Rename @i i
Rename @tr tr
Rename @sp sp
Rename @@ti ti
[FILTER]
Name modify
Match BookingRequests
Rename @t t
Rename @m m
Rename @i i
Rename @tr tr
Rename @sp sp
Rename @@ti ti

[OUTPUT]
Name opensearch
Alias Application
Match Application
Host ${OS_HOST}
Port ${OS_PORT}
AWS_Auth On
AWS_Region ${OS_AWS_REGION}
tls On
Index ${INDEX_PREFIX}-application
Include_Tag_Key Off
Suppress_Type_Name On
Trace_Error On
Trace_Output On
retry_limit 10
workers 1
Buffer_Size 1MB
storage.total_limit_size 1G

[OUTPUT]
Name opensearch
Alias Audit
Match Audit
Host ${OS_HOST}
Port ${OS_PORT}
AWS_Auth On
AWS_Region ${OS_AWS_REGION}
tls On
Index ${INDEX_PREFIX}-audit
Include_Tag_Key Off
Suppress_Type_Name On
Trace_Error On
Trace_Output On
retry_limit 10
workers 1
Buffer_Size 1MB
storage.total_limit_size 500M

Steps to reproduce the issue

Seems when the JSON file is not properly formatted or it doesn't have the json format at all, it throws an error like
[debug] [yyjson->msgpack] read error code=6 msg=unexpected character, expected a JSON value pos=0, but after adding the grep filter to check only files starting with {, it disappears from the logs. But even so the issue persist wihtout that kind of warnings.

With the newer(3.2.3) version, it's the same, but with the aws-for-fluent-bit:2.32.2 version, we haven't had a problem.

Within one ECS task I have 2 containers - this fluent and my application. My application is shutting down gracefully, but the fluent with the 139 exit code.
The only way to workaround this which I have found was storage.type = memory usage.

[BassemKadri]

Hi,

We are experiencing a very similar issue with aws-for-fluent-bit (FireLens) on ECS Fargate, and I’d like to share our context as it might help identify a common root cause.

Context

• Platform: ECS Fargate
• Logging: FireLens (awsfirelens log driver)
• Fluent Bit image: aws-for-fluent-bit:3.x
• Architecture: application container + log router sidecar (Fluent Bit)

Issue

We are observing frequent crashes of the Fluent Bit container with SIGSEGV (exit code 139).

When this happens:

• The Fluent Bit (log_router) container crashes unexpectedly
• Because it is part of the same task, it can lead to task instability or full restart depending on configuration
• The application container itself is stable and shuts down gracefully

This is consistent with logs like:

[engine] caught signal (SIGSEGV)

Observations

• The issue seems unrelated to resource limits (CPU/memory are sufficient)

• We also noticed that malformed JSON logs can trigger parsing issues, but:

  • Even after filtering logs (e.g., using grep to enforce JSON format), the crash still occurs

• Similar to the original report, the issue appears during shutdown or log processing phases

Impact

This is particularly problematic in ECS because:

• A sidecar crash can affect the entire task lifecycle
• It introduces instability in otherwise healthy services
• It may lead to log loss or service disruption

Question

Has there been any confirmed root cause for these SIGSEGV crashes in recent versions (3.x / Fluent Bit v4.x)?

Also, are there recommended configurations or stability guidelines when using FireLens on Fargate to avoid this type of failure?

Thanks in advance for any insights.

[ShelbyZ]

Can you please provide your fluent-bit configuration file to help us determine what plugins are in use and their configuration?

If you are seeing frequent SIGSEGV, it may be worth following guidance at https://github.com/aws/aws-for-fluent-bit/blob/mainline/troubleshooting/debugging.md#segfaults-and-crashes-sigsegv, to capture the corefile

[BassemKadri]

Thanks for your response.

Here is a simplified version of our current setup:

Fluent Bit configuration

[SERVICE]
    Log_Level info
    Parsers_File /fluent-bit/etc/parsers.conf

[FILTER]
    Name         parser
    Match        *
    Key_Name     log
    Parser       json
    Reserve_Data On
    Preserve_Key Off

Dockerfile

FROM public.ecr.aws/aws-observability/aws-for-fluent-bit:3.2.4
COPY extra.conf /extra.conf

ECS container definition (log router)

container_definitions = {
  "log-router" = {
    essential = true
    image     = local.fluentbit_ecr_image_url
    memory    = 128
    cpu       = 0

    firelens_configuration = {
      type = "fluentbit"
      options = {
        enable-ecs-log-metadata = "false"
        config-file-type        = "file"
        config-file-value       = "/extra.conf"
      }
    }
  }
}

Application container log configuration

log_configuration = {
  logDriver = "awsfirelens"
  options = {
    Name                = "cloudwatch_logs"
    region              = local.region
    log_group_name      = local.environments[each.key].service.group
    log_stream_name     = "output.log"
    log_stream_template = "$stream"
    auto_create_group   = "true"
    log_format          = "json"
  }
}

Additional note

The stream field used in log_stream_template is part of the JSON payload inside the log field.

Here are a few more details from our side.

We reproduced the crash with:

• AWS for Fluent Bit 2.32.2
• Fluent Bit 1.9.10
• ECS Fargate + FireLens
• cloudwatch_logs output with log_stream_template = "$stream"

What we consistently see before the crash is:

• multiple PutLogEvents http status=200
• dynamic stream switching in the same sequence
• Retrieving credentials from the HTTP provider..
• keepalive connection reuse
• then [engine] caught signal (SIGSEGV)

Example:

[debug] [output:cloudwatch_logs:cloudwatch_logs.1] PutLogEvents http status=200 
[debug] [output:cloudwatch_logs:cloudwatch_logs.1] Using stream=bff_rest_dispute_client.log 
[debug] [output:cloudwatch_logs:cloudwatch_logs.1] Sending log events to log stream bff_rest_business.log 
[debug] [upstream] KA connection #45 to logs.eu-west-1.amazonaws.com:443 has been assigned (recycled) 
[debug] [aws_credentials] Retrieving credentials from the HTTP provider.. 
[engine] caught signal (SIGSEGV)

One important observation: the crash happens when multiple dynamic streams are used, for example:

• bff_rest_dispute_client.log
• bff_rest_business.log
• bff_rest_dispute_client.log

When we send logs to a single fixed stream instead, the task appears stable.

So the most suspicious combination on our side is:
cloudwatch_logs + dynamic log_stream_template + multiple streams + credential refresh / keepalive reuse.

Does this match any known issue in cloudwatch_logs on ECS Fargate?

[BassemKadri]

We are experiencing the same SIGSEGV crash on Fluent Bit v4.2.2 (aws-for-fluent-bit:3.2.5) on ECS Fargate, during
normal operation (not at shutdown).

Two different crash patterns observed:

Pattern 1 — triggered by a CloudWatch HTTP 400 error:

  [output:cloudwatch_logs] CreateLogStream http status=400
  [output:cloudwatch_logs] Sending log events to log stream ���  ← corrupted pointer
  {"__type":"SerializationException"}
  [engine] caught signal (SIGSEGV)

Pattern 2 — no HTTP error visible, crash occurs after successful PutLogEvents (HTTP 200):

  [output:cloudwatch_logs] PutLogEvents http status=200
  [output:cloudwatch_logs] PutLogEvents http payload={nextSequenceToken...}
  [engine] caught signal (SIGSEGV)

Both patterns affect multiple ECS services across multiple environments.

This suggests a memory corruption issue in the cloudwatch_logs plugin that can be triggered in multiple ways. Could
you confirm if this is fixed in 3.3.0 (Fluent Bit v5.0.3)?

[BassemKadri]

Hi,

We are experiencing recurring Fluent Bit crashes (SIGSEGV) on AWS ECS Fargate. The issue has been happening daily for the past 2
weeks, with 16 crashes on one service alone.

Environment:

• Fluent Bit running as a sidecar container on ECS Fargate
• Output plugin: cloudwatch_logs
• AWS Region: eu-west-1

Observed behavior:

Fluent Bit crashes with SIGSEGV systematically preceded by one of these two errors:

1. InvalidParameterException — empty logStreamName:

  PutLogEvents API responded with error='InvalidParameterException',
  message='1 validation error detected: Value '' at 'logStreamName'
  failed to satisfy constraint: Member must have length greater than or equal to 1'

2. ResourceNotFoundException — log stream does not exist:

PutLogEvents API responded with error='ResourceNotFoundException',
message='The specified log stream does not exist.'

Both errors are immediately followed by a SIGSEGV:

[engine] caught signal (SIGSEGV)
[engine] Shutdown Grace Period=30, Shutdown Input Grace Period=15

Impact:

• Fluent Bit crashes and restarts (via container restart policy) every time one of these API errors occurs
• The crash happens inside the error handling path of the cloudwatch_logs output plugin
• Both tasks of the service crash simultaneously when the trigger log record is processed

Expected behavior:

Fluent Bit should handle InvalidParameterException and ResourceNotFoundException gracefully (log the error, skip the record or
retry) without crashing.

Request:

Could you confirm if this is a known bug and whether a fix is available or planned? Is there a workaround we can apply on our side
(e.g. specific Fluent Bit version, configuration option)?

extra.conf

[SERVICE]
    Flush        1
    Grace        30
    Log_Level    debug
    Parsers_File /fluent-bit/etc/parsers.conf
    Parsers_File /parsers_extra.conf

[FILTER]
    Name         parser
    Match        *
    Key_Name     log
    Parser       appjson
    Reserve_Data On
    Preserve_Key Off

[FILTER]
    Name      modify
    Match     *
    Condition Key_Does_Not_Exist stream
    Set       stream ${DEFAULT_STREAM}

[FILTER]
    Name      modify
    Match     *
    Condition Key_Value_Equals stream ""
    Set       stream ${DEFAULT_STREAM}

ecs.tf

container_definitions = {
        "log-router" = {
          essential          = false
          image              = local.fluentbit_ecr_image_url
          memory_reservation = 256
          cpu                = 128
          restart_policy = {
            enabled              = true
            restartAttemptPeriod = 60
          }
          environment = [
            { name = "DEFAULT_STREAM", value = "bff_rest_out.log" }
          ]
          firelens_configuration = {
            type = "fluentbit"
            options = {
              enable-ecs-log-metadata = "false"
              config-file-type        = "file"
              config-file-value       = "/extra.conf"
            }
          }
        }

Thank you.