kubernetes filter: SIGSEGV when two filter instances enable pod association (Use_Pod_Association) — regression in 5.0.5, still present in 5.0.9

[daikikato9687]

Bug Report

Describe the bug

Fluent Bit crashes with SIGSEGV when two kubernetes filter instances (matching different tags) both enable pod association (Use_Pod_Association On, the alias of aws_use_pod_association). The crash occurs in the TLS session-creation path of the pod-association / entity connection to the CloudWatch agent endpoint (:4311).

This is a regression introduced in v5.0.5: the identical configuration runs indefinitely on v5.0.3, and crashes on v5.0.5 through v5.0.9 (latest at the time of writing).

Two distinct triggers produce the same crash:

1. Endpoint loss: when the pod-association endpoint becomes unreachable (CloudWatch agent pod restart / node replacement), Fluent Bit crashes within minutes.
2. Steady state: even with the endpoint healthy and the node stable for 15+ hours, Fluent Bit crashes roughly every 25 minutes (min ~2 min, max ~1.5 h). Every crash lands on the 60-second connection-refresh cycle aligned to container start time (container started at hh:26:31 → all errors/crashes at hh:mm:32), suggesting corrupted shared state is hit by the periodic reconnect rather than the reconnect itself being the root cause.

To Reproduce

• Rubular link if applicable: N/A
• Example log message if applicable:

[2026/07/23 07:38:32.200] [error] [openssl] could create new SSL context
[2026/07/23 07:38:32.200] [error] [tls] could not create TLS session for tcp://172.20.202.50:4311
[2026/07/23 07:38:32] [engine] caught signal (SIGSEGV)

(The first message is the known mislabeled/typo'd one emitted from tls_session_create() when SSL_new() fails — see #4666. SSL_new() failing is normally very rare, which itself points to a corrupted/freed SSL_CTX. The container exits with code 133 = 128+SIGTRAP because the crash handler traps after logging, on ARM64.)

• Steps to reproduce the problem:

We could not reduce this to a standalone repro outside Kubernetes (the pod-association path requires the CloudWatch agent entity endpoint), but the in-cluster reproduction is deterministic and was confirmed across dozens of crashes on two independent EKS clusters:

1. Deploy the amazon-cloudwatch-observability EKS addon (v6.2.0 or v6.3.0, i.e. Fluent Bit 5.0.5) with a customized Fluent Bit configuration containing two kubernetes filter instances, both with pod association enabled:

# file 1 (application logs)
[FILTER]
    Name                kubernetes
    Match               application.*
    Use_Kubelet         On
    Buffer_Size         0
    Use_Pod_Association On

# file 2 (dataplane logs) — the second instance that triggers the crash
[FILTER]
    Name                kubernetes
    Match               dataplane.tail.*
    Use_Kubelet         On
    Buffer_Size         0
    Use_Pod_Association On

2. Delete the CloudWatch agent pods (kubectl delete pod -n amazon-cloudwatch -l app.kubernetes.io/component=amazon-cloudwatch-agent) so the pod-association endpoint disappears briefly.
3. Fluent Bit crashes within minutes (in steady state it also crashes on its own, roughly every 25 minutes).

Single-variable isolation results (same cluster, same test):

Configuration	Fluent Bit	Result
1 filter instance with pod association (default addon config)	5.0.5	No crash (endpoint loss ×3 + steady state — errors handled cleanly)
2 filter instances, both Use_Pod_Association On	5.0.3	No crash
2 filter instances, both Use_Pod_Association On	5.0.5	Crash (6.4 min after endpoint loss; ~25 min cycle in steady state)
2 filter instances, both Use_Pod_Association On	5.0.9	Crash (same signature, within minutes)
Same, but Use_Pod_Association removed from one of the two filters	5.0.5	No crash under the identical endpoint-loss test

The last two rows differ by exactly one line of configuration.

Expected behavior

Connection failures to the pod-association endpoint should be handled as recoverable errors (as they are in v5.0.3, and in single-filter-instance configurations on v5.0.5+), not crash the process.

Screenshots

N/A

Your Environment

• Version used: Fluent Bit 5.0.5 and 5.0.9 (crash), 5.0.3 (no crash) — via aws-for-fluent-bit 3.4.0 / 3.4.3 / 3.4.9 and 3.3.0
• Configuration: see excerpt above; full config is the amazon-cloudwatch-observability addon default plus a second kubernetes filter with Use_Kubelet On + Use_Pod_Association On on a separate tag
• Environment name and version (e.g. Kubernetes? What version?): Amazon EKS 1.34 / 1.35, deployed via the amazon-cloudwatch-observability EKS addon (v5.4.0 / v6.2.0 / v6.3.0)
• Server type and version: AWS Graviton (ARM64), Spot and On-Demand
• Operating System and version: Bottlerocket OS
• Filters and plugins: filter_kubernetes (×2, with Use_Kubelet + Use_Pod_Association), filter_aws (Enable_Entity true), filter_log_to_metrics, out_cloudwatch_logs (add_entity true), out_prometheus_exporter (log_to_metrics / prometheus_exporter were ruled out by isolation testing)

Additional context

Suspected root cause: v5.0.5 added TLS session/context destroy calls to filter_kubernetes as part of the memory-leak fixes (#11730, #11731, #11738). With two filter instances running pod association concurrently, one instance appears to destroy TLS/pod-association state still referenced by the other (use-after-free), which later manifests as SSL_new() failure followed by SIGSEGV on the periodic reconnect path. This would also explain why v5.0.3 — which leaked instead of destroying — never crashes.

Verified workaround for anyone hitting this signature: ensure only one kubernetes filter instance enables pod association. Removing Use_Pod_Association On from the second filter eliminates the crash while keeping metadata enrichment intact.

We are unblocked by the workaround, so this report is primarily to document the issue for other users searching for this crash signature, and to flag the suspected use-after-free (which may be reachable through paths other than our trigger). Happy to test patched builds, or answer questions.