fluent-bit Use_Pod_Association On is unconditional, causing constant connection-error spam when Application Signals is disabled

[fluffywaffles]

Describe the bug

The chart hardcodes Use_Pod_Association On in the kubernetes filter of fluent-bit's application-log.conf:

helm-charts/charts/amazon-cloudwatch-observability/values.yaml

Line 150 in 222cc32

Use_Pod_Association On

That option makes fluent-bit poll cloudwatch-agent.amazon-cloudwatch:4311 (/kubernetes/pod-to-service-env-map) for Application Signals service/environment entity attribution.

However, the CloudWatch agent's pod-association server on port 4311 only exists when Application Signals is enabled: the operator only adds the cwa-server port (and with it, the cloudwatch-agent Service) when the agent config contains an application_signals section — see getApplicationSignalsReceiversServicePorts in internal/manifests/collector/ports.go.

So with applicationSignals.enabled: false, the cloudwatch-agent Service is never created, and every fluent-bit pod logs this pair roughly every 15 seconds, forever:

[ warn] [net] getaddrinfo(host='cloudwatch-agent.amazon-cloudwatch', err=-2): Name or service not known
[error] [filter:kubernetes:kubernetes.1] [kubernetes] no upstream connections available to cloudwatch-agent.amazon-cloudwatch:4311

Because fluent-bit also tails its own logs, this spam is shipped into /aws/containerinsights/<cluster>/application on every node, indefinitely.

Steps to reproduce

1. Install the chart (or the amazon-cloudwatch-observability EKS addon, observed on v6.4.0-eksbuild.1) with:

   { "applicationSignals": { "enabled": false } }

2. Check any fluent-bit pod's logs in the amazon-cloudwatch namespace.

Expected behavior

Use_Pod_Association should be gated on applicationSignals.enabled (mirroring how the operator gates the port/Service), or the filter should not endlessly retry an endpoint that is configured off.

Environment

• EKS addon amazon-cloudwatch-observability v6.4.0-eksbuild.1 (chart k8s/1.3.17)
• aws-for-fluent-bit 3.4.8
• cloudwatch-agent 1.300070.0b1586
• EKS, Linux nodes (AL2023)

Workaround

Override containerLogs.fluentBit.config.extraFiles."application-log.conf" with a copy of the default config that sets Use_Pod_Association Off.