[processor/k8sattributes] Unbounded memory consumption after upgrading to v0.145.0

[gracework]

Component(s)

processor/k8sattributes

What happened?

Description

We were upgrading from v0.140.1 => v0.147.0 and noticed that the otelcol for Kubernetes apps was repeatedly running out of memory and allocating new pods.

After narrowing it down to a before / after version, it appears that it is v0.145.0 that is correlated with this behaviour. After reviewing the release notes, we hypothesize that this increase is due to this change in v0.145.0:

• processor/k8sattributes: Added processor-specific observability metrics: otelcol.k8s.pod.association with status, pod_identifier, and otelcol.signal attributes (Promote k8sattributes Processor to Stable - Observability #44587)

This appears to correlate with the continually increasing nature of the metric otelcol.k8s.pod.association_total. We are running in a couple of regions. The collector (also running k8sattributes processor, identical configuration) that is much lower in traffic & fewer pods communicating with it does not experience the same growth rate as the higher-traffic one.

In our high-traffic region we have hundreds of (ephemeral) app pods sending data to this OTel Collector, which probably means a lot of IP addresses. The working theory is that pod_identifier is storing all of these and growing in size proportionally to number of pods that send data.

Steps to Reproduce

• v0.145.0
• service.telemetry.level.metrics: detailed
• a bunch of apps / IP addresses sending data to it. I'm not sure how many is necessary to observe the difference. In the screenshot above, the low-traffic region is getting data from 30 pods and you can really only clearly observe it in the otelcol.k8s.pod.association_total growth. Meanwhile the count & heap don't appear out of the ordinary. The high-traffic region is receiving data from over 200 pods during that same time period.

Expected Result

🧘‍♀️ Heap allocation stabilizes after startup

Actual Result

📈 Heap allocation appears to grow in size proportional to the number of pods sending traffic to the otelcol

Collector version

v0.145.0

Environment information

Environment

OS: Ubuntu 22.04.5 LTS

OpenTelemetry Collector configuration

processors:
      k8sattributes:
        auth_type: "serviceAccount"
        passthrough: false

        extract:
          metadata:
            - k8s.namespace.name
            - k8s.node.name
            - k8s.deployment.name
            - k8s.pod.name
            - k8s.pod.uid

        pod_association:
          - sources:
            - from: resource_attribute
              name: k8s.pod.ip
          - sources:
            - from: resource_attribute
              name: k8s.pod.uid
          - sources:
            - from: connection


The k8sattributesprocessor is only included in a single `traces` service pipeline.

Log output

Additional context

No response

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[github-actions]

Pinging code owners:

• processor/k8sattributes: @dmitryax @fatsheep9146 @TylerHelmuth @ChrsMark @odubajDT

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dhruv-shah-sumo]

@dmitryax @fatsheep9146 @TylerHelmuth @ChrsMark @odubajDT I can help with the initial RCA and fix if possible. Issue is fairly new but let me know if anyone else is working on this. Thanks!

[MikeGoldsmith]

I've just done a quick review and I think this bug is still present.

The otelcol.k8s.pod.association metric uses a pod_identifier attribute but the value is built from the actual pod identifier values (IP addresses, hostnames, etc.) via buildPodIdentifierString(). Because each unique value becomes a new metric time series which never evicted, memory grows proportionally to the number of distinct pods seen over the collector's lifetime.

The relevant code in processor.go:

attribute.String("pod_identifier", podIdentifierStr)

I think the pod_identifier attribute should describe how a pod was identified, not what the actual identifier value was. PodIdentifierAttribute already has Source.From (e.g. "resource_attribute", "connection") and Source.Name (e.g. "k8s.pod.ip", "k8s.pod.uid").

As an example, buildPodIdentifierString could be updated to something like:

func buildPodIdentifierSourceString(podIdentifierValue kube.PodIdentifier) string {
    var sources []string
    for i := range podIdentifierValue {
        src := podIdentifierValue[i].Source
        if src.From != "" {
            sources = append(sources, src.From+"/"+src.Name)
        }
    }
    if len(sources) > 0 {
        return strings.Join(sources, ",")
    }
    return "unknown"
}

This would give something like "resource_attribute/k8s.pod.ip" or "connection/k8s.pod.uid".

The metadata.yaml description for the pod_identifier attribute should also be updated to reflect that it now represents the source/type of association rather than the actual identifier value. It may be worth renaming the attribute to pod_association_source or similar to avoid confusion but maybe not worth the change.

Workaround

Until a fix is accepted, users can disable the internal telemetry metric by dropping the otelcol.k8s.pod.association metric in their collector pipeline (e.g. via a filter processor on the collector's own telemetry), or downgrade to a version prior to v0.145.0.

I'm also happy to help put PR for too, or review a PR if someone else wants to take it on.

[gracework]

Hey, thanks for confirming the bug and proposing a solution & workaround. :)

Unfortunately I believe the workaround for us will be remaining on v0.144.0 until the fix has been implemented. We did the filter on the metrics, which does prevent the events from reaching the telemetry backend, but the heap itself is still accumulating information on the pods it talks to. Here's the same type of query as the screenshot above, with the filter in-place, and with both the high-traffic & low-traffic regions:

[dhruv-shah-sumo]

I'm also happy to help put PR for too, or review a PR if someone else wants to take it on.

@MikeGoldsmith I'll take up the PR. Will let you know once it is ready for review.

[ChrsMark]

Thank's folks for checking this!

We can try and fix how otelcol.k8s.pod.association is produced. If that metric ends up being a pain instead of helpful we can consider about removing it entirely. @odubajDT feel free to chime in.

[tredman]

We did the filter on the metrics, which does prevent the events from reaching the telemetry backend, but the heap itself is still accumulating information on the pods it talks to.

Chiming in to say this has also been our experience. If we filter from our metrics pipeline, this avoids the immediate problem of storing these in the backend sink, but due to pod churn in our environment, the number of metrics being exported and filtered grows over time and eventually impacts the collector doing the filtering. We deployed 0.145.0 a few weeks ago and are starting to reach critical memory levels in a number of our environments, so it's a slow buildup but it does become a problem and the only way to intervene right now is to restart the collectors emitting these metrics.

I'm surprised this is being emitted at all because the metadata.yaml suggests this metric is disabled by default. Is this a separate bug or am I misunderstanding how this spec works? https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.145.0/processor/k8sattributesprocessor/metadata.yaml#L173

[ChrsMark]

I'm surprised this is being emitted at all because the metadata.yaml suggests this metric is disabled by default. Is this a separate bug or am I misunderstanding how this spec works? https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.145.0/processor/k8sattributesprocessor/metadata.yaml#L173

@tredman what you highlighted is correct and it's something we should fix. Raised that at open-telemetry/opentelemetry-collector#15164

For now, I'm going to remove the metric from being emitted until we have a proper fix since this affects the component's performance in a critical way: #47798