k8sprocessor is self-referencing?

[cvestre-cisco]

Going around in circles with the k8sprocessor (v0.36.0 currently) in the daemonsset -> collector mode. What we're seeing is that when we configure our daemonset to pull status.podIP and push it down as an environment variable, that the collector simply uses the IP of the daemonset itself to perform K8s API lookups. So what we're getting is the tags for the daemonset instead of the reporting container. We are exporting to Datadog.

We have resourcedetection configured per docs:

• https://docs.datadoghq.com/tracing/setup_overview/open_standards/#kubernetes
• https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/release/v0.36.x/processor/k8sprocessor/testdata/config.yaml

We are reporting traces via the Zipkin exporter and there is no k8s.pod.ip label passed along. We're working to add this label from containers to see if it corrects our lookups.

Any and all suggestions appreciated!

# Daemonset spec:
        env:
          - name: POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
          - name: OTEL_RESOURCE_ATTRIBUTES
            value: "k8s.pod.ip=$(POD_IP)"

# Configmap:
    receivers:
      otlp:
        protocols:
          grpc:
          http:
      zipkin: {}
    exporters:
      otlp:
        endpoint: "http://otel-collector.monitoring.svc.cluster.local:4317"
        insecure: true
        sending_queue:
          num_consumers: 4
          queue_size: 100
        retry_on_failure:
          enabled: true
      logging:
        loglevel: debug
    processors:
      batch:
        timeout: 10s
      memory_limiter:
        # Same as --mem-ballast-size-mib CLI argument
        ballast_size_mib: 165
        # 80% of maximum memory up to 2G
        limit_mib: 400
        # 25% of limit up to 2G
        spike_limit_mib: 100
        check_interval: 5s
      resourcedetection:
        detectors: [env]
        timeout: 5s
        override: false
      k8s_tagger:
        passthrough: true
        filter:
          node_from_env_var: KUBE_NODE_NAME
    extensions:
      health_check: {}
      zpages: {}
    service:
      extensions: [health_check, zpages]
      pipelines:
        traces:
          receivers: [otlp, zipkin]
          processors: [memory_limiter, batch, resourcedetection, k8s_tagger]
          exporters: [otlp, logging]

[cvestre-cisco]

Working with a DataDog rep, ended up adding the following to the OTel Collector (not Agent daemonset):

  processors::
      resource/host:
        attributes:
        - key: host.name
          from_attribute: k8s.node.name
          action: upsert
[...]
    service:
      pipelines:
        traces:
          processors: [batch, k8s_tagger, resource/host]
[...]

Examining the `host.name`, it was actually reporting `pod.name`. Not sure if this was a problem with the Zipkin exporter.

Correcting `host.name` got us performing k8stagger lookups to the correct pod instead of the OTel daemonset.