Skip to content

Opentelemetry own metrics scraping fails #14884

@p53

Description

@p53

Hi,

we are using EDOT with opentelemetry helm-chart 0.150.1, we enabled prometheus receiver, telemetry an in logs we are getting scraping failures, besides those there are not any other warn or error msgs in logs:

{"log.level":"debug","@timestamp":"2026-06-12T10:22:32.324Z","message":"Scrape failed","otelcol.component.kind":"receiver","scrape_pool":"elastic-opentelemetry-collector","target":"http://10.3.8.28:8888/metrics","error":{"message":"server returned HTTP status 500 Internal Server Error"},"ecs.version":"1.6.0","resource":{"host.name":"xxxx","k8s.namespace.name":"elastic-opentelemetry","k8s.node.ip":"10.96.48.144","k8s.node.name":"xxx","k8s.pod.ip":"10.3.8.28","k8s.pod.name":"elastic-opentelemetry-agent-c95bf5f67-j6pvs","service.instance.id":"bec062ec-9726-4b91-b8db-3d4bcbc55992","service.name":"/usr/share/elastic-agent/data/elastic-agent-dd9ee6/components/elastic-otel-collector","service.version":"9.4.2"},"otelcol.component.id":"prometheus","otelcol.signal":"metrics","ecs.version":"1.6.0"}

i execed into pod and tried to curl:

k exec -it -n elastic-opentelemetry elastic-opentelemetry-agent-c95bf5f67-j6pvs -- /bin/bash
[root@elastic-opentelemetry-agent-c95bf5f67-j6pvs elastic-agent]# curl -v http://localhost:8888/metrics
*   Trying ::1:8888...
* Connected to localhost (::1) port 8888 (#0)
> GET /metrics HTTP/1.1
> Host: localhost:8888
> User-Agent: curl/7.76.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 500 Internal Server Error
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Fri, 12 Jun 2026 10:30:58 GMT
< Content-Length: 1437
< 
An error has occurred while serving metrics:

8 error(s) occurred:
* collected metric "otelcol_otelsvc_k8s_pod_table_size" { gauge:{value:0}} was collected before with the same name and label values
* collected metric "otelcol_processor_batch_metadata_cardinality" { label:{name:"processor" value:"batch"} gauge:{value:1}} was collected before with the same name and label values
* collected metric "otelcol_processor_batch_metadata_cardinality" { label:{name:"processor" value:"batch"} gauge:{value:1}} was collected before with the same name and label values
* collected metric "otelcol_otelsvc_k8s_pod_table_size" { gauge:{value:0}} was collected before with the same name and label values
* collected metric "otelcol_processor_batch_metadata_cardinality" { label:{name:"processor" value:"batch"} gauge:{value:1}} was collected before with the same name and label values
* collected metric "otelcol.processor.consumed.items" { label:{name:"otelcol.component.outcome" value:"success"} counter:{value:280}} was collected before with the same name and label values
* collected metric "otelcol.processor.produced.items" { label:{name:"otelcol.component.outcome" value:"success"} counter:{value:280}} was collected before with the same name and label values
* collected metric "otelcol_processor_batch_metadata_cardinality" { label:{name:"processor" value:"batch"} gauge:{value:1}} was collected before with the same name and label values
* Connection #0 to host localhost left intact
[root@elastic-opentelemetry-agent-c95bf5f67-j6pvs elastic-agent]# 

telemetry config section:

      telemetry:
        logs:
          level: DEBUG
        metrics:
          readers:
          - pull:
              exporter:
                prometheus:
                  host: 0.0.0.0
                  port: 8888
                  without_scope_info: true
                  without_type_suffix: true
                  without_units: true
        resource:
          host.name: ${env:OTEL_K8S_NODE_NAME}
          k8s.namespace.name: ${env:OTEL_K8S_NAMESPACE}
          k8s.node.ip: ${env:OTEL_K8S_NODE_IP}
          k8s.node.name: ${env:OTEL_K8S_NODE_NAME}
          k8s.pod.ip: ${env:OTEL_K8S_POD_IP}
          k8s.pod.name: ${env:OTEL_K8S_POD_NAME}

receivers section:

    receivers:
      jaeger:
        protocols:
          grpc:
            endpoint: ${env:MY_POD_IP}:14250
          thrift_compact:
            endpoint: ${env:MY_POD_IP}:6831
          thrift_http:
            endpoint: ${env:MY_POD_IP}:14268
      otlp:
        protocols:
          grpc:
            endpoint: ${env:MY_POD_IP}:4317
            max_recv_msg_size_mib: 80
          http:
            endpoint: ${env:MY_POD_IP}:4318
      prometheus:
        config:
          scrape_configs:
          - job_name: elastic-opentelemetry-collector
            scrape_interval: 10s
            static_configs:
            - targets:
              - ${env:MY_POD_IP}:8888

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions