Unable to Get Internal Metrics for Logs Pipeline via Prometheus Exporter

[shahparthiv]

Component(s)

No response

Describe the issue you're reporting

Hi team,

I'm working with the OpenTelemetry Collector and using a pipeline for processing logs.

Setup:
Receiver: NetFlow receiver (emits logs)
Processor: Custom processor that reads and transforms logs to metrics
Pipeline type: logs
Exporter: debug

Problem:
I’m expecting internal metrics such as:
otelcol_receiver_accepted_log_records
otelcol_processor_incoming_items
otelcol_processor_outgoing_items

…to be exposed via the Prometheus exporter.

However, even after configuring the Prometheus exporter, these internal metrics are not visible at /metrics.

Question:
What’s the correct way to expose internal pipeline metrics for a logs pipeline using the Prometheus exporter?

Do I need to:

Add something specific in the pipeline config?
Enable internal telemetry or metrics level somewhere?
Use a special receiver/exporter combo for logs pipelines?

Current config setup

service:
  telemetry:
    metrics:
      readers:
        - pull:
            exporter:
              prometheus:
                host: '0.0.0.0'
                port: 9991
  pipelines:
    logs:
      receivers:
      - netflow
      processors:
      - custom_netflow_processor
      exporters:
      - debug 

[ChrsMark]

This looks more like a question/issue for https://github.com/open-telemetry/opentelemetry-collector?

Pinging @open-telemetry/collector-approvers

[jade-guiton-dd]

These metrics are not emitted automatically by the Collector itself, they must be implemented by the individual receivers and processors you are using.

The otelcol_receiver_accepted_log_records metric can be emitted using the receiverhelper package (ref), but it looks like the netflow receiver does not use that package. This would be a feature request for its maintainer.

Similarly, the otelcol_processor_incoming_items and otelcol_processor_outgoing_items metrics can be emitted using the processorhelper package (ref). You will need to use this package in your custom processor to emit these metrics.

(Note that the Collector SIG is actively working on implementing new pipeline metrics similar to these, without requiring manual calls by the component code, but this is still work-in-progress.)

[shahparthiv]

@jade-guiton-dd , Thank you so much for your answer. We will try to implement it.