[kube-prometheus-stack] Enable otlp receiver

[truongnht]

Is your feature request related to a problem ?

For new Prometheus 3.0 to receive OTLP metrics, OTLP receiver feature needs to be enabled via cli arg web.enable-otlp-receiver. This option is not available at this moment in helmchart.

Describe the solution you'd like.

Add option to enable otlp-receiver

Describe alternatives you've considered.

n/a

Additional context.

No response

[bartvanackooij]

You could use the additonalArgs on the prometheus.prometheusSpec to enable it:

prometheus:
  prometheusSpec:
    additionalArgs:
      - name: web.enable-otlp-receiver
        value: ""

[truongnht]

@bartvanackooij let me try out, thanks

[kitforbes]

Edit: I misunderstood how Prometheus was using OTLP. The OpenTelemetry Demo was a good reference for configuring this correctly.

What would be required to also open port 4317 and/or 4318 (the typical OTLP ports) on the Prometheus service/pod? I've tried the following with no luck so far.

prometheus:
  prometheusSpec:
    additionalArgs:
    - name: web.enable-otlp-receiver
      value: ""
    additionalConfig:
      otlp:
        promoteResourceAttributes:
        - service.instance.id
        - service.name
        - service.namespace
        - cloud.availability_zone
        - cloud.region
        - container.name
        - deployment.environment.name
        - k8s.cluster.name
        - k8s.container.name
        - k8s.cronjob.name
        - k8s.daemonset.name
        - k8s.deployment.name
        - k8s.job.name
        - k8s.namespace.name
        - k8s.pod.name
        - k8s.replicaset.name
        - k8s.statefulset.name
        translationStrategy: NoUTF8EscapingWithSuffixes
    tsdb:
      outOfOrderTimeWindow: 30m
  service:
    additionalPorts:
    - name: otlp
      port: 4317
      targetPort: 4317

[pieterv-icloud-com]

If followed the instructions here: https://prometheus.io/docs/guides/opentelemetry/#send-opentelemetry-metrics-to-the-prometheus-server

I then configured an opentelemetry otel exporter with a similar url: https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/otlphttpexporter/README.md

The otel exporter complained about an invalid port because it is not expecting the path in the url.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[danielcn]

That is the config that I'm using working fine:

On Prometheus side:

prometheus:
  prometheusSpec:
    scrapeInterval: "15s"  # Sets the global scrape interval
    additionalScrapeConfigs:
      - job_name: 'otel-collector'
        static_configs:
          - targets: ['0.0.0.0:8889'] # or 'otel-collector.monitoring.svc.cluster.local:8889' if inside Kubernetes service into monitoring namespace.

On the Otel-Collector side:

exporters:
  debug:
    verbosity: detailed
  prometheus:
    endpoint: "0.0.0.0:8889"
extensions:
  health_check:
    endpoint: 0.0.0.0:13133
receivers:  
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  prometheus:
    config:
      scrape_configs:
        - job_name: 'otel-collector'
          static_configs:
            - targets: ['0.0.0.0:8888']
service:
  extensions:
  - health_check
  pipelines:
    logs:
      exporters:
      - debug
      receivers:
      - otlp
    metrics:
      exporters:
      - debug
      - prometheus
      receivers:
      - otlp
      - prometheus
    traces:
      exporters:
      - debug
      receivers:
      - otlp

[franco-roura]

@danielcn what method of installation did you use for the otel collector? I'm using the open-telemetry/opentelemetry-collector helm chart, tried your configuration for the Otel-collector side, and it throws error decoding 'exporters': unknown type: "prometheus" for id: "prometheus" (valid values: [otlp otlphttp file loadbalancing otelarrow debug nop])

[diegocejasprieto]

@franco-roura were you able to solve your error? I have the same problem