otlp destination not working

[cmcconnell-relativity]

Hi,

We have a k8s endpoint we are trying to send this data to, and while this chart works fine with sending the data to cloud, it seems not capable of sending to our endpoint. We have lsited explicitly the destinations, there's no auth required, and are confident we've configured it correctly. Could someone take a look?

[mbaykara]

Could you share your respective values.yml and the error you're getting?

[cmcconnell-relativity]

The destinations are in our cluster specific override files:

    - name: otlp-meta-monitoring
      type: otlp
      protocol: grpc
      url: *************************
      metrics: {enabled: true}
      logs: {enabled: false}
      traces: {enabled: false}
      processors:
        transform:
          metrics:
            resource:
              - set(attributes["service.name"], "k8s-monitoring")
              - set(attributes["r1.team.id"], "1234")

We don't see errors we just don't see the data flow

[mbaykara]

What backend are you sending the metrics to? Does it support otlp?

To troubleshoot:
1. Check the grafana-k8s-monitoring-alloy-metrics-0 logs.
2. Make sure your time series database supports otlp and the feature is enabled. For example, Prometheus requires the web.enable-otlp-receiver CLI flag.
3. Check if the URL in the destination above is set correctly. For example, a Prometheus server in the same cluster in default namespace should look like http://prometheus-server.default.svc.cluster.local/api/v1/otlp with http protocol.

[cmcconnell-relativity]

This backend is our otlp collectors - the url is owned by us and is correct with the correct protocol. We also don't see failures in the logs

[mbaykara]

Unfortunately, I am not able reproduce the issue and with given details it is hard to say something.

[cmcconnell-relativity]

We have the destination also outlined here in our values.yaml:

k8s-monitoring:
  integrations:
    collector: alloy-singleton
    destinations:
      - grafanacloud-releye-metrics
      - **otlp-meta-monitoring**
    # attach available node labels to all integration metrics collection
    nodeLabels:
      nodepool: true
      region: true
      availability_zone: true
      node_role: true
      os: true
      architecture: true
      instance_type: true

Is that generally correct? I understand hard to reproduce sorry this is just all I have as well

[petewall]

The destinations list seems fine. Can you look into the generated config map and see if the integrations you've enabled have both metrics destinations? You should see something like this:

      metrics_destinations = [
        prometheus.remote_write. grafanacloud_releye_metrics.receiver,
        otelcol.receiver.prometheus.otlp_meta_monitoring.receiver,
      ]