[TargetAllocator] Support exporting self-telemetry metrics via OTLP

[pirgeo]

Component(s)

target allocator

Is your feature request related to a problem? Please describe.

The TA uses the OTel SDK to collect self-monitoring metrics. These metrics are then exposed via a Prometheus endpoint. I would like to export them via OTLP rather than scraping them from the Prometheus endpoint.

Describe the solution you'd like

It would be valuable to allow the TargetAllocator to push its self-monitoring metrics via OTLP (grpc or http/protobuf) to a configurable endpoint, similar to how the OpenTelemetry Collector already supports this through service::telemetry::metrics::exporters. The TA could be configured in the same way, for example:

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: my-otel
spec:
  targetAllocator:
    telemetry:
      metrics:
        otlp:
          endpoint: https://example.com/api/v2/otlp    # base URL for HTTP; host:port for gRPC
          protocol: http                               # http | grpc (default: grpc)
          temporality: delta                           # delta | cumulative | lowmemory
          exportInterval: 30s                          # default 60s
          timeout: 15s                                 # default 10s
          insecure: false                              # disable TLS (gRPC only, local dev)
          headers:
            Authorization: "Bearer <token>"

or

apiVersion: opentelemetry.io/v1alpha1
kind: TargetAllocator
metadata:
  name: my-ta
spec:
  targetAllocator:
    telemetry:
      metrics:
        otlp:
          endpoint: https://example.com/api/v2/otlp    # base URL for HTTP; host:port for gRPC
          # ... and so on ... 

Describe alternatives you've considered

Using otlpmetrichttp.New(ctx) or otlpmetricgrpc.New(ctx) in the TA binary and injecting the configuration via env vars, e.g. OTEL_EXPORTER_OTLP_METRICS_ENDPOINT. That would be a minimal change with no changes to CRDs, but also no validation or discoverability.

Additional context

I have already played around with this idea and would be happy to contribute the code. The meat of this is pretty simple - just wire up the existing OTLP exporters in the TA binary. The "hard work" is wiring through the config parameters all the way to the binary. Therefore, most of the code is around updating CRDs and related resources.

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[swiatekm]

Sure, sounds like a good idea. Note that for feature parity, we'd need this to also cover the metrics coming from prometheus service discovery. These are currently registered directly, bypassing the otel SDK completely, and I'm not sure how well that would work. It would help if you could put up a draft PR.

[pirgeo]

@swiatekm I have the PR as a draft here: #5068. I think it's mostly done, except for the Prometheus only service discovery metrics, but I'll leave it in draft until we have the open questions worked out. I wonder if we could do the moving of Prometheus only metrics to OTel SDK separately, though. This PR really only wires up the OTel SDK that is already in the binary.

[swiatekm]

We can do the move in a separate PR, but I don't want to have two different ways of consuming target allocator metrics that expose different metric sets.

@open-telemetry/operator-approvers we might want to try to standardize how this is configured. Should we simply put https://pkg.go.dev/go.opentelemetry.io/contrib/otelconf in our CRDs?

[pirgeo]

I think the Prometheus bridge could solve our issue with the missing metrics, right?
https://pkg.go.dev/go.opentelemetry.io/contrib/bridges/prometheus.

I also drafted what it would look like if we used the otelconf package: https://github.com/open-telemetry/opentelemetry-operator/compare/main...dynatrace-oss-contrib:opentelemetry-operator:ta-otlp-otelconf?expand=1
The config would look like:

spec:
  targetAllocator:
    telemetry:
      metrics:
        readers:
          - periodic:
              interval: 30000
              timeout: 15000
              exporter:
                otlp:
                  protocol: http/protobuf
                  endpoint: https://example.com/api/v2/otlp/v1/metrics
                  temporality_preference: lowmemory
                  headers:
                    - name: Authorization
                      value: "Bearer <TOKEN>"

I left the Prometheus exporter as the default to always be there (no change in default behavior).