add option to configure ClusterObservability CR

[frzifus]

Component(s)

cluster observability

Is your feature request related to a problem? Please describe.

Honestly I am not sure if I like the approach or not. I did like to simply overwrite small parts of the CollectorCRs generated from the ClusterObservability CR.

Like disable log collection for a specific service, set another resourcedetection parameter or change the version of the collector image.

One option would be to use kustomize:

  patches:
    - target:
        kind: OpenTelemetryCollector
        name: gateway-agent
      patch: |-
        apiVersion: opentelemetry.io/v1beta1
        kind: OpenTelemetryCollector

I dont like that appraoch, since Ive to wait until the Collector CR gets created and seconds afterwards patched.

On the other hand I think it would blow up the scope if the ClusterObservability CR would expose all these simple nops to e.g. change the image, the image tag, a specific processor config.

Describe the solution you'd like

One option would be to provide some kind of override option on the ClusterObservability CR itself.

  apiVersion: opentelemetry.io/v1alpha1
  kind: ClusterObservability
  metadata:
    name: gateway
    namespace: observability
  spec:
    exporter:
      endpoint: "http://backend.observability.svc.cluster.local:4318"
    agent:
      patch:
        config:
          processors:
            resourcedetection:
              detectors: ["env", "system", "k8snode", "kubeadm"]
              timeout: 2s

That would keep the usage simple and clean.. But also grant a user access to actually all settings.

Describe alternatives you've considered

Another option and maybe the cleanest option could be to add a ClusterObservabilityPatch CR. Similar to https://gateway.envoyproxy.io/docs/tasks/extensibility/envoy-patch-policy/.

Example how to sett or change resource configuration:

apiVersion: opentelemetry.io/v1alpha1
kind: ClusterObservabilityPatch
metadata:
  name: collector-resources
  namespace: observability
spec:
  targetRef:
    name: cluster-observability
  patches:
    - target:
        kind: OpenTelemetryCollector
        name: agent
      type: merge
      value:
        spec:
          resources:
            limits:
              cpu: "500m"
              memory: "512Mi"
            requests:
              cpu: "100m"
              memory: "128Mi"

Additional context

No response

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[frzifus]

cc @jinja2 @atoulme