Add namespace selecting header

[mamari90]

When using OTEL collector to send data to APM it would be helpful to have the possibility to choose the elastic namespace where to save the data (traces, logs and metrics).
In my scenario we use a single cluster to handle the data of multiple environments or sets of applications, and being able to change the namespace used will allow me to fine tune the RBAC on those indices, avoiding security problems.

Proposal:
This could be done using a custom header set on the otel exporter configuration, so that the otel collector on a (let's say) k8s cluster could sent data to a namespace, and another collector on another cluster will send them to a different namespace

extraction of otel configuration

    exporters:
      logging:
        loglevel: warn

      otlphttp/elastic:
        tls:
          insecure: true
        endpoint: "${apm_endpoint}"
        headers:
          Authorization: "ApiKey ${apm_api_key}"
          X-Apm-Namespace: "my.namespace" # APM server will use this value if defined

[inge4pres]

hello @mamari90 👋🏼

when using OTLP you are able to inject the namespace via resource attributes.
We released a feature in 8.13 that allows adding resource level attributes to map data to data streams.

This means that you could either add an environment variable config to the SDKs

OTEL_RESOURCE_ATTRIBUTES="data_stream.namespace=<your-env>"

or create multiple pipelines in the OTel collector config, adding the attributes via attributes processor

You can even add the same attribute as a record-level information, so having multiple namespaces managed inside the application, but this is not coherent with the intent you described of splitting by environment.

As a follow up, we created #16204 to improve the documentation.

[mamari90]

thank you very much @inge4pres for your insight