Component donation: exporter/lgtmexporter

[matt-gp]

Building and distributing outside this repository

• I understand I can use and distribute my component outside of this repository.

Existing component implementation

https://github.com/matt-gp/lgtmexporter

Components covering similar use cases

• extensions/headerssetterextension - Allows you to set static headers or headers using values from contexts or attributes returned from authentication. You can then forward to LGTM using the oltp exporter. However this does not apply the header value from the Resource Attributes.

The purpose and use-cases of the component

Grafana LGTM is a multi-tenant platform that stores Logs (Loki), Metrics (Mimir) and Traces (Tempo), which can then be visualised in Grafana. The method to read/write data is by providing a header (default X-Scope-OrgID) with the tenant id.

LGTM Exporter allows you to forward to Grafana LGTM using a resource attribute as the tenant id. This means that you can run a Collector as a gateway and forward multiple tenants data.

A real life example would be:

• You run a multi-tenant k8s cluster where each tenant gets a namespace.
• You would then collect logs/metrics/traces and apply the namespace as a resource attribute via relabelling.
• You would then configure the LGTM Exporter to use this namespace resource attribute as the tenant id.
• Then each tenant would then have their own data isolated from others in LGTM.

Example configuration for the component

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:

exporters:
  lgtm:
    logs:
      endpoint: http://loki:3100/otlp/v1/logs
      timeout: 30s
      tls:
        insecure: true
    metrics:
      endpoint: http://mimir:8080/otlp/v1/metrics
      timeout: 30s
      tls:
        insecure: true
    traces:
      endpoint: http://tempo:3200/v1/traces
      timeout: 30s
      tls:
        insecure: true
    tenant:
      label: tenant.id
      labels:
        - tenant_id
        - tenantID
        - tenantId
      header: X-Scope-OrgID
      format: "%s-dev"
      default: default
    content_type: protobuf
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 1000

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [lgtm]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [lgtm]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [lgtm]

Telemetry data types supported

traces, metrics, logs

Code Owners

matt-gp

Sponsor (optional)

No response

Additional context

Previously to do this I would use https://github.com/matt-gp/otel-lgtm-proxy and run it as a sidecar.

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[paulojmdias]

What is the value of having this new exporter instead of using the otlp one?

[matt-gp]

The oltp one can't dynamically set the header value depending on the value in the resource attributes.

The header value is what dictates the tenant in LGTM, so it's more than just an arbitrary header.

[paulojmdias]

There is currently a bug open for headersetter extension exactly for that.

I would say it is valuable to have that support within the extension, and for sure, the codeowners would love to have a PR for it.

\cc @VihasMakwana

[matt-gp]

Oh cool, I can look at adding a PR to that instead. Thanks

[paulojmdias]

Oh cool, I can look at adding a PR to that instead. Thanks

I will be really happy to review it! 👍

[matt-gp]

hmm just looking at the code in the extension.

I'm not sure extensions have access to the actual logs/metrics/traces data so not sure if they would be able to read the resource attributes.

[paulojmdias]

Looking into the following comment, it is probably the following PR that should even fix the issue. You could give a try 👍

[matt-gp]

Cool I'll see if it's possible, if not then we could revisit this.

[matt-gp]

Looking at the PR(s) a bit more. I don't think these solve this problem. They allow access to headers from incoming requests, rather than resource attributes.

PR #41879 stipulates enabling users to access client request metadata

It solves the problem that was commented, edge collectors adding a static header, sending into a central collector, central collector can then forward on using the header value that the edge collector applied. But this wouldn't cover situations where you are scraping from exporters, 3rd party exporter or situations where you don't have an incoming http request, such as the example that I gave with a k8s cluster.

So all in all, I don't think that you could just use headerssetter along with the oltp as a replacement for this.

Happy to be wrong though.

[matt-gp]

@paulojmdias am I ok to move forward and try and find a sponsor?

[paulojmdias]

@paulojmdias am I ok to move forward and try and find a sponsor?

I still think that is possible within the headerssetter extension. I can help look into it if is the case.

However, I also don't think it makes sense a single receiver with the same support for OTLP as the generic OTLP receiver supports. Worst case scenario, it should makes more sense a new extension (if by some hard reason we cannot extend the support into headerssetter extension).