Proposal: Metric Aggregation Processor

[huyan0]

Metric Aggregation Processor Proposal

Objectives

The objective of the metric aggregation processor is to provide OpenTelemetry (OT) Collector users the ability to use exporters to send cumulative metrics to backend systems like Prometheus.

Background

Currently, the Prometheus exporter for the OT Collector is not functioning as expected. This is because OTLP Exporter in the SDK is pass through, and currently the Collector doesn't have any aggregation functionality, so the Collector receives deltas aggregations and instantaneous metric events and export them directly without converting them to cumulative values(#1255). In the Go SDK, aggregation of delta values is performed by the Accumulator and the Integrator(SDK Processor), but there is no similar component in the Collector. Any cumulative exporter that may be implemented in the future will encounter the same problem (a proposed exporter: #1150). A processor that maintains states of metrics (time series) and applies incoming deltas to cumulative values solves this problem.

Requirements

The processor should convert instantaneous and delta OTLP metrics to cumulative OTLP metrics. The following table proposes a mapping from instantaneous and delta metric kinds to cumulative metric kinds based on OTel-Proto PR open-telemetry/opentelemetry-collector#168. In the table, red entries are invalid type and kind combinations, white entries correspond to OTLP metrics that are already cumulative(or gauge), and should pass through the processor, and green entries correspond to OTLP metrics that the processor should aggregate.

*Grouping scalar indicates the most recent value to occur in a collection interval.
**Discussions around PR open-telemetry/opentelemetry-collector#168 are still ongoing.This table reflects a current status; will need to be updated when status changes

For green entries, the processor should maintain the cumulative value as a state. Metric state should be identified by a OTLP metric descriptor and a label set. For each metric of a stateful kind, the processor should aggregate cumulative values for every data point by labels.

Design Idea

The metric aggregation processor should be similar to the SDK Controller; it should contain:

• Accumulator, which is the entry point of OTLP metric events and manages incremental changes of each metric
• Integrator, which takes checkpoints of the incremental changes from Accumulator and apply them to cumulative values
• Ticker, which initiates the process of collection and sends metrics to the next metric consumer and
• Custom Aggregators, which maintain the state of metrics; a unique metric is defined by the combination of a metric name, a kind, a type and a label set.

The metric aggregation processor should accept concurrent updates of the metric state by taking in OTLP metrics passed from the pipeline, and perform collection periodically to send cumulative metrics to exporters, acting as a SDK push Controller. The following diagram illustrates the workflow.

cc @alolita @danielbang907 @huyan0 @jmacd

[tigrannajaryan]

We have metrictransform process which does some aggregation. Moving this after GA in case there is more work that we need to do in this area.

[nabam]

I'm happy to help with implementing that functionality. Is someone already working on it? Should such processor belong to core repository or contrib repository?
CC: @huyan0 @tigrannajaryan

[alolita]

Hi @nabam We (@alolita @amanbrar1999 @JasonXZLiu) are working on this. You're welcome to code review and provide feedback when we submit a PR.

[nabam]

@alolita that's great! It's going to be very useful for streaming telemetry from AWS Lambdas. Do you have any time estimates for that? I want to understand if we have to build some temporary custom solution while you are working in it

[amanbrar1999]

I believe that the use cases for this issue have since been solved by changes to the metrics proto, such as the addition of cumulative aggregation temporality. I could be wrong, does anyone know any specific use cases for this processor?
@tigrannajaryan @bogdandrutu @nabam

For example I believe this is a use case that has since been resolved: open-telemetry/opentelemetry-collector#1541

In the last comment of this linked issue it is mentioned that temporality translation exists as of OTLP v0.5, which I believe is what this processor is intended for

[nabam]

@amanbrar1999 one of the use-cases would be collecting metrics from short-living jobs such as AWS lambdas. Exporting uniquely tagged cumulative metrics for every lambda instance will cause high cardinality of time series they produce.

[amitsaha]

Having something like this will make open-telemetry/opentelemetry-python#93 possible. Without this, Python WSGI applications using gunicorn and uwsgi and the like will suffer from not being able to use Prometheus exporter directly. In fact, the lack of statsd exporter (open-telemetry/opentelemetry-specification#374) will make prometheus unusable with opentelemetry for this category of applications.

[pragmaticivan]

Howdy! @huyan0 @alolita, was there any progress towards implementing that proposal?

I'm currently validating a standalone (multiple workloads) Otel collector for single page apps/client/StatsD(Metric) collection but found that was not very clear.

I have taken a look at:

• https://github.com/prometheus/statsd_exporter
• https://github.com/weaveworks/prom-aggregation-gateway

Currently investigating this approach open-telemetry/opentelemetry-js#2118 although that will get affected by cardinality.

[andyzhangdialpad]

Thanks for this amazing proposal! @alolita was there any progress on the implementation? We are considering contributing to this. Don't want to reinvent the wheel of course.