Separate Dimensions for calls and duration metrics in Span Metrics

[EOjeah]

Component(s)

connector/spanmetrics

Is your feature request related to a problem? Please describe.

Currently, the span metrics connector allows users to specify dimensions to extract from span data when converting to metrics. However, these dimensions are applied uniformly to both the calls (counter) and duration (histogram) metrics. This feature request proposes an enhancement to allow users to specify dimensions separately for each metric type, providing more granular control over the metrics configuration.

Describe the solution you'd like

Introduce a configuration option within the span metrics connector that allows users to specify which metrics (calls or duration) the dimensions should be applied to. This would enable users to:

• Apply dimensions extraction to both metrics.
• Apply dimensions extraction only to the calls (counter) metric.
• Apply dimensions extraction only to the duration (histogram) metric.

Before

connectors:
  spanmetrics:
    dimensions:
      - name: "http.method"
      - name: "http.status_code"

After, breaking change since component is still in Alpha status

connectors:
  spanmetrics:
    dimensions:
      global: # Apply to all
        - name: "http.method" 
      calls:  # Optional, specific to calls (counter) metrics
        - name: "host.name"
      duration:  # Optional, specific to duration (histogram) metrics
        - name: "http.status_code"

OR for backward compatibility

connectors:
  spanmetrics:
    dimensions:
      - name: "http.method"
    calls_dimensions:  # Optional, specific to calls (counter) metrics
      - key: "host.name"
    duration_dimensions:  # Optional, specific to duration (histogram) metrics
      - key: "http.status_code"

Describe alternatives you've considered

The current workaround involves using the transform processor to modify the metric type after extraction. While this approach is functional, it adds complexity to the configuration process. By integrating this capability directly into the span metrics connector, users can achieve the desired configuration more intuitively and efficiently.

Additional context

No response

[github-actions]

Pinging code owners:

• connector/spanmetrics: @portertech @Frapschen

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[portertech]

Hey @EOjeah, thanks for the issue 👍 Are you able to provide more examples of cases where you need the separation of dimensions? Also, I would love to see relevant transform processor configuration without this potential feature.

[EOjeah]

Hi @portertech, thank you for reviewing this issue.

Let me explain the use case with examples:

Initial use case, with no added dimension and a single histogram bucket (1ms):

calls_total{} 1
duration{le="1"} 1
duration{le="+Inf"} 1

Use case 1, with extracted 'method' dimension:

calls_total{method="GET"} 1
calls_total{method="POST"} 1
duration{method="GET",  le="1"} 1
duration{method="GET",  le="+Inf"} 1
duration{method="POST", le="1"} 1
duration{method="POST", le="+Inf"} 1

While the calls_total metric's cardinality increases acceptably by the number of methods, the duration metrics increase by {number of buckets} * 'method' cardinality. We may not need histogram quantiles broken down by method.

This increased number of metrics will slow down related Prometheus queries.

With the default setting of about 15 buckets on the span metrics connector, the number of duration metrics would grow substantially compared to this example.

Instead, we'd prefer something like this:

calls_total{method="GET"} 1
calls_total{method="POST"} 1
duration{le="1"} 2
duration{le="+Inf"} 2

We can implement this using the metrics transform processor:

metricstransform: 
  transforms:
    - include: duration
      action: 'update'
      operations:
        - action: aggregate_labels
          label_set: ['method']
          aggregation_type: sum

However, this adds processing overhead to the collector since it must handle all metrics as an extra step in the pipeline. While this isn't problematic for most use cases, it could become an issue with very high volumes of traces/metrics.

[swar8080]

An idea is adding an option to disable the count/sum counters, and then configuring two span metric connectors: one just for counters and the other for histograms.

That'd allow tuning config other than just dimensions for counters vs. histograms. The code change would probably be easier too.

Not sure about the added cpu overhead of running two connectors but my guess is memory usage would be similar

[portertech]

@EOjeah I support this feature, now it's a matter of choosing our configuration approach. It's easy to move forward w/ backward compatibility in mind, however, the result is not as clear (or as explicit) as the hierarchy w/ "global". WDYT @Frapschen? Worthy of a breaking change? I'm thinking so.

@EOjeah I don't have capacity to develop this feature at this time, are you able to take it on?

[portertech]

We also need to account for exclude_dimensions.