Skip to content

[processor/cumulativetodelta] Histogram resets emit corrupted delta with full cumulative value instead of being dropped #48278

Description

@ethuleau

Component(s)

processor/cumulativetodelta

What happened?

Description

The cumulativetodeltaprocessor correctly drops sum and exponential histogram datapoints when a counter reset is detected (the new value is lower than the previous one (restart)). However, when a regular histogram is reset, the processor does not drop the point. Instead, it forwards the new datapoint with the full cumulative bucket counts and sum but tagged as AggregationTemporalityDelta.

This produces a corrupted delta: a one-time spike that includes both pre-reset and post-reset events. Downstream consumers will over count.

Where

processor/cumulativetodeltaprocessor/internal/tracking/tracker.go, in Convert for pmetric.MetricTypeHistogram:

delta := value.Clone()

// Calculate deltas unless histogram count was reset
if valid && delta.Count >= prevValue.Count {
    delta.Count -= prevValue.Count
    delta.Sum -= prevValue.Sum
    for index, prevBucket := range prevValue.BucketCounts {
        delta.BucketCounts[index] -= prevBucket
    }
}

out.HistogramValue = &delta  // when reset, this is the full cumulative value, not a delta

When delta.Count < prevValue.Count the subtraction is skipped but the point is still returned with valid=true, so the processor emits it as a delta.

Steps to Reproduce

Send the following sequence to the processor (one cumulative histogram metric, three points):

t count sum bucket_counts
1 10 100 [10]
2 20 200 [20]
3 5 50 [5]

Expected Result

Match the sum / exponential histogram paths: on reset, mark the point invalid so the processor drops it (keep in memory waiting next datapoint to compute the delta, if >=).

t count sum bucket_counts
2 10 100 [10]
(t=3 dropped)

Actual Result

t count sum bucket_counts
2 10 100 [10]
3 5 50 [5]

I attached a diff file with a unit test to reproduce that bug

cumulativetodelta-histo-reset-bug.patch

Collector version

main

Environment information

Environment

N/A — bug is in the processor logic itself.

OpenTelemetry Collector configuration

processors:
  cumulativetodelta:

Log output

N/A

Additional context

No response

Tip

React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions