Targets in the same group are assigned to the same OTel

[mike9421]

Component(s)

target allocator

Is your feature request related to a problem? Please describe.

otel-allocator allocates multiple container endpoints of the same pod to different OTel. After the relabel operation, different OTel will collect the same metrics. Since I used cumulativeToDelta, the final metric data will be doubled.

Describe the solution you'd like

For example, the same pod detects the following targets:

• 10.22.14.119 --> assigned to OTel-0

• 10.22.14.119:8000 --> assigned to OTel-1

• 10.22.14.119:8080 --> assigned to OTel-2

The address will be changed to 10.22.14.119:8000 after the relabel operation, and all the above OTel will scrape the metric of 10.22.14.119:8000, which causes metric data errors.

Describe alternatives you've considered

After the otel - allocator performs the relabel operation, it unifies and deduplicates the items.
The selection of labels for deduplication is as follows:

• Select the label of the target where the original __address__ completely matches the __address__ after relabeling.

• If no such target exists, the labels of the first target will be used by default, while the metrics with the _meta prefix from the last target will overwrite the labels. This ensures consistency with the behavior of the OTel Prometheus receiver, for detail, see prometheus and prometheus receiver

The following are examples.

Additional context

No response

[swiatekm]

If I understand correctly, you'd like the target allocator to only keep one target with the same __address__? I don't think we can do that, as it's completely valid to scrape the same target multiple times with Prometheus, attaching different labels each time. As far as I know, neither Prometheus nor prometheusreceiver do any target merging based on __address__, and the links you provided do not show this.

[mike9421]

If I understand correctly, you'd like the target allocator to only keep one target with the same __address__? I don't think we can do that, as it's completely valid to scrape the same target multiple times with Prometheus, attaching different labels each time. As far as I know, neither Prometheus nor prometheusreceiver do any target merging based on __address__, and the links you provided do not show this.

Yes, I want the target allocator to retain only one target with the same __address__. And according to target sync code , I think it can be assumed that Prometheus performs deduplication for the scraped targets. An OTel instance can correctly deduplicate the metrics of the targets scraped using kubernetes_sd_configs according to Prometheus logic. However, when otel-allocator distributes targets to multiple OTel instances, since targets with the same address may be assigned to different OTel instances, OTel cannot perform deduplication.

[swiatekm]

And according to target sync code , I think it can be assumed that Prometheus performs deduplication for the scraped targets.

The hash computed in that line depends on all the labels, so it will be different for targets with the same address, but different labels. Target Allocator does something very similar.

To my understanding, Prometheus does not work the way you describe here. Have you tested this? I'm willing to be proven wrong and change target allocator to have behavior consistent with Prometheus, if it currently does not.

[mike9421]

@swiatekm There may be differences in the performance of the target allocator and Prometheus.

This is because the target allocator uses the original address (NewItem), while Prometheus uses the address after relabeling (1. Sync; 2. TargetsFromGroup; 3.PopulateLabels; 4. relabel ).

In this case, after the target allocator distributes targets through its distribution strategy, there may be an issue of duplicate scraping.

[swiatekm]

Ok, that does sound like a bug that you could hit if you're rewriting the address during relabeling. Would you like to submit a PR fixing it, given that you've already investigated the problem down to the code paths responsible?

[mike9421]

Ok, that does sound like a bug that you could hit if you're rewriting the address during relabeling. Would you like to submit a PR fixing it, given that you've already investigated the problem down to the code paths responsible?

@swiatekm I'd be happy to work on this issue. I'll submit a PR soon and would appreciate your review.