This is a feature request to be able to deduplicate HA prometheus samples within a remote-write request.

Current state

Mimir supports receiving duplicated samples from multiple promethei and actually writing only one of the promethei. Docs

When a prometheus sends a remote-write request the distributor looks at the labels only of the first sample. The distributor assumes that all the samples in the remote-write request are from the same prometheus replica.

Feature request

Make the distributor look at each sample in the write request and apply the deduplication logic on each sample separately instead of applying it on all samples in the batch.

Use case

This can be useful when a user has set up prometheus federation with replication. At the top is mimir. Right below it are two per-datacenter promethei (prom_1, prom_2). Below those two pairs of HA promethei (prom_a_1, prom_a_2, prom_b_1, prom_b_2).

prom_a_1 and prom_a_2 scrape the same targets and prom_b_1, prom_b_2 scrape the same targets. All prom_a_1, prom_a_2, prom_b_1, and prom_b_2 remote-write to both prom_1 and prom_2. Only prom_a_* and prom_b_* add HA labels (__replica__ and cluster) to metrics.

Now the remote-write batches of prom_1 and prom_2 to Mimir can contain samples from any of the four bottom promethei. So in this case the batch can contain mixed samples from an elected replica (e.g. prom_a_1) and a secondary replica (e.g. prom_b_1). If the first sample in a batch is from prom_b_1 then the samples from prom_a_1 will also be discarded even though prom_a_1 is the elected replica.

In this case the user wants Mimir to choose one replica from prom_a_1|prom_a_2 and one from prom_b_1|prom_b_2 and consistently accepted samples from each regardless of how the batches are assembled.

Other notes

I have some sort of a POC locally. I will open a PR with some benchmarks soon.