Make labelstore.LabelStore in to a SeriesRefMappingStore

[kgeckhart]

Component(s)

Indirectly impacts all prometheus components

Background

labelstore.LabelStore is deeply integrated in to prometheus pipelines. Below is a sequence diagram of where it sits and its role in the pipelines,

sequenceDiagram
  prometheus.scrape->>fanout: append Ref 0
  fanout->>labelstore: GetOrAddGlobalRef
  labelstore->>fanout: GlobalRef 1
  loop for each remote_write component 0 ... n 
    fanout->>remote_write_n: append Ref 1
    remote_write_n->>labelstore: GetLocalRef for Ref 1
    labelstore->>remote_write_n: LocalRef or 0
    remote_write_n->>WAL: append
    WAL->>remote_write_n: LocalRef 1
    remote_write_n->>labelstore: Add/Update LocalLink (Ref 1 - LocalRef 1)
  end
  fanout->>prometheus.scrape: Ref 1
  Note over prometheus.scrape: Next scrape send same SeriesRef
  prometheus.scrape->>fanout: append Ref 1

Loading

This functionality exists because in alloy we can multi-cast appended samples to 0 or more remote_write components. Each remote_write component comes with a WAL that assumes it is responsible for allocating a globally unique SeriesRef as that is its responsibility in upstream prometheus. LabelStore ensures we have a SeriesRef that is re-usable for multiple calls to append.

The problem we have is that for each GlobalRef we add we are storing a copy of the labels in LabelStore. If the pipeline includes a prometheus.relabel component we have two GlobalRefs one before relabeling and one after with both label sets stored in memory.

The drawbacks to this design are,

1. Each GlobalRef we add we are storing a copy of the labels in LabelStore further increasing memory usage
   • If the pipeline includes a prometheus.relabel component we have two GlobalRefs one before relabeling and one after with both label sets stored in memory
2. Knowing when a ref should be removed from the LabelStore is complicated and requires tracking the staleness of each appended sample (it's complicated enough that there's an indefinite retention bug when used with clustering)
3. We do all of this in many scenarios where it's not necessary like,
   • There are no remote_write components running in alloy
   • One pipeline has a remote_write component and another one does not (both still require LabelStore)
   • A pipeline only contains a single remote_write component - we don't need LabelStore we can pass through the WAL SeriesRef instead

Proposal

Transition LabelStore so that it no longer allocates a GlobalRef for every series appended and instead only allocates a GlobalRef when the Ref returned from fanning out results in two different SeriesRefs. This directly translates to "there's two remote_write components involved in this pipeline and they have conflicting SeriesRef we need map between".

No longer storing any labels dramatically reduces the resource drawback (1), dramatically limits the surface area where we need to consider refs for cleanup (2), and ensures we are only doing this logic when we absolutely have to do it (3).

As a part of doing this we will also need to transition prometheus.relabel away from using the SeriesRef for the cache as that will no longer be reliable. We can instead use labels.Labels.String() or labels.Labels.Bytes() which is suitable to be used a key to a map.

Relevant Issues

Resolves #4020 which is a non-clustered pipeline with a single remote_write component

Tip

_{React with 👍 if this issue is important to you.}