Proposal: Add type and unit metadata labels

[dashpole]

Some screenshots from the Prometheus UI after this PoC: prometheus/prometheus#15683

The PoC updates all PromQL tests to demonstrate that adding type and unit labels doesn't break existing queries.

Querying for __unit__

[ArthurSens]

Great start! I think we need to discuss a bit more whether to handle __* labels in the PromQL engine or the UI. Maybe there's a chance we're doing a breaking change...?

[beorn7]

Thank you very much for proposing this. I love it in general, and as said elsewhere, I had a very similar idea a while ago for completely different reasons (mostly to avoid mixed series with native histograms and float-based sample types). Finding a similar solution for unrelated problems seems like a signal that this could be actually useful. However, I have also run into some conundrums, which was the reason why I put the whole idea on the backburner. I'll try to sketch out my train of thought here:

The __name__ label is the precedent here for "special labels", and as such it gives us a hint of what the issues might be. It's special power is not just about how to display it. It is treated specially for label matching and aggregation, and it is removed by most operations, following the argument that a rate of process_cpu_seconds_total is not process_cpu_seconds_total anymore. We have to ask ourselves the same questions for __unit__ and __type__.

In the brave new world, process_cpu_seconds_total would probably look like process_cpu{__type__="counter", __unit__="seconds"}. If we rate that, the outcome is not a counter anymore (but a gauge), and it is actually unit-less (seconds per second, i.e. it is the CPU usage ratio. In wishful thinking, the outcome should be process_cpu_usage{__type__="gauge", __unit__="ratio"}. Of course, we cannot easily come up with a general procedure to make up new names (which is the reason why current PromQL simply removes the name), and even changing the unit is tough. Changing the type might actually be feasible (we explored that a bit with native histograms, which "know" if they are a counter histogram or a gauge histogram). So maybe we can come up with "type translation rules" for all PromQL operations, and then maybe drop the unit in doubt (when calculating a rate or multiplying) and keep it when it makes sense (a sum aggregation or adding). But you will notice how deep we are in the woods here already.

The next thing is aggregation and label matching. As said, __name__ is also special here, but we probably need to be special in different ways for __unit__ and __type__. For example if we simply do a + b, we probably want to include unit and type in the label match (only add gauges to gauges, only add seconds to seconds etc.). However, a * b is already very different. b might be a scaling factor, i.e. a unit-less gauge. a could be counters or histograms. So in this case, we would like to exclude the unit and type from the label match.

It gets very confusing quickly. I wish we could just add the __unit__ and __type__ as an experiment without further special treatment and see how people cope with it. But right now my worry is it will hit too many road bumps…

[beorn7]

It gets very confusing quickly. I wish we could just add the __unit__ and __type__ as an experiment without further special treatment and see how people cope with it. But right now my worry is it will hit too many road bumps…

Still in brainstorming mode, here is an idea how maybe a step zero could look like: Add the __unit__ and __type__ upon ingestion, but mostly ignore them in PromQL operations, maybe with the following exceptions:

• __unit__ and __type__ can be used in label selectors. This allows the filtering (listed as a goal in this doc) and could also be used to increase readability of a PromQL expression.
• __unit__ and __type__ can still be added via label_replace, e.g. to improve display of a result or to give the result of recording rules the unit and type information.

However, aggregations and label matches would ignore __unit__ and __type__ and would therefore work as before, and any operation would remove the __unit__ and __type__ label (with the exception of label_replace), both meant to circumvent any of the issues laid out above.

From there, we could cautiously move forward with more ideas. A step one might then be to handle __type__ in a more "native" fashion, i.e. attaching the "correct" __type__ label depending on the operation, still allowing the user to override via label_replace (like hard-casting a type in C), but maybe enforce a histogram type on native histograms (and prevent such a type on floats), so that we would avoid mixed series. Also, taking __type__ into account during aggregations might be possible.

[bwplotka]

unit and type can still be added via label_replace, e.g. to improve display of a result or to give the result of recording rules the unit and type information.

Hm, I think it would be also nice to display them when collision actually occurs 🤔

[bwplotka]

Nice work! Looking good, some comments.

[dashpole]

@beorn7 I've added "Aggregations and label matches ignore __unit__ and __type__ and any operation would remove the __unit__ and __type__ label (with the exception of label_replace)." based on your suggestion above. I've also added "Handle type and unit in PromQL operations" as a potential future extension.

[beorn7]

Quick note about federation: To my knowledge, federation still completely ignores metadata, so the exposition format it generates has all metric types as "untyped" and no units and help strings anyway, simply because Prometheus doesn't know about metadata post-ingestion. So being able to funnel unit and type through the Prometheus TSDB via labels would be a direct improvement for federation.

[ArthurSens]

Quick note about federation: To my knowledge, federation still completely ignores metadata, so the exposition format it generates has all metric types as "untyped" and no units and help strings anyway, simply because Prometheus doesn't know about metadata post-ingestion. So being able to funnel unit and type through the Prometheus TSDB via labels would be a direct improvement for federation.

I think it would be better to enable metadata post-scrape/federation than adding type and unit as labels in the exposition format. See #39 (comment)

[beorn7]

Quick note about federation: To my knowledge, federation still completely ignores metadata, so the exposition format it generates has all metric types as "untyped" and no units and help strings anyway, simply because Prometheus doesn't know about metadata post-ingestion. So being able to funnel unit and type through the Prometheus TSDB via labels would be a direct improvement for federation.

I think it would be better to enable metadata post-scrape/federation than adding type and unit as labels in the exposition format. See #39 (comment)

The problem is that federation breaks the assumption that all metrics of the same name (one metric family) all have the same metadata.

[bwplotka]

The problem is that federation breaks the assumption that all metrics of the same name (one metric family) all have the same metadata.

Yea, I wonder if we could lift this with OM 2.0. WDYT @beorn7 ?

Related discussion: #39 (comment)

[beorn7]

The problem is that federation breaks the assumption that all metrics of the same name (one metric family) all have the same metadata.

Yea, I wonder if we could lift this with OM 2.0. WDYT @beorn7 ?

The current structure of both protobuf and text versions of OM doesn't really lend itself to multiple different type, unit, or help entries for metrics with the same name. So we needed a new structural way of specifying unit and type (let's ignore help for now) per metric (rather than per metric family). Ironically, we already have a place for things that are per metric. We call it labels. So let's put type and metric into labels? Wait… that's what this proposal actually proposes. 😁

[bwplotka]

Well yes, but we just discussed that type and unit per label will not be easily parsable in OM/text, so putting it somewhere else actually helps. There might be a big questions there:

1. Is Metric Family concept actually useful these days? Should OM 2.0 simply redefine it, and flat that out? Personally it's only confusing, we don't use this semantic on the storage or PromQL (AFAIK). 💣
2. What's wrong to keep metric families but make them unique per type and unit? (not only name)?

[beorn7]

Well yes, but we just discussed that type and unit per label will not be easily parsable in OM/text, so putting it somewhere else actually helps.

I wouldn't say that this is actually the case. The way we parse the labels doesn't really create a lot of parsing cost if there are more labels. It does create more payload, but not so much in relative terms. The OM text format is designed to have a lot of repetitive information anyway. If we want to avoid repetitive information, we needed a fundamentally different structure.

1. Is Metric Family concept actually useful these days? Should OM 2.0 simply redefine it, and flat that out? Personally it's only confusing, we don't use this semantic on the storage or PromQL (AFAIK). 💣

It correctly models the per-scrape data model. It avoids (a part of the) repetition. Of course you can flat that out, but then you get the repetition that was marked above as "not easily parseable". My claim is that "flattening that out" is pretty much equivalent to "adding that as labels".

2. What's wrong to keep metric families but make them unique per type and unit? (not only name)?

Well, the text format originally was designed to not be ordered in any way. So every line was keyed by the metric family name, i.e. a metric line starts with the metric family name, and all the metadata lines contain the metric family name as the 2nd token after the hash. OM became somewhat dependent on order, but not consequently so (you could avoid a lot of repetition if you did that) (another of the design problems I see with OM). With the original idea of the text format structure, you needed to repeat not just the metric family name on each line, but the combination of metric family name, type, and unit (which, you guessed it, is equivalent to putting type and unit into a label). Alternatively, we could redesign the text format fundamentally and make it really depend on the order everywhere, but then we should do it thoroughly and avoid other repetitions, too (at the very least the metric family name).

Protobuf is a bit different as it is structured, and all the family members are a repeated Metric message inside a MetricFamily message. So keying on more than just the name is relatively easy to accomplish. This is another reason why I think we should "simply" solve the protobuf parsing performance issue and then go back to "if you want efficiency, use protobuf – if you want simplicity, use text" (simplicity as in "easy to create in a valid form", which means kicking out many of the requirements OM text introduced on top of the original text format, like order, whitespace, …).

[bwplotka]

The OM text format is designed to have a lot of repetitive information anyway. If we want to avoid repetitive information, we needed a fundamentally different structure.

I thought it's more the fact we usually need to know what's the metric type (and metric family name) ahead of time (e.g. histograms have a different flow). But you are right, perhaps we could make label based approach work fine. Then it's only human readability question and what's defined in SDK and what's queried.

It correctly models the per-scrape data model.

I think I got it now, FAIK this only makes a difference for unstructured types like classic histogram and summary where metric name you define in SDKs (aka metric family name) is != resulting metric name. In the word with native histograms and perhaps native summaries one day, it would make no different, right?

My claim is that "flattening that out" is pretty much equivalent to "adding that as labels".

Agree.

Well, the text format originally was designed to not be ordered in any way. So every line was keyed by the metric family name, i.e. a metric line starts with the metric family name, and all the metadata lines contain the metric family name as the 2nd token after the hash. OM became somewhat dependent on order, but not consequently so (you could avoid a lot of repetition if you did that) (another of the design problems I see with OM). With the original idea of the text format structure, you needed to repeat not just the metric family name on each line, but the combination of metric family name, type, and unit (which, you guessed it, is equivalent to putting type and unit into a label). Alternatively, we could redesign the text format fundamentally and make it really depend on the order everywhere, but then we should do it thoroughly and avoid other repetitions, too (at the very least the metric family name).

Got it, thank you for detailed explanation! I think we should discuss those options in a clear proposal, make some decision on this in OM 2.0, will add issue for this.

Protobuf is a bit different as it is structured, and all the family members are a repeated Metric message inside a MetricFamily message. So keying on more than just the name is relatively easy to accomplish. This is another reason why I think we should "simply" solve the prometheus/prometheus#14668 and then go back to "if you want efficiency, use protobuf – if you want simplicity, use text" (simplicity as in "easy to create in a valid form", which means kicking out many of the requirements OM text introduced on top of the original text format, like order, whitespace, …).

Nice! Proposing as an intention in our OM 2.0 doc

[bwplotka]

Draft PR for the initial state of a type-and-unit-labels feature flag: prometheus/prometheus#16228

I also went ahead and updated this proposal:

• Added requirements from UTF-8 and Delta angles; mentioned blockers. Removed solution suggestions from Goals. Added goal to not break or negatively impact existing users.

Refreshed goals:

* [Required] Do not break existing suers; do not force them to use more complex PromQL.
* [Required] Handle correctly cases of multiple series with the same metric name but different type and unit.
  * This case can happen already with the native and classic histograms, especially during migrations.
  * Blocker for OTLP no-translation option.
* [Required] Richer Prometheus (e.g. PromQL, relabel, recording) functionality and UX depending on type and unit.
  * Blocker for [delta type](https://docs.google.com/document/d/15ujTAWK11xXP3D-EuqEWTsxWiAlbBQ5NSMloFyF93Ug/edit?tab=t.0#heading=h.5sybau7waq2q).
* [Required] OpenTelemetry users can query for the original names of their metrics.
* [Nice to have] Path for the further innovations around metric identity (e.g. [seamless renames](https://www.google.com/url?q=https://sched.co/1txHv&sa=D&source=docs&ust=1742288864071431&usg=AOvVaw06J0dGuqUNJPPPNF17Eeg8))
* [Nice to have] Improve PromQL UX that errors/warns when using "inappropriate" operations for a metric type.
* [Nice to have] OpenTelemetry users can grep for the original names in the text exposition.

• Added new resources around vision, discussions.
• Updated "WHY" section to focus on other benefits than just Otel users.
• Updated "HOW" (old User Experience) to mention overriding proposal (all user provided type and unit labels will be overridden or dropped). Also added When a query will drop metric name in a effect of an operation or function typeandunit will be also dropped. statement (matching what's written in PromQL changes and what's suggested in https://github.com/prometheus/proposals/pull/39/files#r1933033658).
• Updates alternatives, added OpenQuestions and Considerations (for pros & cons).

I am not sure of the following statement:

Aggregations and label matches ignore __unit__ and __type__

Not going to implement this in the first iteration of the type-and-unit-labels feature flag. Same with UI changes.

[ywwg]

👍🏻 from me from the perspective of allowing Prometheus to expand to support Otel without sacrificing its specificity and preferred way of doing things.

[ArthurSens]

The only concern I have with the proposal is the mention of extra PromQL functionality in the future. I'm not sure we want it and I'm a bit worried if this proposal implies that we'll do it at some point. I'm not blocking the proposal since you mentioned that we might work on it!!!

I've added a few nits here and there, but overall LGTM!

[bwplotka]

Thanks all for comments, will update this week!

[bwplotka]

Addressed comments and slightly adjusted the proposal.

Bigger changes:

• Postponed Prom UI changes to extension. I don't think it's healthy to do this now.
• Clarified classic histograms and summaries challenge.
• Clarified rejection of PromQL short-syntax for now.
• Important: I noticed elephant in the room, so really undefined type and unit values. I expanded in the proposal the challenge and the next steps. I think PromQL should define syntax of type that it supports and normalization for "unknown" type. We could also consider tight definition of units e.g. Otel is requiring ugly and complex but flexible UCUM standard for units..

PTAL for the final (hopefully) review!

@fionaliao @carrieedwards @beorn7 @ArthurSens

[carrieedwards]

The first sentence of this paragraph is a bit confusing.

[ArthurSens]

Some very small comments and questions. Overall LGTM

Thanks Bartek, David and everyone else who worked on this ❤️

[ArthurSens]

I'm not sure if I understand the sentence here... You're saying that PRW v1 will never be able to produce the __unit__ and __type__ labels, and we'll drop them if metrics are sent with those labels even without a proper replacement. Is that correct?

[ArthurSens]

Not sure if I'm missing something here, isn't --query.max-samples an alternative for protection?

[ArthurSens]

I think all parsers included in Prometheus already do this validation, no? I'm not 100% sure, but I think we can't expose metric types that Prometheus doesn't support.. it will fail the scrape with errors

[bwplotka]

TODO: Explain the problem more.

[krajorama]

LGTM with one comment on clarifying what remote write client does (or point me to where it's already defined :) )

[krajorama]

What about the Remote Write clients?
PRW 2.0: options:

• send both __type__ and __unit__ labels and metadata (krajo: this seems wasteful on account of duplicate data),
• do not send __type__, __unit__ labels and rely on receiver to add them back based on the metadata (krajo: my choice, backwards compatible),
• send __type__ and __unit__ labels and no metadata (krajo: not possible because of missing help text).

PRW 1.0:

• send both __type__ and __unit__ labels and metadata (krajo: seems wasteful on account of no symbol table for the labels),
• do not send __type__, __unit__ labels and rely on receiver to add them back based on the metadata (krajo: not possible because metadata is sent out of band),
• send __type__ and __unit__ labels and no metadata (krajo: not possible because of missing help text).

[fionaliao]

Overall LGTM, but I had the same question as Arthur around whether user-provided type and unit labels are always removed like in the case of RW 1.0 (#39 (comment)).