Move preferred approach (__temporality__) to top

Author: fionaliao

proposals/0048-otel_delta_temporality_support.md

@@ -130,34 +130,14 @@ As per [Prometheus documentation](https://prometheus.io/docs/concepts/metric_typ
 
 We propose to add two options as feature flags for ingesting deltas:
 
-1. `--enable-feature=otlp-delta-as-gauge-ingestion`: Ingests OTLP deltas as gauges.
+1. `--enable-feature=otlp-native-delta-ingestion`: Ingests OTLP deltas with a new `__temporality__` label to explicitly mark metrics as delta or cumulative, similar to how the new type and unit metadata labels are being added to series.
 
-2. `--enable-feature=otlp-native-delta-ingestion`: Ingests OTLP deltas with a new `__temporality__` label to explicitly mark metrics as delta or cumulative, similar to how the new type and unit metadata labels are being added to series.
+2. `--enable-feature=otlp-delta-as-gauge-ingestion`: Ingests OTLP deltas as gauges.
 
-We would like to initially offer both options as they have different tradeoffs. The gauge option is more stable, since it's a pre-exisiting type and has been used for delta use cases in Prometheus already. The temporality label option is very experimental and dependent on other experimental features, but is a closer fit to the OTEL model.
+We would like to initially offer both options as they have different tradeoffs. The gauge option is more stable, since it's a pre-exisiting type and has been used for delta use cases in Prometheus already. The temporality label option is very experimental and dependent on other experimental features, but it brings Prometheus more in alignment with the OTEL model. The preferred approach from the Prometheus delta working group is the `__temporality__` option, but we need practical experience to validate this is actually the case.
 
 Below we explore the pros and cons of each option in more detail.
 
-#### Treat as gauge
-
-Deltas could be treated as Prometheus gauges. A gauge is a metric that can ["arbitrarily go up and down"](https://prometheus.io/docs/concepts/metric_types/#gauge), meaning it's compatible with delta data. In general, delta data is aggregated over time by adding up all the values in the range. There are no restrictions on how a gauge should be aggregated over time.
-
-Gauges ingested into Prometheus via scraping represent sampled values for the metric and using `sum_over_time()` for these types of gauges do not make sense. However, there are other sources that can ingest "delta" gauges already for which summing does make sense. For example, `increase()` outputs the delta count of a series over an specified interval. While the output type is not explicitly defined, it's considered a gauge. A common optimisation is to use recording rules with `increase()` to generate “delta” samples at regular intervals. When calculating the increase over a longer period of time, instead of loading large volumes of raw cumulative counter data, the stored deltas can be summed over time.
-
-When ingesting, the metric metadata type will be set to `gauge` / `gaugehistogram`. If type and unit metadata labels is enabled, `__type__="gauge"` / `__type__="gaugehistogram"` will be added as a label.
-
-**Pros**
-* Simplicity - this approach leverages an existing Prometheus metric type, reducing the changes to the core Prometheus data model.
-* Prometheus already uses gauges to represent deltas. For example, `increase()` outputs the delta of a counter series over an specified interval. While the output type is not explicitly defined, it's considered a gauge.
-* Non-monotonic cumulative sums in OTEL are already ingested as Prometheus gauges, meaning there is precedent for counter-like OTEL metrics being converted to Prometheus gauge types.
-
-**Cons**
-* Gauge has different meanings in Prometheus and OTEL. In Prometheus, it's just a value that can go up and down, while in OTEL it's the "last-sampled event for a given time window". While it technically makes sense to represent an OTEL delta counter as a Prometheus gauge, this could be a point of confusion for OTEL users who see their counter being mapped to a Prometheus gauge rather than a Prometheus counter. There could also be uncertainty for the user on whether the metric was accidentally instrumented as a gauge or whether it was converted from a delta counter to a gauge.
-* Scraped Prometheus gauges are usually aggregated in time by averaging or taking the last value, while OTEL deltas are usually summed. Treating both as a single type would mean there wouldn't be an appropriate default aggregation for gauges. Having a predictable aggregation by type is useful for downsampling, or applications that try to automatically display meaningful graphs for metrics.
-* The original delta information is lost upon conversion. If the resulting Prometheus gauge metric is converted back into an OTEL metric, it would be converted into a gauge rather than a delta metric. While there's no proven need for roundtrippable deltas, maintaining OTEL interoperability helps Prometheus be a good citizen in the OpenTelemetry ecosystem.
-
-The cons are generally around it being difficult to tell apart OTEL gauges and counters when ingested into Prometheus. An extension could be to [Add otel metric properties as labels](#add-otel-metric-properties-as-labels), so there is extra information users can use to decide on how to query the metric, while the Prometheus type remains a gauge.
-
 #### Introduce `__temporality__` label
 
 This option extends the metadata labels proposal (PROM-39). An additional `__temporality__` metadata label will be added. The value of this label would be either `delta` or `cumulative`. If the temporality label is missing, the temporality should be assumed to be cumulative.
@@ -176,10 +156,31 @@ Cumulative metrics ingested via the OTLP endpoint will also have a `__temporalit
 **Cons**
 * Introduces additional complexity to the Prometheus data model.
 * Confusing overlap between gauge and `__temporality__="delta"`. As mentioned in [Treat as gauge](#treat-as-gauge), essentially deltas already exist in Prometheus as gauges, and deltas can be viewed as a subset of gauges under the Prometheus definition. The same `sum_over_time()` would be used for aggregating these pre-existing deltas-as-gauges and OTEL deltas with counter type and `__temporality__="delta"`, creating confusion on why there are two different "types".
-  * Pre-existing deltas-as-gauges to counters with `__temporality__="delta"`, to have one consistent "type" which should be summed over time.
+  * Pre-existing deltas-as-gauges could be converted to counters with `__temporality__="delta"`, to have one consistent "type" which should be summed over time.
 * Dependent on the `__type__` and `__unit__` feature, which is itself experimental and requires more testing and usage for refinement.
 * Systems or scripts that handle Prometheus metrics may be unaware of the new `__temporality__` label and could incorrectly treat all counter-like metrics as cumulative, resulting in hard-to-notice calculation errors.
 
+
+#### Treat as gauge
+
+Deltas could be treated as Prometheus gauges. A gauge is a metric that can ["arbitrarily go up and down"](https://prometheus.io/docs/concepts/metric_types/#gauge), meaning it's compatible with delta data. In general, delta data is aggregated over time by adding up all the values in the range. There are no restrictions on how a gauge should be aggregated over time.
+
+Gauges ingested into Prometheus via scraping represent sampled values for the metric and using `sum_over_time()` for these types of gauges do not make sense. However, there are other sources that can ingest "delta" gauges already for which summing does make sense. For example, `increase()` outputs the delta count of a series over an specified interval. While the output type is not explicitly defined, it's considered a gauge. A common optimisation is to use recording rules with `increase()` to generate “delta” samples at regular intervals. When calculating the increase over a longer period of time, instead of loading large volumes of raw cumulative counter data, the stored deltas can be summed over time.
+
+When ingesting, the metric metadata type will be set to `gauge` / `gaugehistogram`. If type and unit metadata labels is enabled, `__type__="gauge"` / `__type__="gaugehistogram"` will be added as a label.
+
+**Pros**
+* Simplicity - this approach leverages an existing Prometheus metric type, reducing the changes to the core Prometheus data model.
+* Prometheus already uses gauges to represent deltas. For example, `increase()` outputs the delta of a counter series over an specified interval. While the output type is not explicitly defined, it's considered a gauge.
+* Non-monotonic cumulative sums in OTEL are already ingested as Prometheus gauges, meaning there is precedent for counter-like OTEL metrics being converted to Prometheus gauge types.
+
+**Cons**
+* Gauge has different meanings in Prometheus and OTEL. In Prometheus, it's just a value that can go up and down, while in OTEL it's the "last-sampled event for a given time window". While it technically makes sense to represent an OTEL delta counter as a Prometheus gauge, this could be a point of confusion for OTEL users who see their counter being mapped to a Prometheus gauge rather than a Prometheus counter. There could also be uncertainty for the user on whether the metric was accidentally instrumented as a gauge or whether it was converted from a delta counter to a gauge.
+* Scraped Prometheus gauges are usually aggregated in time by averaging or taking the last value, while OTEL deltas are usually summed. Treating both as a single type would mean there wouldn't be an appropriate default aggregation for gauges. Having a predictable aggregation by type is useful for downsampling, or applications that try to automatically display meaningful graphs for metrics.
+* The original delta information is lost upon conversion. If the resulting Prometheus gauge metric is converted back into an OTEL metric, it would be converted into a gauge rather than a delta metric. While there's no proven need for roundtrippable deltas, maintaining OTEL interoperability helps Prometheus be a good citizen in the OpenTelemetry ecosystem.
+
+The cons are generally around it being difficult to tell apart OTEL gauges and counters when ingested into Prometheus. An extension could be to [Add otel metric properties as labels](#add-otel-metric-properties-as-labels), so there is extra information users can use to decide on how to query the metric, while the Prometheus type remains a gauge.
+
 ### Metric names
 
 OTEL metric names are normalised when translated to Prometheus by default ([code](https://github.com/prometheus/otlptranslator/blob/94f535e0c5880f8902ab8c7f13e572cfdcf2f18e/metric_namer.go#L157)). This includes adding suffixes in some cases. For example, OTEL metrics converted into Prometheus counters (i.e. monotonic cumulative sums in OTEL) have the `__total` suffix added to the metric name, while gauges do not.
@@ -228,7 +229,7 @@ If we do not modify prometheusremotewritereceiver, then `--enable-feature=otlp-n
 
 ### Querying deltas
 
-For this initial proposal, existing functions will be used for querying deltas.
+For this initial proposal, existing functions will be used for querying deltas. This works for both the `__temporality__` and delta-as-gauge options.
 
 `rate()` and `increase()` will not work, since they assume cumulative metrics. Instead, the `sum_over_time()` function can be used to get the increase in the range, and `sum_over_time(metric[<range>]) / <range>` can be used for the rate. `metric / interval` can also be used to calculate a rate if the collection interval is known.
 
@@ -482,10 +483,6 @@ Users might want to convert back to original values (e.g. to sum the original va
 
 This also does not work for samples missing StartTimeUnixNano.
 
-#### Map non-monotonic delta counters to gauges
-
-Mapping non-monotonic delta counters to gauges would be problematic, as it becomes impossible to reliably distinguish between metrics that are non-monotonic deltas and those that are non-monotonic cumulative (since both would be stored as gauges, potentially with the same metric name). Different functions would be needed for non-monotonic counters of differerent temporalities. 
-
 ### Delta metric type alternatives
 
 #### Add delta `__type__` label values 
@@ -500,6 +497,12 @@ Additionally, combining temporality and type means that every time a new type is
 
 Have a convention for naming metrics e.g. appending `_delta_counter` to a metric name. This could make the temporality more obvious at query time. However, assuming the type and unit metadata proposal is implemented, having the temporality as part of a metadata label would be more consistent than having it in the metric name.
 
+### Monotonicity alternatives
+
+#### Map non-monotonic delta counters to gauges with `__temporality__` option
+
+With the `__temporality__` option, we could map monotonic deltas to the counter type, and non-monotonic counters to gauges. However, it becomes impossible to reliably distinguish between metrics that are non-monotonic deltas and those that are non-monotonic cumulative (since both would be stored as gauges, potentially with the same metric name), without adding [additional otel metric properties as labels](#add-otel-metric-properties-as-labels).
+
 ### Querying deltas alternatives
 
 #### Convert to cumulative on query