Repository navigation
[SPARK-59935] Fix Prometheus _sum and quantiles of histograms and timers - #922
dongjoon-hyun wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
Thanks for the PR, @dongjoon-hyun!
_sum now comes from a running total in SummingHistogram and SummingTimer instead of the reservoir mean times the count. The non-nanos histograms now export the real median and 99.9th percentile. I checked that a SummingTimer created after registerSource reaches the Prometheus handler as itself, so _sum shows up through the real registration path. The one problem is the placeholder ticket in the migration guide. Separately, the operator_sdk_* histograms record whole seconds, so a reconcile under one second adds 0 to the new _sum. That's older than this PR, so I filed SPARK-59949 for it.
Blocking
- 1. Placeholder ticket in the migration guide: The new entry links
SPARK-XXXXXinstead of SPARK-59935. inline
| decrease between scrapes, so `rate()` of it and the average derived from it were wrong. Also, the | ||
| `0.5` and `0.999` quantiles of the `operator_sdk_*` histograms are the median and the 99.9th | ||
| percentile instead of the mean and the 99th percentile. A histogram or timer of a custom metrics | ||
| source has no `_sum` unless it is a `SummingHistogram` or a `SummingTimer` ([SPARK-XXXXX](https://issues.apache.org/jira/browse/SPARK-XXXXX)). |
There was a problem hiding this comment.
Finding 1. The entry still links the SPARK-XXXXX placeholder.
| source has no `_sum` unless it is a `SummingHistogram` or a `SummingTimer` ([SPARK-XXXXX](https://issues.apache.org/jira/browse/SPARK-XXXXX)). | |
| source has no `_sum` unless it is a `SummingHistogram` or a `SummingTimer` ([SPARK-59935](https://issues.apache.org/jira/browse/SPARK-59935)). |
There was a problem hiding this comment.
Thank you for the review, @peter-toth. I applied your suggestion in 27894eb. Thank you for filing SPARK-59949 for the whole seconds of the operator_sdk_* histograms, too.
|
Thank you, @peter-toth ! |
|
Merged to main |
2 participants
What changes were proposed in this pull request?
This PR fixes the Prometheus summaries of Dropwizard histograms and timers in
PrometheusPullModelHandler.SummingHistogramandSummingTimer, which the operator now uses, and export_sumonly for them.0.5and0.999quantiles of non-nanoshistograms, instead of the mean and the 99th percentile.Why are the changes needed?
_sumwas the mean of the reservoir multiplied by the count. Since the mean covers only roughly the last 5 minutes,_sumcould decrease between scrapes, which breaksrate(). This becomes visible once #918 records real latencies.Does this PR introduce any user-facing change?
Yes. The values of
_sumand of the quantiles above change compared to 1.0.0 (2026-07-26), while metric names and types are unchanged. The migration guide is updated.How was this patch tested?
Pass the CIs with the newly added test cases.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5