feat(metrics): make built-in summary objectives configurable#505
Open
tiagomartines11 wants to merge 1 commit into
Open
feat(metrics): make built-in summary objectives configurable#505tiagomartines11 wants to merge 1 commit into
tiagomartines11 wants to merge 1 commit into
Conversation
Add pitaya.metrics.prometheus.objectives to tune the quantiles emitted by
the built-in handler_response_time_ns and handler_handler_delay_ns
summaries, which were hard-coded to {0.7, 0.95, 0.99}. Operators can drop
unused percentiles (or emit none) to cut Prometheus series cardinality.
The map is keyed by the quantile string for YAML/env friendliness and
converted to map[float64]float64 when building SummaryOpts. The option is
intentionally not seeded with a viper default: a seeded default would be
deep-merged back into any partial override, so dropping a percentile via
config would silently fail. Unset resolves to nil, which the reporter maps
to the historical {0.7, 0.95, 0.99}, keeping default behavior identical.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Coverage Report for CI Build 29434363368Coverage increased (+0.5%) to 61.971%Details
Uncovered Changes
Coverage Regressions18 previously-covered lines in 2 files lost coverage.
Coverage Stats
💛 - Coveralls |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Make the objectives (quantiles) of the two built-in Prometheus summary metrics configurable:
pitaya_handler_response_time_ns(ResponseTime)pitaya_handler_handler_delay_ns(ProcessDelay)Both were hard-coded to
{0.7: 0.02, 0.95: 0.005, 0.99: 0.001}. This adds a new config keypitaya.metrics.prometheus.objectivesso operators can drop unused percentiles (or emit none) without forking pitaya.Why
Each summary emits, per label-set, one series per quantile plus
_sumand_count. With the hard-coded three quantiles that is 5 series per label combo. On a high-cardinality handler (e.g. a connector with hundreds of route×status×code combos) the quantile series are ~half of the/metricspayload and push the endpoint well past the Datadog OpenMetrics default 2,000-series cap, which causes truncation and phantom counter spikes.Client-side summary quantiles are per-pod and cannot be aggregated across pods, so teams that only chart
0.95/0.99still pay for0.7. Making objectives configurable lets them drop what they don't use.How
PrometheusConfig.Objectives map[string]float64(keyed by the quantile string, e.g."0.95", for YAML/env friendliness), converted tomap[float64]float64when buildingSummaryOpts.newSummaryVechelper.{0.7, 0.95, 0.99}— byte-for-byte identical to today.Note on the default (important)
The option is intentionally not seeded with a viper default. pitaya's custom
UnmarshalKeydeep-merges seeded default map keys back into any override, so a seeded default would make partial overrides silently regain the dropped percentiles (verified: setting{0.95, 0.99}still produced{0.7, 0.95, 0.99}). Instead, "unset" resolves tonil, andbuildObjectives(nil)supplies the historical default. This makes both "drop0.7" and "emit no quantiles" actually work.Config
{0.7, 0.95, 0.99}{"0.95": .., "0.99": ..}{}(empty)_sum/_countTests
metrics:buildObjectives(nil→default, custom, empty, invalid key) + scrape assertions that the registered summaries emit exactly the configuredquantile="..."series (default / dropped / none), and always_sum/_count.config: resolution through the realNewPitayaConfigviper path — unset→nil, partial-drop, empty-stays-empty (regression guard against re-seeding a default).Scope
spec.Summaries[].Objectives) unchanged — already configurable.ResponseTimehistogram registration is untouched (out of scope).Docs
docs/configuration.rstdocuments the new key, default, YAML shape, and the client-side-quantiles-don't-aggregate caveat.