Add observability for multicursor queue-state resolution loss - #11695
Open
ratdaddy wants to merge 6 commits into
Open
Add observability for multicursor queue-state resolution loss#11695ratdaddy wants to merge 6 commits into
ratdaddy wants to merge 6 commits into
Conversation
yycptt
approved these changes
Aug 21, 2026
|
|
||
| // TODO: this should be generic enough to shrink any predicate type, probably doesn't belong here. | ||
| pendingPerKey := s.pendingPerKey | ||
| metrics.QueueSlicePendingKeys.With(s.metricsHandler).Record(int64(len(pendingPerKey))) |
Member
There was a problem hiding this comment.
nit: just calling out that # of remaining keys is not very interesting for slices that's just created.
| // The counter is a true accumulator; the histogram's _sum is not, since tally's Prometheus | ||
| // reporter replays each sample as its bucket's upper bound, not the recorded value. | ||
| metrics.QueueSliceCountHistogram.With(p.metricsHandler).Record(sliceCount, categoryTag) | ||
| metrics.QueueSliceCountTotal.With(p.metricsHandler).Record(sliceCount, categoryTag) |
Member
There was a problem hiding this comment.
so basically we want to have a more accurate version of histogram's _sum and we can calculate average?
yycptt
enabled auto-merge (squash)
August 21, 2026 05:30
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed?
Adds metrics for how often and by how much queue slices
fail to narrow their predicate, and how large persisted queue state actually is.
queue_slice_pending_keys— histogram, recorded on every narrowing attempt (declined orsucceeded). This is the distribution
queueShrinkPredicateMaxPendingKeysshould be sized against.shard_info_size/queue_state_size— histograms recorded when a shard record is actuallywritten, giving whole-record and per-category size.
queue_state_size_total/queue_slice_count_total— counters paired with the histograms above(and with the existing
queue_slice_count), so an exact bytes-per-slice ratio is possible.queue_slice_countgains atask_categorytag (previously untagged beyondoperation).These are only metrics changes - no behavior changes.
Why?
A slice only narrows its predicate below
queueShrinkPredicateMaxPendingKeys(10) pendingnamespaces; above that it stays universal and re-reads the whole range every time. Raising that
threshold safely requires knowing the pending-key distribution and the persisted size.
This PR is the baseline for evaluating a follow-on encoding.
There are two counters because this server's tally-backed Prometheus reporter doesn't preserve the
true recorded value when a histogram flushes — it replays each sample as its bucket's upper bound,
so a histogram's
_sumhas no more precision than its buckets.How did you test it?
queue_predicate_resolution_lossconfirmed live against a local server under forcednarrowing-decline conditions)
Potential risks