Skip to content

Commit 121932f

Browse files
chanceaclarkclaude
andcommitted
fix: TRAC-1072 Seed idle queue gauges at zero instead of deleting
TRAC-1066 fixed cloudflare_queue_backlog_messages and related gauges freezing at a stale non-zero value by calling DeletePartialMatch on all 7 queue gauges before repopulating them each scrape. This overcorrected: since Cloudflare's queueBacklogAdaptiveGroups and queueConsumerMetricsAdaptiveGroups only emit rows for queues with activity in the scrape window, an idle queue's series got wiped and never repopulated, leaving Grafana with no data for hours at a time instead of a correct zero baseline. For the 3 gauges keyed only by queue_name/account (backlog messages, backlog bytes, consumer concurrency), seed every currently known queue at zero (from fetchQueueNames) immediately after the DeletePartialMatch calls and before applying this scrape's real values. Idle queues now read a continuous zero instead of disappearing. The 4 operation-metric gauges keep the delete-only behavior since their action_type/outcome label combinations aren't enumerable ahead of time. Fixes TRAC-1072 Co-Authored-By: Claude <noreply@anthropic.com>
1 parent b92e920 commit 121932f

1 file changed

Lines changed: 21 additions & 4 deletions

File tree

prometheus.go

Lines changed: 21 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -815,10 +815,8 @@ func fetchQueueAnalytics(account cfaccounts.Account, wg *sync.WaitGroup, deniedM
815815

816816
accountName := strings.ToLower(strings.ReplaceAll(account.Name, " ", "-"))
817817

818-
// Drop this account's previous label combinations so queues that have gone
819-
// idle (or no longer exist) disappear instead of holding their last-seen
820-
// value forever — Cloudflare's queueBacklogAdaptiveGroups only emits rows
821-
// for queues with activity in the scrape window.
818+
// Drop this account's previous label combinations so queues that no longer
819+
// exist disappear instead of holding their last-seen value forever.
822820
queueBacklogMessages.DeletePartialMatch(prometheus.Labels{"account": accountName})
823821
queueBacklogBytes.DeletePartialMatch(prometheus.Labels{"account": accountName})
824822
queueConsumerConcurrency.DeletePartialMatch(prometheus.Labels{"account": accountName})
@@ -827,6 +825,25 @@ func fetchQueueAnalytics(account cfaccounts.Account, wg *sync.WaitGroup, deniedM
827825
queueOperationLagTime.DeletePartialMatch(prometheus.Labels{"account": accountName})
828826
queueOperationRetryCount.DeletePartialMatch(prometheus.Labels{"account": accountName})
829827

828+
// Seed every currently known queue at zero before applying this scrape's
829+
// real values. Cloudflare's queueBacklogAdaptiveGroups/
830+
// queueConsumerMetricsAdaptiveGroups only emit rows for queues with
831+
// activity in the scrape window, so without this an idle queue would
832+
// disappear from /metrics entirely instead of reading zero. The 4
833+
// operation-metric gauges below aren't seeded because their action_type/
834+
// outcome label combinations aren't known ahead of time.
835+
for _, qName := range names {
836+
if !deniedMetricsSet.Has(queueBacklogMessagesMetricName) {
837+
queueBacklogMessages.With(prometheus.Labels{"queue_name": qName, "account": accountName}).Set(0)
838+
}
839+
if !deniedMetricsSet.Has(queueBacklogBytesMetricName) {
840+
queueBacklogBytes.With(prometheus.Labels{"queue_name": qName, "account": accountName}).Set(0)
841+
}
842+
if !deniedMetricsSet.Has(queueConsumerConcurrencyMetricName) {
843+
queueConsumerConcurrency.With(prometheus.Labels{"queue_name": qName, "account": accountName}).Set(0)
844+
}
845+
}
846+
830847
resolveQueueName := func(queueID string) string {
831848
if name, ok := names[queueID]; ok {
832849
return name

0 commit comments

Comments
 (0)