Skip to content

Commit ec650bc

Browse files
chanceaclarkclaude
andcommitted
fix: TRAC-1066 Reset stale queue gauges instead of holding last value
Cloudflare's queueBacklogAdaptiveGroups GraphQL analytics endpoint only returns a row for a queue when there was measured activity in that scrape window. When a queue goes idle, it silently disappears from the response, and fetchQueueAnalytics only calls .Set() for queue IDs present in the response — so the gauge freezes at its last non-zero value forever instead of resetting. This caused MakeswiftStencilWorkerQueueBacklogCritical/Warning to fire in production reporting a 17.44k message backlog while the real queue was confirmed empty via wrangler. Thanos showed the metric flatlined at 691 for 13+ hours, then jumped to exactly 17444 and stayed frozen there on both exporter replicas — the signature of a stale gauge, not real queue depth. Drop each account's previous label combinations for all 7 queue gauges before repopulating them each scrape, mirroring the existing pattern in fetchWorkerWAEAnalytics. Idle or deleted queues now disappear from /metrics instead of holding stale values. Fixes TRAC-1066 Co-Authored-By: Claude <noreply@anthropic.com>
1 parent 98d8160 commit ec650bc

1 file changed

Lines changed: 12 additions & 0 deletions

File tree

prometheus.go

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -805,6 +805,18 @@ func fetchQueueAnalytics(account cfaccounts.Account, wg *sync.WaitGroup, deniedM
805805

806806
accountName := strings.ToLower(strings.ReplaceAll(account.Name, " ", "-"))
807807

808+
// Drop this account's previous label combinations so queues that have gone
809+
// idle (or no longer exist) disappear instead of holding their last-seen
810+
// value forever — Cloudflare's queueBacklogAdaptiveGroups only emits rows
811+
// for queues with activity in the scrape window.
812+
queueBacklogMessages.DeletePartialMatch(prometheus.Labels{"account": accountName})
813+
queueBacklogBytes.DeletePartialMatch(prometheus.Labels{"account": accountName})
814+
queueConsumerConcurrency.DeletePartialMatch(prometheus.Labels{"account": accountName})
815+
queueOperations.DeletePartialMatch(prometheus.Labels{"account": accountName})
816+
queueOperationBytes.DeletePartialMatch(prometheus.Labels{"account": accountName})
817+
queueOperationLagTime.DeletePartialMatch(prometheus.Labels{"account": accountName})
818+
queueOperationRetryCount.DeletePartialMatch(prometheus.Labels{"account": accountName})
819+
808820
resolveQueueName := func(queueID string) string {
809821
if name, ok := names[queueID]; ok {
810822
return name

0 commit comments

Comments
 (0)