Skip to content

Commit b92e920

Browse files
chanceaclarkclaude
andcommitted
fix: TRAC-1066 Reset stale queue gauges instead of holding last value
Cloudflare's queueBacklogAdaptiveGroups GraphQL analytics endpoint only returns a row for a queue when there was measured activity in that scrape window. When a queue goes idle, it silently disappears from the response, and fetchQueueAnalytics only calls .Set() for queue IDs present in the response — so the gauge freezes at its last non-zero value forever instead of resetting. This caused MakeswiftStencilWorkerQueueBacklogCritical/Warning to fire in production reporting a 17.44k message backlog while the real queue was confirmed empty via wrangler. Thanos showed the metric flatlined at 691 for 13+ hours, then jumped to exactly 17444 and stayed frozen there on both exporter replicas — the signature of a stale gauge, not real queue depth. Drop each account's previous label combinations for all 7 queue gauges before repopulating them each scrape, mirroring the existing pattern in fetchWorkerWAEAnalytics. Idle or deleted queues now disappear from /metrics instead of holding stale values. Fixes TRAC-1066 Co-Authored-By: Claude <noreply@anthropic.com>
1 parent 7666c47 commit b92e920

1 file changed

Lines changed: 12 additions & 0 deletions

File tree

prometheus.go

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -815,6 +815,18 @@ func fetchQueueAnalytics(account cfaccounts.Account, wg *sync.WaitGroup, deniedM
815815

816816
accountName := strings.ToLower(strings.ReplaceAll(account.Name, " ", "-"))
817817

818+
// Drop this account's previous label combinations so queues that have gone
819+
// idle (or no longer exist) disappear instead of holding their last-seen
820+
// value forever — Cloudflare's queueBacklogAdaptiveGroups only emits rows
821+
// for queues with activity in the scrape window.
822+
queueBacklogMessages.DeletePartialMatch(prometheus.Labels{"account": accountName})
823+
queueBacklogBytes.DeletePartialMatch(prometheus.Labels{"account": accountName})
824+
queueConsumerConcurrency.DeletePartialMatch(prometheus.Labels{"account": accountName})
825+
queueOperations.DeletePartialMatch(prometheus.Labels{"account": accountName})
826+
queueOperationBytes.DeletePartialMatch(prometheus.Labels{"account": accountName})
827+
queueOperationLagTime.DeletePartialMatch(prometheus.Labels{"account": accountName})
828+
queueOperationRetryCount.DeletePartialMatch(prometheus.Labels{"account": accountName})
829+
818830
resolveQueueName := func(queueID string) string {
819831
if name, ok := names[queueID]; ok {
820832
return name

0 commit comments

Comments
 (0)