fix: TRAC-1072 Seed idle queue gauges at zero instead of deleting#20
Merged
chanceaclark merged 1 commit intoJul 6, 2026
Conversation
TRAC-1066 fixed cloudflare_queue_backlog_messages and related gauges freezing at a stale non-zero value by calling DeletePartialMatch on all 7 queue gauges before repopulating them each scrape. This overcorrected: since Cloudflare's queueBacklogAdaptiveGroups and queueConsumerMetricsAdaptiveGroups only emit rows for queues with activity in the scrape window, an idle queue's series got wiped and never repopulated, leaving Grafana with no data for hours at a time instead of a correct zero baseline. For the 3 gauges keyed only by queue_name/account (backlog messages, backlog bytes, consumer concurrency), seed every currently known queue at zero (from fetchQueueNames) immediately after the DeletePartialMatch calls and before applying this scrape's real values. Idle queues now read a continuous zero instead of disappearing. The 4 operation-metric gauges keep the delete-only behavior since their action_type/outcome label combinations aren't enumerable ahead of time. Fixes TRAC-1072 Co-Authored-By: Claude <noreply@anthropic.com>
chanceaclark
deleted the
chancellorclark/ltrac-1105-queue-backlog-metrics-show-gaps-in-grafana-after-trac-1066
branch
July 6, 2026 18:29
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request Template
Description
Follow-up to #19, which fixed
cloudflare_queue_backlog_messagesand related gauges freezing at a stale non-zero value by callingDeletePartialMatch({"account": accountName})on all 7 queue gauges infetchQueueAnalyticsbefore repopulating them each scrape.That overcorrected: a queue that's idle most of the time will have no rows in Cloudflare's
queueBacklogAdaptiveGroups/queueConsumerMetricsAdaptiveGroupsresponses, since those endpoints only emit rows for queues with activity in the scrape window. The delete-then-rebuild wiped the series on every idle scrape and it never got repopulated, so downstream dashboards showed no data at all for hours at a stretch instead of a correct zero baseline. Confirmed in production: after the #19 deploy, an idle queue's backlog/concurrency series had one brief real reading, then many hours of zero data points, while the exporter itself stayed healthy and scraping normally.Fix: for the 3 gauges keyed only by
queue_name/account(queueBacklogMessages,queueBacklogBytes,queueConsumerConcurrency), seed every currently known queue (fromfetchQueueNames) at0immediately after theDeletePartialMatchcalls and before applying this scrape's real values. Idle queues now read a continuous0instead of disappearing. The 4 operation-metric gauges (queueOperations,queueOperationBytes,queueOperationLagTime,queueOperationRetryCount) keep the delete-only behavior, since theiraction_type/outcomelabel combinations aren't enumerable ahead of time the wayqueue_nameis.Jira: TRAC-1072
Type of Change
Testing
make pr-testsand all tests passgo build ./...andgofmt -l .are clean, and the pre-commitgolangci-linthook passed.make pr-tests'stesttarget (run_e2e.sh) requires live Cloudflare API credentials via.envandvenom, not run here.Manual verification post-deploy: watch a normally-idle queue's
cloudflare_queue_backlog_messages/cloudflare_queue_consumer_concurrencyseries — it should read a continuous0during idle periods instead of showing gaps or a stuck non-zero value.Code Quality
Additional Notes
Requires a downstream rebuild/redeploy trigger once this merges.