Skip to content

fix: TRAC-1072 Seed idle queue gauges at zero instead of deleting#20

Merged
chanceaclark merged 1 commit into
masterfrom
chancellorclark/ltrac-1105-queue-backlog-metrics-show-gaps-in-grafana-after-trac-1066
Jul 6, 2026
Merged

fix: TRAC-1072 Seed idle queue gauges at zero instead of deleting#20
chanceaclark merged 1 commit into
masterfrom
chancellorclark/ltrac-1105-queue-backlog-metrics-show-gaps-in-grafana-after-trac-1066

Conversation

@chanceaclark

@chanceaclark chanceaclark commented Jul 6, 2026

Copy link
Copy Markdown

Pull Request Template

Description

Follow-up to #19, which fixed cloudflare_queue_backlog_messages and related gauges freezing at a stale non-zero value by calling DeletePartialMatch({"account": accountName}) on all 7 queue gauges in fetchQueueAnalytics before repopulating them each scrape.

That overcorrected: a queue that's idle most of the time will have no rows in Cloudflare's queueBacklogAdaptiveGroups/queueConsumerMetricsAdaptiveGroups responses, since those endpoints only emit rows for queues with activity in the scrape window. The delete-then-rebuild wiped the series on every idle scrape and it never got repopulated, so downstream dashboards showed no data at all for hours at a stretch instead of a correct zero baseline. Confirmed in production: after the #19 deploy, an idle queue's backlog/concurrency series had one brief real reading, then many hours of zero data points, while the exporter itself stayed healthy and scraping normally.

Fix: for the 3 gauges keyed only by queue_name/account (queueBacklogMessages, queueBacklogBytes, queueConsumerConcurrency), seed every currently known queue (from fetchQueueNames) at 0 immediately after the DeletePartialMatch calls and before applying this scrape's real values. Idle queues now read a continuous 0 instead of disappearing. The 4 operation-metric gauges (queueOperations, queueOperationBytes, queueOperationLagTime, queueOperationRetryCount) keep the delete-only behavior, since their action_type/outcome label combinations aren't enumerable ahead of time the way queue_name is.

Jira: TRAC-1072

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Code refactoring
  • Other (please describe):

Testing

  • I have run make pr-tests and all tests pass
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

go build ./... and gofmt -l . are clean, and the pre-commit golangci-lint hook passed. make pr-tests's test target (run_e2e.sh) requires live Cloudflare API credentials via .env and venom, not run here.

Manual verification post-deploy: watch a normally-idle queue's cloudflare_queue_backlog_messages/cloudflare_queue_consumer_concurrency series — it should read a continuous 0 during idle periods instead of showing gaps or a stuck non-zero value.

Code Quality

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation

Additional Notes

Requires a downstream rebuild/redeploy trigger once this merges.

TRAC-1066 fixed cloudflare_queue_backlog_messages and related gauges
freezing at a stale non-zero value by calling DeletePartialMatch on
all 7 queue gauges before repopulating them each scrape. This
overcorrected: since Cloudflare's queueBacklogAdaptiveGroups and
queueConsumerMetricsAdaptiveGroups only emit rows for queues with
activity in the scrape window, an idle queue's series got wiped and
never repopulated, leaving Grafana with no data for hours at a time
instead of a correct zero baseline.

For the 3 gauges keyed only by queue_name/account (backlog messages,
backlog bytes, consumer concurrency), seed every currently known queue
at zero (from fetchQueueNames) immediately after the DeletePartialMatch
calls and before applying this scrape's real values. Idle queues now
read a continuous zero instead of disappearing. The 4 operation-metric
gauges keep the delete-only behavior since their action_type/outcome
label combinations aren't enumerable ahead of time.

Fixes TRAC-1072
Co-Authored-By: Claude <noreply@anthropic.com>
@chanceaclark
chanceaclark merged commit 79886af into master Jul 6, 2026
10 checks passed
@chanceaclark
chanceaclark deleted the chancellorclark/ltrac-1105-queue-backlog-metrics-show-gaps-in-grafana-after-trac-1066 branch July 6, 2026 18:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant