fix: TRAC-1066 Reset stale queue gauges instead of holding last value

[chanceaclark]

Pull Request Template

Description

Two production alerts (MakeswiftStencilWorkerQueueBacklogCritical/Warning) fired for the makeswift-stencil-publish-revalidation-production-us queue, reporting a 17.44k message backlog, while the real queue (confirmed via wrangler) was empty.

Root cause: Cloudflare's queueBacklogAdaptiveGroups GraphQL analytics endpoint only returns a row for a queue when there was measured activity in that scrape window. When a queue goes idle, it silently disappears from the response. fetchQueueAnalytics only calls .Set() for queue IDs present in the response, so the gauge freezes at its last non-zero value forever instead of resetting. Confirmed via Thanos: the metric flatlined at 691 for 13+ hours, then jumped to exactly 17444 and stayed frozen there on both exporter replicas — the signature of a stale gauge, not real queue depth.

Fix: fetchQueueAnalytics now calls DeletePartialMatch({"account": accountName}) on all 7 queue-related gauges (queueBacklogMessages, queueBacklogBytes, queueConsumerConcurrency, queueOperations, queueOperationBytes, queueOperationLagTime, queueOperationRetryCount) before repopulating them each scrape, mirroring the existing pattern already used in fetchWorkerWAEAnalytics for workerOperationDuration. Idle or deleted queues now disappear from /metrics instead of holding stale values.

Verified the downstream alert rules in prometheus-alerts/rules/store_cells/trac/makeswift-stencil-worker.yml.erb use plain max(metric{...}) > threshold comparisons with no absent() dependency, so a missing series simply won't match — no alerting-rule changes needed.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Code refactoring
• Other (please describe):

Testing

• I have run make pr-tests and all tests pass
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

go build ./... and gofmt -l . are clean, and the pre-commit golangci-lint hook passed. make pr-tests's test target (run_e2e.sh) requires live Cloudflare API credentials via .env and venom, which weren't run here — recommend CI or a reviewer with credentials confirm /metrics still serves correctly end-to-end. No unit tests exist for fetchQueueAnalytics specifically in this repo today.

Manual verification recommended post-deploy: watch cloudflare_queue_backlog_messages{queue_name="makeswift-stencil-publish-revalidation-production-us"} in Thanos — it should either disappear (queue idle) or track real fluctuating values, not stay pinned at 17444.

Code Quality

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation

Additional Notes

Jira: TRAC-1066