Commit b92e920
fix: TRAC-1066 Reset stale queue gauges instead of holding last value
Cloudflare's queueBacklogAdaptiveGroups GraphQL analytics endpoint only
returns a row for a queue when there was measured activity in that
scrape window. When a queue goes idle, it silently disappears from the
response, and fetchQueueAnalytics only calls .Set() for queue IDs
present in the response — so the gauge freezes at its last non-zero
value forever instead of resetting.
This caused MakeswiftStencilWorkerQueueBacklogCritical/Warning to fire
in production reporting a 17.44k message backlog while the real queue
was confirmed empty via wrangler. Thanos showed the metric flatlined
at 691 for 13+ hours, then jumped to exactly 17444 and stayed frozen
there on both exporter replicas — the signature of a stale gauge, not
real queue depth.
Drop each account's previous label combinations for all 7 queue gauges
before repopulating them each scrape, mirroring the existing pattern
in fetchWorkerWAEAnalytics. Idle or deleted queues now disappear from
/metrics instead of holding stale values.
Fixes TRAC-1066
Co-Authored-By: Claude <noreply@anthropic.com>1 parent 7666c47 commit b92e920
1 file changed
Lines changed: 12 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
815 | 815 | | |
816 | 816 | | |
817 | 817 | | |
| 818 | + | |
| 819 | + | |
| 820 | + | |
| 821 | + | |
| 822 | + | |
| 823 | + | |
| 824 | + | |
| 825 | + | |
| 826 | + | |
| 827 | + | |
| 828 | + | |
| 829 | + | |
818 | 830 | | |
819 | 831 | | |
820 | 832 | | |
| |||
0 commit comments