Commit ec650bc
fix: TRAC-1066 Reset stale queue gauges instead of holding last value
Cloudflare's queueBacklogAdaptiveGroups GraphQL analytics endpoint only
returns a row for a queue when there was measured activity in that
scrape window. When a queue goes idle, it silently disappears from the
response, and fetchQueueAnalytics only calls .Set() for queue IDs
present in the response — so the gauge freezes at its last non-zero
value forever instead of resetting.
This caused MakeswiftStencilWorkerQueueBacklogCritical/Warning to fire
in production reporting a 17.44k message backlog while the real queue
was confirmed empty via wrangler. Thanos showed the metric flatlined
at 691 for 13+ hours, then jumped to exactly 17444 and stayed frozen
there on both exporter replicas — the signature of a stale gauge, not
real queue depth.
Drop each account's previous label combinations for all 7 queue gauges
before repopulating them each scrape, mirroring the existing pattern
in fetchWorkerWAEAnalytics. Idle or deleted queues now disappear from
/metrics instead of holding stale values.
Fixes TRAC-1066
Co-Authored-By: Claude <noreply@anthropic.com>1 parent 98d8160 commit ec650bc
1 file changed
Lines changed: 12 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
805 | 805 | | |
806 | 806 | | |
807 | 807 | | |
| 808 | + | |
| 809 | + | |
| 810 | + | |
| 811 | + | |
| 812 | + | |
| 813 | + | |
| 814 | + | |
| 815 | + | |
| 816 | + | |
| 817 | + | |
| 818 | + | |
| 819 | + | |
808 | 820 | | |
809 | 821 | | |
810 | 822 | | |
| |||
0 commit comments