Skip to content

Commit 66083d0

Browse files
authored
fix: avoid zero-rate queue reader throttling (#9619)
## Summary - When `persistenceMaxQPS` is set to `0` (meaning "unlimited"), the queue reader rate limiter receives a zero rate, causing all queue readers to fail with "burst size is smaller than required token count" and never process tasks. - This fixes the rate function to return a high default (100,000) when persistence QPS is unlimited, and adds a missing `return` after the error log to prevent falling through to an invalid lock/read path. ## Bug Details `NewHostRateLimiterRateFn` computes `float64(persistenceMaxRPS()) * persistenceMaxRPSRatio`. When `persistenceMaxRPS()` returns 0 (unlimited), this produces rate=0, which creates a rate limiter with burst=0. The reader then cannot acquire even 1 token, logging the error but continuing into `r.Lock()` without tasks — effectively a tight error loop. ## Testing - Verified with benchmarks that queue readers function correctly with `persistenceMaxQPS: 0` in dynamic config. - Existing unit tests pass.
1 parent bc0354e commit 66083d0

2 files changed

Lines changed: 13 additions & 4 deletions

File tree

service/history/queue_factory_base.go

Lines changed: 12 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -226,9 +226,17 @@ func NewHostRateLimiterRateFn(
226226
return float64(maxPollHostRps)
227227
}
228228

229-
// ensure queue loading won't consume all persistence tokens
230-
// especially upon host restart when we need to perform a load
231-
// for all shards
232-
return float64(persistenceMaxRPS()) * persistenceMaxRPSRatio
229+
if pMaxRPS := persistenceMaxRPS(); pMaxRPS > 0 {
230+
// ensure queue loading won't consume all persistence tokens
231+
// especially upon host restart when we need to perform a load
232+
// for all shards
233+
return float64(pMaxRPS) * persistenceMaxRPSRatio
234+
}
235+
236+
// persistenceMaxQPS=0 means "unlimited" — use a high default to avoid
237+
// producing a zero rate which would block all queue readers.
238+
// NOTE: Do not use math.MaxFloat64 here; int(math.MaxFloat64) overflows
239+
// to math.MinInt64, which breaks burst calculation in the rate limiter.
240+
return 100_000
233241
}
234242
}

service/history/queues/reader.go

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -431,6 +431,7 @@ func (r *ReaderImpl) loadAndSubmitTasks() {
431431

432432
// this should never happen
433433
r.logger.Error("Queue reader rate limiter burst size is smaller than required token count")
434+
return
434435
}
435436

436437
r.Lock()

0 commit comments

Comments
 (0)