[Feature]: Throttle Worker Pool Workers by Gate Budget

[jtechapps]

Problem Statement

Currently gate Budget() does not throttle the number of active workers in a worker pool it is only used for a binary decision to admit/block/redeliver requests.

If saturation is hit on the worker group then all messages are parked/redelivered, and then when the gate becomes unsaturated, all workers are resumed at the same time which leads to a burst of requests past the saturation.

Example Test

I was able to create a benchmark which shows that the saturation can exceed the configured saturation due to traffic bursting when all workers are woken up when the endpoint state goes from saturated to unsaturated.

https://github.com/jtechapps/llm-d-async/tree/samplebenchmark

When the gate is on the worker pool (workerPools), the queue gate defaults to wide-open (ConstOpenGate). The queue worker pops a full batch of 64 requests every 50ms and feeds them directly into memory.

• No Proportional Throttling: Each of the 64 independent worker goroutines evaluates poolGate.Apply() as a binary verdict (ActionContinue vs. ActionWait).
• Parallel Traffic Pump: Because there is no proportional batch-size shrinking, all 64 parallel workers continue firing requests at EPP at full speed (~10ms per request) as long as the last-scraped Prometheus saturation metric remains < 0.8. During the Prometheus scrape delay window, those 64 workers effortlessly pump ~174 requests into EPP before the binary switch flips to ActionWait.

When the gate is placed on the queue (queuesConfig), it is evaluated by a single queue worker goroutine before pulling messages from Redis.

• Proportional Scaling: Instead of a simple yes/no switch, gate.Budget(ctx) returns a continuous fraction representing remaining capacity. As EPP saturation approaches 0.8, budget smoothly shrinks toward 0.0.
• Dynamic Batch Sizes: The queue worker multiplies its batchSize (64) by this budget. As EPP fills up, the queue worker automatically slows down ingestion (e.g., popping 30 requests, then 10, then 2) before saturation ever reaches 0.8. When saturation hits 0.8, batchSize becomes 0, and the worker stops popping messages from Redis entirely.

Metric / Attribute	Benchmark 1: Queue-Level Gate	Benchmark 2: Worker-Pool-Level Gate
Gate Placement	Message Queue (queuesConfig)	Worker Pool (workerPools)
Gate Type	prometheus-saturation	wait-on-refuse (wrapping prometheus-saturation)
Throttling Mechanism	Proportional ingestion scaling (batchSize * budget)	Binary open/close evaluation across 64 parallel workers
Saturation Verdict Action	ActionRefuse (Re-enqueues back to Redis sorted set)	ActionWait (Parks goroutine in memory, polling every 50ms)
Total Requests Processed	5,000	5,000
Total Queue Clearing Time	54.13s	1m05.02s (65.02s)
Average Saturation Level	0.8118	1.7375
Behavior Under Load	Smooth ingestion throttling before saturation hits 0.8; keeps EPP queue stable at target threshold.	Initial burst overshoot due to Prometheus scrape lag across 64 workers, followed by in-memory parking until EPP drains.

The desired behavior is that both benchmarks should have average saturation around 0.8.

Proposed Solution

Implement a mechanism for sleeping/blocking a fraction of workers within a worker pool based on the gates Budget() similar to how Budget() will control batch size for queue level gates.

Alternatives Considered

The user can lower the number of a workers so that saturation cannot be overshot. This will work assuming homogenous requests, and no interference from other systems.

Willingness to Contribute

Yes, I can submit a PR

Additional Context

No response

[jtechapps]

After rerunning the test in https://github.com/jtechapps/llm-d-async/tree/samplebenchmark I was getting average saturation of 1-3 for the queue saturation gate benchmark, this means the saturation is also being overshot when queue gate throttles batch size. The issue with the pool gate throttling is still an issue though.