Question - how to introduce waiting timeout in queue

[domik82]

I have introduced this limiter in my app but what I see as a consequence is that requests simply respond slowly after rate limit is hit waiting for their turn. Now that is fine and rate limit is properly applied but what if I would like to timeout request waiting in queue after 2s?

In ideal world I would like to throw 429 'Too Many Requests' when some timeout of waiting in queue is being hit.
This would preserve resources on the server (HTTP connections) and the client could implement retry logic after backing up a bit.

Any ideas how to do it based on this lib?

In below example I have set the rate limit to be 20 req/s

The load test runs with 20 users at rate 1 req/s per user. The moment I will cross 20 rps resp blows up but there are no errors despite
user waiting for response for 6s or longer.

I can do

        if rate_limiter.has_capacity():
            await rate_limiter.acquire()
            # do stuff here
        else:
            return web.json_response(
                data={'Rate limit failure': 'Too many requests are sent to service.'},
                status=HTTPStatus.TOO_MANY_REQUESTS,
            )

But then this is a really hard limit and I wondered if I could avoid it by applying some soft queue so that request would be served in next second in this example but with increased latency.

Update:
I guess the easiest way to achieve what I want is:

    try:
        # Wait for capacity with a timeout (defaulting to 2 seconds)
        await asyncio.wait_for(rate_limiter.acquire(), timeout=2)
        # do stuff

    except asyncio.TimeoutError:
        return web.json_response(
            data={'Rate limit failure': 'Too many requests are sent to service.'},
            status=HTTPStatus.TOO_MANY_REQUESTS,
        )

[mjpieters]

Instead of rate_limiter.acquire() you could also just check for capacity immediately with AsyncLimiter.has_capacity()

if not rate_limiter.has_capacity():
    return web.json_response(
            data={'Rate limit failure': 'Too many requests are sent to service.'},
            status=HTTPStatus.TOO_MANY_REQUESTS,
    )

This is a synchronous check, and it's the same check that rate_limiter.acquire() makes. If has_capacity() returns True, you can safely call rate_limiter.acquire() next (or use async with rate_limiter:) and not be blocked.

There is no point in waiting before telling your clients they exceeded the rate limit.

(Incidentally, the inverse is not quite true, if has_capacity() returns False, there is a very small chance that enough time passes to free up enough capacity for a subsequent await rate_limiter.acquire() to not block).

[domik82]

Thanks for response. I wanted to allow some bursts of requests limiting those with delay - this is why I wanted timeout.