Support rate limiting by concurrent requests

[ehfd]

Description:

Currently, Envoy AI Gateway utilizes token-based rate limiting (https://aigateway.envoyproxy.io/docs/capabilities/traffic/usage-based-ratelimiting). This is useful for commercial API endpoints that pay by token but not for self-hosted API endpoints like vLLM or SGLang.

As an alternative, there should be rate limiting applied directly in Envoy AI Gateway to rate limit by concurrent requests per Envoy AI Gateway API tokens (user). This is much more useful for restricting concurrent requests per Envoy API key in self-hosted multi-tenant environments.

[nacx]

If the rate limit is not about used tokens, couldn't you just use the existing local or global rate limit mechanisms?

[ehfd]

Would we be able to do this?

1. Rate limit concurrent API connections globally and per model, as well as per API key.
2. Rate limit all aggregate tokens per API key and/or all keys in a certain team (partially implemented in Envoy AI Gateway).
3. Exceptions and different rate limits for different teams/API keys.