Setting global rate limit with request header

[Stephen-X]

Hi there,

I'm inquiring if there's currently a way to set the global rate limit dynamically using a request header.

This is how we normally use Envoy Gateway to set a 3 RPH global rate limit on every unique users:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy 
metadata:
  name: policy-httproute
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: http-ratelimit
  rateLimit:
    type: Global
    global:
      rules:
      - clientSelectors:
        - headers:
          - type: Distinct
            name: x-user-id
        limit:
          requests: 3
          unit: Hour

In our use case, Envoy Gateway is used internally to protect an internal service, which is utilized only by an external service that hosts all the user configurations. Instead of building a delta user configuration reload pipeline for EG, we are wondering whether we could pass user configurations to EG via request headers instead.

For example, we could pass to EG this request header x-envoy-ratelimit-global-limit-requests: 3hr, which sets the RPH in the BackendTrafficPolicy or override the existing default configuration for a unique user ID value.

I came across the Envoy Header-To-Metadata filter and the concept of "dynamic metadata", which appears to be close to how it could be done generally but I'm currently unsure.

[arkodg]

is the limit dynamic or the usage ?
here's an example around dynamic usage based ratelimiting https://github.com/envoyproxy/gateway/blob/main/test/e2e/testdata/ratelimit-usage-ratelimit.yaml

[Stephen-X]

We're looking for the limit itself to be dynamic.

[arkodg]

can you share your use case, as to which clients need different limits, and why/how they are dynamic, I can try and help solution the ratelimit config

[Stephen-X]

Thank you for the kind offer @arkodg ! 😃 I briefly described our use case in the main question, but to elaborate:

We run a high-traffic service that hosts thousands of users and are working on enhancing some of the features with GenAI. We are currently exploring using Envoy Gateway with custom extproc sidecar as an internal AI Gateway to a few of our LLM endpoints. This is the high level flow:

[Some frontend service that also manages user configs] -----> [Envoy Gateway] -----> [LLM backends]

To support different configs for each user, EG requires creating separate K8s custom resources, but that would require a lot of work for us to build a dynamic config loading pipeline for EG (and also we haven't learnt how to do that just yet). We are hoping to simplify development by letting the frontend service control Envoy Gateway behavior dynamically for each user.

To start with, we'd like to have basic user-based rate limiting utilizing existing functionalities to get things running before we implement custom AI Gateway features. We're trying to see if it's possible for the frontend, which already has user configs loaded, to control EG with request headers.

For example, say a user set a maximum rate limit of 11 RPS, then our frontend could be sending requests to the backends with the following 3 headers:

x-user-id: xxxxx
x-user-limit-requests: 11
x-user-limit-unit: Second

And EG could be comparing the RPS limit against the counter value in Redis to decide if the current request should be throttled.

Is this scenario possible with the current EG? If it's not readily possible, we are wondering if we could still utilize the existing rate limit module as much as possible with a bit extra dev work.

[Stephen-X]

Got my answer from envoyproxy/ai-gateway#557, not supported by Envoy in general.