GEP-5160: Rate limit API

[yanavlasov]

What type of PR is this?

/kind gep

What this PR does / why we need it:

This proposal introduces a standardized, provider-agnostic Rate Limit API for defining rate limit buckets, expressions for assigning requests to buckets and optionally request costs computation, addressing the fragmentation caused by vendor-specific CRDs.

Which issue(s) this PR fixes:

Fixes #5160

Does this PR introduce a user-facing change?:

NONE

[kubernetes-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: yanavlasov
Once this PR has been reviewed and has the lgtm label, please assign thockin for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• geps/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[kubernetes-prow]

Welcome @yanavlasov!

It looks like this is your first PR to kubernetes-sigs/gateway-api 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/gateway-api has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[kubernetes-prow]

Hi @yanavlasov. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[kubernetes-prow]

@youngnick: Closed this PR.

Details

In response to this:

@yanavlasov, this is not how we work here.

When requesting a change to the API, we follow the GEP process, which involves creating a document which outlines what you are proposing, why we should do it, and who it will affect. Please see https://gateway-api.sigs.k8s.io/geps/overview/ for the correct process.

At an absolute minimum, for a feature like this, I would expect to see some discussion on the issue, followed by a GEP PR targeting a Provisional level, outlining some of the asks you had in the issue, combined with a review of any options offered by current Gateway API implementations, and what options are available in the various dataplanes they use.

Thanks for the enthusiasm, but I'm closing this PR.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[youngnick]

Sorry for closing @yanavlasov, I've been doing a lot of reviews and made an error here. I'll give this a proper review now.

[youngnick]

I'd rather use Policy Attachment as a last resort, can you think of any other ways we could do this?

Maybe in a prior art/existing implementation review, we could keep track of where this config is set, and see if Policy is the right option?

In particular, I'm wondering is it common for different routes/paths/backends to have different Rate Limiting settings?

If the config does live at the Gateway level, would we also need config at the Route level? Where would the policy attach?

Detecting same-level policy conflicts is actually shockingly complex, and once you add the possibility for the same Policy to be used at multiple levels, things get much worse.

This is not to say that Policy is off the table, just that it needs to be a last resort, as it has significant discoverability tradeoffs.

Luckily, this sort of thing is exactly what Provisional is intended to draw out.

[rikatz]

came here to say exactly the same, I would prefer a strong arguing about a policy other than relying on a "XBackend" attribute to limit against a service (including the fact that, as we are speaking about token rate limit you may want to limit it when your route is trying to reach some external service), or other specific cases.

I think arguing in favor of policy due to a new definition of persona ("It allows different personas to manage Gateway infrastructure independently") is not really a good reasoning in this case IMO, but rather we have the 3 personas well defined already, and we should try to think on:

• How to allow each of them to configure their rate limits on their scoped resources
• How to allow each of them to query about rate limits configuration that may be impacting their services

[yanavlasov]

These are good questions.
@youngnick yes, it is common for different attachment points to have different rate limit policies. For example you may have one rate limit for the entire Gateway to protect your infra and separate rate limits on routes to different services, based on expected traffic and service capacities.

Further down the pipeline, there could be policies protecting. In this case I would want to attach different policies to a Service or maybe a deployment based on their capacity. But this is a much more complex case if there is autoscaling involved.

The same applies to token rate limits as well. There could be different limits for different routes depending on their value to the operator and yet separate limits to backends based on their capacity (i.e. if the same backend serves multiple limits).

I'm not thinking of this from the perspectives of personas so much from the perspective of different components in the serving stack needing different policies.

Also, vendor specific APIs allow multiple backends to target a route to implement failover. In this case the backends can also have different rate limits.

[usize]

I think we can partially resolve this with the suggested survey of existing implementations. If every AI Gateway supports global/gateway level rate limits that compose with per backend / per route policies... that could influence how highly we weight the need for this.

If we are going to use policy attachment, my instinct is that it would be easier to start with it now than to backtrack into it later.

[youngnick]

Maybe, but I would encourage everyone thinking about a Policy thinking about how a HTTPRoute owner would determine what rate limits apply to their service?

How would they know what rate limit policies they are being affected by? Do they need to have permissions to go and get the Policy that's attached to the central Gateway their route rolls up to? If they don't have that, how do they know?

How would they calculate what the actual rate limit is if they overlap? If they have their own ratelimit on the backend the HTTPRoute points to, how do they know if it's in effect?

What are the conflict resolution rules? Do the settings get atomically overridden, or on a per-field basis?

All of these questions need to be asked every time we have a Policy that could attach at multiple levels.

Like I said, I'm not saying no, I'm saying, we need to be really, really sure that a Policy is the only way to do this.

I'm sure @kflynn will back me on this one.

[kflynn]

Well... yeah, I will. 🙂 In particular, I think @youngnick's first comment is spot on:

I'd rather use Policy Attachment as a last resort, can you think of any other ways we could do this?

and this goes double, in my mind, since we own all the resources in play here (GatewayClass, Gateway, Route, and Backend assuming Backend proceeds as we expect), so we should all but certainly first be thinking in terms of how to change those resources rather than relying on policy attachment.

So I'd really much rather start by exploring giving HTTPRoute a rate limit filter (or even a rate limit stanza), and then adding explicit defaults to Gateway (and/or GatewayClass?) instead of using multiple levels of attachment to provide defaults.

[youngnick]

I'd like to see some review of what implementations are doing in this area at the moment. I know Envoy Gateway has some support, and I'm pretty sure others do too. In particular, we need to consider what is the common set of capabilities

For each of those knowing things like:

• What's the config model?
• If it's a Policy, where does it attach? How are conflicts handled?
• What sort of bucketing is used in the implementation? Static, dynamic? How are requests bucketed? Per backend, per-Gateway, something else?

Ideally, someone not knowing much about Rate Limiting should be able to read the section and have a reasonable understanding of the problem space, so that we can talk more usefully about it when it comes time for API design. Having a good idea about the capabilities of various data planes also helps us make sure that whatever API we end up designing will actually be implementable across data planes.

My rough feeling here is that this is an area where data planes may substantially differ in functionality.

[yanavlasov]

Here is an overview of some implementations. If there are some other specific ones you'd like me to explore, please let me know.

All surveyed implementations use a policy (or its equivalent). All policies can attach to Gateway or HTTPRoute and some can also attach to Service or backends. All policies allow static and dynamic bucket definitions.

Behavior of multiple policies matching request is split into two classes. Most specific config wins or the fullest bucket wins - i.e. request is rejected if any of the matching buckets is above limit.

I might be wrong, but buckets MUST be attached to the same component that the policy that created them is attached. I do not think it will work otherwise.

So it seems like there is a fairly big cross section. Multiple policy resolution could be an option.

Implementation	Config Model	Attachment	Bucket Definition	Conflict Resolution
Envoy-gateway	Policy	LocalPolicyTargetReferenceWithSectionName gateway, HTTRoute and backend attachments	Static and Dynamic	Fullest matched bucket wins. Request is matched to all buckets defined for GW and matched route. If any matched bucket is above the limit, the request is rejected.
Envoy AI Gateway	Policy	LocalPolicyTargetReferenceWithSectionName gateway, HTTRoute and backend attachments	Static and Dynamic	Fullest matched bucket wins. Request is matched to all buckets defined for GW and matched route and backend. If any matched bucket is above the limit, the request is rejected.
Solo agentgateway	Policy	LocalPolicyTargetReferenceWithSectionName gateway, HTTRoute can be targets	Static and Dynamic	Most specific matching policy wins.
Kuadrant	Policy	LocalPolicyTargetReferenceWithSectionName gateway, HTTRoute can be targets	Static and Dynamic	Not sure. Most likely the same as Envoy-Gateway as it is the same dataplane implementation.
Kong	Plugin	Gateway, Route and Service can be targets	Static and Dynamic	Most specific policy wins.
Apache	Plugin	Gateway and Route	Static and Dynamic	Fullest matched bucket wins.

[youngnick]

This seems like a good start, but what can the underlying data planes do? What is available in Envoy itself? In NGINX? In HAProxy? In Traefik? At the end of the day, the relevant detail here is: what can the
data plane underying the Gateway implementation actually do? It seems like all the Gateway implementations can do static and dynamic bucketing, but what is that common against all the data planes? This is one where you can reach out to maintainers of implementations that use other data planes and ask for help, as well.

The plugin or policy part is good to know, but I don't think we can make the call about policy or filter based on what implementations are doing at the moment - Policy is the tool that can do a lot of what people want relatively easily, so it's what people reach for. I'd like the upstream spec to have a higher standard here, and be sure we are getting enough value out of using Policy to justify it, especially compared to other options. The Policy v Filter discussion can also (and probably should) be kept to the API design phase, after we've reached agreement on the problems we are solving here.

[youngnick]

Oh, I missed mentioning that this table or something like it should be included in the GEP.

[yanavlasov]

This seems like a good start, but what can the underlying data planes do? What is available in Envoy itself? In NGINX? In HAProxy? In Traefik? At the end of the day, the relevant detail here is: what can the data plane underying the Gateway implementation actually do? It seems like all the Gateway implementations can do static and dynamic bucketing, but what is that common against all the data planes? This is one where you can reach out to maintainers of implementations that use other data planes and ask for help, as well.

Can you be more specific about what you want to know, please? What specific behavior you want to clarify?

[youngnick]

That's totally fair. Let me be specific.

Firstly - the comparison table above should be included in the GEP.

Secondly - there should be a similar comparison table, but comparing the underlying dataplane or proxy. Off the top of my head, that covers Envoy, NGINX, HAProxy, Traefik, and agentgateway, I think? I seem to recall something about agentgateway moving to their own proxy, but I could be misremembering.

For each of those and for the Gateway API implementations that use them, we need to know:

• Can the implementation do rate-limiting (this is almost certainly a yes, but it's better to call it out specifically)
• What kind of rate-limiting can it do? Local to just that instance of the proxy or dataplane, or shared across the whole Gateway? The former is way easier, but may not be in scope for this GEP.
• What kind of bucketing is available. You've got this one already, it's a great callout.
• How is rate-limiting currently configured? At what level is it configured, Gateway, Route, or other?
• How does conflict resolution work? Is that configurable by that dataplane or implementation?

Once we have a better idea about what the various data planes and implementations can do, it makes it possible for us to build a minimum viable feature set that will work across most if not all of them. That's the goal here, is to converge on a feature set that's broadly implementable across data planes, and can be widely implemented and conformance tested.

Already, the info you have says that it's likely that we'll need to provide a way to configure both static and dynamic bucketing, for example.

Edit: if you think of any other relevant information, particularly if it varies across proxies and implementations, please feel free to add it.

[rikatz]

Suggested change

	1. Enforce consistent rate limjti rules across different infrastructure providers.
	1. Enforce consistent rate limit rules across different infrastructure providers.

[rikatz]

I would like to have the user stories a bit more focused on our current personas.

As an example, I am missing a differentiation of what a Gateway owner can do (eg.: establish connections/s on a listener, tho I am not caring much now about the metrics but on what is the owner desire) vs the application developer (eg.: can the application developer also request a rate limit on their own route? Is it per path? Is it per backend?)

Immediately one could think even that:

• Infra admin - Cares about reqs/s on the infra loadbalancer
• Cluster admin - Cares about reqs/s on the Gateway
• App developer - Cares about reqs/s for each backend

[rikatz]

Suggested change

	Rate Limnit is proposed to use the Policy Attachment model as the most effective approach to meet the stated goals, primarily for two reasons:
	Rate limit is proposed to use the Policy Attachment model as the most effective approach to meet the stated goals, primarily for two reasons:

[rikatz]

how would this work, from a user perspective? Given the rate limit is enforced by the dataplane, is there some kind of expectation that the dataplane would add a header signaling a ratelimit violation, or what?

We don't really need to cover the "how" now but I am curious on how this would be achieved

[snorwin]

/cc

[bittrance]

I suggest being a little bit more targeted here. What types of rate limiting are we trying to support? Here are some types that I can come up with:

• DoS defence (typically TCP/UDP level rate limiting)
• circuit-breakers (buckets will primarily be decided by properties of request and response)
• reserved capacity (some clients should be guaranteed access when they need it)
• quotas (each user or customer gets a number of requests per hour or month)

Supporting all of these types will require sophisticated extraction of metadata on which to make rate limiting decisions. They would also engage all the roles in the persona gallery, leading to a large number of points where rate limiting decisions could be made.

[nirrozenbaum]

there are multiple types of routes, use Route as general term

Suggested change

	While the Gateway API specification has unified how traffic is routed via `HTTPRoute` and `Gateway`, there is no standard way of configuring rate limit policies for traffic flowing through Gateway. The absence of
	While the Gateway API specification has unified how traffic is routed via `Route` and `Gateway`, there is no standard way of configuring rate limit policies for traffic flowing through Gateway. The absence of