Rate Limit Receiver Extension

[MovieStoreGuy]

Is your feature request related to a problem? Please describe.
There is two scenarios that I would like to see some amount of rate limiting happening:

1. Running the collector in a gateway deployment style (many stateless collectors running as deployment) can be accepting 3x traffic from Service A which can impact delivery of service B sending <1x scale.

2. Running the collector as a stand alone / sidecar deployment (Single node, potentially stateful with only a single service sending data).

If in either of the cases, the service sends an erroneous amount of traffic, it impact others trying to send through the same pipeline.

Describe the solution you'd like
A new extension made to the collector contrib that can potential rate limit of header fields or HTTP meta data information.

Ideally it should follow an exiting rate limiting convention that is widely adopted so that effort to update instrumentation is low touch.

Considering Option 1 case, the collector deployment will need to send data among each other to ensure that traffic is correctly rate limited (with some tolerance for eventually consistent).

Describe alternatives you've considered
Each collector could have an in memory counter that is of fix number B over T time.
Therefore the considered rate limit would be N nodes X B buckets / (2T) (assuming a drift of T between each node)
Which could work, but the drift might not be tolerable but adopters.

Additional context
There is some public projects that attempt to make generic solutions to this:

• https://github.com/envoyproxy/ratelimit
• https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/other_features/global_rate_limiting

[MovieStoreGuy]

@tigrannajaryan , @jpkrohling , @bogdandrutu ,

This is a large potential piece of work and I was wondering if it could be brought up for discussion for the next Collector SIG?
Edit: More specific what rate limiting strategy to use, what should be configurable by a user, should the collector use some sort of consensus protocol among each node, or use a third party store like redis, dynamo, etc...

Within Atlassian, we've built an internal version of this that works for our own needs however, I am convinced that we can contribute back to an extension that could be used by all adopters of the collector.

[tigrannajaryan]

Could rate limiter be implemented as an auth extension? Alternatively, perhaps it could be a processor if the processor has access to the data that is needed to make a rate limiting decision.

Considering Option 1 case, the collector deployment will need to send data among each other to ensure that traffic is correctly rate limited (with some tolerance for eventually consistent).

What do you mean by "correctly rate limited"? Are you for example looking for rate limiting traces by dropping spans from one trace?

Definitely worth discussing in the SIG.

[Aneurysm9]

I'd second the suggestion to model rate limiting as an auth extension, that should provide the clearest path to integrating it with as many receivers as possible as early in the pipeline as possible. I think using a processor might be insufficient as all the work of the receiver has been performed at that point, reducing the amount of work that can be avoided on rate-limited requests.

In the past I've found it simplest to avoid coordination between nodes and simply set the desired rate limit to limit / #nodes where it is expected that traffic will be evenly distributed across nodes. Is that a workable assumption here, or is coordination truly required?

The thing I worry about with implementing a rate limiter is losing data. OTLP has a built-in mechanism for telling senders to hold off on retries, which should help for correctly implemented OTLP senders, but what about other protocols that do not account for backpressure?

[MovieStoreGuy]

Could rate limiter be implemented as an auth extension? Alternatively, perhaps it could be a processor if the processor has access to the data that is needed to make a rate limiting decision.

I would need to double check the interface for Auth extensions, but I thought it would be a nice addition if any receiver could be wrapped by it so that it is one thing that we can extract out from each of those components.

How we had done it internally is that we created a new receiver that wrapped the default otlp http receiver due. It wasn't the cleanest solution but it meant that we could those large deployments from breaking our pipeline.

I also realise while typing that I am assuming that I have access to the raw request which may not be the case in some scenarios. (For example, It might be a use case in future to rate limit host collected metrics from collectd. Out there example but I could see that happening with things like JMX or expvar)

Long winded point of saying that a processor might be a better option here since it could apply to more use cases over time.

What do you mean by "correctly rate limited"? Are you for example looking for rate limiting traces by dropping spans from one trace?

Yes sorry, I had meant that the rate limiting buckets are kept in sync with each node, so each node knows the expiry and reset interval.

However, there is several things I can think of that we may (and currently do internally) attempt to rate limit on:

• Number of Requests (it is unusual for someone to experiment with new tracing or metrics instrumentation that happens to be high volume)
• Cumulative Content Length (Not something we currently do but could be handy for those who use the collector to front some data store such as Kinesis, Kafka)

And then after it starts getting into introspection of that data:

• Number of Resource Values (Number of Spans, Datapoints, Logs)
• Number of Resource Types (Have different thresholds for Tracing, Metrics, logs)

I'd second the suggestion to model rate limiting as an auth extension, that should provide the clearest path to integrating it with as many receivers as possible as early in the pipeline as possible. I think using a processor might be insufficient as all the work of the receiver has been performed at that point, reducing the amount of work that can be avoided on rate-limited requests.

I am okay with this approach, however, it does feel a misuse of the Auth interface to a degree?

In the past I've found it simplest to avoid coordination between nodes and simply set the desired rate limit to limit / #nodes where it is expected that traffic will be evenly distributed across nodes. Is that a workable assumption here, or is coordination truly required?

So working off the assumption that collector is only given an interval duration and makes no effort to start on nearest duration window (ie. 1m interval wouldn't try to start at 08:44.00000 but would start the interval when the component was ready)

I really think it would come down to:

• Could the downstream services withstand nearly limit / #nodes * 2 worth of traffic?
  • This could be a result of a new deployment to the gateway (a green, green deployment manor)
  • The nodes starting being ready at the boundaries of interval duration
• Does there need to be a max number of traffic that can be processed by your deployment?
  • In the event that the gateway deployment needs to scale, the potential limit on a service could be greater that would could be consume by the downstream.

I think Atlassian falls into the second bucket, where we are rather happy to scale as needed to meet demand however, we need to enforce a max limit consume by the entire deployment before it starts impacting our delivery to Kinesis / Kafka.

(We currently have plans for effectively a ETL on Tracing to Metric data that is going to be done within the gateway so it could force a scale up event within the ASG and raise the allowed limits for the service).

The thing I worry about with implementing a rate limiter is losing data. OTLP has a built-in mechanism for telling senders to hold off on retries, which should help for correctly implemented OTLP senders, but what about other protocols that do not account for backpressure?

I think this would be a good chance to further helps those exporters by expanding the exporthelper package that would help account for these situations instead of those extensions needed to reinvent having to handle back pressure?
I am not sure of the best patterns to do so but there is already the Retry helper, Queue helper which do help with this but we could add in for RateLimit so that the user could decided on what was to happen in the event of rate limit event:

• Drop latest
• Drop earliest
• Rollup data (not sure permissible within the collector to mutate data)

However, I suspect for those exporter that do no currently handle cases of back pressure already experience it and offering further helpers to assist with that would drastically reduce effort required by those components.

This is more tangental, but if we do go with an uncoordinated effort (say as a default mode without any external resource to force a sync) that we should recommend routing policies to be round robin instead of least connections or ip hash (assuming that nginx rate limiting naming would work across different ingress controllers)

[jpkrohling]

Long winded point of saying that a processor might be a better option here

The auth capabilities were also a processor at first, but for security-related tasks, and I would include this one in the mix, it's better to accept/reject a request before much processing has been done. So, something that runs at the same level as authenticators would certainly make sense. I don't think an authenticator is the solution either, as there can only be one authenticator per receiver.

I need to think a bit more about this, but I'll certainly forget everything by the time I come back next year... So, ping me again in a couple of weeks :-)

[MovieStoreGuy]

If it makes it easier, I can limit the scope of the initial feature to be on receivers that accept a http / RPC instead of all receivers.

We can look resolving the issue for host / pull based receivers in a future issue if that would help?

[Aneurysm9]

I am okay with this approach, however, it does feel a misuse of the Auth interface to a degree?

It doesn't feel like a misuse to me. I view rate limits as an authorization rule. Once a requestor has exceeded their quota they are no longer authorized to have further requests processed until their quota is again positive.

[MovieStoreGuy]

I had honestly never thought about Rate Limiting like @Aneurysm9 , but yeah, that makes sense

[jpkrohling]

Perhaps we need authorization in addition to authentication? While we can have only one authenticator, we could have multiple authorizers.

[MovieStoreGuy]

@jpkrohling , pining you here to help get some ideas flowing :)

[jpkrohling]

How about something like this:

extensions:
  oidc:

  ratelimiter:
    qps: 1000

receivers:
  otlp:
    protocols:
      grpc:
        auth:
          authenticator: oidc
          authorizers:
          - ratelimiter

processors:

exporters:
  logging:
    logLevel: debug

service:
  extensions: [oidc, ratelimiter]
  pipelines:
    traces:
      receivers: [otlp]
      processors: []
      exporters: [logging]

[MovieStoreGuy]

I really like the structure of that configuration file, I think it works well for what as an example authorizer.

[jpkrohling]

@MovieStoreGuy do you think you have the necessary information to get started with this? If so, we might need a change or two as part of the core before implementing the first authorizer here in the contrib, in which case, we need to get clearance from @bogdandrutu and/or @tigrannajaryan.

[MovieStoreGuy]

Yeah, i think I have enough of an idea on how to get started.

My thoughts on this was to do the following:

• Initial Draft Authorizer interface with an nop type within the core repo
• Update the configuration types as needed
• Add in some harness testing for future Authorizer's following the existing life cycle pattern
• Add an in memory rate limiter as the initial rate limter

In the mean time, I would want to draft up some idea ons how best to implement a global sync of rate limits to allow a gateway style of deployment to scale as required without requiring the downstream services to have the same elasticity.

I believe I had also seen a pattern where this work starts in the contrib and then moves into core, I am also happy to follow that development pattern.