Add the ability to limit the number of requests to metrics

[Satbek]

Under normal operating conditions, metrics are requested from an instance every 10/15 seconds or once per minute, which does not impact performance.

However, if multiple requests per second are sent to the metrics, significant performance degradation begins.

Therefore, it would be beneficial to have the ability to limit the number of requests to metrics. If the limit is exceeded, the server should return a 429 Too Many Requests response.

Here is a link to this research on the impact of metric requests on performance. Adding a rate limiter for metric requests significantly reduces performance degradation.

Here is a code snippet demonstrating how this was implemented:

local clock = require('clock')
local last_called = {}
local timeout = 1

local too_many_func = function ()
    return {
        status = 429,
        headers = { ['content-type'] = 'text/plain; charset=utf8' },
        body = 'Too Many Requests',
    }
end

local http_handlers = {
    json = function(req)
        local json_exporter = require('metrics.plugins.json')
        return req:render({ text = json_exporter.export() })
    end,
    prometheus = function(req)
        if last_called[req.peer] == nil then
            last_called[req.peer] = clock.monotonic()
        elseif clock.monotonic() - last_called[req.peer] < timeout then
            return too_many_func()
        else
            last_called[req.peer] = clock.monotonic()
        end

        local http_handler = require('metrics.plugins.prometheus').collect_http
        return http_handler(req)
    end,
}

[Satbek]

I don't think it really matters how the rate limit is implemented, as long as it exists and works by default. I believe this is not an option that people will fine-tune extensively.

We should set some reasonable defaults, like no more than 1 request per second.

It could be configured like this:

roles.metrics-export:
    http:
        ratelim_rps: 1|0.2|null

• 1 means no more than 1 request per second per peer
• 0.2 means no more than 1 request every 5 seconds
• null means no rate limiting

By default, it should be enabled since nobody needs a high RPS for metrics.