overload manager: overload signals based on number of downstream connections and active requests

[antoniovicente]

It would be helpful to add resource monitors to track the number of sockets used for (1) downstream connections, (2) active requests and (3) possibly upstream connections in order to provide better protection against resource attacks against configured fd rlimits. Tracking these counts proxy-wide should be sufficient. The motivations behind this enhancement include a desire for more consistent configuration of resource limits and actions we can take when approaching overload, recently introduced parameters to limit the max number of client connections (globally or per listener) to protect against fd rlimit attacks, and the introduction of more graceful options to handle increases in resource usage, including the introduction of adaptive HTTP request timeouts in #11427.

Possible future enhancements:

• A resource pressure signal based on number of upstream connections would benefit from some additional overload actions specific to upstream connections, including reductions in idle upstream connection timeouts, disabling of keep-alive or adjustments to prefetch logic.
• Being able to configure thresholds and apply overload manager actions to client connections associated with specific listeners in order to prioritize connections to specific listeners. That said, I don't think our deployment scenarios have an immediate need for such a hook.

[antoniovicente]

cc: @nezdolik

[mattklein123]

@nezdolik once you accept the org invite we can assign the issue. Thank you for working on this!

[nezdolik]

Thanks @mattklein123 @antoniovicente, I believe issue assignment should work now.

[akonradi]

Do we have any established patterns for aggregating counts of objects across multiple workers efficiently? We probably have stats for most of the metrics we'd want to use but I don't know if the flush frequency is high enough for the overload manager to make use of them.

[antoniovicente]

It may make sense to keep an atomic counter with the aggregate number of active client connections across all listeners and clusters.

It may make sense to think of that number as separate from stats.

[akonradi]

What about having aggregate counts in the individual worker threads via a TLS object, then having the resource monitor aggregate those using an atomic counter when the overload manager initiates a poll?

[nezdolik]

@mattklein123 @antoniovicente @akonradi please take a look: https://docs.google.com/document/d/109p2eWereP9i_0XbuiVhsIWoJ6oI_jUhsjG-rHCDjyg/edit?usp=sharing

[mattklein123]

Thanks @nezdolik! I will take a look next week. OOO for a bit.

[nezdolik]

I think this task needs to be splitted into 3 parts:

• Introduce reactive checks into overload manager
• Plug OM into dispatcher
• Write the actual resource monitor & migrate code from TCP Listener