Feature Request: allow duplicate queries to be rejected when consolidator is overwhelmed

[tanjinx]

Feature Description

Adding a new vttablet config parameter --consolidator-query-waiter-cap-method, which can be:

• fallthrough: current and default behavior, when the waitercap is exceeded, falling back to the regular connection pool
• reject: new behavior, when the waitercap is exceeded, rejecting the queries, and returning an error like ErrResourceExhausted: consolidator waiter cap (%d) exceeded

Use Case(s)

When the tablet is overloaded, there could be a ddos of duplicated queries which are usually non-critical/sheddable workloads, with the current behavior, falling back to the regular query processing could easily make the connection pool exhausted, resulting in higher latency or timeouts for those more-critical workloads.

And when the waiter cap was exceeded, it likely implicates:

• the query is expensive, taking long time to return the result
• or there were flooding dup queries
• or the tablet itself was overloaded, everything was running slower

so allowing to reject those duplicated queries would prevent making the overload worse.

[aviralgarg05]

I would like to solve this issue