Add circuit breaker for response sizes

[williamrandolph]

If a coordinating node can determine that a response will exceed a configured size, this should trip a response size circuit breaker. This, in turn, should cancel the request so that it doesn't continue to run on other nodes.

The issue for cancelling a request as a result of its tripping a circuit breaker is here: #67473

This issue was originally raised as part of #62457 .

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[williamrandolph]

We discussed this in today's core-infra team meeting, and this is something we would like to do. #66196, although closed, gives a good example of the problem a circuit breaker like this would solve. Further, if a user is repeating queries whose assembled results could crash a node, bailing out early, and having a circuit breaker explain why, is better than letting expensive queries pile up.

We will almost certainly have to use SearchTransportService#cancelSearchTask, added in #63520, as part of the solution
here.

https://github.com/dnhatn/elasticsearch/blob/f590d4b0507fd423b1faf82640ffa6af40df78de/server/src/main/java/org/elasticsearch/action/search/SearchTransportService.java#L434

Since we expect that it will be a fair amount of work to implement a solution and we don't yet have a slot for this work in our roadmap, I'm adding the "high hanging fruit" label so that we will review this issue for prioritization.

[DaveCTurner]

I am not sure we can do this in a generic fashion because we do not have a uniform way of accumulating partial responses on the coordinating node, so measuring their size is difficult. The discussion above focusses mostly on searches, but it might also apply to other areas such as stats.

On the stats side we have taken a slightly different approach. We made all the relevant APIs cancellable (#55550), and then made the coordinating node discard the intermediate results eagerly on cancellation (#82337), and finally converted all the relevant APIs over to using chunked-encoding to serialise the response in O(1) memory (#89838). Together these changes mean that in practice we don't see much memory pressure from stats responses any more, even if the final JSON output is 100s of MiB in size.

I think a similar approach would be good on the search side, but this needs work from the search team because (as mentioned above) there is no generic framework for measuring the size of a partially-accumulated response. Are there other APIs that need similar effort? If so maybe a good way forward here would be to identify them and break out issues for the relevant teams.