Circuit breakers should be able to cancel search requests

[williamrandolph]

Currently, if one node trips a circuit breaker the request continues to run on other nodes. If using a coordinating only node, that node will still continue to receive responses from other nodes. By sending a cancel request to all other nodes that contain the request, the node could be saved in the event that many small requests would send the node over the limit. ex. 75 nodes, each sending a response in the neighborhood of 1.9 gb, circuit breaker trips on one request at 2.1gb, still another 74 nodes sending 1.9gb, which could be cancelled thereby saving the node from going unresponsive until all request finish.

This could possibly be implemented as a kind of circuit breaker, for example:

breaker.search.cancel_request - This should cancel the search request across all nodes if a circuit breaker trips due to a search or aggregation.

This issue was originally mentioned as part of #62457

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[dnhatn]

I think we implemented this in #63520, where we cancel a search request if we hit a shard level failure, and it does not accept partial results. Also, we will be working to retry search requests when they hit circuit breakers.

[williamrandolph]

@lanerjo Does the change in #63520 address your concern about cancelling search requests?