[Enhancement] Multi-Search (_msearch) endpoint doesn't operate the way users would expect it to

[epugh]

Is your feature request related to a problem? Please describe

The way Multi-Search is described in the documentation:

multi-search operation lets you bundle multiple search requests into a single request.

This is great for when you are dealing with browser limits on parallel requests, or just know you need multiple searches to build a final answer. However, just in the past two weeks I've personally run into TWO situations where the _msearch end point didn't work with search use cases that aren't that exotic, which violates the Principle of Least Surprise. This had led me to wondering if Multi-Search is actually not something I can depend on, and that I may be setting myself up for future pain by using it. Right now Multi-Search is described as a simple feature, but the reality is that it has many warts. I'd like to see a clear path for the future of this capability.

Here are links to the two tickets just recently opened related to _msearch, are there many more bugs out there in various code paths waiting to get us? How can we get ahead of this versus responding one by one?

• [Feature Request] Support "search-pipeline" in Multi-Search API  #15748
• [FEATURE] Handle multisearch requests user-behavior-insights#29

Describe the solution you'd like

I'd like to see Multi-Search either enhanced to meet the description as written in the documentation (my preferred!), or be more narrowly defined as "An internal API for OpenSearch Dashboards to use", or even deprecated and removed in OS 3.0.

If we embrace Multi-Search, I can imagine some powerful enhancements for example using the understanding of ALL the user's queries to come up with the most optimal order, i.e a Query Planner that figures out which filters or queries to run first. We could think about enhancing the response format to stream individual queries as they become ready, instead of waiting till all are done, saving time and memory.

Related component

Search

Describe alternatives you've considered

1. Move this whole _msearch concept over to being a client only thing.

2. Build a wrapper layer (proxy) in OpenSearch that passes through a list of queries to individual _search endpoints, and doesn't apply filtering or any logic on the parameters. No specific Java code and having a special code path.

Additional context

We can keep fixing the Multi-Search bugs as they come up, but it would be good if every enhancement to Search had a Multi-Search Validation step.