[Docs] Documentation gap: Max response size

[Swiddis]

Is your feature request related to a problem? Please describe

In side-channels, multiple people have asked on different occasions what the maximum size for a single OpenSearch HTTP response is. There's the http.max_content_length setting which applies to the request size. No similar option for the response size seems to exist.

Here's my attempt at an answer based on combining a few sources and envelope math:

The response body doesn't have a hard limit¹ since it's in principle enforced by Jetty (which supports arbitrarily large responses). But it can be bound by several things:

• The cluster needs to have enough heap memory to compute and store the response, otherwise you'll get a CircuitBreakerException.
• If the cluster is servicing multiple requests at once, they'll interfere with each other's memory usage, leading to earlier circuit breakers.
• It also likely depends on the exact query: something like computing a large histogram will take more memory than simply listing a bunch of documents.
• Other background operations on the cluster (e.g. shard management, especially if there's many shards) also will take some of that heap.

Very roughly, you can expect a max response size of about 10% of the cluster's available heap, up to the exact usage profile. The most surefire way to know is to test the cluster with realistic request load for your use case.

Describe the solution you'd like

Determine a conclusive answer (even this one if it's sufficiently accurate) and document it somewhere.

This question should be avoidable with pagination, but the topic comes up often enough regardless, so there may be use case gaps. We can link this or a similar page from the answer.

Related component

Search

Describe alternatives you've considered

N/A

Additional context

N/A

Footnotes

1. We can compute a hard upper limit for listing records: there's a 10k records response limit, and each record is limited by OpenSearch to be at most 1 MB. So the largest response you could get is 10GB. In practice, this will be significantly reduced by the above factors. ↩

[mch2]

@jainankitk @kaushalmahi12 Wondering if either of you have experience with this from the wlm work, do you know how the system responds here so we can document correctly?

[kaushalmahi12]

As per my understanding response size is indirectly limited by following

• Maximum number of docs returned in the single response but this is taken from request so not sure if this can assume any value.
• max_search_buckets: Which dictates max buckets an agg can have at both shard level and co-ordinator level results
  From the WLM's point of view we do not consider response size to cancel/reject requests.