Slow performance with large HTTP/2 streams on Jetty 12.x

[nseidan]

Hi,
I am looking into this issue (https://lists.apache.org/thread/x3fv574g4y2645nxoxsc5cojorg48p24), slow performance with large HTTP/2 streams on Jetty 12.x.
Is there a known issue or tuning for such large cases (params such as initialStreamRecvWindow, initialSessionRecvWindow ) where multiple large streams are multiplexed concurrently on a single connection ?

[sbordet]

Can you please detail the "largeness" of the flow control windows? Both the apache and jira issues refer to generic "large" amounts of data, but it's not clear how large is "large".

It is true, and it is a well known issue, that HTTP/1.1 over multiple connections can be slightly better than HTTP/2 on a single connection with the same number of multiplexed streams.
This comes from the DATA frame overhead (9 bytes every 16 KiB), and the WINDOW_UPDATE notifications from the receiving peer.

Having said that, a well tuned HTTP/2 system would allow sending at full speed, meaning that the server never stalls.

In our experience, the server is never the problem; the problem is typically that clients are much slower to read and process the data than the server is able to send.
Because clients cannot consume data fast enough, the send of WINDOW_UPDATE frames is delayed, and eventually this results in the server stalling.

What client are you using?

The typical solution is to make sure that the client is fast at consuming data.
This may mean to offload blocking/slow work to other threads, so that the "network" thread never slows down.
However, the right solution depends on what client you are using, and what feature the clients offer.

Also of note, it is the client that must be configured with large "recv" windows, not the server, when the problem is that the server stalls the sends, like you describe in the mentioned tickets.

I would suggest deriving the configuration from calculations.
For a 10 Gib network, and 25 ms latency between client and server, the bytes in flight are 32 MiB.
Jetty's HTTP2Client default is 16 MiB, so not that far away from the theoretical max in the above conditions.
What I'm trying to say is that it is of little use to configure the client with enormous windows, but keeping the slow processing: it is better to fix the slow processing first, and then look for flow control window enlargements.

You must also look into network limits settings of your OS, for example in Linux net.ipv4.tcp_rmem.
Other limits include the CPU interrupts rate and memory bus speed.

Finally, you have to benchmark it, for example using h2load.
Create a benchmark for your environment where you download from the server large amounts of data, and the client does nothing else than receiving and discarding the data.
See what happens there: is the server stalling? is the GC running too often? what's the max data the client can spool per second?
Jetty reports the stall times of the session and the streams, so it would be easy to know if the server stalled due to flow control.

It is an interesting topic, keep us posted and let us know how it goes.

[nseidan]

The “client” is a Solr node (SolrJ/JettyHttpClient) talking to other Solr nodes.

Based on this benchmark PR,

• Index recovery case: ~20 GiB per shard, 12 shards in total.
• Streaming search case: rows=docCount=10,000 with docSizeBytes=10,024 and 25 text fields; estimate for returned document content is docCount * docSizeBytes = ~96 MiB (maybe more, json/field overhead).

Also tested with larger client flow control windows:
-Dsolr.http2.initialStreamRecvWindow=8000000 and -Dsolr.http2.initialSessionRecvWindow=96000000

[sbordet]

Do you have a single request downloading 96 MiB of response content, or many concurrent requests each downloading 96 MiB?

In the HTTP/1.1 case, how many connections do you have?

[nseidan]

StreamingSearch case: with numShards=12, StreamingSearch query fans out to 12 shard requests concurrently over the same SolrJ/Jetty HTTP/2 client session, so there are ~12 concurrent response streams multiplexed on one HTTP/2 connection. 96MiB is across all 12 shards.

[sbordet]

For future record, here's the math to avoid server stalls.

Let's assume the network bandwidth 1 GiB/s and the latency (half round-trip) is 100 micros.
The bandwidth-delay-product (BDP) is then about 100 MiB; this is the number of bytes in flight.

Assuming the client can process data at zero overhead, then:

• The server can send a full BDP before the client sees the first byte.
• When the client sends a WINDOW_UPDATE to the server, the server can send another full BDP before seeing the WINDOW_UPDATE.
• The rate at which the client sends WINDOW_UPDATEs must match the bandwidth, so the client must send an update of 10 MiB every 10 micros, or an update of 50 MiB every 50 micros, etc.

The above means that the client window should be: W=2*BDP+U where U is the window update in bytes that the client sends.

The update sent by the client must be what the BufferingFlowControlWindow sends, so U=W*f, where f is the BufferingFlowControlWindow factor, from which it follows:

W=2*BDP+W*f
W=2*BDP/(1-f)

This means that in the example above for f=0.2 the window is W=250 and U=50.
For f=0.5, then W=400 and U=200.

You may want to give a little room to the client window W and make it a little larger.

Note that the above works under the assumption that the client can process data at zero overhead.
If the client introduces data processing overhead, then both HTTP/1.1 and HTTP/2 will eventually stall, as the server can send data faster than the client can process.

In conclusion, it is possible to configure HTTP/2 so that the server does not stall, once the BDP is known; and that should mimic the behavior of HTTP/1.1.

If the client receive window is smaller than the calculations above, then HTTP/2 will stall a lot more than HTTP/1.1 and the download performance will be a lot worse.

[nseidan]

Thank you, this BDP-based sizing explanation is very helpful for the next step, benchmarking, a dump client to isolate pure transport and validate BDP-based client receive window sizing.

[sbordet]

Make sure that both the client and the server are the same that Solr uses. While using h2load may be quicker, eventually you have to deal with the actual implementation.

Once you have the "pure transport" data, introduce data processing, and see how's affecting the results.
Then you can decide whether to offload processing if it is too costly, but that should be the same for HTTP/1.1 and HTTP/2.