How to get performance on POST InputStream using Quarkus

[fredericBregier]

In various project, we need to get or post an InputStream (not a file) which could be huge (more than 10 GB). Those files are streamed from/to another service (for instance a S3 service) on the server receiving this GET or POST request.

As most of our project are running in Pods (conatiners), it seems to us that of course Memory is not the option and using local disk as "buffer" is not a valid option.

Using Quarkus, we are able to do both ways (GET and POST), but POST is somehow too slow:

• GET of 1 GB : 459 MB/s
• POST of 1 GB : 25 MB/s

Those times are taken from an example here: https://github.com/fredericBregier/TestInputStreamQuarkus

If we use Netty for the POST, the times are quite better: (same Quarkus Server behind, just the client changes)

• POST with Netty of 1 GB : 387 MB/s

The example project shows the differences, but we can't find a way to configure Quarkus in order to use the default HTTP Quarkus client to reach or approach the Netty native performances. Of course, we understand that a high level client has its own footprint, even if it is using Netty under the surface (Quarkus Client => Vertx => Netty), but it is there a factor of 10.

Have you any idea?

[gsmet]

It’s definitely worth an investigation.

Could you add more information to the README on how we can reproduce the issue easily from the command line?

Also if you are interested in digging by yourself, the following guide is usually helpful: https://github.com/quarkusio/quarkus/blob/main/TROUBLESHOOTING.md

[fredericBregier]

The simplest way is to run the unique test...

[fredericBregier]

I updated the Readme. Just use the test, adapting if necessary the size (StaticValues) or the iteration (20 to whatever) ;-)

[fredericBregier]

I tried to generate and examine Flame graph, but in comparaison (Netty vs Quarkus client) almost all is "cached" by the acceleration and show mainly the server side, which is unchanged so unuseful.
If I take a look on the Method list:

• io.netty.channel.ChannelOutboundBuffer.setWritable(boolean) seems to be the most difference (15K% more with Netty)

So it seems to me that Quarkus/Vertx does not claim enough quickly the next item, such that it probably waits too much between 2 chunks.

QuarkusInputStreamTest_flame.zip

[gsmet]

@geoand this is about Quarkus REST so I thought it might be of interest to you.

[geoand]

🙏🏽

[geoand]

I'll try and have a look in a few days

[geoand]

I am fairly confident that the difference is due to the different chunk size used. In Quarkus the chunk size is 2048 while in your Netty example you are using 65536.
Do you mind trying 2048 and seeing if the results in Netty fall close to those of Quarkus? If they do, we probably need to a configuration option that allows users to adjust this.

[fredericBregier]

If you look at the configuration, the chunk size is 65536 for both (and default is 8K).

Here is the application.properties of src/test/ressources, the one used by the test ;-)

quarkus.log.console.format=%d{HH:mm:ss} %-5p [%c{2.}] [%l] (%t) %s%e%n
quarkus.log.console.level=INFO
quarkus.log.level=INFO

quarkus.http.so-reuse-port=true
quarkus.http.tcp-cork=true
quarkus.http.tcp-quick-ack=true
quarkus.http.tcp-fast-open=true
quarkus.http.limits.max-chunk-size=65536
quarkus.http.limits.max-body-size=100G

quarkus.vertx.prefer-native-transport=true

quarkus.rest-client.max-chunk-size=65536
quarkus.rest-client."org.example.client.TestClientService".url=http://localhost:8081
quarkus.rest-client."org.example.client.TestClientService".scope=jakarta.inject.Singleton
quarkus.rest-client.read-timeout=300000

quarkus.rest.input-buffer-size=65536
quarkus.rest.output-buffer-size=65536

quarkus.resteasy.vertx.response-buffer-size=65536

[fredericBregier]

I try with the value of 2048 right now...

[fredericBregier]

I changed all 65536 to 2048 everywhere.
2048 changed everywhere (application.properties and code) using 1GB file with 5 iteration: (still same Quarkus Server Rest behind)

• Quarkus POST : 26.1 MB/s
• Netty POST: 51.0 MB/s, so x2
• Quarkus READ: 43.4 MB/s, again almost x2

8192 with same context:

• Quarkus POST : 26.5 MB/s
• Netty POST: 176.6 MB/s, so x6.7
• Quarkus READ: 149.3 MB/s

65536 with same context:

• Quarkus POST : 26.4 MB/s
• Netty POST: 218.2 MB/s, so x8.3
• Quarkus READ: 419.1 MB/s

I don't know about this 2048 value (not seen in configuration but you know the code so...).
It seems indeed it ignores any values from the various configuration entries (whatever 8K or 64K).

[geoand]

I'll make a change to have that value taken into account

[fredericBregier]

Here?

• quarkus/independent-projects/resteasy-reactive/client/runtime/src/main/java/org/jboss/resteasy/reactive/client/impl/InputStreamReadStream.java

  Line 27 in 7843777

  private static final int CHUNK_SIZE = 2048;

• quarkus/independent-projects/resteasy-reactive/common/runtime/src/main/java/org/jboss/resteasy/reactive/common/providers/serialisers/FileBodyHandler.java

  Line 67 in 7843777

  final byte[] buf = new byte[2048];

[fredericBregier]

Even if in this case, only the first is related since I do not used file. But both are quite a limitation (even the JDK change the default to 8K for instance for CipherInputStream).

[geoand]

Here?

• quarkus/independent-projects/resteasy-reactive/client/runtime/src/main/java/org/jboss/resteasy/reactive/client/impl/InputStreamReadStream.java

  Line 27 in 7843777

  private static final int CHUNK_SIZE = 2048;

• quarkus/independent-projects/resteasy-reactive/common/runtime/src/main/java/org/jboss/resteasy/reactive/common/providers/serialisers/FileBodyHandler.java

  Line 67 in 7843777

  final byte[] buf = new byte[2048];

Nope, that's not the place. I'll open a PR tomorrow as I'm on the phone now and I can't really find where the 2048 constant is

[fredericBregier]

No problem. I suspect anyway this 2 could be changed at least to 8K anyway... ;-)
I tried to look for 2048 values, and that were the only relevant ones for me, but of course not 100% sure ;-)

[geoand]

Mind trying #46739?

[fredericBregier]

I will give it a try tomorrow morning ;-)
I suppose of course I need to compile at least resteasy-reactive/client module, or do I need to compile full Quarkus project ?

[geoand]

Best checkout the PR and build the entire project. There are some tips here

[geoand]

Thanks!

[fredericBregier]

Whouah ! I didn't wait for tomorrow morning. ;-)

Still using 64K as buffer size.

Before (3.19.2):

• Quarkus POST : 26.4 MB/s
• Netty POST: 218.2 MB/s, so x8.3
• Quarkus READ: 419.1 MB/s

Version 999-SNAPSHOT from your git/commit:

• Quarkus POST : 359.6 MB/s
• Netty POST: 419.4 MB/s
• Quarkus READ: 436.1 MB/s

Great !!

I would suggest to change also the 2048 buffer size in FileBodyHandler, as it can probably be the same (I do not have this case, since we do not use direct files but streaming).

Also the link in the PR is not related (the same number but as this was not an issue but a discussion ;-))

Really Thank you !

[geoand]

Thanks a lot for checking!

I will check your suggestion tomorrow and likely apply it