degroff/virtual threads update #33

robotdan · 2025-05-16T21:20:42Z

Summary

bug fixes
additional logging
logging updates
performance updates
consideration for 1.0.0

-      if (ch == 'H' || ch == 'T' || ch == 'P' || ch == '/' || ch == '1' || ch == '.') {
+      // While this server only supports HTTP/1.1, allow the request protocol to be parsed for any valid version.
+      // - The supported version will be validated elsewhere.
+      if (ch == 'H' || ch == 'T' || ch == 'P' || ch == '/' || ch == '.' || (ch >= '0' && ch <= '9')) {


This allows us to parse HTTP/1.0 so we can allow this protocol.

robotdan · 2025-05-30T20:02:07Z

src/main/java/io/fusionauth/http/server/internal/HTTPServerThread.java

@@ -82,7 +82,7 @@ public HTTPServerThread(HTTPServerConfiguration configuration, HTTPListenerConfi
    }

    socket.setSoTimeout(0); // Always block
-    socket.bind(new InetSocketAddress(listener.getBindAddress(), listener.getPort()));
+    socket.bind(new InetSocketAddress(listener.getBindAddress(), listener.getPort()), configuration.getMaxPendingSocketConnections());


Performance.

The default backlog on the Server Socket .bind() is 50. This meant when I hammered the crap out of it, or ran ab (Apache Benchmark)and used> 50` workers and they all started at the same time, I would get socket exceptions because the server would reject them.

So this was a bottleneck to ramping up load testing.

Apache Tomcat calls this acceptCount in their config, and it defaults to and defaults to 100.

https://tomcat.apache.org/tomcat-9.0-doc/config/http.html#:~:text=Description-,acceptCount,-The%20maximum%20length

I have our default at 200 - which in theory is adequate. This is essentially the internal server socket queue and then we are pulling off pending connections as client sockets and creating a virtual thread to work them.

I have been able to run load tests with a 100 or more, and achieve 100k RPS with this config.

robotdan · 2025-05-30T20:06:32Z

src/main/java/io/fusionauth/http/server/internal/HTTPServerThread.java

+        //        Tomcat also has 'maxThreads' which defaults to 200 - this is per "connector", not sure how this is different than maxConnections.
+        //        Reading doc, perhaps 'maxConnections' is across all listeners (connectors)? So maybe we go close to the maxThreads config which is 200.
+        //        Without a config, we may be vulnerable to a DOS attack, so if we did want to do something, I would suggest using a blocking queue
+        //        for the clients collection and cause it to block when we reach capacity.


Tomcat has a config for maxThreads which defaults to 200.

https://tomcat.apache.org/tomcat-9.0-doc/config/http.html#Attributes_Standard%20Implementation_maxThreads

Since we are using virtual threads, in theory we don't need this. But we may want to consider how many virtual threads we have running at any given time, or how many requests per worker thread we allow.

For example, some HTTP servers just kill of workers after n requests or n amount of time. This would sort of be a maximum lifetime of a virtual thread regardless of the keep alive state. This can be useful to prevent DOS attacks where sockets are just kept open forever.

This would be super easy to implement if we wanted to do so. I am already keeping track of the create instant, and the number of handled requests by an HTTP worker.

robotdan · 2025-05-30T20:07:31Z

src/main/java/io/fusionauth/http/util/HTTPTools.java

@@ -96,6 +106,14 @@ public static boolean isTokenCharacter(byte ch) {
   */
  public static boolean isURICharacter(byte ch) {
    // TODO : Fully implement RFC 3986 to accurate parsing
+    //        https://www.rfc-editor.org/rfc/rfc3986#section-1.3


No change here, just added some doc.

robotdan · 2025-05-30T20:07:49Z

src/main/java/io/fusionauth/http/util/HTTPTools.java

+   * @param state the current parser state
+   * @return a throwable exception
+   */
+  public static ParseException makeParseException(byte b, Enum<? extends Enum<?>> state) {


This is just to allow for a consistent ParseException thrown from various places.

robotdan · 2025-05-30T20:12:26Z

src/main/java/io/fusionauth/http/server/internal/HTTPWorker.java

-        var handler = configuration.getHandler();
-        state = State.Process; // Transition to processing
-        handler.handle(request, response);
+        // TODO : Daniel : Review : If we must drain the InputStream prior to calling close, it seems like this would be a problem since the request handler


Would like to discuss this with some other folks. It seems to me that we must drain the InputStream prior to calling close() on the OutputStream.

I can demonstrate the effect of moving the drain past the close().

In practice, I think this is ok - however - because we can't control what the request handler does with close() this seems brittle.

We could remove the ability for a request handler to call close() as one option so we control it.

robotdan · 2025-05-30T20:14:58Z

src/main/java/io/fusionauth/http/server/internal/HTTPWorker.java

+      // - Close the connection, unless we drain it, the connection cannot be re-used.
+      // - Treating this as an expected case because if we are in a keep-alive state, no big deal, the client can just re-open the request. If we
+      //   are not ina keep alive state, the request does not need to be re-used anyway.
+      // TODO : Daniel : Review : We could add something to the instrumenter? In theory we could try and write back a failure, but that will most likely


Would like to discuss this as well.
I think this is the right thing to do - the JDK server caps the drain byte sat something super small, like 16K I think.

But if we do hit this, there isn't a great way to let the client know. But in a general sense it seems like the request handler should be reading these bytes.

One use case would be if an app server wanted to cap the number of bytes it reads in for safety reasons... in practice, it could look at the Content-Length header, or if chunked, it could read until it gets to a threshold.

In that case, it could return not having read all of the bytes, and we could hit this condition. It is plausible to think the app wanted to tell the client the payload was too big - but if we just close the socket, we are not going to communicate anything to the caller.

mmanes · 2025-05-30T22:07:47Z

src/main/java/io/fusionauth/http/server/ThreadSafeCountingInstrumenter.java

@@ -47,6 +49,11 @@ public void acceptedConnection() {
    connections.incrementAndGet();
  }

+  @Override
+  public void acceptedRequests() {


I think for consistency this should be named acceptedRequest().

Yes, good point. Thank you.

mmanes · 2025-05-30T22:14:49Z

load-tests/self/src/main/java/io/fusionauth/http/load/LoadHandler.java

+    res.setContentLength(response.length);
+    res.setContentType("text/plain");
+    try {
+      res.getOutputStream().write(response);


Most of the other handlers are calling flush(), but not this one. I'm assuming it's called downstream, so it's likely redundant.

Good point, I can take a closer look. We always call close() on the HTTPOutputStream in the HTTPWorker after we invoke the handler, so I suppose I could remove flush() in the other locations.

robotdan · 2025-05-30T22:33:29Z

src/main/java/io/fusionauth/http/server/HTTPServerConfiguration.java

+   */
+  @Override
+  public HTTPServerConfiguration withMaxPendingSocketConnections(int maxPendingSocketConnections) {
+    this.maxPendingSocketConnections = maxPendingSocketConnections;


Daniel note: Need to validate this is not less than some number. The server socket default is 50 - this could be a good minimum.

robotdan · 2025-05-30T22:35:51Z

src/main/java/io/fusionauth/http/server/Configurable.java

@@ -77,6 +77,8 @@ default T withContextPath(String contextPath) {
   * @param validator The validator.
   * @return This.
   */
+  // TODO : Daniel : Review : It would be cool to offer some additional methods such as withOptionalExpectValidator?
+  //                 Or would there be another way that we could allow the caller to designate null as don't set?


I initially was not going to allow this to be set to null. I am still thinking that is maybe the better option.

This would means we force you to be explicit and provide an expect handler. If it is null then there is a "default" behavior in the code instead of it being prescribed.

The reason I took it out initially was we had a test that just passed in null and expected it to set this to the default behavior - hence the comment about the withOptional.

Leaning towards requiring a non-null value here.

robotdan added 14 commits May 13, 2025 17:19

update some logging, add comments

43e9767

working

55a2aba

working

5a2c059

working

981f84e

working

22dc35c

Add logging, and tests

f128fbc

Tests

dfdc104

Tests

8c8c3ec

Tests

6836604

Tests

c709bbb

Working

309e8c3

Tests

32746a9

Tests

8356101

Tests

af92fe8

robotdan commented May 16, 2025

View reviewed changes

robotdan added 13 commits May 16, 2025 15:28

copyright

8ae5cf1

Working

2f7c476

Working

2376537

Working

b23099d

Working

d425944

Working

78e4c2e

Working

daaa59c

Working

2e0aa12

Working

409b40f

Working

55f14b3

Working

3d322ac

Working

9dc897f

Working

f7148a0

robotdan added 7 commits May 29, 2025 19:48

Working

a468f6b

Working

713ccb9

Tests

91db717

Working

e659c0e

Working

a0691f0

Get Tomcat setup

797ed9c

README

598729b

robotdan commented May 30, 2025

View reviewed changes

robotdan added 2 commits May 30, 2025 13:49

review edits

31a8e6f

build updates

993eb99

robotdan commented May 30, 2025

View reviewed changes

robotdan mentioned this pull request May 30, 2025

Add support for using Java or Restify HTTP clients FusionAuth/fusionauth-load-tests#20

Open

mmanes reviewed May 30, 2025

View reviewed changes

robotdan commented May 30, 2025

View reviewed changes

                       bodyBytes = null;
                     }
                   } else {
-                    read = delegate.read(buffer);
+                    read = delegate.read(buffer, offset, length);

-                    double result = ((double) numberOfBytesRead / (double) millis) * 1_000;
-                    return Math.round(result);
+                    // Always zero
+                    return numberOfBytesWritten;

-                      bodyBytes = inputStream.readAllBytes();
-                    } catch (IOException e) {
-                      throw new BodyException("Unable to read the HTTP request body bytes", e);
+                  if (bodyBytes == null) {

                   if (bodyBytes != null) {
-                    read = Math.min(bodyBytes.length, length);
+                    int remaining = bodyBytes.length - bodyBytesIndex;
+                    read = Math.min(remaining, length);

degroff/virtual threads update #33

Are you sure you want to change the base?

degroff/virtual threads update #33

Uh oh!

Conversation

robotdan commented May 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related

Uh oh!

robotdan May 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

robotdan May 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

robotdan May 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

robotdan commented May 16, 2025 •

edited

Loading

robotdan May 16, 2025 •

edited

Loading

robotdan May 16, 2025 •

edited

Loading

robotdan May 16, 2025 •

edited

Loading