Optimizing IO Throughput

[TonyValenti]

I'm not sure if this is related or not but we recently upgraded to .NET 7.0.

I have an IO intensive application that seems to be running into a bottleneck and I'm not sure how to address it.

The application does the following with DataFlow:

• Read records from a local database
• Call a webservice with info from the records
• Update the database with results from the web service call.

My goal is for the application to consume all available system resources. I'd like to see CPU or network (or both!) fully maxed out.

I have some logging in my app and I notice that a line like this:

            var Content = await This.Message.Content.ReadAsStringAsync(Token)
                .ConfigureAwait(false)
                ;

occasionally takes 10-20 seconds to complete even when fetching only 1kb of text. However, watching network activity through Fiddler, it seems like the request actually is processed really quickly but it just takes .NET 10-20 seconds to realize the content is available.

When I pause my app in VS, I see lots of threads on these two methods:

System.Private.CoreLib.dll!Interop.Kernel32.GetQueuedCompletionStatus
System.Net.Http.dll!System.Net.Http.HttpConnectionPool.TryGetPooledHttp11Connection

I have read a bit about IOCompletion ports and the thread pool and I suspect I'm running into something related to that but I'm not sure what to do or how to address it.

Does anyone have any thoughts?