Need help with SocketException (125)

[Yalpe]

Hello!

I need help troubleshooting a SocketException that happens fairly often in our production environment (obviously nowhere else).

System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
 ---> System.TimeoutException: The operation was canceled.
 ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
 ---> System.IO.IOException: Unable to read data from the transport connection: Operation canceled.
 ---> System.Net.Sockets.SocketException (125): Operation canceled
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
   at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](TIOAdapter adapter, Memory1 buffer)
   at System.Net.Http.HttpConnection.<CheckUsabilityOnScavenge>g__ReadAheadWithZeroByteReadAsync|44_0()
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)

Wrapped inside is this:

This exception is logged from an HttpClient call that times out. I don't know exactly why the socket would be closed (by whom?). The code is running in docker on alpine and on .NET 6.0.

Is it happening because the server is throttling the request and not returning a 429 in under 100 seconds and then the task gets cancelled? Is there anything that we can do on our end to handle this better?

To me it seems not related to throttling because we had this happen over multiple different servers in all our pods. I'm thinking there's something wrong with our connections (maybe at the OS level?).

[denisp13]

@Yalpe we are facing the same issue as described by you..
Have you got any solution bynow?

cheers

[Robert-Ed-Smith]

Did you manage to resolve this?

We are also observing a very similar error, when we access a Mongo database.

Again this happens only on a production environment.

This has only started since migrating to Net 7.

MongoDB.Driver.MongoConnectionException: An exception occurred while receiving a message from the server.
---> System.IO.IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request..
---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request.

I am uncertain why this has started, and of the best way to resolve (it is not clear we can simply retry the Mongo query, as the write - in this case - could have been partially successful.)

[Yalpe]

No we have not resolved this as it is mostly an annoyance and not critical (at least, we hope its not). We are also using MongoDB but on .NET 6.0. You don't have the same kernel error code however (quick google search shows its related to IO completion ports?). It does not seem related to my issue at all.

[MihaZupan]

System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.

This indicates that the server didn't respond for a long time.
The exact socket exception that is reported shows where exactly the cancellation occurred, but the trigger is still HttpClient's timeout.