Description
I was doing some testing of my app in which I have restc_cpp as the client to an HTTP server (in this case, InfluxDB). In testing my app's resilience, I restarted the HTTP server and noticed that the app didn't resume successfully sending HTTP requests to the server. Instead, it kept trying to reuse the Connection, and the Connection kept being persisted.
I have been able to reproduce this in unit tests by:
- Starting up the HTTP server and successfully sending it a requests
- Terminating the HTTP server for some time and attempting more requests
- Starting the HTTP server again and attempting a request
I need to dive into the code further to try and see what's going wrong, because I know the Connection logic is such that ~ReplyImpl() is usually good about closing a Connection when necessary. But I wanted to get the conversation going and enlist some help.
I'm wondering if either ConnectionPoolImp::GetConnection or ConnectionPoolImpl::OnRelease would be an appropriate place to destroy a connection with a defunct socket based on the following logs I'm seeing:
Nov 08 17:49:48.754 [ATCS|REST] TRACE (00007f9d85e58700) Reusing connection from cache {Connection f6b26435-3b20-4f70-8f01-e1dea96d2599 {Socket socket# e {std exception: remote_endpoint: Transport endpoint is not connected}}{Socket (unused/closed)}}
Nov 08 17:49:48.754 [ATCS|REST] WARN (00007f9d85e58700) Write failed with exception type: N5boost6system12system_errorE, message: Broken pipe
Nov 08 17:49:48.755 [ATCS|REST] TRACE (00007f9d85e58700) Canceled timer SendRequestPayload
Nov 08 17:49:48.755 [ATCS|REST] TRACE (00007f9d85e58700) Recycling {Entry {Key http://127.0.0.1:8086} {Connection f6b26435-3b20-4f70-8f01-e1dea96d2599 {Socket socket# e {std exception: remote_endpoint: Transport endpoint is not connected}}{Socket (unused/closed)}}} after use
Nov 08 17:49:48.756 [ATCS|REST] TRACE (00007f9d85e58700) Reusing connection from cache {Connection f6b26435-3b20-4f70-8f01-e1dea96d2599 {Socket socket# e {std exception: remote_endpoint: Transport endpoint is not connected}}{Socket (unused/closed)}}
The problem may also be that Cache Cleanup does not reap this connection
Nov 08 17:49:48.605 [ATCS|REST] TRACE (00007f9d85e58700) Cleaning cache...
Nov 08 17:49:48.606 [ATCS|REST] TRACE (00007f9d85e58700) Keeping << {Connection f6b26435-3b20-4f70-8f01-e1dea96d2599 {Socket socket# e {std exception: remote_endpoint: Transport endpoint is not connected}}{Socket (unused/closed)}} expieres in 3a seconds
In IoTimer::Create the broken socket is also apparent:
Nov 08 17:49:48.754 [ATCS|REST] TRACE (00007f9d85e58700) Created timer SendRequestPayload for {Connection f6b26435-3b20-4f70-8f01-e1dea96d2599 {Socket socket# e {std exception: remote_endpoint: Transport endpoint is not connected}}{Socket (unused/closed)}}