running out of ports when SkyWalking server unavailable

[evanyang85]

Hello,

We have found some issue on our API after the SkyWalking server unavailable:
Exception: One or more errors occurred. (An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full.
after analysis the source code, we found the root cause:

when creating GrpcChannel, use default GrpcChannelOptions
DisposeHttpClient property in this object default vaule is false
_channel = GrpcChannel.ForAddress(_server);

when shutdown the channel on ConnectAsync
there are no any implementation for ShutdownAsync on GrpcChannel
only empty virtual method there, see ShutdownAsyncCore
when assign new GrpcChannel to _channel, _channel is not shutdown and no release by GC in some scenario.

[a526672351]

已收到！

[evanyang85]

After a graceful closure or an abrupt closure of a session, after a period of 4 minutes (default), the port used by the process or application would be released back to the available pool. During this 4 minutes, the TCP connection state will be TIME_WAIT state. In a situation where you suspect port exhaustion, an application or process won't be able to release all the ports that it has consumed and will remain in the TIME_WAIT state.

[evanyang85]

After a graceful closure or an abrupt closure of a session, after a period of 4 minutes (default), the port used by the process or application would be released back to the available pool. During this 4 minutes, the TCP connection state will be TIME_WAIT state. In a situation where you suspect port exhaustion, an application or process won't be able to release all the ports that it has consumed and will remain in the TIME_WAIT state.

If SkyWalking server unavailable, the connection is not Ready, exception will through, but there has a timer in ExecutionService, the Period is hard code: 15 sec.

That means if SkyWalking server unavailable, every 15 sec will create new GrpcChannel, the tcp connection waiting .net runtime / OS release every 4 minutes (default), release speed cannot catch up creation speed, that's why running out of TCP ports.

we use below PowerShell to check the issue, found more than 1K+ Bound state on TCP connection.
Get-NetTCPConnection | Group-Object -Property State, OwningProcess | Select -Property Count, Name, @{Name="ProcessName";Expression={(Get-Process -PID ($_.Name.Split(',')[-1].Trim(' '))).Name}}, Group | Sort Count -Descending

[evanyang85]

The #609 is not the best approach, but it resolve the issue, "Bound" state still increasing, but will be release in 4 minutes.

the best approach is using incremental period for the ConnectService / simple connection testing for create new GrpcChannel, But I know it not simple, more scenario should be considering.
I am new to this project, no any background of SkyWalking, please advice or help propose better solution for next release, thanks.