Yamux: Zombie connection when the client's network is not stable

[marverlous811]

I am running a service to proxy the request from the application to the embedded devices. But I have some problems with Yamux. It creates many connections in some devices that have network problems, despite enabling keep-alive. When I use the tcpdump command to check packets, only the real connection has a keep-alive packet between the client and server; all the other connections do not have anything. I think some things are stuck in the keep-alive poll. Could anyone please give me some suggestions to fix it?

Currently, the server has 8000 Yamux sessions. The zombie connection is still increasing, which causes a memory leak on my server.

My server

OS: Ubuntu
Hardware: 2 cores and 2 GB of RAM.

[driftluo]

Your description is vague, but I suspect your server or client may be under attack. If clearer code were provided, it might be more direct.

Yamux's keep-alive only executes at the session level; sub-streams do not have keep-alive operations and completely rely on session-level keep-alive support.

When your sub-streams are constantly increasing, it's reasonable to suspect there are continuous streaming operations in your code or that you’re being attacked. This can be addressed by checking the code or setting these parameters to limit it:

• max_stream_count: The maximum number of sub-streams allowed to be opened.
• accept_backlog: The maximum number of sub-streams allowed to be pending.

Any newly opened sub-streams will be immediately closed if these settings are exceeded.

By the way, the trigger priority of keep-alive is indeed higher, but it does not affect the communication of sub-streams. If there are unlimited sub-stream opening requests(this is the attack), and without limiting of max_stream_count, it will prioritize opening requests and temporarily not process the communication requests of sub-streams.(This behavior is expected. Prioritize session-level tasks, followed by sub-stream tasks.)

The following operations should not exist in your code:

loop {
  control.open_stream().await?
}

or

loop{
  yamux::Session.open_stream().unwrap()
}

[marverlous811]

@driftluo I don't think the sub-stream creates a new TCP socket, or I am under a network attack. My server is running on port 50001. The result of the command on the server netstat -tan | grep 50001 | grep 10.10.0.106 shows more than one TCP connection that is established. Meanwhile, in the client, it shows only one connection.

Example:

root@ubuntu:~# netstat -tan | grep 50001 | grep 10.10.0.106
tcp        0      0 10.10.0.105:50001    10.10.0.106:47016    ESTABLISHED
tcp        0      0 10.10.0.105:50001    10.10.0.106:52516    ESTABLISHED
tcp        0      0 10.10.0.105:50001    10.10.0.106:54516    ESTABLISHED
tcp        0      0 10.10.0.105:50001    10.10.0.106:52525    ESTABLISHED
tcp        0      0 10.10.0.105:50001    10.10.0.106:42516    ESTABLISHED

root@ubuntu:~# netstat -tan | grep 50001
tcp        0      0 10.10.0.106:47016    10.10.0.105:50001    ESTABLISHED

In my opinion, the keep-alive logic is having some trouble. It does not detect the disconnected event of the old session. It's difficult to simulate it with a small number of connections and a local network.

My main loop

let mut session = Session::new_client(in_stream, Default::default());
loop {
  select! {
    control = control_rx.recv() => match control {
       Some(Command::CreateStream(tx)) => match session.open_stream() {
         Ok(stream) => {
             tx.send(stream)
         }
         Err(err) => {
            log::error("{:?}", err);
         }
       }
    }
    accept = session.next() => match accept {
       Some(_) => {}
       Some(None) => {
         break; 
      }
      Err(e) => {
         break;
      }
    }
  }
}

[driftluo]

@marverlous811 Yes, the sub-stream doesn't create a new TCP socket. I'm not sure why your server has extra TCP connections. It may be attacked.

However, I wrote a test and found that the keep_alive in the yamux implementation has a scenario where the waker is lost. This is an implementation error. I added a test and submitted a PR to fix it. Thanks to your report.

However, if the TCP connection is not packaged into a yamux session, this fix will be invalid.

[marverlous811]

Yes, I also think the keep_alive interval lost the waker. So it doesn't release the closed session.

Thanks for your support

[driftluo]

#412 has been merged and published

[marverlous811]

I will try it and report for you soon

[marverlous811]

@driftluo thank you very much. After 4 days, while I was using your update patch, I didn't see any zombie connections in my server. It's worked. I will close this issue.

Before After