Deadlock in WebSocket when there is a receive in progress

[Nemirtingas]

There is a deadlock that can occur if the WebSocket is receiving a message right when another one is sent.
Here is the callstack of the receive:

Here is the callstack of the send:

[Nemirtingas]

Send lock order:

1. WsTransport::sendFrame
   Locks WsTransport::mSendMutex
2. TlsTransport::send
   Locks TlsTransport::mSslMutex

Receive lock order:

1. TlsTransport::doRecv
   Locks TlsTransport::mSslMutex
2. WsTransport::sendFrame
   Locks WsTransport::mSendMutex

[paullouisageneau]

Thank you for reporting. It looks like the bug is more subtle than that: it affects only MbedTLS, and is triggered specifically when a zero-sized message (caused by a read timeout) is received when a message is sent. The exact issue is the recv() from the MbedTLS ReadCallback.

[Nemirtingas]

Does GnuTLS also has the issue?

libdatachannel/src/impl/tlstransport.cpp

Line 262 in 46f8e7c

t->recv(std::move(message)); // Pass zero-sized messages through

[paullouisageneau]

Does GnuTLS also has the issue?

No, because GnuTLS is thread safe so it doesn't require the big mSslMutex to guard API calls.