[Bug Report] Unable to connect device client using MQTT_WS when there is high network latency

[sesu-bio]

Context

• OS and version used: BuildRoot Linux 2022.02.7
• Java runtime used: openjdk version "1.8.0_352"
• SDK version used: iot-device-client 1.31.0, can reproduce with 2.1.3

Description of the issue

Hi, Several of our devices in Australia are unable to connect to Azure via 3G. We think that the problems are related to a higher network latency there. When we simulate a high latency here, the logs look the same as the ones from Australia. To do that, we use the following command on Linux:
tc qdisc add dev usb0 root netem delay 200ms

Note that the higher latency only causes problems when using the MQTT_WS protocol. When using MQTT, the connection can be established without problems in our test setup.

Code sample exhibiting the issue

We can reproduce the issue with the following code. It uses iot-device-client 2.1.3. The command above needs to be executed first to reproduce the issue.

SSLContext sslContext = new AzureSslContextFromCredentials().apply(PUB_KEY_CERT, PRIV_KEY, null);
ClientOptions clientOptions = ClientOptions.builder().sslContext(sslContext).build();
DeviceClient client = new DeviceClient(CONNECTION_STRING, PROTOCOL, clientOptions);
try {
	client.open(false);
	System.out.println("Connected successfully!");
} catch (IotHubClientException e) {
	System.out.println("Error connecting!");
	e.printStackTrace();
}
System.out.println("finished");

Console log of the issue

error_log.txt

[timtay-microsoft]

Do you see this same pattern when using AMQPS vs AMQPS_WS?

[sesu-bio]

Thanks for the quick reply! AMQPS works fine even with the delay. AMQPS_WS always fails in my test setup because it doesn't support authentication.

[timtay-microsoft]

And how consistently can you repro this behavior from MQTT_WS in your latency induced environment? Is it a 100% repro, or just an occasional repro?

[sesu-bio]

It's a 100% repro as far as I can tell.

[sesu-bio]

I did some more testing now with different latency settings.
At 200ms added latency, I did see it connect successfully twice out of many attempts. The chance must be very slim (<5%). Side note: The successful connection attempts happened when I used a patched version of Paho where I adjusted the MqttConnectOptions but I'm pretty sure that was just a coincidence.
At 175ms, the success rate rises to about 50% (10 attempts).
At 150ms, it was 90% (10 attempts).

The successful connect attempts at 200ms happened relatively close together. At 175ms, four of the successful attempts happened in succession at the end. It seems like there are phases with a higher likelihood of success, possibly related to the "natural" latency. The azure servers don't respond to my pings. The latency to google.com is 35-55ms.

[sesu-bio]

To rule out all factors related to our device, I reproduced the problem in an Ubuntu 22.04 VM with OpenJDK 17 in a LAN environment. The latency I needed to add before I could reproduce the problem was notably higher. At 250ms added latency, I still had a 100% success rate. At 260ms, I could rarely get a successful connection attempt. I suspect that the tipping point for the total latency is somewhere between 270 and 290ms.

[timtay-microsoft]

We'll try to take a look into this when we get the chance, but it likely is an issue with the MQTT library we use. I'd suggest filing an issue on that library as well

[pizerg]

We've been also having issues with MQTT_WS that are probably related to this issue. In our case random disconnections from the service happened several times per day using SDK 1.34.3

The only workaround we could find was to switch to AMQPS_WS which seems to be working fine

[Xerxekyran]

Reporting this issue within paho.mqtt.java repo did nothing, as it seems that there is no activity there anymore. So MQTT_WS is not really useable for now. Due to bug #728 AMQPS_WS is not an option for me, as X509 is not supported. The HTTPS implementation does lead to a severe higher work load on the client as it reopens connections all the time just for querying if there are any new messages available.
Can microsoft not offer any solution / fix here after nearly a full year? Due to possible needed proxy chaining I am bound to X_WS or HTTPS protocols. I would really highly appreciate any progress here.