Intermittent MQTTS Connection Failures (IDFGH-13979)

[vicatcu]

Checklist

• Checked the issue tracker for similar issues to ensure this is not a duplicate
• Read the documentation to confirm the issue is not addressed there and your configuration is set correctly
• Tested with the latest version to ensure the issue hasn't been fixed

How often does this bug occurs?

rarely

Expected behavior

MQTTS connection should consistently work

Actual behavior (suspected bug)

When connecting to my MQTT Server (self-hosted, Mosquitto), I intermittently experience the failures to connect with MQTTS.

Error logs or terminal output

When it fails to connect, I get the following output from my code:


E (15049) mqtt_client: esp_mqtt_handle_transport_read_error: transport_read(): EOF
E (15050) mqtt_client: esp_mqtt_handle_transport_read_error: transport_read() error: errno=119
[MQTT EVENT] reported from esp-tls (0x8008)
[MQTT EVENT] Last errno string (Success)
E (15065) mqtt_client: esp_mqtt_connect: mqtt_message_receive() returned -2
E (15068) mqtt_client: MQTT connect failed

Steps to reproduce the behavior

My code effectively looks like this:

  static esp_mqtt_client_config_t mqtt_cfg;
  mqtt_cfg.broker.address.hostname = settings->server;
  mqtt_cfg.broker.address.transport = MQTT_TRANSPORT_OVER_SSL;
  mqtt_cfg.broker.address.port = 8883;
  mqtt_cfg.credentials.username = settings->username;
  mqtt_cfg.credentials.client_id = settings->identity;
  mqtt_cfg.credentials.authentication.password = settings->password;
  mqtt_cfg.broker.verification.certificate = root_ca;
  esp_mqtt_client_handle_t _client = esp_mqtt_client_init(&mqtt_cfg);
  esp_mqtt_client_register_event(_client, (esp_mqtt_event_id_t) ESP_EVENT_ANY_ID, mqtt_event_handler, NULL);
  const esp_err_t ret = esp_mqtt_client_start(_client);
  boolean connected = false;
  if (ESP_OK == ret) {
      connected = waitForMqttConnected();
      // polls for global _mqttIsConnected global variable set in mqtt event handler to become true
      // case MQTT_EVENT_CONNECTED sets it to true
      // case MQTT_EVENT_DISCONNECTED sets it to false
      // case MQTT_EVENT_ERROR sets it to false 
      // or times out after 15 seconds
  }

  ... event handler code that is printing out the [MQTT EVENT] log messages looks like this:
    case MQTT_EVENT_ERROR:
      lastMqttEventId = MQTT_EVENT_ERROR;
      _mqttIsError = true;
      _mqttIsConnected = false;
      esp_tls_stack_err = 0;
      if (event->error_handle->error_type == MQTT_ERROR_TYPE_TCP_TRANSPORT) {
        if (event->error_handle->esp_tls_last_esp_err  != 0) {
          Serial.printf("[MQTT EVENT] reported from esp-tls (0x%x)\r\n", event->error_handle->esp_tls_last_esp_err);
        }
        if (event->error_handle->esp_tls_stack_err != 0) {
          esp_tls_stack_err = event->error_handle->esp_tls_stack_err;
          Serial.printf("[MQTT EVENT] reported from tls stack (0x%x)\r\n", event->error_handle->esp_tls_stack_err);
        }
        if (event->error_handle->esp_transport_sock_errno != 0) {
          Serial.printf("[MQTT EVENT] captured as transport's socket errno (0x%x)\r\n", event->error_handle->esp_transport_sock_errno);
        }
        Serial.printf("[MQTT EVENT] Last errno string (%s)\r\n", strerror(event->error_handle->esp_transport_sock_errno));
      }
      break;
   ...

Project release version

idf-release_v5.1-632e0c2a

System architecture

Intel/AMD 64-bit (modern PC, older Mac)

Operating system

Linux

Operating system version

Ubuntu 22.04

Shell

Bash

Additional context

No response

[vicatcu]

Cross-posted from espressif/arduino-esp32#10541

[VojtechBartoska]

@euripedesrocha Can you please help with triage? Thanks

[vicatcu]

@euripedesrocha just checking in, what do you think? how can I help us solve this problem?

[euripedesrocha]

@vicatcu first I need more information. Could you provide logs?
If the disconnection is happening rarely and mqtt client is able to reconnect, the behavior is as expected. Is it possible that the disconnection is a local network failure?
The error presented is typical from disconnection from the broker, I suggest you also look into the broker side logs also.

[vicatcu]

@euripedesrocha the logs were included in the OP:

E (15049) mqtt_client: esp_mqtt_handle_transport_read_error: transport_read(): EOF
E (15050) mqtt_client: esp_mqtt_handle_transport_read_error: transport_read() error: errno=119
[MQTT EVENT] reported from esp-tls (0x8008)
[MQTT EVENT] Last errno string (Success)
E (15065) mqtt_client: esp_mqtt_connect: mqtt_message_receive() returned -2
E (15068) mqtt_client: MQTT connect failed

This is certainly not typical of the output in the usual connect-report-disconnect case...

[euripedesrocha]

@vicatcu I saw this, I meant more logs to identify if we have an issue.

Why would you say it's not usual disconnect? The logs state that the socket was closed by the peer ( I know that they are not great in clarifying that).

As I suggest you could find your issue by looking into your broker's log.

[vicatcu]

I would say it's not a normal disconnect on the basis that my code doesn't act like an MQTT_EVENT_CONNECTED event ever happened. The trace I do get suggests its something related to TLS that's the problem. But to be fair, I don't have an explicit printf output in that switch-case, so I should probably add printf outputs to every switch-case in the event handler to get a better trace of what exactly is happening on the device side.

It's difficult for me to get broker-logs as this is a production server with lots of clients connecting and disconnecting all the time - a bit of a fire hose. If you have advice on how I might instrument that, I'm open to exploring / examining it.

My solution space is:

• identifying if there is a bug in the library and helping squash it if possible
• alternatively, identifying a strategy for coping with / recovering from whatever this intermittent behavior actually is

The behavioral pattern in my code is actually this if it helps:

• power up
• connect to Wi-Fi
• connect to MQTT broker (via MQTTS on TCP port 8883)
• publish some messages
• disconnect from MQTT broker
• shut down

And to be clear, when I say "power up" and "shutdown" I don't mean sleep modes, I mean power-off / power-on.

[euripedesrocha]

@vicatcu,
as you mention, the connected event is not happening because it doesn't complete. In such scenarios, the broker is rejecting the connect control message and closing.
Few things that might trigger such cases:

• Authentication is incorrect
• Repeated client id

If you are using a self-hosted mosquitto in production, you maybe enable the logs to the sys channel and subscribe to it to monitor. It will make the identification of the root cause easy to look to those logs.

[vicatcu]

@euripedesrocha I captured a bit more output with additional logging enabled in my application:

// the following two lines must be in response to esp_mqtt_client_start(_client)
E (14350) mqtt_client: esp_mqtt_handle_transport_read_error: transport_read(): EOF
E (14350) mqtt_client: esp_mqtt_handle_transport_read_error: transport_read() error: errno=119

// then the MQTT_EVENT_ERROR fires and these messages print out
[MQTT EVENT] MQTT_EVENT_ERROR
[MQTT EVENT] reported from esp-tls (0x8008)
[MQTT EVENT] Last errno string (Success)
E (14366) mqtt_client: esp_mqtt_connect: mqtt_message_receive() returned -2
E (14372) mqtt_client: MQTT connect failed

// then the MQTT_EVENT_DISCONNECTED fires
[MQTT EVENT] MQTT_EVENT_DISCONNECTED

I do have access to the sys channel and can subscribe to it to monitor it... but what exactly should I be looking for?

[vicatcu]

I'm also seeing sometimes that I get the MQTT_EVENT_CONNECTED messages after I have given up and timed out (after 15 seconds)... that seems like a long time to connect

[vicatcu]

For what it's worth, I tried implementing a "retry strategy" (up to three attempts) before giving up and I got this behavior:

// first attempt (same as before)
E (19477) mqtt_client: esp_mqtt_handle_transport_read_error: transport_read(): EOF
E (19477) mqtt_client: esp_mqtt_handle_transport_read_error: transport_read() error: errno=119

[MQTT EVENT] MQTT_EVENT_ERROR
[MQTT EVENT] reported from esp-tls (0x8008)
[MQTT EVENT] Last errno string (Success)
E (19492) mqtt_client: esp_mqtt_connect: mqtt_message_receive() returned -2
E (19498) mqtt_client: MQTT connect failed

[MQTT EVENT] MQTT_EVENT_DISCONNECTED
 Failed, no connect

// second attempt (first retry)
[MQTT EVENT] MQTT_EVENT_CONNECTED

My application is battery powered, though, so it's a trade-off whether to stay on and keep trying or to give up quickly, save state, and try again later. When it works, it works quickly, when it doesn't it takes a bit of time to fail.

[vicatcu]

If I subscribe to $SYS/broker/log/# on the broker, and filter messages down to those containing the MQTT ID of my development unit, are pairs of messages like this:

1731447729: New client connected from 74.69.54.130 as XXXXXXXXXXXXXXXX (c1, k120, u'XXXXXXXXXXXXXXXX').
1731447729: Client XXXXXXXXXXXXXXXX disconnected.

Do I need to widen the net on what I'm looking at somehow?

[vicatcu]

I've captured about 19 hours of logs from my firmware captured in PuTTY and run some post-hoc analysis on it. The summary of my preliminary analysis is that about 53 of the 232 cycles (23%), at power up cycles of about every 5 minutes - hard shutdown between attempts, required one or more retries. Among those, 46 cycles (~20%) successfully connected with 1-2 retries (37 required one retry, 9 required 2 retries), and 7 (~3%) failed to connect within 2 retries. These statistics don't seem that good to me... is this what one would expect?

I'm running another data capture over the weekend without using MQTTS and will report back on the results. I have a hypothesis that this behavior is only evident when using TLS. I'm not sure where that leaves me if the hypotheses proves to be true...