OTA firmware update causes watchdog timer to bite

[andrewclapham]

Hey all.

Really appreciate the effort that's gone into the autoconnect project.

I'm using Arudino IDE and an ESP32 for a fun project that I'm working on.

When I upload new firmware, I get bitten by the watchdog timer before the file finishes uploading.

My .bin file is 1.2MB

Here's my serial output from the ESP32:

Start OTA Updating
OTA Progress: 1436(1436)
OTA Progress: 2872(1436)
(...)
OTA Progress: 281456(1436)
E (59065) task_wdt: Task watchdog got triggered. The following tasks did not reset the watchdog in time:
E (59106) task_wdt:  - IDLE0 (CPU 0)
E (59107) task_wdt: Tasks currently running:
E (59107) task_wdt: CPU 0: establishWifiTa
E (59107) task_wdt: CPU 1: IDLE1
E (59107) task_wdt: Aborting.
abort() was called at PC 0x40178984 on core 0

ELF file SHA256: 0000000000000000

Backtrace: 0x40089660:0x3ffbfa00 0x400898dd:0x3ffbfa20 0x40178984:0x3ffbfa40 0x40087d3d:0x3ffbfa60 0x400859f9:0x3ffb3e80 0x400886f7:0x3ffb3ea0 0x4008884a:0x3ffb3ec0 0x4018335d:0x3ffb3ee0 0x400f0d7d:0x3ffb3f00 0x400ecd31:0x3ffb3f20 0x400ecee5:0x3ffb3f40 0x400e4f57:0x3ffb3f60 0x400e01a9:0x3ffb3f90 0x4018c626:0x3ffb3fd0 0x400e3426:0x3ffb3ff0 0x400dcae7:0x3ffb4060 0x4018c3aa:0x3ffb4080 0x400e83e1:0x3ffb40a0 0x400d819a:0x3ffb40d0 0x400d8de5:0x3ffb4100 0x400d9c69:0x3ffb4300 0x400dc709:0x3ffb44c0 0x400df7f7:0x3ffb4510 0x400dfcda:0x3ffb4530 0x400dfcff:0x3ffb4670 0x400d26dd:0x3ffb4690 0x4008a8ee:0x3ffb46f0

Rebooting...
ets Jun  8 2016 00:22:57

rst:0xc (SW_CPU_RESET),boot:0x12 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0018,len:4
load:0x3fff001c,len:1044
load:0x40078000,len:10124
load:0x40080400,len:5856
entry 0x400806a8
Setup() code is running on core: 1
(...)

Here's the relevant section of my task:

void establishWifi(void * parameter)
{

  Serial.print("                    # establishWifiTask is running on core: ");
  Serial.println(xPortGetCoreID());

  yield();

  if (setupmode)
  {
    // Setup Wifi
    Serial.println("                    # WIFISETUP_PARADIGM ENABLED. Going directly to captive portal");
    acConfig.immediateStart = true;
    acConfig.ota = AC_OTA_BUILTIN;
  }
  else
  {
    Serial.println("                    # Not in WIFISETUP_PARADIGM. Search for saved wifi network.");
    acConfig.autoReconnect = true; // Just try to connect to the saved Wifi networks
    acConfig.autoRise = false; // Do not show the captive portal for any reason.
  }

  Serial.println("                    # Establishing server.");

  // Setup Wifi AutoConnect page
  Server.on("/", rootPage);

  acConfig.apid = "Example-" + String((uint32_t)(ESP.getEfuseMac() >> 32), HEX);
  acConfig.psk = THIS_DEVICE_WIFI_PASSWORD;
  acConfig.title = "Setup wifi";

  acConfig.boundaryOffset = 200;

  Portal.onDetect(startCP);

  Portal.config(acConfig);
  // portal.begin();
  
  Portal.onOTAStart(OTAStart);
  Portal.onOTAEnd(OTAEnd);
  Portal.onOTAProgress(OTAProgress);
  Portal.onOTAError(OTAError);
  

  Serial.println("                    # About to start wifi");

  if (Portal.begin())
  {

    Serial.println("                    # !!!!!!");
    Serial.println("                    # WiFi connected. IP: " + WiFi.localIP().toString());

  }
  else
  {
    Serial.println("                    # Connecting to WiFi failed.");
  }

  Serial.println("                    # After establishWifi main work.");

      // Now, kick off the maintainWifiTask

  xTaskCreatePinnedToCore(
                  maintainWifi,   /* Task function. */
                  "maintainWifiTask",     /* name of task. */
                  10000,       /* Stack size of task */
                  NULL,        /* parameter of the task */
                  1,           /* priority of the task */
                  &maintainWifiTask,      /* Task handle to keep track of created task */
                  0);          /* pin task to core 0 */


  vTaskDelete(NULL);
}

void maintainWifi( void * parameter)
{

  Serial.print("                   $ maintainWifiTask started running on core: ");
  Serial.println(xPortGetCoreID());

  for(;;)
  {
      // Don't let the watchdog timer bite
      yield();

      // Start wifi connection checking


      Serial.println("                   $ Maintaining wifi.");

      if (WiFi.status() == WL_CONNECTED)
      {
          Serial.println(F("                   $ WIFI is connected"));

      }
      else
      {
          Serial.println(F("                   $ WIFI connection is not present."));

      }


      // End checking
      Portal.handleClient();
      delay(1000);
  }

} // End of maintainWifi

Has anyone else encountered this? Would love any high level advice or hints.

Could I perhaps disable the watchdog timer while the firmware is uploading, and then re-enable it dynamically?

Thanks all!
Andrew

[andrewclapham]

I suspect that a solution has been found!

I added some superstitious yield(); commands in the offending task, but more importantly, I switched running establishWifiTask from core 0 to core 1.

Suddenly, the captive portal is a lot more responsive (faster between page loads) and the firmware upgrade now works, the WDT doesn't bite.

I've read that core 0 handles Wifi and BLE, and that if you're using Wifi or BLE, that it's advisable to keep all your application logic on the other core, core 1. So that seems to have done the trick!

Hope someone else is helped by this possible solution.

Thanks again for making and maintaining the library all.
Andrew

[dumarjo]

I have the same problem with an ESP32-S2-wroover module. This is a single core, so cannot switch the portal to another core. I try to find the offending core in the update firmware code, but cannot find it.

I have lower the loop priority to the IDLE priority and I have the same problem. On esp32-s2 the watchdog is biting in a ISR context, So I don't know if this is a AutoConnect problem OR an IDF problem.

Guru Meditation Error: Core  0 panic'ed (Interrupt wdt timeout on CPU0).                                                                                                                                                                        
C                                                                                                                                                                                             
PC      : 0x4002593e  PS      : 0x00060034  A0      : 0x400253bd  A1      : 0x3ffc4040  A2      : 0x3ffe5c9c  A3      : 0x00000000  A4      : 0x00000000  A5      : 0x4002cc90
A6      : 0x3ffc4294  A7      : 0x00000000  A8      : 0x8002593e  A9      : 0x3ffc4010
A10     : 0x00000000  A11     : 0x00060021  A12     : 0x3ffc4044  A13     : 0x00000000
A14     : 0x3ffe5de0  A15     : 0x3ffe5e34  SAR     : 0x00000001  EXCCAUSE: 0x00000005
EXCVADDR: 0x00000000  LBEG    : 0x3ffc4044  LEND    : 0x00000000  LCOUNT  : 0x40027068
Core  0 was running in ISR context:
EPC1    : 0x400b82af  EPC2    : 0x00000000  EPC3    : 0x00000000  EPC4    : 0x4002593e


Backtrace:0x4002593b:0x3ffc40400x400253ba:0x3ffc4070 0x4015643c:0x3ffe5700 0x400af6df:0x3ffe5720 0x400af8d2:0x3ffe5770 0x400ac063:0x3ffe5850 0x400a6dce:0x3ffe5890 0x400a6eb1:0x3ffe58d0 0x400a72ab:0x3ffe58f0 0x400a744d:0x3ffe5930 0x400a6f6d:0x3ffe5950 0x4008c87e:0x3ffe5980 0x401555e2:0x3ffe5a00 0x40089626:0x3ffe5a20

[Hieromon]

@andrewclapham @dumarjo Sorry for the late reply.

First, in response to @andrewclapham; It might be worth trying delay instead of yield(). What happens if you change delay(1000); immediately after handleClient() to delay(1); and exclude yield()?

And to @dumarjo; Can you let me know more about the conditions under which the phenomenon you are encountering occurs? Usually, the ISR needs to degenerate to a quite few steps. Otherwise, it will be banned by the WDT. Also, processes that prevent thread switching, such as while (1) break; loops, are also trapped by the WDT.

[dumarjo]

@Hieromon

I think the problem is inside the library (the esp-IDF), the ESP32-S2 is a single core and this look like the WDT bite on erase sector. Probably the ISR are disabled in this time and the erase sector is a bit too long for the time set in the SDT (300 ms) by default I think. When I continue to work on this project, I will do more testing.

Regards,

[andrewclapham]

Hey @Hieromon no problem at all, thanks for the utility, I'm getting a huge amount of value out of it.

If the problem arises again, I'll replace yield with delay. Thanks again!