watchdog_reboot delay in flashFirmware

[websterelec]

I found that with larger programs, the delay in watchdog_reboot (500ms) isn't long enough for the initial flash_range_erase call that erases the target memory area to complete. I measured 730ms for a size of 0x35000. I increased the delay to 2 seconds for my application.

For some reason that I don't understand, when I had a usb connection to the pc (used for loading .uf2 files) and the watchdog expired due to insufficient time, things behaved differently and I couldn't get the board to recover. I noticed this during panicked troubleshooting so I'm not certain if the usb connection really had an impact.

The 500ms in the original code is correct for the sample applications, but I wanted to leave a note that might be helpful to someone in the future.

[rhulme]

Hi,

Thanks for that. Yes, if you have a large area to erase, it can take longer than 500ms. I hadn't encountered that in the scenarios I've used it.

The cleanest solution is probably to erase smaller blocks in a loop and reset the watchdog on each pass. Then it doesn't make any difference how big the application is. I'll look at making that change as soon as I have time.

The issue you saw after the watchdog expired is very strange. I'll see if I can reproduce it.

[websterelec]

I can't see how having the USB connection should impact anything. Possibly due to my application including pico_stdlib that includes stdio.h, but I have pico_enable_stdio_usb(${PROJECTNAME} 0) in my cmakelist file. After increasing the watchdog timeout, I did not see the issue again. Since the USB connection won't be accessible in the final product, this would only show up for me in development.

Thanks again for sharing your code. It has saved me a significant amount of time.

[websterelec]

I compared the erase times between boards. The board that successfully programs with the timeout at 500ms takes 1.1 seconds to erase. The board that fails and goes to the standard bootloader takes 1.2 seconds. I changed the timeout from 500ms to 700ms and repeated with both boards and they both succeeded. So I'm guessing I must be on the edge with the retries somehow. For now, I'm going to leave the timeout at 2 seconds. I haven't seen an issue with it and trying to debug further to nail down exactly what is happening hopefully isn't worth the time.

[rhulme]

OK. I fixed the main timeout issue by erasing the flash area one block at a time and then retriggering the watchdog timer. That should be scalable too so even if your application becomes even bigger over time, there shouldn't be any problems.

It took some fiddling but I also found out why it worked after power-cycling the board. The hardware watchdog module needs to know how fast the main clock is running so that it can scale appropriately. Because the flashloader wasn't initialising it, it was running at a much slower speed and therefore didn't trigger a reset (the time take to erase flash is independent of the speed the RP2040 is running at).

If the normal application is started and this uses the standard SDK initialisation code, the watchdog is initialised correctly so the timeout was correct for every warm start afterwards.

On a cold start where there was no valid application, the timeout would be much longer and the update process had time to run through.

[websterelec]

That makes sense. Thanks for digging into it. I'll update my code with the new versions and let you know how it goes after I've done some testing. I'm using your code to allow both firmware updates locally, through a USB interface, and remotely, via cloud transfer. Remote updates are both comforting (fix bugs remotely) and concerning (brick every unit in the field).

[rhulme]

When you say "bricking every unit in the field", do you mean there is the potential to brick them, or that they are being bricked?
There's not a huge amount that a flashloader can do to stop you flashing buggy software. It can be theoretically mitigated by having some kind of fallback mechanism that either flashes the previous "known good" software or if the flashloader itself can have the ability to fetch a new update from somewhere but the main problem is determining that the new software has a problem. There are any number of scenarios where problems could be caught and just as many where it it's incredibly difficult to somehow programmatically recognise issues. Then it mainly comes down to testing as much as possible before deploying and managing the roll-out so not every device immediately updates to the latest software.

If you don't already, I would seriously consider using the 'urloader' branch to allow you to also update the flashloader in the field if required.

[websterelec]

When I said "bricking every unit in the field", I'm just worrying that something could happen during the update process that causes a failure where the board locks up or ends up in the standard bootloader. I understand that should never happen based on how your code works. It has a a checksum to make sure the new code is valid and if power is lost at any point during the update, it will just start over when power is restored. I've updated firmware hundreds of times locally through a USB drive and the only issue I had was the wd timeout. In case of bugs in my application, I am testing as much as possible and intend to only allow updates on a site-by-site basis initially to make sure things go smoothly. I'll check out the urloader branch and hope to have time to implement it. Thanks for all your help.