Runtime errors hang controller even after power restart

[scottdky]

I am programming an RP2040. If during the course of operating the program, it encounters some sort of error, sometimes the micro hangs, even after a hard restart or even a power off. The only way to recover is to boot into bootloader mode, mount as USB, unmount, and restart. Then the program runs normally.

There are no errors in startup, and the fact that going into bootloader mode fixes things, makes me believe there is some sort of persistent state that is only reset when mounting as a USB drive. Am I correct? What is the best way to deal with this situation? This project will be in an industrial product, so going into a persistent wacky state would be a major problem.

[scruss]

Unless you're reflashing when you mount as USB (aka BOOTSEL mode on the RP2040), it shouldn't affect flash state for subsequent reboots. Could the code be affecting boot.py or main.py when it's running?

Code that runs at startup, then fails after a while may likely have a subtle runtime error. How's your memory allocation? If you're networked, are you closing all requests you open? Are you using threads? Not that threads are inherently unsafe, but they add complexity that can pull the rug from under you in many unexpected ways. Again, those shouldn't cause unrecoverable errors.

[scottdky]

That is what is so weird, other than using a thread, I am not doing any of those things. The runtime errors I get are in response to a user input over USB serial. It may be a code error or an out of memory condition. But whether I push reset (I have a reset button on my board, in addition to bootsel), or actually remove the USB/power connection, the code will sometimes refuse to startup after one of these errors, even though the code in boot and main simply load working code.

It is as if there is some sort of state preserved across rebooting, and only entering bootsel mode and mounting as a USB drive will reset it. I have also noticed that out-of-memory conditions when loading a lot of libraries seem to be random. In other words, sometimes the same code will run just fine, and other times it runs out of memory during imports.

[kjm1102]

1. Is it board specific, do you have another 2040 you could try?
2. Is it power related, can you plug a lipo in to prevent usb power sags?
3. Are you using an IDE. I've noticed some some of my programs will do weird stuff running in the IDE but behave normally when I actually download them to the board.
4. Are you using deepsleep? Deepsleep on the 2040 in upython has issues.

[scottdky]

Thanks so much for the suggestions. While I haven't tried another board, I do know there aren't any power sags. I am using rshell, so that rules out the IDE, and no to using sleep mode.

I do believe I may have found the cause though. Since originally posting, I have experienced more state issues where for example, the serial device wouldn't post to my USB connection. When I pulled the power for just a second or two, it still wouldn't post. When I pulled the power for several seconds, everything was restored to normal. So either there is an expiration time for enumerating USB devices in Linux or there is some residual energy in the capacitors or power supplies on my board that continue to keep the RP2040 running even after the power is withdrawn and any power LEDs are extinguished. I am assuming the latter for now.

Other than some minor issues as described, I don't seem to be having any more problems. If I do, I'll just make sure I unplug for several seconds. That seems to resolve things.

[davefes]

residual energy in the capacitors

I was putting 1000uF electros on the 3V3 line and run into issues with deepsleep causing reboot issues. Reading the ESP32 docs there seems to be a requirement that the 3V3 line drops below 0.6V (I am recalling) before you can successfully re-boot the device.

When you put the device into light or deepsleep it can take quite awhile for the 3V3 to drop, with a big cap on it.