Update to Micropython 1.25, ESP-IDF 5.4.1

[Molive-0]

Description

Updates the entire repository to use the latest stable Micropython and ESP-IDF release.
Note that free flash space decreases from 12% to 9% with this update.

Also:

• Readme now recommends the build container be run with your own user rather than root (as per the ESP-IDF docs), which fixes some issues with build artifact permissions.
• Adds scripts/hard-reset.sh, which deletes all build artifacts and returns the submodules to how they were before scripts/firstTime.sh was run.
• Removes CONFIG_FREERTOS_ENABLE_STATIC_TASK_CLEAN_UP=y from the sdkconfig in order to fix the compile. Need to check if this is safe.

See Jonty's comment below for a quick summary of the Micropython changelogs.

Todo

• Check codebase for sys.exit() and raise SystemExit (from Jonty's comment). A quick Ctrl-f on the codebase seems to suggest that TildagonOS never does either of these things (aside from sys.exit()s in the sim), so we should be fine. Doesn't rule out any apps doing this, but I see no reason why they would.
• Is it safe to remove CONFIG_FREERTOS_ENABLE_STATIC_TASK_CLEAN_UP=y from the sdkconfig, or is that integral to some part of TildagonOS? (see Update to Micropython 1.25, ESP-IDF 5.4.1 #200 (comment)) Config option was added in this commit by Matthew Wilkes, but documentation suggests it's only for if you have a vPortCleanUpTCB function, and I don't think we do.

[Jonty]

Quickly skimming the .24 and .25 release notes for things we need to check or should take advantage of:

v1.24.0

To check

• The behaviour of sys.exit() and raise SystemExit (which are equivalent) has been changed: previously this would terminate the running script and drop to the REPL, but now it terminates the running script and triggers a soft reset of the device. This is more consistent with the unix port, which exits completely on a SystemExit exception. This change in behaviour is a breaking change and may impact certain applications; see commit micropython/micropython@69c25ea for details.

Interesting

• In the networking sub-system, a new API for configuring IP addresses and related settings has been added: the global network.ipconfig() function, and a nic.ipconfig() method on individual network interfaces. This new API supports both IPv4 and IPv6, with much more control compared to the original nic.ifconfig() method; the latter is still available for backwards compatibility, but ipconfig() is now preferred moving forward.

v1.25.0

Interesting

• After more than three years in development, the "ROMFS" feature has been finalised, its filesystem format specified, and the VFS driver and supporting code are included in this release of MicroPython. This feature builds on bytecode version 6 (available for many years now), which supports executing bytecode in-place, that is, without the need to copy it to RAM. ROMFS defines a read-only, memory-mappable, extensible filesystem that can contain arbitrary resources, including precompiled mpy files, and allows executing bytecode directly from the filesystem. This makes importing significantly faster and use a lot less memory. Also, data resources such as fonts can be used in-place on a ROMFS without loading into RAM.
• The MicroPython native linker mpy_ld.py now includes support for linking in static libraries automatically. This allows the native-module build scripts to look for required symbols from libraries such as libgcc and libm that are provided by the compiler. This now makes it possible to use standard C functions like exp() in native modules. Also, native modules now support 32-bit RISC-V code.
• Dynamic USB device support is now enabled on ESP32-S2 and ESP32-S3 MCUs, allowing configuration of the USB device at runtime. I2S has been enabled on all ESP32-C3 boards, the Pin.toggle() method has been added, and the I2C bus identifier is now an optional argument (by default, bus 0 is used). Additionally, memory management has been improved for the allocation of TLS sockets to attempt to automatically free any existing unused TLS memory when needed.

[Molive-0]

Annoyingly, the CI has failed to compile this. I'll look into it tomorrow.

[Molive-0]

This branch should now build in CI, as a test I ran on my laptop locally seems to work. It did require removing the tildagon_usb driver, however it seems that it never did anything in the first place (let alone anything to do with USB)

[ChrisDick]

There's been a change to the idf where freertos.h no longer includes idf_additions.h. I think #include "freertos/idf_additions.h" should be added instead of removing the usb from the build as the dynamic usb device support that this change brings in may fix the issues with the usb branch.

Also the docker pull isn't working, probably needs @MatthewWilkes to create it.

[Molive-0]

I am personally unsure on the ramifications of this change to the sdkconfig. It was required to make the compile work, and the firmware appears to behave, but I can't help but feel that this is needed somewhere.
All changes made as part of this review were to fix existing features during the upgrade rather than remove them, but no matter what I did this did not seem to work.

[Molive-0]

Looking at documentation, this config setting only seems to be useful if the function vPortCleanUpTCB is defined, and as far as I am aware this project does not do that.

[ChrisDick]

there's a warning when building with previous: #warning "CONFIG_FREERTOS_ENABLE_STATIC_TASK_CLEAN_UP is deprecated. Use CONFIG_FREERTOS_TASK_PRE_DELETION_HOOK instead." for mp1.25 the pre deletion hook is set to y in sdkconfig.base. I think it's right to remove the task clean up one.

[Molive-0]

Hm.
I just rebased against main, because I wanted the PR to be up to date but didn't want to add a merge commit. I thought this would be cleaner but it's just its own brand of scuffed.
I'll refrain from doing that again in the future.

[ChrisDick]

Since this is now conflicted( I can't resolve on your branch), can it be updated to Micropython v1.26 and ESP-IDF v5.4.2?

[Molive-0]

Since this is now conflicted( I can't resolve on your branch), can it be updated to Micropython v1.26 and ESP-IDF v5.4.2?

PR is now for Micropython v1.26 and ESP-IDF v5.4.2. I will test on a real badge shortly.

[MatthewWilkes]

Thank you!

[Molive-0]

It appears that the 1.26 update doesn't work immediately, and will take time to debug the issues around it.
To this end I suggest that I return this PR back to 1.25, and then create a separate PR with updates to 1.26.

[ChrisDick]

yeah, good suggestion, thanks for trying it.

[Jonty]

Just skimmed through this and I'm unclear on the current state - does this actually work? Can we merge and upgrade to 1.25?

[Molive-0]

Just skimmed through this and I'm unclear on the current state - does this actually work? Can we merge and upgrade to 1.25?

As far as I'm aware it still works. I'll check it again after work

[ChrisDick]

We're waiting until we've done another release before merging this one. The plan is to release this in v2.0 as it breaks the mpy files used on hexpansions and we have another small break with the power supply capabilities function to go in with it. This worked when I tested it and I'm happy the todos can be marked as complete.

[Molive-0]

todos are now marked complete