ESP32: Significant size increase with MICROPY_ROM_TEXT_COMPRESSION and external modules

[ricksorensen]

I do no know if this is a micropython code issue or from the external modules.

This is a reprise of #17225 which I prematurely marked as closed. If I compile a GENERIC_C3 or GENERIC_S3 board with ulab, the size increased significantly from v1.25 to the v1.26-preview versions. It appears the MICROPY_ROM_TEXT_COMPRESSION setting contributes to the issue. Compiling today (20250611 11AM CST approx) I see these sizes for micropython.bin, compilint with IDF 5.2.2 for the C3: (same is true for S3)

ROM_TEXT_COMPRESSION	ExtModule	micropython.bin size
ON	none	1815264
ON	examples/usercmodule	1816928
ON	st7789_mpy	2019408
ON	ulab	3891184
OFF	none	1818448
OFF	examples/usercmodule	1820112
OFF	st7789_mpy	1870704
OFF	ulab	2132064

When text compression is enabled, the st7789 and the ulab builds increase in size significantly, while the `examples/usercmodule' build decreases in size. Neither RP2 nor unix modules seem to exhibit this growth,

I cannot figure out why this is happening- the scripts look okay (to me). I expect there is something that should be modified in the ulab/st7789 code to allow this compression to work correctly, but I am not sure if that is true.

Note- the compression cannot be disabled without changing 'esp32_common.cmake which forces theMICROPY_ROM_TEXT_COMPRESSIONtoON. To test I created a new version to force it OFF`.

I do not use cmake often but I expect the conditional should check for a defined variable, not just True/False.

The build process I used was on ubuntu24,04. This is the build script:

make -j 1 V=1 BOARD=ESP32_GENERIC_C3 2>&1 >buildC3.default.on
mpybinsz=$( stat --format=%s "build-ESP32_GENERIC_C3/micropython.bin" )
echo "micropython.bin:,ON,none,$mpybinsz"
mv build-ESP32_GENERIC_C3 build-ESP32_GENERIC_C3.default.on
make -j 1 V=1 BOARD=ESP32_GENERIC_C3 USER_C_MODULES=$MYDIR/micropython/examples/usercmodule/micropython.cmake 2>&1 >buildC3.examp.on
mpybinsz=$( stat --format=%s "build-ESP32_GENERIC_C3/micropython.bin" )
echo "micropython.bin:,ON,examples/usercmodule,$mpybinsz"
mv build-ESP32_GENERIC_C3 build-ESP32_GENERIC_C3.examp.on
make -j 1 V=1 BOARD=ESP32_GENERIC_C3 USER_C_MODULES=$MYDIR/st7789_mpy/st7789/micropython.cmake 2>&1 >buildC3.st7789.on
mpybinsz=$( stat --format=%s "build-ESP32_GENERIC_C3/micropython.bin" )
echo "micropython.bin:,ON,st7789_mpy,$mpybinsz"
mv build-ESP32_GENERIC_C3 build-ESP32_GENERIC_C3.st7789.on
make -j 1 V=1 BOARD=ESP32_GENERIC_C3 USER_C_MODULES=$MYDIR/ulab/code/micropython.cmake 2>&1 >buildC3.ulab.on
mpybinsz=$( stat --format=%s "build-ESP32_GENERIC_C3/micropython.bin" )
echo "micropython.bin:,ON,ulab,$mpybinsz"
mv build-ESP32_GENERIC_C3 build-ESP32_GENERIC_C3.ulab.on

cp /work/domicropy/mybuilder/esp32_common.compoff.cmake esp32_common.cmake
echo "check esp32_common.cmake"
git diff esp32_common.cmake     # show state of compression variable
make -j 1 V=1 BOARD=ESP32_GENERIC_C3 2>&1 >buildC3.default.off
mpybinsz=$( stat --format=%s "build-ESP32_GENERIC_C3/micropython.bin" )
echo "micropython.bin:,OFF,none,$mpybinsz"
mv build-ESP32_GENERIC_C3 build-ESP32_GENERIC_C3.default.off
make -j 1 V=1 BOARD=ESP32_GENERIC_C3 USER_C_MODULES=$MYDIR/micropython/examples/usercmodule/micropython.cmake 2>&1 >buildC3.examp.off
mpybinsz=$( stat --format=%s "build-ESP32_GENERIC_C3/micropython.bin" )
echo "micropython.bin:,OFF,examples/usercmodule,$mpybinsz"
mv build-ESP32_GENERIC_C3 build-ESP32_GENERIC_C3.examp.off
make -j 1 V=1 BOARD=ESP32_GENERIC_C3 USER_C_MODULES=$MYDIR/st7789_mpy/st7789/micropython.cmake 2>&1 >buildC3.st7789.off
mpybinsz=$( stat --format=%s "build-ESP32_GENERIC_C3/micropython.bin" )
echo "micropython.bin:,OFF,st7789_mpy,$mpybinsz"
mv build-ESP32_GENERIC_C3 build-ESP32_GENERIC_C3.st7789.off
make -j 1 V=1 BOARD=ESP32_GENERIC_C3 USER_C_MODULES=$MYDIR/ulab/code/micropython.cmake 2>&1 >buildC3.ulab.off
mpybinsz=$( stat --format=%s "build-ESP32_GENERIC_C3/micropython.bin" )
echo "micropython.bin:,OFF,ulab,$mpybinsz"
mv build-ESP32_GENERIC_C3 build-ESP32_GENERIC_C3.ulab.off

[jonnor]

Hi @ricksorensen - this seems like a real bug. Have you filed an issue about it? I tried searching but could not find.

I believe I am experiencing the same in emlearn-micropython when using that with USER_C_MODULES on esp32 - the build with these modules added take an additional 300 kB - even though the code is just around 20 kB when built as .mpy files. Using -Os instead of -O2 seems to reduce the usage - however that will likely have negative effects on execution time, so not a good fix...

As you mention in the previous discussion, this seems to be related to the MICROPY_ROM_TEXT_COMPRESSION function. I think it is being inlined many times, which causes immense bloating.

[jonnor]

This has been reported in ulab - v923z/micropython-ulab#748 - which also links to #18880

[jonnor]

There seems to be a fix available in #18884 - can you test that @ricksorensen ?

[ricksorensen]

I am currently compiling C3 and S3 xiao boards with full ULAB without modifying the MICROPY_ROM_TEXT_COMPRESSION setting, sot the #18884 fix works for my appliication.

For my work arounds on the XIAO C3 I had tried:

• decreasing the the flash filesystem size (increasing flash size for firmware). This worked
• eliminating files from manifest.py. This worked at some loss of capabilitye

Now I use the original manfest.py with my added full ulab and st7789 modules with asyncio in manifest successfully.