esp32-s3: crash on os.sync() with freezefs mounted

[ned-pcs]

I've just added freezefs to my project, and am having problems with os.sync() crashing when it's called right after my freezefs module is loaded. If I os.umount() my freezefs module first, I can call os.sync() with no crash. I'm using v1.22.2 and have frozen my freezefs module. Unfortunately, I am not yet set up for debugging with this device.

I've traced the stack dump here:

py/obj.c:60
        return types[(uintptr_t)o_in & 0xf];

py/runtime.c:709
    const mp_obj_type_t *type = mp_obj_get_type(fun_in);

py/runtime.c:729
    return mp_call_function_n_kw(args[0], n_args + adjust, n_kw, args + 2 - adjust);

extmod/vfs_blockdev.c:124
                    mp_call_method_n_kw(0, 0, self->u.old.sync);

extmod/vfs_fat_diskio.c:121
        ret = mp_vfs_blockdev_ioctl(&vfs->blockdev, bp_op, 0);

extmod/modos.c:80 (discriminator 3)
        disk_ioctl(MP_OBJ_TO_PTR(vfs->obj), CTRL_SYNC, NULL);

[bixb922]

Thank you for the detailed error report.

I have been able to reproduce the error on my ESP32-S3 too. However, I haven't been able to fix it so far.

The VfsFrozen class is modeled after the RemoteFS class in mpremote (see https://github.com/micropython/micropython/blob/master/tools/mpremote/mpremote/transport_serial.py). The RemoteFS class is sent to the microcontroller to implement the mpremote mount command.

The stack trace shows that MicroPython believes that this is an "old style" Virtual File System driver, and tries to call the "sync" method. New style VFS drivers implement the "ioctl" method at the VFS level.

I have been comparing VfsFrozen with RemoteFS again and tried to make the interface identical, but to no avail. Also, I tried by implementing VfsFrozen.ioctl() or the old style VfsFrozen.sync() and VfsFrozen.count() methods, but the problem does not disappear. The symptoms vary a bit sometimes, either the microcontroller crashes, or some exception like `TypeError: 'int' object isn't callable´ occuring inside the MicroPython runtime.

However, MicroPython's RemoteFS seems to be stable, and `os.sync() works with RemoteFS. I found no problem there.

I will continue to look at the problem, but I ran a bit out of ideas at this point :-(

Any idea will be highly welcome....

[bixb922]

os.sync() is treating the VfsFrozen object as if it had a underlying block device (just like a flash or sdcard), but there is no block device here, so a pointer from random location gets called. If that location holds a int or str, TypeError is raised, else the microcontroller crashes, or sometimes os.sync() seems to work. So the symptoms vary with the exact position of the VfsFrozen code in memory.

I don't see a change I can make in freezefs to correct that. freezefs is pure Python code, no C code. I think this would need a change in the MicroPython binaries, which would need a special image to use freezefs, and that's a bit inconvenient. The MicroPython team is planning to release PR #8381 some time in the near future, which will probably render freezefs obsolete. I don't think there will be much interest in solving this issue, since #8381 will be the official way to store non-Python files in the frozen image. (BTW take a look at that PR, it's interesting! I don't have any clue when that will be released. Perhaps version 1.24?)

Looking at the trace you sent, surprisingly the ESP32 flash partitions seem to do nothing with a os.sync(). The sync does not even get to LittleFS2 but is passed to the underlying block device object. Searching for MP_BLOCKDEV_IOCTL_SYNC in esp32_partition.c and even machine_sdcard.c shows that these function just return "ok" and don't really do something. I have posted a question on MicroPython Discord asking what os.sync() really does, to confirm this.

[bikeNomad]

Thanks for looking at this. For now I can eliminate calls to os.sync(), and keep using your freezefs solution (thanks for this!).

[bixb922]

Glad to be of help!

Just to complete the information about os.sync, see issue #11449. There the developers clarify that os.sync() is implemented only for the STM32 and Renesas RA ports and is a no-op for filesystems other than FAT.

[gampam2000]

Hi,

I came across your very interesting project while reading PR #8381. Your following statement is not clear to me:

The MicroPython team is planning to release PR #8381 some time in the near future, which will probably render freezefs obsolete. I don't think there will be much interest in solving this issue, since #8381 will be the official way to store non-Python files in the frozen image.

I'm not so sure about that, but maybe I did not understand PR #8381. I use an ESP32-S3 with OTA updates.
Your Freeze FS makes it possible so that I can compile the non python files into the firmware image and thus all the OTA Updates only need to flash the firmware image, where the ESP32 has a good support for OTA updates in Micropython with OTA0 and OTA1 partitions and rollback support etc..

When using the romfs of PR #8381, it is in its own partition on the flash and not included in the actual firmware. At least this is what I understood how it works. So How to update then the romfs partition? Would i need an extra update process for this? If that's the case I would stick with your solution, because it is for this use case much simpler.

What are your thought about this?

Best regards