Flashing multiple esp chips at the same time with progress (ESPTOOL-1104)

[shipljos]

Is your feature request related to a problem?

I am a developer of a python app built on pyqt6 that allows me to communicate with and flash our esp32s3 chip. Currently I can flash one device at a time with progress being displayed using the esptool library. I cannot get multiple devices to flash with the progress because I cannot figure out how to differentiate where each progress update is coming from.

Describe the solution you'd like

I would like there to be some sort of method to update multiple devices and update all their progress bars separately.

Describe alternatives you've considered

No response

Additional context

No response

[github-actions]

Hi @shipljos! Please be aware that (1) the following suggestions are generated by a bot and haven't been fact-checked by Espressif Systems, (2) burning eFuses and enabling security features are irreversible operations and can damage your ESP32 device. We hope that this message will help you until an Espressif Engineer looks at your issue.

---

Thank you for your feature request and for describing your use case in detail.

Currently, the public API of esptool allows you to programmatically flash multiple ESP chips by creating separate connections to each device. However, the API does not provide built-in mechanisms to differentiate or tag progress updates for each device when flashing multiple devices in parallel. The progress reporting is global and not designed for multi-device scenarios out of the box, which makes it difficult to map progress updates to individual devices in a multi-threaded or multi-process environment.

What is possible today:

• You can instantiate separate esptool sessions (e.g., in different threads or processes), each targeting a different serial port.
• Each session will output its own progress, but esptool’s progress reporting is not natively designed to be redirected or tagged per device for integration with custom UI elements like multiple progress bars.
• The upcoming esptool v5 release introduces improved progress bars and more user-friendly output, but there is no mention in the documentation of per-device progress callbacks or hooks for multi-device flashing in a single process or application context. The improvements are focused on the command-line interface and do not address this specific use case for the Python API yet esptool: Updates about the upcoming v5 major release.

Suggested approach:

• As a workaround, you can run each esptool flashing operation in a separate process and capture their outputs independently, then parse and display the progress in your UI. This is not as clean as having a per-device progress callback, but it is currently the most reliable method.
• If you need more granular control, you may consider contributing to esptool or opening a feature request for a per-device progress callback or event system in the Python API, which would allow better integration with multi-device flashing UIs Contributions Guide.

Summary:
At this time, esptool does not natively support per-device progress reporting for simultaneous multi-device flashing via its Python API. The recommended workaround is to manage each flashing process separately and capture their outputs. For a more robust solution, a feature request or contribution to esptool would be needed to add this capability.

If you have further questions or would like to discuss possible implementation strategies, please let us know!

[radimkarnis]

Hello @shipljos,

Since esptool v5, implementing a custom logger is supported, please see the docs. Possibly, you could extend the progress_bar() method to also print which device on which port the progress relates to.

Then, I believe, it should be just a matter of spawning more esptool processes or launching more threads.

I cannot figure out how to differentiate where each progress update is coming from

Please correct me if I am wrong - does this mean you have a working solution of flashing multiple chips at the same time, and your only issue now is the logging part?

[shipljos]

Yes, I can flash multiple at the same time. The issue is that there is no easy way for the gui to know which progress is for which device. What ends up happening is that when I am updating multiple devices, the progress bars on each one gets updated anytime there is a callback to my pyqtSignal that updates my progress bar. They aren't treated as separate.

It would be nice if those functions in the custom logging could be overwritten or I could insert a callback function for them (i.e. connect my progress bar function to the TemplateLogger progress_bar() function). When I have tried doing anything of the sort it exits out of the write_flash function immediately and the flash does not work.

[radimkarnis]

It would be nice if those functions in the custom logging could be overwritten

The linked docs article describes exactly how to do this. There is even an example of a custom logger implementation. You can just copy that and rewrite the progress_bar() method to your liking.

When I have tried doing anything of the sort it exits out of the write_flash function immediately

Did you receive any output or indication as to why the exit occurs?

[shipljos]

The main problem is that the progress bars get all messed up when you try to flash multiple. I am using the custom class that is shown in that documentation and the problem I think is that it is an abstractmethod so there are no class member variables. There is no such thing as multiple instances of the the TemplateClass. Every instance of the logger is not a copy of the same one, it is the same one. There is no output or indication as to why the exit occurs when it jumps out.

[shipljos]

Any updates on this yet?

[radimkarnis]

Sorry, no progress yet. We will look into this and think about allowing multiple logger instances or creating an example of how to achieve this.

[law-ko]

@radimkarnis This is crazy, as now we can't do multiple flashing at once with progress bar showing the progress. I suggest to have at least a progress callback that we can put into write_flash is more ideal.

I found this article it is providing example with progress callbacks, but not sure why is has been removed in 5.1 release on pip.