Adding native modules to MicroPython (ESP32)

The following instructions were tested under Ubuntu 20.04 (64 bit), using these sources of information:

Note: The details of the build process may change relatively quickly. The following recipies are also subject to change.

Building MicroPython v1.14 with ulab included
Building MicroPython v1.15 and higher with ulab included
Adding own C code modules to the MicroPython firmware
Generating a .mpy MicroPython library from C code ...

Building MicroPython v1.14 with `ulab` included

Note: With version 1.14, MicroPython switched to cmake on the ESP32 port, thus breaking compatibility with "standard" user modules. ulab can, however, still be compiled with release 1.14. The following describes how to do this.

Setting up the build chain

First update and upgrade your system

sudo apt-get update
sudo apt-get upgrade

Currently, the easiest way to install the build chain for MicroPython and to include ulab is the script esp32.sh available from the ulab repository. Copy the script from the repository in to a file called esp32.sh and execute it:
```
cd ~
bash ./esp32.sh
```
This should successfully build the MicroPython firmware with ulab included.

Configure the build chain (`umqtt`, easy re-building etc.)

The following steps modify the installation from 2., including umqtt and making it easier to re-build the firmware without always re-running the whole script. We assume here that /micropython/ is located in the home folder (~).

Include the umqtt related modules:

cd ~
git clone https://github.com/micropython/micropython-lib
mkdir ~/micropython/ports/esp32/modules/umqtt
cp ~/micropython-lib/umqtt.robust/umqtt/robust.py ~/micropython/ports/esp32/modules/umqtt/robust.py
cp ~/micropython-lib/umqtt.simple/umqtt/simple.py ~/micropython/ports/esp32/modules/umqtt/simple.py

Add the following path to the end of the ~/.profile file (e.g. using nano ~/.profile)
```
export PATH="$HOME/micropython/esp32/esp-idf/xtensa-esp32-elf/bin:$PATH"
```
Important: Under Windows 10, restart the Ubuntu/Linux subssystem to activate the change. Under Ubuntu/Linux, restarting the shell is not sufficient, here log-out and -in again.

OPTIONAL: To use double precision, change in file mpconfigport.h in folder micropython/ports/esp32 the following line:

#define MICROPY_FLOAT_IMPL    (MICROPY_FLOAT_IMPL_FLOAT)

to:

#define MICROPY_FLOAT_IMPL    (MICROPY_FLOAT_IMPL_DOUBLE)

Generate the following file as makefile in micropython/ports/esp32 (this configuration is for the HUZZAH32, assuming the device is connected under Windows to COM4):
```
ESPIDF = $(HOME)/micropython/esp32/esp-idf
BOARD = GENERIC
PORT = /dev/ttyS4
BAUD = 460800
FLASH_MODE = dio
FLASH_SIZE = 4MB
#CROSS_COMPILE = xtensa-esp32-elf-
USER_C_MODULES = $(HOME)/ulab

include Makefile
```
In the Ubuntu/Linux subsystem, the transmission fails if the baud rate is too high. If 460800 does not work, try a lower baud rate, such as 230400.
Allow user to access the HUZZAH32 via USB (replace <username>):
```
sudo adduser <username> dialout
```
Build and deploy MicroPython
```
cd ~/micropython/ports/esp32
make -j 8 submodules
make -j 8  
```
Note: -j 8 instructs to compiler to use up to 8 cores, which depending on the system, can accelerate the build process.
The resulting firmware (firmware.bin) then resides in micropython/ports/esp32/build-GENERIC/. To deploy it, run the following commands from the esp32 folder:
```
make erase
make deploy
```
To access it from the Windows host system, run:
```
explorer.exe .
```

Building MicroPython v1.15 and higher with `ulab` included

The following procedure builds the MicroPython firmware for ESP32 microcontrollers based on the latest version of MicroPython (currently 1.15) and includes ulab.

It has been tested with Ubuntu 20.04 installed in the Linux subsystem of Windows 10 (64 bit).

Note: The only limitation I noticed is that deploying the firmware seems not to work with the standard baud rate 460800, however, reducing this to 230400 works reliably (see below).

The first part largely follows the instructions for the ESP32 in the MicroPython repository.

export BUILD_DIR=$(pwd)

git clone https://github.com/v923z/micropython-ulab.git ulab
git clone https://github.com/micropython/micropython.git

cd $BUILD_DIR/micropython/

git clone -b v4.2.1 --recursive https://github.com/espressif/esp-idf.git

cd esp-idf
./install.sh    
. ./export.sh

cd $BUILD_DIR/micropython/
make -C mpy-cross
cd $BUILD_DIR/micropython/ports/esp32
make submodules

Run make to check if the installation was successful.

In the following steps, we generate a configuration file and a script to facilitate re-building the firmware and to include ulab:

Still in $BUILD_DIR/micropython/ports/esp32 create a makefile e.g. using nano makefile:
```
BOARD = GENERIC
PORT = /dev/ttyS4
BAUD = 460800
USER_C_MODULES = $(BUILD_DIR)/ulab/code/micropython.cmake

include Makefile
```
BOARD selects the board type (e.g. GENERIC or UM_TINYPICO etc.); PORT needs to be changed according to COM port your board connects to (here, e.g. COM4 under Windows). Now, MicroPython can be re-built with ulab and deployed:
```
make clean
make
make deploy
```
If deploy fails, try changing the baud rate in makefile to 230400. This seems to be an issue when deploying from the Windows 10 Linux subsystem.

The export of the esp-idf paths is here not permanent. To facilitate re-building the firmware in a new session, the following script helps to set the stage. In $BUILD_DIR create a file e.g. using nano prep.sh containing:
```
export BUILD_DIR=$(pwd)
cd $BUILD_DIR/micropython/esp-idf
source export.sh
cd $BUILD_DIR/micropython/ports/esp32
```
When starting a new session, source prep.sh from your build folder (. prep.sh), which will export the esp-idf paths and bring you directly to $BUILD_DIR/micropython/ports/esp32.

TINYPICO

When selecting BOARD=UM_TINYPICO, the firmware is built but deploy fails, because with ulab, the firmware is too large for the standard partition. To address this issue:

Add a new partition table file named partitions_ulab.cvs to $BUILD_DIR/micropython/ports/esp32/. It increases the size of the partition factory by 128 kB and decreases the size vfs by the same amount.

# Notes: the offset of the partition table itself is set in
# $ESPIDF/components/partition_table/Kconfig.projbuild and the
# offset of the factory/ota_0 partition is set in makeimg.py
# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     0x9000,  0x6000,
phy_init, data, phy,     0xf000,  0x1000,
factory,  app,  factory, 0x10000, 0x200000,
vfs,      data, fat,     0x220000, 0x180000,

Change sdkconfig.board in $BUILD_DIR/micropython/ports/esp32/boards/UM_TINYPICO by adding following two lines:
```
CONFIG_PARTITION_TABLE_CUSTOM=y
CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="partitions_ulab.csv"
```
Finally, make sure that makefile in $BUILD_DIR/micropython/ports/esp32/ contains the correct board type (BOARD=UM_TINYPICO).

Adding own C code modules to the MicroPython firmware

Note: These instructions need to be updated.

Follow the instructions in the official documentation. The prerequisite is, of course, that you were able to successfully build the firmware without any additional modules first (see above). A couple of hints:

If the build process does not throw any errors but the new module cannot be imported, delete the module's folder from micropython/ports/esp32/build-GENERIC/ and start a clean built (make clean, and then the make line below).
An example for a module with a more complex function is blob.

When everything is in place, build the firmware by running from the micropython/ports/esp32 folder:

make -j 16 USER_C_MODULES=../../../modules CFLAGS_EXTRA=-DMODULE_BLOB_ENABLED=1 all
make erase
make deploy

Generating a .mpy MicroPython library from C code

Note: These instructions need to be updated.

While this is a very elegant method for adding fast (e.g. C code), custom modules to MicroPython without having to compile an own firmware version, there are currently a number of limitations, in particular with respect to memory access. These need to be considered before deciding whether to add a module to the firmware (see above) or to generate a native .mpy file. For the latter, MicroPython contains a number of examples with increasing complexity that may serve as templates.

In the following, the example from the documentation is used.

Make a new folder factorial in micropython/examples/natmod and add the following files:

(a) factorial.c

// Include the header file to get access to the MicroPython API
#include "py/dynruntime.h"

// Helper function to compute factorial
STATIC mp_int_t factorial_helper(mp_int_t x) {
  if (x == 0) {
    return 1;
  }
  return x * factorial_helper(x - 1);
}

// This is the function which will be called from Python, as factorial(x)
STATIC mp_obj_t factorial(mp_obj_t x_obj) {
  // Extract the integer from the MicroPython input object
  mp_int_t x = mp_obj_get_int(x_obj);
  // Calculate the factorial
  mp_int_t result = factorial_helper(x);
  // Convert the result to a MicroPython integer object and return it
  return mp_obj_new_int(result);
}
// Define a Python reference to the function above
STATIC MP_DEFINE_CONST_FUN_OBJ_1(factorial_obj, factorial);

// This is the entry point and is called when the module is imported
mp_obj_t mpy_init(mp_obj_fun_bc_t *self, size_t n_args, size_t n_kw, mp_obj_t *args) {
  // This must be first, it sets up the globals dict and other things
  MP_DYNRUNTIME_INIT_ENTRY

  // Make the function available in the module's namespace
  mp_store_global(MP_QSTR_factorial, MP_OBJ_FROM_PTR(&factorial_obj));

  // This must be last, it restores the globals dict
  MP_DYNRUNTIME_INIT_EXIT
}

(b) Makefile

MPY_DIR = ../../..

# Name of module
MOD = factorial

# Source files (.c or .py)
SRC = factorial.c

# Architecture to build for (x86, x64, armv7m, xtensa, xtensawin)
ARCH = xtensawin

# Include to get the rules for compiling and linking the module
include $(MPY_DIR)/py/dynruntime.mk

Build the module:

cd ~/micropython/examples/natmod/factorial
make

Copy factorial.mpy onto the HUZZAH32 as usual. The test it using the REPL. Run on the HUZZAH32:
```
import factorial
factorial.factorial(6)
```

Home

Board
Installation
Zusammenbau
3.1. Übersicht
3.2. Bauteile
3.3. HexBug-Spinne zerlegen
3.4. Adapter und Sensorarm
3.5. Akku und Platine
"robotling"-Code hochladen und Demo starten
"Verhalten"
Sensoren etc.
Erweiterungen und Modifikationen
Telemetrie über MQTT
Adding native modules to MicroPython (Englisch)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Adding native modules to MicroPython (ESP32)

Building MicroPython v1.14 with `ulab` included

Setting up the build chain

Configure the build chain (`umqtt`, easy re-building etc.)

Building MicroPython v1.15 and higher with `ulab` included

TINYPICO

Adding own C code modules to the MicroPython firmware

Generating a .mpy MicroPython library from C code

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally

Adding native modules to MicroPython (ESP32)

Building MicroPython v1.14 with ulab included

Setting up the build chain

Configure the build chain (umqtt, easy re-building etc.)

Building MicroPython v1.15 and higher with ulab included

TINYPICO

Adding own C code modules to the MicroPython firmware

Generating a .mpy MicroPython library from C code

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally

Building MicroPython v1.14 with `ulab` included

Configure the build chain (`umqtt`, easy re-building etc.)

Building MicroPython v1.15 and higher with `ulab` included