diff --git a/docs/usages/run-on-different-esp-target.rst b/docs/usages/run-on-different-esp-target.rst new file mode 100644 index 00000000..b21d6097 --- /dev/null +++ b/docs/usages/run-on-different-esp-target.rst @@ -0,0 +1,285 @@ +############################################ + Tests in ESP-IDF Projects with ESP Targets +############################################ + +.. note:: + + This guide only applies to ``--embedded-services esp,idf``. + +For projects which depend on ESP-IDF, a common use case is to run tests on different ESP targets. This guide will show you how to run tests locally, and on CI, on different ESP targets. + +********************************************* + Getting Started - Select the Target via CLI +********************************************* + +Here's an example on a minimal project which based on ESP-IDF, with the following folder structure: + +.. code:: bash + + project/ + ├── CMakeLists.txt + └── main/ + ├── CMakeLists.txt + └── main.c + +We could add a test case to the project. Usually we put it under the ``tests`` directory: + +.. code:: python + + # test_hello_world.py + def test_hello_world(dut): + dut.expect('Hello World!') + +The current folder structure looks like this: + +.. code:: bash + + project/ + ├── CMakeLists.txt + ├── main/ + │ ├── CMakeLists.txt + │ └── main.c + └── tests/ + └── test_hello_world.py + +To run the test on ESP targets, we shall build the project first: + +.. code:: bash + + idf.py set-target esp32 build + +The current folder structure looks like this: + +.. code:: bash + + project/ + ├── CMakeLists.txt + ├── build/ # built binaries for esp32 + ├── main/ + │ ├── CMakeLists.txt + │ └── main.c + └── tests/ + └── test_hello_world.py + +Then we can run the test on ESP32 by passing the CLI argument: + +.. code:: bash + + pytest --target esp32 --embedded-services esp,idf + +When running the same test on ESP32S2, we need to build the project again. In order to not overwrite the previous build, we can create a new build folder: + +.. code:: bash + + idf.py -B build_esp32s2 set-target esp32s2 build + +The current folder structure looks like this: + +.. code:: bash + + project/ + ├── CMakeLists.txt + ├── build/ # built binaries for esp32 + ├── build_esp32s2/ # built binaries for esp32s2 + ├── main/ + │ ├── CMakeLists.txt + │ └── main.c + └── tests/ + └── test_hello_world.py + +Then we can run the test on ESP32S2 by passing the CLI argument: + +.. code:: bash + + pytest --target esp32s2 --build-dir build_esp32s2 --embedded-services esp,idf + +The ``--build-dir`` argument is used to specify the build directory to flash to the target automatically. + +********************************************** + Simplify the CLI Call - Using ``pytest.ini`` +********************************************** + +You may notice that in the above example, we need to specify the ``-embedded-services esp,idf`` every time we run the test on a different target. To simplify this, we can create a ``pytest.ini`` file under the project's root directory: + +.. code:: ini + + [pytest] + addopts = + --embedded-services esp,idf + +The current folder structure looks like this: + +.. code:: bash + + project/ + ├── CMakeLists.txt + ├── build/ # built binaries for esp32 + ├── build_esp32s2/ # built binaries for esp32s2 + ├── main/ + │ ├── CMakeLists.txt + │ └── main.c + ├── tests/ + │ └── test_hello_world.py + └── pytest.ini + +Then we can run the test on ESP32 by passing the CLI argument: + +.. code:: bash + + pytest --target esp32 + +**************************************************** + One Step Further - Customizing the Build Directory +**************************************************** + +You may notice that in the above example, we need to specify the build directory every time we run the test on a different target. To simplify this, we can customize the a conftest.py file to set the build directory based on the target: + +.. code:: python + + # conftest.py + def build_dir(request, target): + return f"build_{target}" + +Now when we're running the test case on ESP32S2, we can simply run: + +.. code:: bash + + pytest --target esp32s2 + +The build directory will be automatically set to ``build_esp32s2``. + +Run the test case on ESP32 will fail, since we built it under the ``build`` directory and the test case is expecting the binaries under the ``build_esp32`` directory. Don't forget to pass the ``-B`` argument to build the project under a different build directory, as what we did in the previous example for ESP32S2. + +*************** + Running on CI +*************** + +To run the test on CI, you can use the same approach as running locally. Basically, you need two types of jobs: "build jobs" and "test jobs". + +In the build jobs, you build the project for different targets. Usually you can run it on linux machine, and build the project with the ESP-IDF docker image. + +Here we provide a GitHub Actions example: + +.. code:: yaml + + name: Build and Test Application + + on: + pull_request: + paths: + - "src/**/*.c" + push: + branches: + - main + + env: + IDF_PATH: /opt/esp/idf + + defaults: + run: + shell: bash + + jobs: + build: + name: Build Test App + strategy: + fail-fast: false + matrix: + # choose the version of ESP-IDF to use for the build + idf-branch: + - release-v5.0 + - latest + # choose the target to build for + target: + - esp32 + - esp32c3 + runs-on: ubuntu-22.04 + container: + image: espressif/idf:${{ matrix.idf-branch }} + steps: + - uses: actions/checkout@v3 + - name: Build Test Application with ESP-IDF + run: | + . $IDF_PATH/export.sh + idf.py -B build_${{ matrix.target }} set-target ${{ matrix.target }} build + - name: Upload files to artifacts for run-target job + uses: actions/upload-artifact@v4 + with: + name: built_binaries_${{ matrix.target }}_${{ matrix.idf-branch }} + path: | + **/build**/bootloader/bootloader.bin + **/build**/partition_table/partition-table.bin + **/build**/*.bin + **/build**/*.elf + **/build**/flasher_args.json + if-no-files-found: error + +In the test jobs, you run the test cases on different targets. Usually you need to register a self-hosted runner that connects to the ESP targets, and run the test cases on the runner. + +Here we provide a GitHub Actions example: + +.. code:: yaml + + name: Build and Test Application + + on: + pull_request: + paths: + - "src/**/*.c" + push: + branches: + - main + + env: + IDF_PATH: /opt/esp/idf + test_dirs: examples + + defaults: + run: + shell: bash + + jobs: + build: + .. # same as above + target-test: + name: Run Test App on ESP Target + needs: build + strategy: + fail-fast: false + matrix: + # choose the version of ESP-IDF to use for the build + idf-branch: + # - release-v5.0 + # - release-v5.1 + - latest + # choose the target to build for + target: + - esp32 + - esp32c3 + runs-on: ubuntu-22.04 + container: + image: hfudev/qemu:main + steps: + - uses: actions/checkout@v3 + - uses: actions/download-artifact@v2 + with: + name: built_binaries_${{ matrix.target }}_${{ matrix.idf-branch }} + - name: Install Python packages + run: | + . $IDF_PATH/export.sh + pip install \ + pytest-embedded-idf \ + pytest-embedded-qemu + - name: Run Test App on target + run: | + . $IDF_PATH/export.sh + pytest ${{ env.test_dirs }} \ + --target ${{ matrix.target }} \ + --embedded-services idf,qemu \ + --junit-xml test_${{ matrix.target }}_${{ matrix.idf-branch }}.xml \ + --build-dir build_${{ matrix.target }}_${{ matrix.idf-branch }} + - uses: actions/upload-artifact@v2 + if: always() + with: + name: test_${{ matrix.target }}_${{ matrix.idf-branch }}_junit + path: test_${{ matrix.target }}_${{ matrix.idf-branch }}.xml