Serial bus No. 2: Update over Serial (#182)

* added base version of serial bus

* freertos 1000 hz

* added ram debugging

* add  receive answers

* updated logic of serial bus coordinator and peers and documentation

* add ota

* update ota

* update to work with bus connected to expander

* update lower chunk size default

* update docs

* update remove unnecessary information

* removed debug

* update minimal fixes

* remove debug

* code review

* refactor

* refactor & error handling

* Agent Review fix

* default handling & relay simplification

* refactoring

* refactoring serial_bus_ota

* update and refactor code

* documentation

* AI Review

* simplify verification and just use esp_ota_end() to verify

* add line breaks in Markdown files

* cleanup whitespace, reduce diff

* simplify CSV parsing

* adressing review

* remove obsolete OTA server

* remove outdated "OTA" references

* tiny formatting improvement

* remove `parse_ota_partition` and use default range

* simplify OTB protocol and otb_update.py

* update otb & readd sliding window (4x speed)

* changes based on review

* match const pattern of other files

* fix docs

* minor Markdown improvements

* improve some comments

* refactor OTB responses from polled buffer to send callback

* auto-format otb.cpp

* remove bus_reset_session from header file

* avoid re-implementing base64 decoder

* resove the `abort_flash` parameter

* remove redundant check

* respond with an error instead of silently logging an unknown message type

* handle session mismatch consistently

* call millis() inside the function

* make more variables const

* use human-readable error messages

* added handling for error and ack msgs

---------

Co-authored-by: Falko Schindler <falko@zauberzeug.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

Author: 3 people

.cursor/rules/python-tools.mdc

@@ -16,14 +16,14 @@ Python scripts in this project are **build/flash/monitor tools**, not the main a
 
 ## Key Scripts
 
-| Script | Purpose |
-|--------|---------|
-| `build.py` | Build for ESP32/ESP32-S3 (wraps `idf.py build`) |
-| `flash.py` | Flash firmware to device |
-| `monitor.py` | Serial monitor |
-| `espresso.py` | OTA deployment tool |
-| `configure.py` | Project configuration |
-| `core_dumper.py` | Analyze core dumps |
+| Script           | Purpose                                         |
+| ---------------- | ----------------------------------------------- |
+| `build.py`       | Build for ESP32/ESP32-S3 (wraps `idf.py build`) |
+| `flash.py`       | Flash firmware to device                        |
+| `monitor.py`     | Serial monitor                                  |
+| `espresso.py`    | Deployment tool                                 |
+| `configure.py`   | Project configuration                           |
+| `core_dumper.py` | Analyze core dumps                              |
 
 ## Pattern
 

AGENTS.md

@@ -93,10 +93,10 @@ lizard/
 │   ├── parser.h            # Generated parser (from language.owl via gen_parser.sh)
 │   ├── compilation/        # DSL compilation (expressions, variables, routines, rules)
 │   ├── modules/            # Hardware modules (motors, sensors, I/O, CAN, etc.)
-│   └── utils/              # Utilities (UART, OTA, timing, string helpers)
+│   └── utils/              # Utilities (UART, OTB updates, timing, string helpers)
 ├── components/             # ESP-IDF components and submodules
 ├── docs/                   # MkDocs documentation
-├── examples/               # Usage examples (ROS integration, OTA, trajectories)
+├── examples/               # Usage examples (ROS integration, trajectories)
 ├── build.py               # Build script (wraps idf.py)
 ├── flash.py               # Flash script
 ├── monitor.py             # Serial monitor

docs/module_reference.md

@@ -32,22 +32,16 @@ It is automatically created right after the boot sequence.
 | `core.print(...)`                | Print arbitrary arguments to the command line      | arbitrary    |
 | `core.output(format)`            | Define the output format                           | `str`        |
 | `core.startup_checksum()`        | Show 16-bit checksum of the startup script         |              |
-| `core.ota(ssid, password, url)`  | Starts OTA update on a URL with given WiFi         | 3x `str`     |
 | `core.get_pin_status(pin)`       | Print the status of the chosen pin                 | `int`        |
 | `core.set_pin_level(pin, value)` | Turns the pin into an output and sets its level    | `int`, `int` |
 | `core.get_pin_strapping(pin)`    | Print value of the pin from the strapping register | `int`        |
+| `core.pause_broadcasts()`        | Pause property broadcasts (all modules)            |              |
+| `core.resume_broadcasts()`       | Resume property broadcasts                         |              |
 
 The output `format` is a string with multiple space-separated elements of the pattern `<module>.<property>[:<precision>]` or `<variable>[:<precision>]`.
 The `precision` is an optional integer specifying the number of decimal places for a floating point number.
 For example, the format `"core.millis input.level motor.position:3"` might yield an output like `"92456 1 12.789"`.
 
-The OTA update will try to connect to the specified WiFi network with the provided SSID and password.
-After initializing the WiFi connection, it will attempt an OTA update from the given URL.
-Upon successful updating, the ESP will restart and attempt to verify the OTA update.
-It will reconnect to the WiFi and try to access URL + `/verify` to receive a message with the current version of Lizard.
-The test is considered successful if an HTTP request is received, even if the version does not match or is empty.
-If the newly updated Lizard cannot connect to URL + `/verify`, the OTA update will be rolled back.
-
 `core.get_pin_status(pin)` reads the pin's voltage, not the output state directly.
 
 ## Bluetooth
@@ -90,6 +84,11 @@ The serial bus module lets multiple ESP32s share a UART link with a coordinator
 | `bus.send(receiver, payload)`       | Send a single line of text to a peer `receiver` (0-255)    | `int`, `str` |
 | `bus.make_coordinator(peer_ids...)` | Set the list of peer IDs, making this node the coordinator | `int`s       |
 
+**Firmware Updates:**
+Peers on the serial bus can be updated remotely via the coordinator.
+Use the `otb_update.py` tool to push new firmware to any peer node.
+See [OTB Update](tools.md#otb-update) for details.
+
 ## Input
 
 The input module is associated with a digital input pin that is be connected to a pushbutton, sensor or other input signal.

docs/tools.md

@@ -36,6 +36,82 @@ You can also use an SSH monitor to access a microcontroller via SSH:
 
 Note that the serial monitor cannot communicate while the serial interface is busy communicating with another process.
 
+### OTB Update
+
+`otb_update.py` pushes firmware to a peer over a `SerialBus` coordinator using the OTB (Over The Bus) protocol.
+
+```bash
+./otb_update.py build/lizard.bin --port /dev/ttyUSB0 --target <peer_id> [--bus <name>] [--expander <name>]
+```
+
+| Argument     | Description                                           |
+| ------------ | ----------------------------------------------------- |
+| `firmware`   | Path to the firmware binary (e.g. `build/lizard.bin`) |
+| `--port`     | Serial port (default: `/dev/ttyUSB0`)                 |
+| `--baud`     | Baudrate (default: `115200`)                          |
+| `--target`   | Bus ID of the target node (required)                  |
+| `--bus`      | Name of the SerialBus module (default: `bus`)         |
+| `--expander` | Expander name when coordinator is behind an expander  |
+
+**Expander chains:**
+
+When the SerialBus coordinator sits behind an expander (e.g. `p0`), pass `--expander p0`.
+The script will pause broadcasts on that expander via `core.pause_broadcasts()` before the transfer
+and resume them afterwards to keep the UART link clear.
+
+Example:
+
+```bash
+./otb_update.py build/lizard.bin --port /dev/ttyUSB0 --target 1 --expander p0
+```
+
+This flashes node 1 through expander `p0`.
+The target node will reboot with the new firmware after a successful transfer.
+
+**OTB Protocol:**
+
+The OTB (Over The Bus) protocol uses these message types:
+
+| Host → Target              | Description                                                |
+| -------------------------- | ---------------------------------------------------------- |
+| `__OTB_BEGIN__`            | Begin firmware update session                              |
+| `__OTB_CHUNK_<seq>__:data` | Send base64-encoded firmware chunk (incl. sequence number) |
+| `__OTB_COMMIT__`           | Commit update and set boot partition                       |
+| `__OTB_ABORT__`            | Cancel the update session                                  |
+
+| Host ← Target             | Description                               |
+| ------------------------- | ----------------------------------------- |
+| `__OTB_ACK_BEGIN__`       | Acknowledge begin                         |
+| `__OTB_ACK_CHUNK_<seq>__` | Acknowledge chunk (incl. sequence number) |
+| `__OTB_ACK_COMMIT__`      | Acknowledge commit                        |
+| `__OTB_ERROR__:reason`    | Error response with reason code           |
+
+Flow:
+
+```
+Host                            Target
+  |                                |
+  |--- __OTB_BEGIN__ ------------->|
+  |<-- __OTB_ACK_BEGIN__ ----------|
+  |                                |
+  |--- __OTB_CHUNK_<0>__:... ----->|
+  |<-- __OTB_ACK_CHUNK_<0>__ ------|
+  |                                |
+  |--- __OTB_CHUNK_<1>__:... ----->|
+  |<-- __OTB_ACK_CHUNK_<1>__ ------|
+  |                                |
+  |       ... more chunks ...      |
+  |                                |
+  |--- __OTB_CHUNK_<N-1>__:... --->|
+  |<-- __OTB_ACK_CHUNK_<N-1>__ ----|
+  |                                |
+  |--- __OTB_COMMIT__ ------------>|
+  |<-- __OTB_ACK_COMMIT__ ---------|
+  |                                |
+```
+
+On error at any point, the target responds with `__OTB_ERROR__:<message>` with a human-readable error message.
+
 ### Configure
 
 Use the configure script to send a new startup script to the microcontroller.

espresso.py

@@ -4,8 +4,8 @@
 import sys
 import time
 from contextlib import contextmanager
-from typing import Generator
 from pathlib import Path
+from typing import Generator
 import argparse
 
 try:
@@ -101,6 +101,7 @@ def release(self) -> None:
 parser.add_argument('--swap', action='store_true', help='Swap En and G0 pins for piggyboard version lower than v0.5')
 parser.add_argument('--firmware', default='build/lizard.bin', help='Path to firmware binary')
 parser.add_argument('--chip', choices=['esp32', 'esp32s3'], default='esp32', help='ESP chip type')
+parser.add_argument('--reset-partition', action='store_true', help='Reset to default OTA partition after flashing')
 parser.add_argument('-d', '--dry-run', action='store_true', help='Dry run')
 parser.add_argument('--device', nargs='?', default=DEFAULT_DEVICE, help='Serial device path (auto-detected on Jetson)')
 
@@ -201,6 +202,22 @@ def erase() -> None:
                 raise RuntimeError('Failed to erase flash.')
 
 
+def reset_partition() -> None:
+    """Reset the OTA partition to the default state."""
+    print_bold('Resetting partition to "ota_0"...')
+    success = run(
+        'esptool.py',
+        '--chip', args.chip,
+        '--port', DEVICE,
+        '--baud', '115200',
+        'erase_region',
+        '0xf000',  # otadata partition offset (default ESP-IDF partition table)
+        '0x2000',  # otadata partition size (default ESP-IDF partition table)
+    )
+    if not success:
+        raise RuntimeError('Failed to reset OTA partition.')
+
+
 def flash() -> None:
     """Flash the microcontroller."""
     print_bold('Flashing...')
@@ -224,6 +241,8 @@ def flash() -> None:
             )
             if not success:
                 raise RuntimeError('Flashing failed. Use "sudo" and check your parameters.')
+            if args.reset_partition:
+                reset_partition()
 
 
 def run(*run_args: str) -> bool:

examples/ota_example/main.py

@@ -25,6 +25,7 @@ idf_component_register(
         esp_timer
         esp_wifi
         lwip
+        mbedtls
         nvs_flash
         spi_flash
 )

main/CMakeLists.txt

@@ -18,7 +18,6 @@
 #include "rom/gpio.h"
 #include "rom/uart.h"
 #include "storage.h"
-#include "utils/ota.h"
 #include "utils/tictoc.h"
 #include "utils/timing.h"
 #include "utils/uart.h"
@@ -420,12 +419,6 @@ void app_main() {
         echo("error while loading startup script: %s", e.what());
     }
 
-    try {
-        xTaskCreate(&ota::verify_task, "ota_verify_task", 8192, NULL, 5, NULL);
-    } catch (const std::runtime_error &e) {
-        echo("error while verifying OTA: %s", e.what());
-    }
-
     printf("\nReady.\n");
 
     while (true) {

main/main.cpp

@@ -1,7 +1,6 @@
 #include "core.h"
 #include "../global.h"
 #include "../storage.h"
-#include "../utils/ota.h"
 #include "../utils/string_utils.h"
 #include "../utils/timing.h"
 #include "../utils/uart.h"
@@ -80,14 +79,6 @@ void Core::call(const std::string method_name, const std::vector<ConstExpression
             checksum += c;
         }
         echo("checksum: %04x", checksum);
-    } else if (method_name == "ota") {
-        Module::expect(arguments, 3, string, string, string);
-        auto *params = new ota::ota_params_t{
-            arguments[0]->evaluate_string(),
-            arguments[1]->evaluate_string(),
-            arguments[2]->evaluate_string(),
-        };
-        xTaskCreate(ota::ota_task, "ota_task", 8192, params, 5, nullptr);
     } else if (method_name == "get_pin_status") {
         Module::expect(arguments, 1, integer);
         const int gpio_num = arguments[0]->evaluate_integer();
@@ -163,6 +154,14 @@ void Core::call(const std::string method_name, const std::vector<ConstExpression
             echo("Not a strapping pin");
             break;
         }
+    } else if (method_name == "pause_broadcasts") {
+        Module::expect(arguments, 0);
+        Module::broadcast_paused = true;
+        echo("broadcasts paused");
+    } else if (method_name == "resume_broadcasts") {
+        Module::expect(arguments, 0);
+        Module::broadcast_paused = false;
+        echo("broadcasts resumed");
     } else {
         Module::call(method_name, arguments);
     }

main/modules/core.cpp

@@ -229,7 +229,7 @@ void Expander::deinstall() {
 }
 
 void Expander::send_proxy(const std::string module_name, const std::string module_type, const std::vector<ConstExpression_ptr> arguments) {
-    static char buffer[256];
+    static char buffer[512];
     int pos = csprintf(buffer, sizeof(buffer), "%s = %s(", module_name.c_str(), module_type.c_str());
     pos += write_arguments_to_buffer(arguments, &buffer[pos], sizeof(buffer) - pos);
     pos += csprintf(&buffer[pos], sizeof(buffer) - pos, "); ");
@@ -238,14 +238,14 @@ void Expander::send_proxy(const std::string module_name, const std::string modul
 }
 
 void Expander::send_property(const std::string proxy_name, const std::string property_name, const ConstExpression_ptr expression) {
-    static char buffer[256];
+    static char buffer[512];
     int pos = csprintf(buffer, sizeof(buffer), "%s.%s = ", proxy_name.c_str(), property_name.c_str());
     pos += expression->print_to_buffer(&buffer[pos], sizeof(buffer) - pos);
     this->serial->write_checked_line(buffer, pos);
 }
 
 void Expander::send_call(const std::string proxy_name, const std::string method_name, const std::vector<ConstExpression_ptr> arguments) {
-    static char buffer[256];
+    static char buffer[512];
     int pos = csprintf(buffer, sizeof(buffer), "%s.%s(", proxy_name.c_str(), method_name.c_str());
     pos += write_arguments_to_buffer(arguments, &buffer[pos], sizeof(buffer) - pos);
     pos += csprintf(&buffer[pos], sizeof(buffer) - pos, ")");