hwci: retry flash+boot on the AMYboard's intermittent early-boot flake#1029
Conversation
The AMYboard bench occasionally fails with "board MIDI never appeared": the board panic-loops in the ESP-IDF 2nd-stage bootloader (Guru Meditation, StoreProhibited) before USB-MIDI enumerates. This is NOT a firmware bug -- the crashing image's bootloader is byte-identical to builds that boot fine (diffing the two full images over the bootloader region turns up only the embedded __TIME__ build string + the derived checksum/SHA), so it's a transient flash-read/power/reset glitch after the ~61 s USB flash. Investigated via shorepine/amy#748. Wrap flash+boot in a retry loop (--boot-attempts, default 3). On "MIDI never appeared", stop the attempt's boot logger (freeing the dongle so esptool can drive reset), re-flash -- which forces ROM download mode, a clean known-good reset -- and wait again. A genuinely broken build still fails every attempt, so real regressions aren't masked; each failed attempt's crash log is kept in the combined serial log. wait_for_board() now returns None on timeout instead of sys.exit. Verified the loop with mocked flash/wait: all-fail does N re-flashes then exits; a boot on attempt 2 flashes twice then proceeds.
🔌 AMYboard PR previewEditor + flasher: https://amyboard-pr-1029.vercel.app/editor/ This preview bundles this PR's firmware — its flasher only flashes this build (not the release). Rebuilt on every push; removed when the PR closes. The hardware CI has been kicked off and should return within a few minutes, stand by! |
🎛️ HW CI (physical bench)AMYboard (USB-MIDI + AMY Tulip (TULIP4_R11; serial-REPL audio + WiFi screenshot): ✅ PASS — flashed this PR’s firmware; all checks matched the references. ⬇️ Artifacts: recordings · screenshot · serial logs · run logs Self-hosted bench. Audio spectral-compared to |
Problem
The AMYboard HW CI intermittently fails with
[boot] board MIDI never appeared (didn't boot?). The board panic-loops in the ESP-IDF 2nd-stage bootloader (Guru Meditation StoreProhibited,EXCVADDR 0xb0, registers full of0xaaaaaaaa) right afterentry 0x403c8890, before USB-MIDI enumerates — so a single transient glitch fails the whole run.This is a bench flake, not a firmware bug (proven)
Investigated under shorepine/amy#748. The same amy SHA built via the
amyboard-amypr-<N>(dispatch) path booted + passed, while theamyboard-pr-<N>(pull_request) build of that identical SHA boot-looped.esptool image_info+cmpof the two full images over the bootloader region shows the bootloader machine code is byte-identical — the only differences are 4 bytes of the embedded__TIME__build-time string (the two compiled at different wall-clock times) and the derived checksum/SHA-256. Identical code can't be a deterministic code bug, so it's a transient flash-read / power / reset glitch after the ~61 s USB flash.Fix
Wrap flash+boot in a retry loop (
--boot-attempts, default 3):MIDI never appeared, stop the attempt's boot logger (freeing the dongle so esptool can drive the reset lines), re-flash — which forces ROM download mode, a clean known-good reset that clears the glitch — and wait again.[boot N/M]), so a persistent failure is still fully debuggable.wait_for_board()now returnsNoneon timeout instead ofsys.exit.Testing
py_compileclean. Verified the loop with mockedflash/wait_for_board:🤖 Generated with Claude Code