Skip to content

hwci: retry flash+boot on the AMYboard's intermittent early-boot flake#1029

Merged
bwhitman merged 1 commit into
mainfrom
hwci-boot-retry
Jun 22, 2026
Merged

hwci: retry flash+boot on the AMYboard's intermittent early-boot flake#1029
bwhitman merged 1 commit into
mainfrom
hwci-boot-retry

Conversation

@bwhitman

Copy link
Copy Markdown
Collaborator

Problem

The AMYboard HW CI intermittently fails with [boot] board MIDI never appeared (didn't boot?). The board panic-loops in the ESP-IDF 2nd-stage bootloader (Guru Meditation StoreProhibited, EXCVADDR 0xb0, registers full of 0xaaaaaaaa) right after entry 0x403c8890, before USB-MIDI enumerates — so a single transient glitch fails the whole run.

This is a bench flake, not a firmware bug (proven)

Investigated under shorepine/amy#748. The same amy SHA built via the amyboard-amypr-<N> (dispatch) path booted + passed, while the amyboard-pr-<N> (pull_request) build of that identical SHA boot-looped. esptool image_info + cmp of the two full images over the bootloader region shows the bootloader machine code is byte-identical — the only differences are 4 bytes of the embedded __TIME__ build-time string (the two compiled at different wall-clock times) and the derived checksum/SHA-256. Identical code can't be a deterministic code bug, so it's a transient flash-read / power / reset glitch after the ~61 s USB flash.

Fix

Wrap flash+boot in a retry loop (--boot-attempts, default 3):

  • On MIDI never appeared, stop the attempt's boot logger (freeing the dongle so esptool can drive the reset lines), re-flash — which forces ROM download mode, a clean known-good reset that clears the glitch — and wait again.
  • A genuinely broken build still fails every attempt, so real regressions aren't masked.
  • Each failed attempt's crash log is retained in the combined serial log (labeled [boot N/M]), so a persistent failure is still fully debuggable.
  • Happy path is unchanged: boots first try → one flash, no extra time. CI passes no new flags (default 3).

wait_for_board() now returns None on timeout instead of sys.exit.

Testing

py_compile clean. Verified the loop with mocked flash/wait_for_board:

  • all attempts fail → exactly 3 re-flashes then a clear exit;
  • boot succeeds on attempt 2 → 2 flashes, then proceeds into the test.

🤖 Generated with Claude Code

The AMYboard bench occasionally fails with "board MIDI never appeared": the
board panic-loops in the ESP-IDF 2nd-stage bootloader (Guru Meditation,
StoreProhibited) before USB-MIDI enumerates. This is NOT a firmware bug --
the crashing image's bootloader is byte-identical to builds that boot fine
(diffing the two full images over the bootloader region turns up only the
embedded __TIME__ build string + the derived checksum/SHA), so it's a
transient flash-read/power/reset glitch after the ~61 s USB flash.
Investigated via shorepine/amy#748.

Wrap flash+boot in a retry loop (--boot-attempts, default 3). On "MIDI never
appeared", stop the attempt's boot logger (freeing the dongle so esptool can
drive reset), re-flash -- which forces ROM download mode, a clean known-good
reset -- and wait again. A genuinely broken build still fails every attempt,
so real regressions aren't masked; each failed attempt's crash log is kept in
the combined serial log.

wait_for_board() now returns None on timeout instead of sys.exit. Verified the
loop with mocked flash/wait: all-fail does N re-flashes then exits; a boot on
attempt 2 flashes twice then proceeds.
@github-actions

Copy link
Copy Markdown

🔌 AMYboard PR preview

Editor + flasher: https://amyboard-pr-1029.vercel.app/editor/

This preview bundles this PR's firmware — its flasher only flashes this build (not the release). Rebuilt on every push; removed when the PR closes.

The hardware CI has been kicked off and should return within a few minutes, stand by!

@github-actions

Copy link
Copy Markdown

🎛️ HW CI (physical bench)

AMYboard (USB-MIDI + AMY zP → audio): ✅ PASS — flashed this PR’s firmware; all checks matched the references.

Tulip (TULIP4_R11; serial-REPL audio + WiFi screenshot): ✅ PASS — flashed this PR’s firmware; all checks matched the references.

⬇️ Artifacts: recordings · screenshot · serial logs · run logs

Self-hosted bench. Audio spectral-compared to ref/hwci_basic.wav + ref/tulip_basic.wav; Tulip screenshot pixel-compared to ref/tulip_screenshot.png. Both analog outs share one capture card, so the tests run sequentially.

@bwhitman bwhitman merged commit 082cb4d into main Jun 22, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant