rasprover: add hardware-backed task watchdog - #71
Merged
Conversation
rasprover's recorded failure modes are hangs, not CPU faults: a WiFi boot auto-connect that livelocks the driver, and an LVGL mono display path that hangs on the ESP32 OLED. A fatal-error handler catches neither; a fed watchdog catches a hang in main's own call path. app_watchdog wraps Zephyr's task_wdt in four functions (init, register, unregister, feed) and degrades to no-ops when the subsystem is unavailable, so call sites need no guards. On ros_driver/esp32 the TIMG0 MWDT is exposed as the watchdog0 alias, so CONFIG_TASK_WDT_HW_FALLBACK backs the kernel timer with real hardware; targets without that alias pass NULL to task_wdt_init() and get the software-only path. Coverage starts at the top of main(), not after init, because the init sequence is where the motivating hangs occur. A "boot" channel covers it at 60 s -- 2x app_net_connect()'s 30 s NET_CONNECT_TIMEOUT, the longest legitimate single step, since the channel is fed between steps rather than sized off their total. Once boot completes the channel is swapped for a "main" one whose timeout is derived from get_loop_delay_s() (3x the loop period plus 10 s) rather than hardcoded. This covers only what main() calls directly. A hang inside a workqueue handler (the LVGL display work) or during pre-main driver init is not detected, since main keeps running and keeps feeding; that would need a channel fed by the offending thread. A failed task_wdt_init() logs loudly rather than silently leaving the board unprotected. Guarded by CONFIG_APP_WATCHDOG (default y). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds stall detection to
rasprover, ported from the pattern in Protocentral/healthypi-move-fw. No app in this workspace had a watchdog before this.Why a watchdog, not a fatal handler
rasprover's recorded failure modes are hangs, not CPU faults — a WiFi boot auto-connect that livelocks the driver, and an LVGL mono display path that hangs on the ESP32 OLED.
k_sys_fatal_error_handlercatches neither. A fed watchdog catches a hang in main's own call path.What it does
app_watchdogwraps Zephyr'stask_wdtin four functions (init/register/unregister/feed) that degrade to no-opstatic inlines whenCONFIG_APP_WATCHDOG=n, so call sites carry no#ifdefs.Coverage starts at the top of
main(), not after init, because the init sequence is where the motivating hangs occur:"boot"channel, 60 s — covers the init sequence, fed between steps. Sized off the longest legitimate single step rather than the total:app_net_connect()atNET_CONNECT_TIMEOUT(30 s), doubled for a driver that overruns its own bound. Everything else is either non-blocking (app_time_start()andapp_display_init()only submit work) or bounded well under it (z_open()at 10 s, IP-literal locator so no DNS leg)."main"channel — swapped in once boot completes; timeout derived fromget_loop_delay_s()(3x the loop period + 10 s) rather than hardcoded, so it cannot silently break if that setting changes.On
ros_driver/esp32the TIMG0 MWDT is exposed as thewatchdog0alias, soCONFIG_TASK_WDT_HW_FALLBACKbacks the kernel timer with real hardware. Targets without that alias passNULLtotask_wdt_init()and get the documented software-only path.Known coverage limits
Documented in the header and Kconfig help rather than papered over. Coverage is per registered thread, and
main()is the only registrant:app_display_init()only submits — leavesmainrunning and feeding, so it is not detected.main()driver init hang predates the first feed.Catching either needs a channel registered and fed by that thread. The concrete win here is
app_net_connect().Notes
task_wdt_init()logs loudly (*** RUNNING UNPROTECTED ***) instead of silently leaving the board unprotected. Degrade-to-no-op behaviour retained.TASK_WDT_MIN_TIMEOUT(100 ms) wakes ak_timerto feed the hardware WDT even while the app sleeps 60 s at a time. Harmless here; that symbol is the knob if power draw ever matters.__noinitcrash breadcrumb — deliberately descoped, since.noinitsurvival across the ESP32 second-stage bootloader is unverified.Verification
mise run agent-build rasprover --sysbuild— succeeds, no new warnings.native_simnot re-run: fails atarch/posixCMake configure on macOS before compiling anything (known host limitation, unrelated). The Kconfig stage ahead of it resolves the newselects cleanly on a board with no watchdog device.🤖 Generated with Claude Code