Skip to content

rasprover: add hardware-backed task watchdog - #71

Merged
rosterloh merged 1 commit into
mainfrom
feat/rasprover-task-watchdog
Aug 14, 2026
Merged

rasprover: add hardware-backed task watchdog#71
rosterloh merged 1 commit into
mainfrom
feat/rasprover-task-watchdog

Conversation

@rosterloh

Copy link
Copy Markdown
Owner

Adds stall detection to rasprover, ported from the pattern in Protocentral/healthypi-move-fw. No app in this workspace had a watchdog before this.

Why a watchdog, not a fatal handler

rasprover's recorded failure modes are hangs, not CPU faults — a WiFi boot auto-connect that livelocks the driver, and an LVGL mono display path that hangs on the ESP32 OLED. k_sys_fatal_error_handler catches neither. A fed watchdog catches a hang in main's own call path.

What it does

app_watchdog wraps Zephyr's task_wdt in four functions (init / register / unregister / feed) that degrade to no-op static inlines when CONFIG_APP_WATCHDOG=n, so call sites carry no #ifdefs.

Coverage starts at the top of main(), not after init, because the init sequence is where the motivating hangs occur:

  • "boot" channel, 60 s — covers the init sequence, fed between steps. Sized off the longest legitimate single step rather than the total: app_net_connect() at NET_CONNECT_TIMEOUT (30 s), doubled for a driver that overruns its own bound. Everything else is either non-blocking (app_time_start() and app_display_init() only submit work) or bounded well under it (z_open() at 10 s, IP-literal locator so no DNS leg).
  • "main" channel — swapped in once boot completes; timeout derived from get_loop_delay_s() (3x the loop period + 10 s) rather than hardcoded, so it cannot silently break if that setting changes.

On ros_driver/esp32 the TIMG0 MWDT is exposed as the watchdog0 alias, so CONFIG_TASK_WDT_HW_FALLBACK backs the kernel timer with real hardware. Targets without that alias pass NULL to task_wdt_init() and get the documented software-only path.

Known coverage limits

Documented in the header and Kconfig help rather than papered over. Coverage is per registered thread, and main() is the only registrant:

  • A hang inside a workqueue handler — including the LVGL display work, which app_display_init() only submits — leaves main running and feeding, so it is not detected.
  • A pre-main() driver init hang predates the first feed.

Catching either needs a channel registered and fed by that thread. The concrete win here is app_net_connect().

Notes

  • A failed task_wdt_init() logs loudly (*** RUNNING UNPROTECTED ***) instead of silently leaving the board unprotected. Degrade-to-no-op behaviour retained.
  • With HW fallback, TASK_WDT_MIN_TIMEOUT (100 ms) wakes a k_timer to feed the hardware WDT even while the app sleeps 60 s at a time. Harmless here; that symbol is the knob if power draw ever matters.
  • No __noinit crash breadcrumb — deliberately descoped, since .noinit survival across the ESP32 second-stage bootloader is unverified.

Verification

  • mise run agent-build rasprover --sysbuildsucceeds, no new warnings.
  • native_sim not re-run: fails at arch/posix CMake configure on macOS before compiling anything (known host limitation, unrelated). The Kconfig stage ahead of it resolves the new selects cleanly on a board with no watchdog device.
  • Not runtime-verified — no hardware in this session. The reboot-on-stall path has never executed on a board.

🤖 Generated with Claude Code

rasprover's recorded failure modes are hangs, not CPU faults: a WiFi boot
auto-connect that livelocks the driver, and an LVGL mono display path that
hangs on the ESP32 OLED. A fatal-error handler catches neither; a fed
watchdog catches a hang in main's own call path.

app_watchdog wraps Zephyr's task_wdt in four functions (init, register,
unregister, feed) and degrades to no-ops when the subsystem is unavailable,
so call sites need no guards. On ros_driver/esp32 the TIMG0 MWDT is exposed
as the watchdog0 alias, so CONFIG_TASK_WDT_HW_FALLBACK backs the kernel
timer with real hardware; targets without that alias pass NULL to
task_wdt_init() and get the software-only path.

Coverage starts at the top of main(), not after init, because the init
sequence is where the motivating hangs occur. A "boot" channel covers it at
60 s -- 2x app_net_connect()'s 30 s NET_CONNECT_TIMEOUT, the longest
legitimate single step, since the channel is fed between steps rather than
sized off their total. Once boot completes the channel is swapped for a
"main" one whose timeout is derived from get_loop_delay_s() (3x the loop
period plus 10 s) rather than hardcoded.

This covers only what main() calls directly. A hang inside a workqueue
handler (the LVGL display work) or during pre-main driver init is not
detected, since main keeps running and keeps feeding; that would need a
channel fed by the offending thread. A failed task_wdt_init() logs loudly
rather than silently leaving the board unprotected.

Guarded by CONFIG_APP_WATCHDOG (default y).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@rosterloh
rosterloh merged commit b8e72a9 into main Aug 14, 2026
8 checks passed
@rosterloh
rosterloh deleted the feat/rasprover-task-watchdog branch August 14, 2026 08:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant