Skip to content

memfault.md

Latest commit

 

History

History
90 lines (59 loc) · 4.91 KB

memfault.md

File metadata and controls

90 lines (59 loc) · 4.91 KB
Raw

Memfault remote debugging

Memfault is a device observability platform that complements on-device debugging. It collects crash coredumps, reboot events, stack/heap metrics, and logs from deployed devices so you can diagnose issues without physical access — especially useful for sporadic faults or problems that only appear on real networks.

The 91m1_ppp application forwards Memfault data through the existing nRF Cloud CoAP connection (same DTLS session and JWT as cloud messaging). No separate Memfault credentials or HTTP upload path is required.

Prerequisites

Complete Getting started first — the device must be onboarded to your nRF Cloud account and successfully connect (Cloud connected in the log). Memfault data is routed to the Memfault project linked to that nRF Cloud account.

Setup

  1. Open Memfault from nRF Cloud — Log in to nRF Cloud and click Memfault in the left sidebar. This opens the Memfault project linked to your account.

  2. Upload the firmware symbol file — Memfault needs the build's zephyr.elf to decode crash addresses into function names and line numbers. Upload it once per firmware build, before or as soon as devices start reporting data:

    • In the Memfault UI: Software → Symbol Files → Upload Symbol File
    • Select build/zephyr/zephyr.elf from your west build directory (the default output when building from applications/91m1_ppp).

    Alternatively, upload from the command line with the Memfault CLI:

    memfault \
  --org-token <token> \
  --org <org> \
  --project <project> \
  upload-mcu-symbols build/zephyr/zephyr.elf

  3. Verify data is flowing — After the device connects to nRF Cloud, periodic Memfault uploads run in the background (CONFIG_MEMFAULT_PERIODIC_UPLOAD). Look for:

    <inf> mflt: Periodic background upload scheduled - initial delay=... period=...

    To trigger a manual upload from the shell:

    uart:~$ mflt test heartbeat
uart:~$ mflt post_chunks

Viewing device data

In the Memfault UI:

  1. Click Devices in the left toolbar to see devices that have reported in.
  2. Select a device to inspect its coredumps, metrics, and event history.

Coredumps are captured automatically on crashes (RAM-backed, 3 KB) and are truncated if a crash needs more space than that. Without an uploaded symbol file, traces appear with a Symbols Missing label and cannot be decoded. Memfault matches a coredump to its symbol file by the GNU build ID that the application logs at boot (<inf> mflt: GNU Build ID: ...), so the symbol file has to come from the exact build running on the device; rebuilding the same version produces a different build ID.

Because TF-M owns the fault handlers for HardFaults, only BusFaults and SecureFaults originating in non-secure code reach Memfault's handler (CONFIG_TFM_ALLOW_NON_SECURE_FAULT_HANDLING=y). A mflt test hardfault is trapped by TF-M, which halts the core without collecting a coredump; use mflt test busfault instead.

Testing

Trigger test faults from the nRF54L15 shell to verify the full pipeline:

uart:~$ mflt test busfault
uart:~$ mflt test assert
uart:~$ mflt test usagefault

After a test fault the device reboots, reconnects to nRF Cloud, and uploads the coredump. Confirm the decoded trace appears under Issues or on the device's page in Memfault.

Software watchdog

Each module thread is monitored by a Zephyr task watchdog. When a thread stops feeding its watchdog, the timeout callback calls MEMFAULT_SOFTWARE_WATCHDOG(), which captures a coredump tagged Software Watchdog before rebooting. To trigger this manually, suspend a monitored thread from the kernel shell so it can no longer feed its watchdog:

uart:~$ kernel thread list
uart:~$ kernel thread suspend <thread_id>

Use the 0x... id printed next to the main thread. After the watchdog timeout (CONFIG_APP_MAIN_WATCHDOG_TIMEOUT_SECONDS) elapses, the device captures the coredump, reboots, and uploads it. The trace appears in Memfault with the Software Watchdog reason.

What gets collected

Data Description
Coredumps Register state, stack, and memory snapshot on crash
Reboot events Unexpected reboots and boot reason
Stack metrics Thread stack high-water marks (CONFIG_MEMFAULT_NCS_STACK_METRICS)
Heap stats Heap usage snapshots (CONFIG_MEMFAULT_HEAP_STATS)
Logs Recent log buffer captured around faults (CONFIG_MEMFAULT_LOGGING_ENABLE)

References