fs_usage watcher dies silently — no log when the sensor pipeline stops

[LiorFink00]

Summary

When the fs_usage watcher pipeline dies, the agent logs nothing. watch_fs_usage is fs_usage | grep | while read …; done and then return 0:

agent/thumper_agent.sh:454   $cmd 2>/dev/null | grep --line-buffered -F "$@" | while read -r line; do
agent/thumper_agent.sh:470   return 0

If fs_usage exits (e.g. it lost the kdebug trace to another agent — see fs_usage-singleton issue), the while read loop hits EOF, the pipeline ends, the backgrounded watcher subshell exits, and watch_fs_usage returns 0 as if nothing happened. No error, no warning, no "watcher stopped" line anywhere.

Evidence (observed live)

thumper-dev's full log after its watcher was knocked out:

[11:16:58] heartbeat every 60s (pid 99333)
[11:16:58] watching 1 bait file(s) via fs_usage

That is the last line. The watcher, fs_usage, and grep are all gone, but the log gives zero indication the sensor stopped.

Impact

An operator has no way to know read detection has stopped. The endpoint looks fine. Diagnosing this required manually walking the process tree.

Proposed direction

• After the pipeline ends, log loudly at error level: watcher exited unexpectedly (fs_usage stopped) with the exit status of fs_usage.
• Distinguish "fs_usage never started / not permitted" (return 1, line 450) from "fs_usage started then died" — both are currently invisible in the sync-loop path.
• Consider treating an immediate exit as fatal-or-fallback rather than return 0.

Related: fs_usage-singleton, dead-watcher-not-restarted, exit-logs.

[LiorFink00]

Part of a cluster found while debugging two Thumper agents on one macOS host. All related:

• macOS fs_usage is a single global sensor — two agents on one host knock each other's watcher out #94 — fs_usage is a single global sensor (root cause)
• Singleton lock is per-install, not per-host — two agents run on one host and collide on fs_usage #95 — singleton lock is per-install, not per-host
• fs_usage watcher dies silently — no log when the sensor pipeline stops #96 — watcher dies silently (no log)
• No exit/shutdown logs — agent stops with nothing written to the log #97 — no exit/shutdown logs
• Heartbeat reports healthy while the watcher is dead — endpoint blind but shown alive #98 — heartbeat reports healthy while watcher is dead
• Dead watcher is never restarted in live-sync mode — no sensor crash recovery #99 — dead watcher never restarted in live-sync mode

[LiorFink00]

Still valid under #100 (FIFO-only v1), re-scoped: fs_usage is removed, but the FIFO serving loop can also die/EOF silently (and FIFO adds new modes — SIGPIPE on a reader that closes early, or a blocked open()). The fix is the same shape: log loudly when the watcher pipeline stops, and distinguish 'never started' from 'started then died'. Update the references from watch_fs_usage to the FIFO watcher.

[LiorFink00]

Closing: fs_usage is being replaced by the per-bait FIFO / inotify / atime sensors (epic #100, PRs #123/#160/#164), so this fs_usage-specific problem no longer applies. The general watcher-health concerns live on in #98/#99.