macOS fs_usage is a single global sensor — two agents on one host knock each other's watcher out

[LiorFink00]

Summary

On macOS, fs_usage reads the kernel trace facility (kdebug), which is effectively a single global, single-consumer resource (shared with dtrace, latency, sc_usage). When two Thumper agents run on the same host and both take the fs_usage path in watch_fs_usage, only one can hold the trace. The agent that starts fs_usage last wins, and the earlier agent's fs_usage is terminated — silently disabling its read detection.

Evidence (observed live)

Two agents on one Mac, both root, byte-identical thumper_agent.sh:

• thumper-demo (/usr/local/thumper) — started 11:17 — has a live fs_usage:

  99418  fs_usage -w -f filesys     (child of thumper-demo watcher)
  

• thumper-dev (/usr/local/thumper-2) — started 11:16, logged watching 1 bait file(s) via fs_usage — has no fs_usage, no grep, no watcher subshell left. Its watcher pipeline collapsed when demo's fs_usage took over the trace.

System-wide there is exactly one fs_usage process, belonging to the agent that started last.

Where

• agent/thumper_agent.sh:437 watch_fs_usage() — runs fs_usage … | grep … | while read
• agent/thumper_agent.sh:521 start_watcher() — chooses fs_usage on darwin+root

Impact

A host with two enrolled installs silently loses read detection on one of them. The bait still exists; a read of it goes undetected. High severity because read detection is the entire product.

Proposed direction

• Detect that fs_usage failed to acquire / lost the trace (pipeline EOF immediately after start) and surface it loudly (see the "watcher dies with no log" issue).
• Coordinate a single shared fs_usage sensor per host that fans out to all local installs, OR
• Fall back to the atime poll for the loser instead of going blind, OR
• At minimum, refuse/warn when a second agent on the same host would contend for fs_usage (see the "double agent on one host" issue).

Related: double-agent host guard, watcher-dies-silently, heartbeat-false-healthy, dead-watcher-not-restarted.

[LiorFink00]

Part of a cluster found while debugging two Thumper agents on one macOS host. All related:

• macOS fs_usage is a single global sensor — two agents on one host knock each other's watcher out #94 — fs_usage is a single global sensor (root cause)
• Singleton lock is per-install, not per-host — two agents run on one host and collide on fs_usage #95 — singleton lock is per-install, not per-host
• fs_usage watcher dies silently — no log when the sensor pipeline stops #96 — watcher dies silently (no log)
• No exit/shutdown logs — agent stops with nothing written to the log #97 — no exit/shutdown logs
• Heartbeat reports healthy while the watcher is dead — endpoint blind but shown alive #98 — heartbeat reports healthy while watcher is dead
• Dead watcher is never restarted in live-sync mode — no sensor crash recovery #99 — dead watcher never restarted in live-sync mode

[LiorFink00]

Mechanism verification

The single-consumer claim was initially inferred from one observation (one fs_usage alive, one dead). Checked against documentation — it holds:

kdebug is available for only one consumer at a time; sc_usage(1), fs_usage(1) and latency(1) all use kdebug, so you cannot run multiple fs_usage simultaneously. (HackTricks — macOS apps inspecting/debugging; The Apple Wiki — Kdebug)

Nuance (to avoid confusion): DTrace supports multiple concurrent consumers, and modern ktrace(1) coordinates kdebug access — but classic fs_usage still takes kdebug exclusively. The single-consumer constraint is specific to fs_usage/latency/sc_usage, which is exactly what the agent uses.

Local repro (decisive)

With one agent already watching via fs_usage (holding kdebug), start a second consumer:

sudo fs_usage -w -f filesys 2>&1 | head -5

Expected on a single-consumer system: it errors / produces no events while the first holds the trace. If it streams normally, this issue is invalid and should be closed.

[LiorFink00]

✅ Confirmed on-host

Ran the repro with one agent already watching via fs_usage (holding kdebug). Starting a second fs_usage fails immediately:

fs_usage: ... resource busy

This empirically confirms the single-consumer kdebug constraint on this macOS version — two fs_usage instances cannot coexist. Root cause for the two-agents-on-one-host blindness is verified, not inferred.

[LiorFink00]

Remediation tracked in #100 — move read detection to EndpointSecurity (eslogger near-term, signed ES system extension long-term). ES is multi-client, so it resolves the single-consumer fs_usage contention at the root.

[LiorFink00]

Superseded by the v1 decision in #100: FIFO (named-pipe) bait replaces fs_usage as the sole read sensor. FIFO uses no kdebug and each bait is its own per-install pipe, so two agents on one host no longer contend for a single global trace — the root cause here is removed, not worked around. Recommend closing once #100 lands. (The "single shared fs_usage per host" / fan-out direction proposed here was investigated and rejected: kdebug is single-consumer and can't be shared from userspace — a PATH shim of fs_usage can't reach launchd/daemon callers or Instruments/dtrace, and would make us the permanent kdebug hog.)

[LiorFink00]

Closing: fs_usage is being replaced by the per-bait FIFO / inotify / atime sensors (epic #100, PRs #123/#160/#164), so this fs_usage-specific problem no longer applies. The general watcher-health concerns live on in #98/#99.