Fall back to polling-based file watching on inotify/FD exhaustion

[kriszyp]

Summary

Closes harper#488.

When chokidar's native watchers hit ENOSPC (inotify limit) or EMFILE (file descriptor limit), they emit an error and stop firing change events. Harper's three watcher classes — EntryHandler (per-component file tree), OptionsWatcher (per-component config.yaml), RootConfigWatcher (root harperdb-config.yaml) — now detect those exhaustion errors and transparently re-open the affected watcher with usePolling: true. Polling doesn't consume per-watch inotify handles or file descriptors, so Harper continues to function (with somewhat higher CPU and longer change-detection latency) until the operator raises the OS limit.

Why

The original symptom in #488 was an ENOSPC thrown from an OptionsWatcher trying to add a single config-file watch — the system-wide watch pool was already exhausted by the time it tried. Increasing fs.inotify.max_user_instances resolves the immediate failure, but Harper was effectively dead at that point: subsequent watcher activity was unrecoverable without restart. This PR makes that scenario self-healing.

This is one of three independent mitigations for #488 (alongside PR #809 which tightens the ignore list, and a forthcoming PR that suppresses auto-reload during deploy).

Where to look

• utility/watcherFallback.ts — shared module: isWatcherExhaustionError, POLLING_FALLBACK_OPTIONS (1s/2s for single-file watchers), DIRECTORY_POLLING_FALLBACK_OPTIONS (3s/5s for tree watchers, to bound CPU), warnWatcherFallback (process-wide one-time warning that includes the OS sysctl knobs and notes the CPU trade-off).
• Each of the three watcher classes:
  • Splits chokidar setup into #openWatcher() (or reuses existing #watch() for EntryHandler)
  • Catches exhaustion errors in the existing error handler, swaps to polling
  • Tracks #usingPolling and #closed flags to make the close→reopen sequence race-safe

Cross-model review notes

• Codex flagged two real bugs (both fixed):
  1. After flipping #usingPolling, subsequent exhaustion errors during the same recovery cycle were still being emitted to consumers. Now swallowed.
  2. If close() is called while the failed-watcher close() is still pending, the finally callback was reopening a watcher after teardown. Now guarded by #closed.
• Gemini flagged three findings (all addressed):
  1. EntryHandler watching deep trees at 1s polling could burn meaningful CPU — moved to a 3s/5s interval set for directory watchers.
  2. The close-during-recovery test used a 1.5s setTimeout to assert "no event fired" — replaced with a deterministic _openCountForTests getter check (test now 102ms).
  3. void .close().finally(...) could surface unhandled promise rejections — added .catch(() => {}) to swallow teardown rejections (these aren't actionable on an already-failed watcher).

Potential concerns to focus on

• Test-only API surface: each watcher exposes _simulateWatcherErrorForTests / _usingPollingForTests (and _openCountForTests on OptionsWatcher). They're underscore-prefixed and *ForTests-suffixed. Open to renaming if there's an existing Harper convention.
• Silent EMFILE-in-polling-mode: once we're already polling, additional exhaustion errors are swallowed. In the (rare) case the polling watcher's fs.stat calls themselves can't acquire an FD, we'd silently miss changes until pressure subsides. The alternative (re-emitting after first warn) seems noisier than helpful given the operator already has the actionable info from the first warning.
• Polling-and-atomic-renames: chokidar's polling mode can miss the brief deletion phase of an atomic save. For a config.yaml-style file that's typically written via atomic rename by editors, this can cause an occasional missed reload while in fallback mode. Acceptable for a degraded-mode behavior.
• Generated by an LLM (Claude Opus 4.7).

[claude]

Reviewed; no blockers found.