Skip to content

Deploy drift: ~/.local/lib/brainlayer helpers are never synced from scripts/launchd — a fix merged 2026-08-08 was still undeployed on 08-24 and killed the watcher 991 times #737

Description

@EtanHey

Problem

The launchd helper scripts in ~/.local/lib/brainlayer/ are not kept in sync with
scripts/launchd/, nothing detects it, and on 2026-08-24 that cost 991 process kills.

scripts/launchd/throughput-watchdog.py gained a starvation guard in
PR #667 "fix: prevent checkpoint starvation and watchdog freeze"progress_slo_seconds: int = 180
plus operational progress tracking (offset_bytes / drain_cycles / drained_total), so a watcher
that is advancing its file offsets but slow to commit a chunk is not counted as stalled.

The copy that actually executes did not contain the string progress_slo_seconds at all:

installed  ~/.local/lib/brainlayer/throughput-watchdog.py   2026-07-18
repo       scripts/launchd/throughput-watchdog.py           2026-08-08   (PR #667)
$ grep -c progress_slo_seconds ~/.local/lib/brainlayer/throughput-watchdog.py
0

Merged 2026-08-08, still not deployed 2026-08-24. Meanwhile
~/.local/share/brainlayer/health-check-state.json recorded:

"restart_count": 991, "restart_attempt_count": 1071

That stale artifact launchctl kickstart -k'd (SIGKILL) the watcher 991 times, because it could
not tell "wedged" from "CPU-starved". The fix for that had been sitting in main for over two weeks.

It is not one file

file installed repo
throughput-watchdog.py 2026-07-18 2026-08-08 STALE
backup-daily.sh 2026-08-15 2026-08-17 STALE
jsonl-backup.sh 2026-06-26 2026-06-18 STALE
tier0-watchdog.sh 2026-07-10 2026-07-13 STALE
brainlayer-env-run.sh same

The brainlayer CLI itself was fine (1.5.9 = repo main), so watch/drain ran current code. Only
these helpers drifted — which is exactly why nobody noticed.

Cross-machine: the M1 Pro has a different drift

Evidence-only probe of locals-macbook-pro (Apple M1 Pro) the same day:
~/.local/lib/brainlayer/ contains backup-daily.sh, brainlayer-env-run.sh,
hotlane_brainbar_daemon.py, jsonl-backup.sh — and throughput-watchdog.py is absent entirely
(no watchdog installed at all). Also note it does have an installed hotlane_brainbar_daemon.py,
where the M4 runs that daemon straight from the dev working tree (see #733 comment). So a drift check
must be per-host, and "installed" cannot be assumed to mean "the same set".

Acceptance criteria

  • A check that diffs ~/.local/lib/brainlayer/* against scripts/launchd/* and fails loudly on
    drift, including missing files, not just different ones.
  • It runs somewhere it will actually be seen (CI cannot see this host; a launchd job or a doctor
    subcommand can).
  • Deploying the watchdog fix is a prerequisite for re-enabling com.brainlayer.watch — bringing the
    watcher back under the stale watchdog restarts the 991-kill loop.

Related: #675 (deployed binary pinned/stale — same "merged ≠ deployed" class, different artifact),
#733 (LaunchAgent interpreter drift), #734 / #611 / #672 / #732 (the loop this caused).

— brainlayerClaude (lead) · claude-code/claude-opus-5

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions