Skip to content

Commit d6c5fce

Browse files
edgeheroedgehero
authored
Merge pull request #13 from edgehero/plan-exec/2026-07-17-worker-daemon-pause-resume
Background service: daemonize the worker + durable pause/resume switch
2 parents a8060b5 + b0db38c commit d6c5fce

20 files changed

Lines changed: 756 additions & 21 deletions

File tree

‎.github/workflows/deploy-lint.yml‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
# Syntax lint for the deploy/ daemon artifacts.
2+
#
3+
# These files are hand-written examples an operator adapts; nothing exercises them in the normal test
4+
# suite (deploy/** is not in any other workflow's path filter), so a typo'd directive or a malformed
5+
# plist ships silently and only fails on the operator's box. This workflow is the cheap gate: it checks
6+
# STRUCTURE, not runtime behaviour.
7+
#
8+
# - worker.service / receiver.service : `systemd-analyze verify` parses the units. The PLACEHOLDER
9+
# paths (/opt/pi-dispatch, /usr/bin/node) do not exist on the runner, so path-existence warnings are
10+
# EXPECTED and ignored; only genuine syntax errors (unknown directives, parse failures) fail.
11+
# - com.pi-dispatch.worker.plist : `xmllint --noout` proves the plist is well-formed XML. plutil is
12+
# macOS-only, so xmllint (libxml2-utils) is the Linux stand-in for well-formedness.
13+
# - worker-env-wrapper.sh : `sh -n` is a POSIX shell parse check (no execution).
14+
#
15+
# The `.cmd` wrappers (worker-env-wrapper.cmd, nssm-install.cmd) have no practical Linux linter and are
16+
# not checked here -- deliberately, rather than with a flaky approximation.
17+
18+
name: deploy lint
19+
20+
on:
21+
push:
22+
paths:
23+
- "deploy/**"
24+
- ".github/workflows/deploy-lint.yml"
25+
pull_request:
26+
paths:
27+
- "deploy/**"
28+
- ".github/workflows/deploy-lint.yml"
29+
workflow_dispatch:
30+
31+
jobs:
32+
lint:
33+
name: deploy/ artifacts are syntactically valid
34+
runs-on: ubuntu-latest
35+
steps:
36+
- uses: actions/checkout@v4
37+
38+
- name: Install linters
39+
run: sudo apt-get update && sudo apt-get install -y libxml2-utils
40+
41+
- name: systemd units parse (syntax only; PLACEHOLDER path warnings ignored)
42+
run: |
43+
# verify exits non-zero for the missing PLACEHOLDER exec paths, which is expected -- so gate on
44+
# the CONTENT of the diagnostics, not the exit code. A real defect is an unknown directive or a
45+
# parse failure; a missing /opt/pi-dispatch or /usr/bin/node is not.
46+
systemd-analyze verify --no-pager deploy/worker.service deploy/receiver.service > verify.log 2>&1 || true
47+
cat verify.log
48+
if grep -Eiq 'Unknown (key name|lvalue|section)|Failed to parse|Invalid setting|not a valid|expected ' verify.log; then
49+
echo "::error::systemd-analyze found a syntax error in a unit file. See the log above."
50+
exit 1
51+
fi
52+
echo "OK: unit files are syntactically valid (path-existence warnings ignored)"
53+
54+
- name: plist is well-formed XML
55+
run: |
56+
xmllint --noout deploy/com.pi-dispatch.worker.plist \
57+
|| { echo "::error::deploy/com.pi-dispatch.worker.plist is not well-formed XML."; exit 1; }
58+
echo "OK: plist well-formed"
59+
60+
- name: POSIX wrapper parses
61+
run: |
62+
sh -n deploy/worker-env-wrapper.sh \
63+
|| { echo "::error::deploy/worker-env-wrapper.sh has a shell syntax error."; exit 1; }
64+
echo "OK: worker-env-wrapper.sh parses"

‎.plan-artifacts/2026-07-16-github-trigger-pluggable-auth-plan/drift-snapshot.json‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
{
2+
"generated": "2026-07-16T00:00:00Z",
3+
"planSlug": "2026-07-16-github-trigger-pluggable-auth-plan",
4+
"exitCode": 0,
5+
"reqStates": {}
6+
}

‎.plan-artifacts/2026-07-17-cron-trigger-plan/drift-snapshot.json‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
{
2+
"generated": "2026-07-17T12:57:17Z",
3+
"planSlug": "2026-07-17-cron-trigger-plan",
4+
"exitCode": 0,
5+
"reqStates": {}
6+
}

‎.plan-artifacts/2026-07-17-worker-daemon-pause-resume-plan/drift-snapshot.json‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
{
2+
"generated": "2026-07-17T19:14:18Z",
3+
"planSlug": "2026-07-17-worker-daemon-pause-resume-plan",
4+
"exitCode": 0,
5+
"reqStates": {}
6+
}

‎README.md‎

Lines changed: 82 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,86 @@ flowchart LR
7272

7373
Read [`SECURITY.md`](SECURITY.md) before you rely on it — it states plainly what is and is not defended.
7474

75+
## Run as a service
76+
77+
`pi-dispatch worker` is a long-running process — run it in a terminal, or hand it to your OS's service
78+
manager so it starts on boot and restarts on a crash. The units in [`deploy/`](deploy/) are **per-host
79+
templates, not turnkey**: each carries `<PLACEHOLDER>` paths you fill in for your machine. The systemd
80+
unit's *structure* is checked by `systemd-analyze`; the launchd and nssm units are worked examples. All
81+
three run the worker on the **host** — it drives the `docker` CLI and is not itself containerised — so
82+
they need the AOF-enabled Valkey from [`deploy/docker-compose.yml`](deploy/docker-compose.yml) running
83+
alongside, which is what makes the queue **and the pause state** survive a reboot.
84+
85+
**Steer the running worker** without stopping it — these commands talk to Valkey, so they work whether the
86+
worker runs in a terminal or under a service manager:
87+
88+
- `pi-dispatch pause` — stop taking new jobs. **Durable**: the pause lives in the queue and survives a
89+
worker restart, so a paused worker comes back paused after a reboot. Jobs still enqueue; they just wait.
90+
- `pi-dispatch resume` — start taking jobs again.
91+
- `pi-dispatch status` — prints `{ pausedState, waiting, active, paused, delayed, failed }`. `pausedState`
92+
is the switch; `paused` is the backlog **count** of jobs that piled up while paused (they land in the
93+
`paused` list, not `waiting`).
94+
95+
### Linux (systemd)
96+
97+
Edit [`deploy/worker.service`](deploy/worker.service): set `WorkingDirectory`, `EnvironmentFile`, `User`,
98+
and the `node` path to your clone. Then install and start it:
99+
100+
```bash
101+
sudo cp deploy/worker.service /etc/systemd/system/
102+
sudo systemctl daemon-reload
103+
sudo systemctl enable --now worker
104+
```
105+
106+
`systemctl stop worker` sends **SIGTERM** — the worker stops accepting jobs and lets the in-flight
107+
container drain before it exits.
108+
109+
### macOS (launchd)
110+
111+
Edit [`deploy/com.pi-dispatch.worker.plist`](deploy/com.pi-dispatch.worker.plist) and its wrapper
112+
[`deploy/worker-env-wrapper.sh`](deploy/worker-env-wrapper.sh): set the repo-root and log paths (launchd
113+
has no `EnvironmentFile`, so the wrapper loads `.env` at runtime). Then bootstrap it:
114+
115+
```bash
116+
launchctl bootstrap gui/$(id -u) deploy/com.pi-dispatch.worker.plist
117+
```
118+
119+
`launchctl bootout gui/$(id -u)/com.pi-dispatch.worker` sends **SIGTERM** for the same graceful drain.
120+
121+
### Windows (nssm)
122+
123+
Put `nssm.exe` on PATH ([nssm.cc](https://nssm.cc)), set `SERVICE` / `REPO` / `LOGDIR` in
124+
[`deploy/nssm-install.cmd`](deploy/nssm-install.cmd), then run it and start the service:
125+
126+
```
127+
deploy\nssm-install.cmd
128+
nssm start pi-dispatch-worker
129+
```
130+
131+
The wrapper [`deploy/worker-env-wrapper.cmd`](deploy/worker-env-wrapper.cmd) loads `.env` at runtime. A
132+
console-stop (`nssm stop pi-dispatch-worker`) sends the worker a signal it handles, so it drains
133+
gracefully.
134+
135+
### Drain before a planned restart
136+
137+
A planned restart should abort no in-flight job. Pause, wait for the queue to go idle, restart, then
138+
resume:
139+
140+
```bash
141+
pi-dispatch pause # stop taking new jobs (durable)
142+
pi-dispatch status # repeat until "active": 0 — nothing in flight
143+
sudo systemctl restart worker # (or the launchctl / nssm equivalent)
144+
pi-dispatch resume # take jobs again
145+
```
146+
147+
Because the pause is durable, the worker comes back paused even if the restart outruns your `resume`, so
148+
nothing slips through in the gap.
149+
150+
**Windows caveat**: stop the service with nssm's **console-stop** (`nssm stop`), which delivers a signal
151+
the worker handles and drains gracefully. Task Scheduler is a weaker fallback — it stops a task with a
152+
hard kill, giving the worker no chance to drain; a job killed mid-flight leaves a stray container that the
153+
worker's **boot reaper** clears on the next start, rather than draining cleanly.
154+
75155
## Scheduling recurring jobs
76156

77157
A cron schedule is a trigger, not a new job kind: each entry runs a local folder through a flow on a cron
@@ -159,7 +239,8 @@ minutes.
159239
## Status
160240

161241
The local-folder path (image, worker, `pi-dispatch run` / `worker`), the GitHub webhook path
162-
(receiver → queue → clone → PR), and scheduled (cron) triggers for local folders are built and work. The
242+
(receiver → queue → clone → PR), and scheduled (cron) triggers for local folders are built and work; the
243+
worker runs in a terminal or as an OS service on Linux, macOS or Windows (see **Run as a service**). The
163244
admin panel is in progress. The design is specified in
164245
[`specs/`](specs/) — start with [`specs/constitution.md`](specs/constitution.md) for the non-negotiables
165246
and [`specs/design.md`](specs/design.md) for the decisions and what was rejected.

‎deploy/com.pi-dispatch.worker.plist‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
3+
<!--
4+
UNTESTED EXAMPLE: a starting point for the macOS (launchd) worker daemon, not a shipped, verified
5+
unit. Adapt it. The Linux/systemd equivalent is deploy/worker.service.
6+
7+
ProgramArguments points at deploy/worker-env-wrapper.sh; that wrapper is what loads `.env`, because
8+
launchd has no EnvironmentFile mechanism. NO secrets are inlined here: there is deliberately no
9+
EnvironmentVariables dict, since that would commit credentials into this file. The wrapper reads
10+
`.env` at runtime instead (note the ANTHROPIC_OAUTH_TOKEN over ANTHROPIC_API_KEY precedence trap
11+
documented in the wrapper).
12+
13+
Graceful shutdown needs NO macOS-specific code: `launchctl bootout` sends SIGTERM, which reaches node
14+
through the wrapper's `exec`, and the worker drains in-flight work on SIGTERM. ExitTimeOut 30 gives it
15+
room (at least the 5s docker-stop grace) before launchd escalates to SIGKILL.
16+
17+
KeepAlive restarts on a crash (SuccessfulExit false) but NOT on a clean exit, so a deliberate stop
18+
stays stopped.
19+
20+
Per-host PLACEHOLDERS: replace /opt/pi-dispatch (the repo root, used in ProgramArguments and
21+
WorkingDirectory) and /opt/pi-dispatch/logs (StandardOutPath, StandardErrorPath) with your paths.
22+
23+
One worker per host (DES-CONCURRENCY-3): parallelism is PI_CONCURRENCY inside the one process.
24+
Requires the AOF-enabled Valkey from deploy/docker-compose.yml.
25+
-->
26+
<plist version="1.0">
27+
<dict>
28+
<key>Label</key>
29+
<string>com.pi-dispatch.worker</string>
30+
31+
<key>ProgramArguments</key>
32+
<array>
33+
<string>/bin/sh</string>
34+
<string>/opt/pi-dispatch/deploy/worker-env-wrapper.sh</string>
35+
</array>
36+
37+
<key>WorkingDirectory</key>
38+
<string>/opt/pi-dispatch</string>
39+
40+
<key>RunAtLoad</key>
41+
<true/>
42+
43+
<key>KeepAlive</key>
44+
<dict>
45+
<key>SuccessfulExit</key>
46+
<false/>
47+
</dict>
48+
49+
<key>ExitTimeOut</key>
50+
<integer>30</integer>
51+
52+
<key>StandardOutPath</key>
53+
<string>/opt/pi-dispatch/logs/worker.out.log</string>
54+
55+
<key>StandardErrorPath</key>
56+
<string>/opt/pi-dispatch/logs/worker.err.log</string>
57+
</dict>
58+
</plist>

‎deploy/nssm-install.cmd‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
@echo off
2+
REM UNTESTED EXAMPLE -- a starting point for the Windows worker service, not a shipped, verified unit.
3+
REM Adapt it. The Linux/systemd equivalent is deploy/worker.service; the macOS one is
4+
REM deploy/com.pi-dispatch.worker.plist.
5+
REM
6+
REM Registers the pi-dispatch worker as a Windows service via nssm (the Non-Sucking Service Manager).
7+
REM nssm is an operator-downloaded binary (https://nssm.cc) -- it is documented here, NOT vendored into
8+
REM this repo; put nssm.exe on PATH before running this. The service's Application is the `.cmd` wrapper
9+
REM (deploy/worker-env-wrapper.cmd), which loads `.env` at runtime -- so NO secrets are inlined here or
10+
REM passed via AppEnvironmentExtra. `.env` is gitignored; nothing in this file is a credential.
11+
REM
12+
REM Why nssm over Task Scheduler: Task Scheduler stops a task with TerminateProcess (a hard kill), which
13+
REM gives the worker no chance to drain the in-flight job -- it then relies on the queue's boot reaper to
14+
REM recover the orphaned container. nssm's AppStopMethodConsole sends a real Ctrl-C first, which node
15+
REM receives as SIGINT for a graceful drain. Task Scheduler is a weaker fallback, not the recommended
16+
REM path.
17+
REM
18+
REM One worker per host (DES-CONCURRENCY-3): parallelism is PI_CONCURRENCY inside the one process, not
19+
REM multiple services. Requires the AOF-enabled Valkey from deploy/docker-compose.yml.
20+
REM
21+
REM Per-host PLACEHOLDERS: set SERVICE / REPO / LOGDIR below for your host before running.
22+
23+
setlocal
24+
25+
set "SERVICE=pi-dispatch-worker"
26+
set "REPO=C:\pi-dispatch"
27+
set "LOGDIR=C:\pi-dispatch\logs"
28+
29+
REM Application is the wrapper (loads `.env`), not node directly and not an env dict with real values.
30+
nssm install %SERVICE% "%REPO%\deploy\worker-env-wrapper.cmd"
31+
nssm set %SERVICE% AppDirectory "%REPO%"
32+
nssm set %SERVICE% AppStdout "%LOGDIR%\worker.out.log"
33+
nssm set %SERVICE% AppStderr "%LOGDIR%\worker.err.log"
34+
35+
REM Stop = send Ctrl-C (node SIGINT, graceful drain), wait 15000ms (>= the 5s docker-stop grace) before
36+
REM nssm escalates to a hard kill.
37+
nssm set %SERVICE% AppStopMethodConsole 15000
38+
39+
REM StartLimit analogue: pause 5000ms between restarts so a crash loop does not spin the provider bill
40+
REM (mirrors StartLimitIntervalSec/StartLimitBurst + RestartSec in deploy/worker.service).
41+
nssm set %SERVICE% AppThrottle 5000
42+
43+
REM Restart on a crash by default...
44+
nssm set %SERVICE% AppExit Default Restart
45+
REM ...but exit 2 is EXIT_POLICY: a determinate config/budget refusal, never retried. Do NOT restart it
46+
REM (mirrors RestartPreventExitStatus=2 in deploy/worker.service).
47+
nssm set %SERVICE% AppExit 2 Exit
48+
49+
echo Installed service "%SERVICE%". Start it with: nssm start %SERVICE%
50+
51+
endlocal

‎deploy/receiver.service‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,10 @@
1-
# UNTESTED EXAMPLE (design.md:483-484) -- a starting point, not a shipped, verified unit. Adapt it.
1+
# UNTESTED EXAMPLE (DES-WORKER-ON-HOST) -- a starting point, not a shipped, verified unit. Adapt it.
22
#
33
# The receiver runs on the HOST as the public edge: it verifies GitHub deliveries and enqueues jobs.
44
# Requires node >=22.19.0 on PATH for User=pi -- or pin an absolute `ExecStart=/usr/bin/node ...`.
5-
# This is a Linux/systemd unit; a launchd (macOS) / nssm (Windows) equivalent is left to the operator.
5+
# This is a Linux/systemd unit. The worker ships cross-platform daemon artifacts under deploy/
6+
# (com.pi-dispatch.worker.plist for launchd, nssm-install.cmd for Windows); receiver daemonization is
7+
# out of scope, but a launchd/nssm receiver would follow the same .env-wrapper pattern those use.
68
#
79
# NAT / tunnel: the receiver binds `RECEIVER_BIND` (default 0.0.0.0) and must be reachable by GitHub's
810
# webhook delivery. On a home machine behind NAT, put it behind a tunnel (cloudflared / ngrok /

‎deploy/worker-env-wrapper.cmd‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
@echo off
2+
REM UNTESTED EXAMPLE -- a starting point for a Windows service, not a shipped, verified unit. Adapt it.
3+
REM
4+
REM pi-dispatch worker launcher for Windows service managers (nssm; see deploy/nssm-install.cmd). Windows
5+
REM services have no `.env` mechanism, so this wrapper loads `.env` from the repo root itself, then
6+
REM launches node. It reads ONLY the declared `.env` (see `.env.example`), never the host user profile:
7+
REM the container-boundary rules require an explicit, auditable variable set. Nothing here contains a
8+
REM credential -- the secrets live in `.env`, which is gitignored and read at runtime.
9+
REM
10+
REM TRAP: inside pi, ANTHROPIC_OAUTH_TOKEN silently takes precedence over ANTHROPIC_API_KEY. Set exactly
11+
REM one in `.env`.
12+
REM
13+
REM `.env` FORMAT for this loader: KEY=VALUE, one per line. Values MUST be UNQUOTED -- cmd's `set` keeps
14+
REM surrounding quotes as part of the value. `eol=#` skips `#` comment lines; blank lines are ignored.
15+
REM `tokens=1,* delims==` splits on the FIRST `=` only, so values containing `=` (base64, API keys)
16+
REM survive intact.
17+
REM
18+
REM One worker per host (DES-CONCURRENCY-3): parallelism is PI_CONCURRENCY inside the single process, not
19+
REM multiple services. Requires the AOF-enabled Valkey from deploy/docker-compose.yml.
20+
21+
setlocal
22+
23+
REM Resolve repo root relative to this script (deploy\ is one level down).
24+
cd /d "%~dp0.." || exit /b 1
25+
26+
if not exist ".env" (
27+
echo worker-env-wrapper: .env not found in "%CD%" 1>&2
28+
exit /b 1
29+
)
30+
31+
for /f "usebackq eol=# tokens=1,* delims==" %%A in (".env") do set "%%A=%%B"
32+
33+
node worker\src\cli.mjs worker
34+
exit /b %ERRORLEVEL%

‎deploy/worker-env-wrapper.sh‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
#!/bin/sh
2+
# UNTESTED EXAMPLE -- a starting point for a non-systemd daemon, not a shipped, verified unit. Adapt it.
3+
#
4+
# pi-dispatch worker launcher for daemon managers that have NO EnvironmentFile mechanism. systemd reads
5+
# `.env` for you via `EnvironmentFile=` (see deploy/worker.service); launchd (macOS) has no equivalent --
6+
# a plist's ProgramArguments cannot name a `.env`. This wrapper closes that gap: launchd execs THIS
7+
# script, which loads the explicit `.env` from the repo root and then hands off to node.
8+
#
9+
# It sources ONLY the declared `.env` (see `.env.example`), never the host login shell: the
10+
# container-boundary rules require an explicit, auditable variable set, not whatever the operator's
11+
# profile happens to export. Nothing here contains a credential -- the secrets live in `.env`, which is
12+
# gitignored and read at runtime.
13+
#
14+
# TRAP: inside pi, `ANTHROPIC_OAUTH_TOKEN` silently takes precedence over `ANTHROPIC_API_KEY`. Set exactly
15+
# one in `.env`; this wrapper only ADDS the `.env` vars on top of the current environment, it does not
16+
# clear a stray pre-existing one, so a leaked host `ANTHROPIC_OAUTH_TOKEN` would still win.
17+
#
18+
# `exec` is load-bearing: it REPLACES this shell with node, so SIGTERM (e.g. from `launchctl bootout`)
19+
# reaches node directly for a graceful drain instead of dying at the shell and orphaning the worker.
20+
#
21+
# One worker per host (DES-CONCURRENCY-3): parallelism is PI_CONCURRENCY inside the single process, not
22+
# multiple daemons. Requires the AOF-enabled Valkey from deploy/docker-compose.yml.
23+
24+
# Resolve repo root relative to this script (deploy/ is one level down).
25+
cd "$(dirname "$0")/.." || exit 1
26+
if [ ! -f .env ]; then echo "worker-env-wrapper: .env not found in $(pwd)" >&2; exit 1; fi
27+
set -a; . ./.env; set +a
28+
exec node worker/src/cli.mjs worker

0 commit comments

Comments
 (0)