Skip to content

Commit 278c9e4

Browse files
edgeheroclaude
andcommitted
chore(plan-exec): 2026-07-17-worker-daemon-pause-resume — phase "Phase 4: Docs + spec lockstep" passed gates
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NGgfho2J6YfRSQN1bSDhj6
1 parent 3fb835f commit 278c9e4

3 files changed

Lines changed: 100 additions & 7 deletions

File tree

README.md

Lines changed: 82 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,86 @@ flowchart LR
7272

7373
Read [`SECURITY.md`](SECURITY.md) before you rely on it — it states plainly what is and is not defended.
7474

75+
## Run as a service
76+
77+
`pi-dispatch worker` is a long-running process — run it in a terminal, or hand it to your OS's service
78+
manager so it starts on boot and restarts on a crash. The units in [`deploy/`](deploy/) are **per-host
79+
templates, not turnkey**: each carries `<PLACEHOLDER>` paths you fill in for your machine. The systemd
80+
unit's *structure* is checked by `systemd-analyze`; the launchd and nssm units are worked examples. All
81+
three run the worker on the **host** — it drives the `docker` CLI and is not itself containerised — so
82+
they need the AOF-enabled Valkey from [`deploy/docker-compose.yml`](deploy/docker-compose.yml) running
83+
alongside, which is what makes the queue **and the pause state** survive a reboot.
84+
85+
**Steer the running worker** without stopping it — these commands talk to Valkey, so they work whether the
86+
worker runs in a terminal or under a service manager:
87+
88+
- `pi-dispatch pause` — stop taking new jobs. **Durable**: the pause lives in the queue and survives a
89+
worker restart, so a paused worker comes back paused after a reboot. Jobs still enqueue; they just wait.
90+
- `pi-dispatch resume` — start taking jobs again.
91+
- `pi-dispatch status` — prints `{ pausedState, waiting, active, paused, delayed, failed }`. `pausedState`
92+
is the switch; `paused` is the backlog **count** of jobs that piled up while paused (they land in the
93+
`paused` list, not `waiting`).
94+
95+
### Linux (systemd)
96+
97+
Edit [`deploy/worker.service`](deploy/worker.service): set `WorkingDirectory`, `EnvironmentFile`, `User`,
98+
and the `node` path to your clone. Then install and start it:
99+
100+
```bash
101+
sudo cp deploy/worker.service /etc/systemd/system/
102+
sudo systemctl daemon-reload
103+
sudo systemctl enable --now worker
104+
```
105+
106+
`systemctl stop worker` sends **SIGTERM** — the worker stops accepting jobs and lets the in-flight
107+
container drain before it exits.
108+
109+
### macOS (launchd)
110+
111+
Edit [`deploy/com.pi-dispatch.worker.plist`](deploy/com.pi-dispatch.worker.plist) and its wrapper
112+
[`deploy/worker-env-wrapper.sh`](deploy/worker-env-wrapper.sh): set the repo-root and log paths (launchd
113+
has no `EnvironmentFile`, so the wrapper loads `.env` at runtime). Then bootstrap it:
114+
115+
```bash
116+
launchctl bootstrap gui/$(id -u) deploy/com.pi-dispatch.worker.plist
117+
```
118+
119+
`launchctl bootout gui/$(id -u)/com.pi-dispatch.worker` sends **SIGTERM** for the same graceful drain.
120+
121+
### Windows (nssm)
122+
123+
Put `nssm.exe` on PATH ([nssm.cc](https://nssm.cc)), set `SERVICE` / `REPO` / `LOGDIR` in
124+
[`deploy/nssm-install.cmd`](deploy/nssm-install.cmd), then run it and start the service:
125+
126+
```
127+
deploy\nssm-install.cmd
128+
nssm start pi-dispatch-worker
129+
```
130+
131+
The wrapper [`deploy/worker-env-wrapper.cmd`](deploy/worker-env-wrapper.cmd) loads `.env` at runtime. A
132+
console-stop (`nssm stop pi-dispatch-worker`) sends the worker a signal it handles, so it drains
133+
gracefully.
134+
135+
### Drain before a planned restart
136+
137+
A planned restart should abort no in-flight job. Pause, wait for the queue to go idle, restart, then
138+
resume:
139+
140+
```bash
141+
pi-dispatch pause # stop taking new jobs (durable)
142+
pi-dispatch status # repeat until "active": 0 — nothing in flight
143+
sudo systemctl restart worker # (or the launchctl / nssm equivalent)
144+
pi-dispatch resume # take jobs again
145+
```
146+
147+
Because the pause is durable, the worker comes back paused even if the restart outruns your `resume`, so
148+
nothing slips through in the gap.
149+
150+
**Windows caveat**: stop the service with nssm's **console-stop** (`nssm stop`), which delivers a signal
151+
the worker handles and drains gracefully. Task Scheduler is a weaker fallback — it stops a task with a
152+
hard kill, giving the worker no chance to drain; a job killed mid-flight leaves a stray container that the
153+
worker's **boot reaper** clears on the next start, rather than draining cleanly.
154+
75155
## Advanced: GitHub automation
76156

77157
pi-dispatch can also be triggered by GitHub — label an issue, and a container works it on a fresh clone,
@@ -133,7 +213,8 @@ minutes.
133213
## Status
134214

135215
The local-folder path (image, worker, `pi-dispatch run` / `worker`) and the GitHub webhook path
136-
(receiver → queue → clone → PR) are built and work. The admin panel and scheduled (cron) triggers are in
216+
(receiver → queue → clone → PR) are built and work; the worker runs in a terminal or as an OS service on
217+
Linux, macOS or Windows (see **Run as a service**). The admin panel and scheduled (cron) triggers are in
137218
progress. The design is specified in
138219
[`specs/`](specs/) — start with [`specs/constitution.md`](specs/constitution.md) for the non-negotiables
139220
and [`specs/design.md`](specs/design.md) for the decisions and what was rejected.

specs/design.md

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -192,6 +192,12 @@ money with no upstream turn limit (`REQ-RUNNER-TURN-BUDGET`).
192192
throttles concurrent streams and tokens-per-minute long before Docker runs out of memory. A **config
193193
knob rather than a constant** because one of the two inputs is an unmeasured guess (`OQ-002`), so
194194
re-tuning after measurement should be a deploy, not a code change.
195+
- **Boot-reaper invariant — single worker per docker daemon**: The knob is parallelism *within* one worker
196+
process; the design assumes exactly one worker per docker daemon. The boot reaper clears every stray
197+
`pi-job-*` container on start (a leaked container keeps spending, so it must go before any new job
198+
launches) — a co-located second worker sharing the daemon would read the first's in-flight container as
199+
its own to remove and kill a live job. Per-host is the common case, but the docker daemon, not the host,
200+
is the true boundary.
195201
- **Traces to**: `OQ-002`, `REQ-QUEUE-BURST-NO-DROP`
196202

197203
## DES-CRON-VIA-BULLMQ-SCHEDULER
@@ -481,7 +487,8 @@ pi-dispatch/
481487
flows/ # frontend-fix.md, bug-fix.md, triage.md — DEFAULTS, seeded into the data volume
482488
persona/ # hard rules; baked into the image. Not runtime-editable
483489
deploy/ # docker-compose runs Valkey only; worker/receiver/panel are host Node processes
484-
# (DES-WORKER-ON-HOST). systemd units ship as untested examples.
490+
# (DES-WORKER-ON-HOST). The systemd unit is a verified-structure per-host template;
491+
# launchd (.plist) and Windows (nssm) units are added as untested examples.
485492
.env.example # provider key, spend/concurrency knobs, VALKEY_URL, PI_JOB_IMAGE
486493
docs/
487494
```
@@ -498,8 +505,11 @@ it is where the SDK traps in `INT-SDK-SESSION-OPTIONS` live, and they are cheape
498505
else in the frame.
499506

500507
**Platform**: Windows, macOS and Linux, wherever Docker runs. `docker-compose` is the supported
501-
deployment; systemd units ship as untested examples. Two consequences are not incidental and are tracked
502-
where they bite: a containerised worker talking to the Docker socket resolves bind-mount paths in the
508+
deployment; the systemd unit is a verified-structure per-host template — its structure statically checked
509+
by `systemd-analyze`, its placeholders unresolved, so it is neither turnkey nor end-to-end tested — and
510+
the launchd (`.plist`) and Windows (nssm) units are added as untested examples. Two consequences are not
511+
incidental and are tracked where they bite: a containerised worker talking to the Docker socket resolves
512+
bind-mount paths in the
503513
*daemon's* namespace, not its own; and a home machine behind NAT cannot receive GitHub webhooks without
504514
a tunnel.
505515

specs/requirements.md

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -219,9 +219,11 @@ local), the credential (a short-lived scoped token for GitHub jobs vs none for l
219219
nothing. Streaming the container output is not a debug nicety — on the operator's own machine, watching
220220
the agent work on their own folder is the primary feedback surface, and a missing completion line is
221221
what tells them a run did nothing.
222-
- **Note on logs**: this is the operator's own terminal for their own folder, not a persistent multi-user
223-
log; `no-pii-in-logs` still applies to any *stored* worker logs (log the stable job id and outcome,
224-
not task bodies).
222+
- **Note on logs**: in a terminal this is the operator's own console for their own folder, not a
223+
persistent multi-user log. **Under a service manager it becomes one**: the console is the manager's
224+
captured, persistent log (systemd's journald, launchd's `StandardOutPath`, nssm's `AppStdout`), so
225+
`no-pii-in-logs` applies to it directly — not only to hypothetical *stored* logs. Log the stable job id
226+
and outcome, not task bodies.
225227
- **Traces to**: `CONST-PI-VERSION-PINNED`, `DES-CLI-TRIGGER-FOR-LOCAL`, `INT-RUNNER-EXIT-CODE-PROTOCOL`
226228
- **Acceptance**: Given a local job reaching a terminal state, the worker console shows exactly one
227229
completion or failure line carrying the job id and outcome; during the run, the container's output is

0 commit comments

Comments
 (0)