Skip to content

Commit cada20c

Browse files
committed
Fix spec/code drift the conformance scan found; make code satisfy the new REQ
A spec-conformance scan (spec-scan agent) caught three real drifts between the committed specs and what the code does. This project treats spec/code drift as a defect, so: 1. WORKER ON HOST -- the serious one: code contradicted a committed spec. design.md line 276 listed worker-on-host as REJECTED ("abandons compose") and DES-JOB-FILES-VIA-VOLUME-SUBPATH mandated a containerised worker + socket-proxy -- while deploy/docker-compose.yml, the README, and run-container.mjs all run the worker on the host. The decision was made during Phase 2 (verified: the docker CLI translates host paths, the daemon does not, and the VM prefix moved between Docker Desktop versions; local-folder jobs also REQUIRE a host bind mount a named volume cannot give) but only written to the plan, never the specs. Added DES-WORKER-ON-HOST with the evidence; marked DES-JOB-FILES-VIA-VOLUME-SUBPATH SUPERSEDED in place (its subpath/socket-proxy research stays relevant if a GitHub-only deploy ever re-containerises the worker). Fixed the repo-layout deploy line. 2. CLI TRIGGER unspecified -- built (user-directed) but no spec mentioned it. Added DES-CLI-TRIGGER-FOR-LOCAL, including the verified check that CONST-BUDGET-BEFORE-TOKENS still holds: the cap is enforced in the processor immediately before the container, not in the trigger, so a producer cannot bypass it. 3. SCOPE stale -- requirements.md still said "triggers on GitHub issue activity" and never mentioned local folders/CLI/cron, though local is now first-class and built. Rewrote Scope as trigger x target. Scoped REQ-JOB-STATUS-COMMENTS to GitHub jobs explicitly (a local job has no issue). Added REQ-LOCAL-JOB-VISIBILITY -- and made the code satisfy it: startWorker now logs one job_completed/job_failed line per job with the id and outcome, the local counterpart of the issue comment and the signal for CONST-PI-VERSION-PINNED's silent-no-op mode. drift clean, no dangling IDs, all three new IDs resolve. The gap-scan agent's MISSING list is now stale -- everything it flagged (runContainer, config, entrypoint, enqueue) was built this session; the loop runs.
1 parent bfffe7b commit cada20c

3 files changed

Lines changed: 131 additions & 9 deletions

File tree

specs/design.md

Lines changed: 79 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -236,8 +236,79 @@ money with no upstream turn limit (`REQ-RUNNER-TURN-BUDGET`).
236236
- **Traces to**: `CONST-RETRY-INFRA-ONLY`, `CONST-BUDGET-BEFORE-TOKENS`, `REQ-RUNNER-TURN-BUDGET`,
237237
`REQ-QUEUE-BURST-NO-DROP`
238238

239+
## DES-CLI-TRIGGER-FOR-LOCAL
240+
241+
- **Decision**: Local-folder jobs are triggered by a **CLI** (`pi-dispatch run <folder> --task … [--flow …]`)
242+
as the first interface, in addition to the panel that `DES-PANEL-SEPARATE-FROM-RECEIVER` describes. Both
243+
are producers that call one `enqueueLocalJob`; the CLI is the leaner, dev-native path and the panel
244+
reuses the same enqueue.
245+
- **Why**: For a self-hosted tool that mostly runs on people's own machines, the terminal is the natural
246+
first interface — no web server, no bigger build before anything runs — and it is what makes the local
247+
path usable ahead of the panel. The spec build order reaches a usable *local* experience only at the
248+
panel, which sits after the receiver; a CLI closes that gap without the GitHub-webhook receiver a local
249+
user does not need.
250+
- **Consistency check** (this was verified, not assumed): the CLI trigger violates no constraint.
251+
`CONST-TRIGGER-AUTHOR-GATE` is webhook/comment-scoped by construction, and local jobs are ungated by
252+
design (`SECURITY.md`: panel/CLI access *is* the trust boundary for local). Critically,
253+
**`CONST-BUDGET-BEFORE-TOKENS` still holds**: the cap is checked and incremented in the *worker's
254+
processor*, immediately before the container starts — never in the trigger. A producer that enqueues a
255+
job cannot bypass the budget, because the budget gate lives on the consumer side, after prepare and
256+
before `runContainer`. The CLI is only a producer; it spends nothing.
257+
- **Safety**: a local job edits the folder in place with no undo, so the CLI refuses a **dirty git working
258+
tree** unless `--force` — a cheap guard the panel should mirror.
259+
- **Traces to**: `CONST-BUDGET-BEFORE-TOKENS`, `DES-PANEL-SEPARATE-FROM-RECEIVER`,
260+
`REQ-LOCAL-JOB-VISIBILITY`
261+
262+
## DES-WORKER-ON-HOST
263+
264+
- **Decision**: The worker runs **on the host** (a Node process, `pi-dispatch worker` / `npm start`), not
265+
in a container. It launches job containers by shelling out to the real `docker` CLI.
266+
`docker-compose` runs **only Valkey**; the worker is a host process alongside it.
267+
- **Why**: This is the reversal of `DES-JOB-FILES-VIA-VOLUME-SUBPATH` (below, superseded), forced by two
268+
findings and the local-first target.
269+
**(1) The `docker` CLI translates host paths; the daemon does not.** On Docker Desktop the daemon is a
270+
*Linux* daemon behind a Windows named pipe (`docker context` shows `npipe://…` with `docker info`
271+
reporting `linux/x86_64`), so `C:\Users\…` is not a path a Linux kernel can bind-mount. Translation is
272+
client-side — corroborated by compose's `COMPOSE_CONVERT_WINDOWS_PATHS` rewriting paths even against a
273+
remote non-Windows daemon. So a **containerised** worker calling the Engine API must construct the
274+
VM-internal path itself, and **that prefix has already moved between Docker Desktop versions**
275+
(`/host_mnt/c/…``/run/desktop/mnt/host/c/…`). Pinning bespoke path math to an undocumented,
276+
*moving* internal is `CONST-PI-VERSION-PINNED`'s failure class with a different vendor — it would break
277+
on a silent Docker Desktop auto-update. A host worker shelling out to `docker` inherits Docker's own
278+
cross-platform-tested translation for free. This is `library-first` one level up: do not reimplement
279+
Docker Desktop's path translation.
280+
**(2) Local-folder jobs *require* a host bind mount.** The named-volume trick in the superseded entry
281+
dissolves the path problem only because no host path crosses the boundary — which is exactly what a
282+
local-folder job cannot do: the operator's own folder must be bind-mounted as `/workspace`, edited in
283+
place. There is no volume to hide behind. Since local folders are the primary self-hosted experience,
284+
the deployment model must support the bind mount, and the host worker does so with zero path math.
285+
**Isolation is unaffected.** `CONST-ISOLATION-CONTAINER-PER-JOB` is about the **job** container being
286+
the boundary — pi still never runs on the host. And a container holding `/var/run/docker.sock` is
287+
already root-equivalent on the host, so containerising the worker bought *no* isolation; it only bought
288+
deployment tidiness, which is what this trades away.
289+
- **What this deletes**: the named-volume + `volume-subpath` machinery, the `≥26.1.0` Engine floor, the
290+
socket mount, and the `docker-socket-proxy`. The worker binds `/job:ro` and `/workspace` (the folder)
291+
directly.
292+
- **Accepted cost, stated plainly**: Node on the host, not only Docker. `docker compose up` alone no
293+
longer runs everything; the operator also runs `pi-dispatch worker`. That is the honest price of
294+
local-folder jobs working on Windows/macOS/Linux without fragile path math. The receiver and panel are
295+
Node too, so it is one install story (`npm ci`), with Docker running Valkey and the job containers.
296+
- **Evidence**: verified first-hand this session — `docker context` (`npipe` endpoint, `linux/x86_64`
297+
daemon) · `moby/for-win#14271` (VM prefix `/run/desktop/mnt/host/…`) and `docker/compose#5563`
298+
(older `/host_mnt/…`), i.e. the prefix moved · `docker/compose#4240`
299+
(`COMPOSE_CONVERT_WINDOWS_PATHS` is unconditional client-side rewriting) · a constructed `docker run`
300+
argv launched the real job image with a host-native folder path, cross-platform, with no translation.
301+
- **Rejected**: containerised worker + `volume-subpath` (cannot bind-mount a local folder; pins moving
302+
path math) — see the superseded entry for its full reasoning, kept as the record of why it was tried.
303+
- **Traces to**: `INT-CONTAINER-JOB-INPUTS`, `INT-CONTAINER-RUNTIME-CONTRACT`,
304+
`CONST-ISOLATION-CONTAINER-PER-JOB`
305+
239306
## DES-JOB-FILES-VIA-VOLUME-SUBPATH
240307

308+
- **Status**: **SUPERSEDED by `DES-WORKER-ON-HOST`.** Kept in place because IDs are permanent and its
309+
research (subpath semantics, the socket-proxy hardening) stays relevant if a GitHub-only deployment
310+
ever re-containerises the worker. The decision below is **not** what the code does: the worker runs on
311+
the host and bind-mounts directly.
241312
- **Decision**: The worker hands `/workspace` and `/job` to the job container via a **plain named volume
242313
per job** plus `--mount type=volume,…,volume-subpath=…`. Never a host bind mount. Docker Engine
243314
**≥26.1.0**. A `tecnativa/docker-socket-proxy` sits between the worker and the Docker socket.
@@ -273,8 +344,10 @@ money with no upstream turn limit (`REQ-RUNNER-TURN-BUDGET`).
273344
Docker Desktop FAQ: *"Mac and Windows WSL 2 users can connect via Unix socket at
274345
`unix:///var/run/docker.sock`"* · `Tecnativa/docker-socket-proxy` README (per-API-section allowlist;
275346
`POST`/`AUTH`/`SECRETS` revoked by default)
276-
- **Rejected**: same-path bind mount (breaks on native Windows) · worker-on-host (abandons compose) ·
277-
`docker cp` (cannot enforce read-only `/job`; kept as fallback)
347+
- **Rejected** *(at the time of this superseded entry)*: same-path bind mount (breaks on native Windows) ·
348+
worker-on-host — **this rejection was itself reversed by `DES-WORKER-ON-HOST`**, which found that
349+
local-folder jobs make a host bind mount unavoidable and that the containerised alternative pins moving
350+
path math · `docker cp` (cannot enforce read-only `/job`; kept as fallback)
278351
- **Open**: `readonly` combined with `volume-subpath` is documented as an orthogonal field but **no worked
279352
example was found combining them** — smoke-test it in CI before relying on it, because
280353
`INT-CONTAINER-JOB-INPUTS` is a security boundary, not a convenience. Likewise `--rm` is believed not to
@@ -407,7 +480,9 @@ pi-dispatch/
407480
image/ # Dockerfile + entrypoint + /runner (SDK job runner)
408481
flows/ # frontend-fix.md, bug-fix.md, triage.md — DEFAULTS, seeded into the data volume
409482
persona/ # hard rules; baked into the image. Not runtime-editable
410-
deploy/ # docker-compose (Valkey + receiver + worker + panel); systemd units as examples
483+
deploy/ # docker-compose runs Valkey only; worker/receiver/panel are host Node processes
484+
# (DES-WORKER-ON-HOST). systemd units ship as untested examples.
485+
.env.example # provider key, spend/concurrency knobs, VALKEY_URL, PI_JOB_IMAGE
411486
docs/
412487
```
413488

@@ -436,3 +511,4 @@ a tunnel.
436511
|---|---|
437512
| 2026-07-15 | Initial. Extracted from `DESIGN.md` v0.1 (2026-07-14, local, uncommitted) §2, §3, §4, §5, §9, §11. That document recorded "50 claims adversarially verified: 48 confirmed, 2 refuted" — **verified against documentation**. Source-verification at `earendil-works/pi @ 5e336cf` subsequently corrected ~7 points. `DES-PERSONA-VIA-APPEND-SYSTEM-MD` is materially rewritten: the source doc's decisions #1 and #2 were mutually exclusive as written. `DES-NAME-KEEP-PI-DISPATCH` is new. `pi-harness` and `pi-sentry` were absent from the source doc's alternatives and are added. §5.7's "caches roll at midnight" caveat is **dropped** — 0.80.7 removed the date from the default system prompt. |
438513
| 2026-07-15 | An admin panel and cross-platform (Windows/macOS/Linux + Docker) added to scope. Two new decisions and one **security correction**. `DES-PANEL-SEPARATE-FROM-RECEIVER`: the source doc mounted Bull Board on the receiver — defensible for a read-only dashboard, **not** once the same surface sets the model and rewrites flows, because the receiver is the one process that must be internet-reachable. The panel and the receiver have opposite reachability requirements and cannot share a port. `DES-FLOWS-ARE-DATA-PERSONA-IS-CODE`: the panel requirement collided with keeping flows as reviewed repo markdown; resolved by observing that one file was carrying two jobs — hard rules need immutability, task recipes need editability. Architecture diagram and repo layout updated; the public edge is now drawn explicitly. Build order extended with panel and deploy. |
514+
| 2026-07-16 | **Resolved a spec/code contradiction.** `DES-WORKER-ON-HOST` added and `DES-JOB-FILES-VIA-VOLUME-SUBPATH` marked SUPERSEDED: the worker runs on the host (the `docker` CLI translates host paths, the daemon does not, and the VM prefix moved between Docker Desktop versions; local-folder jobs also *require* a host bind mount a named volume cannot give). The committed spec had rejected worker-on-host while the code already did it -- caught by a spec-conformance scan. `DES-CLI-TRIGGER-FOR-LOCAL` added: the CLI producer was built (user-directed) but unspecified; recorded with the check that `CONST-BUDGET-BEFORE-TOKENS` still holds because the cap is enforced in the processor, not the trigger. Repo-layout `deploy/` line corrected (compose runs Valkey only). |

specs/requirements.md

Lines changed: 42 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -7,9 +7,21 @@ Evidence convention as in `constitution.md`: `Evidence (upstream)` is authoritat
77

88
## Scope
99

10-
Run pi as an always-on automation harness that triggers on GitHub issue activity, follows a predefined
11-
flow, executes each job in an isolated container, supports frontend work with visual verification, and
12-
survives burst load without dropping work.
10+
Run pi as a self-hosted harness that executes each job in an isolated container, follows a predefined
11+
flow, supports frontend work with visual verification, and survives burst load without dropping work.
12+
13+
A job is a **trigger × target** (see `DES-CRON-VIA-BULLMQ-SCHEDULER`):
14+
15+
- **Targets**: a **local folder** on the operator's machine (edited in place — the primary self-hosted
16+
use, needs only a provider key), or a **GitHub repo** (cloned, worked, opened as a PR — needs a GitHub
17+
App).
18+
- **Triggers**: the **CLI** / **panel** (operator-initiated, `DES-CLI-TRIGGER-FOR-LOCAL`), a **webhook**
19+
(GitHub issue activity), or **cron** (a schedule).
20+
21+
Everything below the trigger is identical: budget check → `/job:ro` inputs → one container → the runner
22+
→ an exit code. What differs is authz (a label/collaborator gate for webhooks vs panel/CLI access for
23+
local), the credential (a 1h scoped token for GitHub jobs vs none for local), and the completion signal
24+
(an issue comment vs the console/panel — see `REQ-JOB-STATUS-COMMENTS` and `REQ-LOCAL-JOB-VISIBILITY`).
1325

1426
**Out of scope**: being a hosted service; multi-tenancy; merging anything.
1527

@@ -165,15 +177,38 @@ survives burst load without dropping work.
165177

166178
## REQ-JOB-STATUS-COMMENTS
167179

168-
- **Statement**: Each job shall comment on its triggering issue at start, and on completion or failure.
180+
- **Statement**: Each **GitHub-backed** job shall comment on its triggering issue at start, and on
181+
completion or failure.
182+
- **Scope**: GitHub jobs only. A local-folder job has no issue to comment on; its equivalent is
183+
`REQ-LOCAL-JOB-VISIBILITY`. Stated explicitly because the original requirement assumed every job is a
184+
GitHub issue — it is not.
169185
- **Why**: State must be visible where the human already is. The queue dashboard sits behind basic auth
170186
on a home box and nobody opens it; the issue thread is where the requester is already looking, and is
171187
the only surface a non-maintainer ever sees. It is also the **only** signal for
172188
`CONST-PI-VERSION-PINNED`'s silent-no-op failure mode: if an upstream break makes every job a no-op,
173189
the queue still reports success — a missing completion comment is what a human would actually notice.
174190
- **Traces to**: `CONST-MERGE-NEVER-AUTOMATIC`, `CONST-PI-VERSION-PINNED`
175-
- **Acceptance**: Given any job reaching a terminal state, exactly one completion or failure comment
176-
exists on the issue.
191+
- **Acceptance**: Given any GitHub job reaching a terminal state, exactly one completion or failure
192+
comment exists on the issue.
193+
194+
## REQ-LOCAL-JOB-VISIBILITY
195+
196+
- **Statement**: A local-folder job shall surface its outcome where the operator is already looking — the
197+
worker's console — at start and on completion or failure, and (once built) in the panel's job view. The
198+
container's own output shall stream to that console during the run.
199+
- **Why**: The local counterpart of `REQ-JOB-STATUS-COMMENTS`, and it carries the same load: it is the
200+
signal for `CONST-PI-VERSION-PINNED`'s silent-no-op failure mode. A local job has no issue thread, so
201+
without a console signal a broken run would still report success to the queue and a human would notice
202+
nothing. Streaming the container output is not a debug nicety — on the operator's own machine, watching
203+
the agent work on their own folder is the primary feedback surface, and a missing completion line is
204+
what tells them a run did nothing.
205+
- **Note on logs**: this is the operator's own terminal for their own folder, not a persistent multi-user
206+
log; `no-pii-in-logs` still applies to any *stored* worker logs (log the stable job id and outcome,
207+
not task bodies).
208+
- **Traces to**: `CONST-PI-VERSION-PINNED`, `DES-CLI-TRIGGER-FOR-LOCAL`, `INT-RUNNER-EXIT-CODE-PROTOCOL`
209+
- **Acceptance**: Given a local job reaching a terminal state, the worker console shows exactly one
210+
completion or failure line carrying the job id and outcome; during the run, the container's output is
211+
visible there.
177212

178213
---
179214

@@ -195,3 +230,4 @@ wait-list working as designed, not a failure — see `README.md`.
195230
| Date | Change |
196231
|---|---|
197232
| 2026-07-15 | Initial. Extracted from `DESIGN.md` v0.1 §1, §5.1–5.2, §5.6, §7, §8. `REQ-RUNNER-TURN-BUDGET` and `REQ-UPSTREAM-CONTRACT-TESTS` are **new** — both exist because source-verification refuted design assumptions the doc had marked "verify". §8's failure-mode table was the richest source; one of its rows ("verify: pi max-turns option") was wrong. |
233+
| 2026-07-16 | **Scope de-GitHub-ified.** It said "triggers on GitHub issue activity" and never mentioned local folders, the CLI/panel, or cron -- stale, since local is now first-class and built. Rewritten as trigger × target. `REQ-JOB-STATUS-COMMENTS` scoped to GitHub jobs explicitly (a local job has no issue). New `REQ-LOCAL-JOB-VISIBILITY`: local jobs surface their outcome on the worker console (and later the panel) -- the local counterpart of the issue comment and the same signal for `CONST-PI-VERSION-PINNED`'s silent-no-op mode. Code updated to match: startWorker now logs one terminal line per job. |

worker/src/start.mjs

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,16 @@ export function startWorker(env = process.env) {
3636
},
3737
});
3838

39+
// REQ-LOCAL-JOB-VISIBILITY: exactly one terminal line per job, carrying the job id and outcome,
40+
// where the operator is already looking. This is the local counterpart of the GitHub issue
41+
// comment and the signal for CONST-PI-VERSION-PINNED's silent-no-op mode -- a missing line is
42+
// what tells a human a run did nothing. The container's own output already streams via
43+
// runContainer's onOutput during the run.
44+
worker.on("completed", (job, result) => log("job_completed", { jobId: job?.id, outcome: result?.outcome }));
45+
worker.on("failed", (job, err) =>
46+
log("job_failed", { jobId: job?.id, attempt: job?.attemptsMade, reason: String(err?.message ?? err).slice(0, 120) }),
47+
);
48+
3949
log("worker_started", {
4050
queue: "pi-jobs",
4151
concurrency: config.concurrency,

0 commit comments

Comments
 (0)