What
Two Mac minis (or Linux boxes) should be able to serve one pi-dispatch deployment: one queue, one budget, one admin view — sandboxes running on whichever host has capacity. The shape is a worker per machine sharing one Valkey, which the architecture already half-supports: BullMQ happily splits one queue across workers, and everything that lives in Valkey is already global. What's missing is everything that silently assumes one host.
What already works (don't redo this)
VALKEY_URL is remote-capable (host/port/auth/db all honored; the systemd unit even notes Valkey "may be remote"). Two workers on one queue split jobs correctly today.- All the shared state is already shared: jobId dedup and the semantic window are Redis-side; the budget windows and the daily token counter are atomic INCRs on shared keys ("verified against a real Valkey under 20 parallel increments"); the pause/resume kill switch, scheduler stall counters, and the
dispatch_run hourly limit are all Valkey state. Two workers share one cap regime for free.
- The boot reaper is per-daemon, not per-queue —
docker ps --filter name=pi-job- only sees its own machine, so two workers on two machines don't kill each other's containers (the documented single-worker invariant is per docker daemon; co-located workers remain forbidden).
- Container names embed globally-unique jobIds — no cross-host collision.
Gap 1 — divergent triggers files are a mutual-teardown loop
reconcile prunes every resident scheduler not named in this worker's config. Two workers with different triggers.json files each delete the other's schedulers on every boot and every file-watch reload. Idempotence only holds when the files are identical — and nothing checks that they are. Divergence must refuse loudly (e.g. a triggers-file hash in Valkey; mismatching worker refuses cron ownership) or reconcile must be owned by exactly one elected worker.
Gap 2 — local folders are host paths with no affinity
run.folder must exist on whichever host pops the job — but jobs land wherever BullMQ sends them. Worse, folder existence is validated at boot, so a worker whose host lacks one configured folder refuses to start entirely. Multi-host needs host affinity for local/cron work: name the host on the trigger and route (per-host queue or pop-time requeue), while github jobs stay any-host — their workspace is a fresh per-job clone.
Gap 3 — run history splits across hosts, invisibly
Each worker writes sidecars to its own logsDir; the admin reads exactly one, returns [] for anything missing, and shows a partial runs list with no hint that a second host exists. previousRunAt in a cron job's event.json scans the local dir only, so a scheduler alternating hosts reports the wrong previous fire. Records need a merged view. This grinds against DES-RUN-HISTORY-FLAT-FILES-NO-DB — state that tension openly: the least-new-machinery answer is mirroring the (already PII-free, already bounded) records into Valkey with a TTL, which is queue-state-shaped, not a database; the files stay the durable source on each host.
Gap 4 — per-host config drifts silently
The settings overlay, pause-windows file, and triggers file are all local files with per-host fs.watch; the admin writes to its own host's copy. Two hosts diverge silently on concurrency, caps, model, and quiet hours. Same medicine as Gap 1: one declared source of truth, or divergence detection that refuses instead of drifting.
Gap 5 — nothing knows which host did what
No host identity exists anywhere: run records carry no host field, log lines carry no host field, and getWorkers degrades to a bare count ("unknown" on providers without CLIENT SETNAME). Minimum viable: a worker name (env-declared, defaulting to hostname) stamped on log lines and run records (additive, nullable), shown in the runs list and RUN_DETAIL.
Gap 6 — image parity is unverified
pi-job:latest is a per-host local tag; doctor checks presence per host, and nothing notices two hosts running different digests — same flow, different behavior, undebuggable. Doctor (or the worker boot line) should surface the image digest so mismatch is at least visible.
Explicitly NOT proposed
- One worker driving remote Docker daemons (
DOCKER_HOST/contexts). DES-WORKER-ON-HOST's finding kills it: the docker CLI translates client-side host paths for bind mounts; a remote daemon doesn't — every /job, /workspace, and local-folder mount breaks. The superseded DES-JOB-FILES-VIA-VOLUME-SUBPATH is the record of that road.
- Kubernetes or any orchestrator. Two Macs and a queue is the target, and BullMQ already is the scheduler.
- Relaxing the one-worker-per-daemon invariant. Multi-host, not co-located workers.
Acceptance
Two machines, one Valkey: github jobs run on either host; a cron/local job only ever runs on the host that has its folder, and a missing-folder configuration fails that trigger's routing loudly instead of refusing worker boot for unrelated work; schedulers install once and divergent triggers files refuse rather than prune each other; the admin shows a merged runs view labeled by host, and cron previousRunAt is correct across hosts; budget/token caps and the kill switch remain global; doctor surfaces per-host image digests; a single-host deployment is byte-identical to today.
What
Two Mac minis (or Linux boxes) should be able to serve one pi-dispatch deployment: one queue, one budget, one admin view — sandboxes running on whichever host has capacity. The shape is a worker per machine sharing one Valkey, which the architecture already half-supports: BullMQ happily splits one queue across workers, and everything that lives in Valkey is already global. What's missing is everything that silently assumes one host.
What already works (don't redo this)
VALKEY_URLis remote-capable (host/port/auth/db all honored; the systemd unit even notes Valkey "may be remote"). Two workers on one queue split jobs correctly today.dispatch_runhourly limit are all Valkey state. Two workers share one cap regime for free.docker ps --filter name=pi-job-only sees its own machine, so two workers on two machines don't kill each other's containers (the documented single-worker invariant is per docker daemon; co-located workers remain forbidden).Gap 1 — divergent triggers files are a mutual-teardown loop
reconcileprunes every resident scheduler not named in this worker's config. Two workers with differenttriggers.jsonfiles each delete the other's schedulers on every boot and every file-watch reload. Idempotence only holds when the files are identical — and nothing checks that they are. Divergence must refuse loudly (e.g. a triggers-file hash in Valkey; mismatching worker refuses cron ownership) or reconcile must be owned by exactly one elected worker.Gap 2 — local folders are host paths with no affinity
run.foldermust exist on whichever host pops the job — but jobs land wherever BullMQ sends them. Worse, folder existence is validated at boot, so a worker whose host lacks one configured folder refuses to start entirely. Multi-host needs host affinity for local/cron work: name the host on the trigger and route (per-host queue or pop-time requeue), while github jobs stay any-host — their workspace is a fresh per-job clone.Gap 3 — run history splits across hosts, invisibly
Each worker writes sidecars to its own
logsDir; the admin reads exactly one, returns[]for anything missing, and shows a partial runs list with no hint that a second host exists.previousRunAtin a cron job'sevent.jsonscans the local dir only, so a scheduler alternating hosts reports the wrong previous fire. Records need a merged view. This grinds againstDES-RUN-HISTORY-FLAT-FILES-NO-DB— state that tension openly: the least-new-machinery answer is mirroring the (already PII-free, already bounded) records into Valkey with a TTL, which is queue-state-shaped, not a database; the files stay the durable source on each host.Gap 4 — per-host config drifts silently
The settings overlay, pause-windows file, and triggers file are all local files with per-host
fs.watch; the admin writes to its own host's copy. Two hosts diverge silently on concurrency, caps, model, and quiet hours. Same medicine as Gap 1: one declared source of truth, or divergence detection that refuses instead of drifting.Gap 5 — nothing knows which host did what
No host identity exists anywhere: run records carry no host field, log lines carry no host field, and
getWorkersdegrades to a bare count ("unknown" on providers without CLIENT SETNAME). Minimum viable: a worker name (env-declared, defaulting to hostname) stamped on log lines and run records (additive, nullable), shown in the runs list and RUN_DETAIL.Gap 6 — image parity is unverified
pi-job:latestis a per-host local tag;doctorchecks presence per host, and nothing notices two hosts running different digests — same flow, different behavior, undebuggable. Doctor (or the worker boot line) should surface the image digest so mismatch is at least visible.Explicitly NOT proposed
DOCKER_HOST/contexts).DES-WORKER-ON-HOST's finding kills it: the docker CLI translates client-side host paths for bind mounts; a remote daemon doesn't — every/job,/workspace, and local-folder mount breaks. The supersededDES-JOB-FILES-VIA-VOLUME-SUBPATHis the record of that road.Acceptance
Two machines, one Valkey: github jobs run on either host; a cron/local job only ever runs on the host that has its folder, and a missing-folder configuration fails that trigger's routing loudly instead of refusing worker boot for unrelated work; schedulers install once and divergent triggers files refuse rather than prune each other; the admin shows a merged runs view labeled by host, and cron
previousRunAtis correct across hosts; budget/token caps and the kill switch remain global; doctor surfaces per-host image digests; a single-host deployment is byte-identical to today.