Skip to content

Commit 141ba1d

Browse files
V3RONclaude
andauthored
feat(gateway,core,contract,daemon): fleet queue, routing, and lease + exec forwarding (ADR 0005, #118) (#130)
Third of the four PRs under [ADR 0005](https://github.com/callstackincubator/simlock/blob/claude/adr-0005-120-docs/docs/adr/0005-gateway-and-worker-modes.md), closing #118. Stacked on #129. A gateway stops being a read-only view of the fleet and starts routing work through it: one fleet-wide queue, a routing policy that picks the worker, and lease and `device.exec` calls forwarded to whichever worker owns the device. ## What lands **New modules under `src/gateway/`** - `queue.ts` — `FleetQueue`, a thin wrapper over `core/wait-queue.ts`'s `WaitQueue`. Reused rather than forked; `WaitQueue#list()` was added upstream because the fleet has to walk the whole FIFO to pass over a request no worker can serve, where the worker's single-resource model only ever advances the head. - `lease-index.ts` — `FleetLeaseIndex`, the gateway's record of the leases it issued. `rebuildFromWorker` is deliberately upsert-only so removal has exactly one source of truth (the worker's own `lease.released`/`lease.expired`), which is what keeps a fresh grant from racing a stale view refresh. - `routing.ts` — `RoutingPolicy` plus a registry mirroring `capacity/strategy.ts`, with one built-in `warm-then-free`: eligibility, then a warm matching device, then most free capacity (§13). - `fleet-coordinator.ts` — admission, dispatch and forwarding. All forwarding goes through one `#forwardToWorker` chokepoint. - `owner-routed-facts.ts` — replaces the inert placeholder, resolving a relayed fact's real owner from the index. **Reshaped:** `dispatcher.ts` (the six lease/exec operations become real; `lease.list`/`list.get` rewrite gateway-issued lease ids), `aggregate.ts`, `boundary.test.ts` (core is no longer wholesale forbidden — four modules are explicitly allowlisted, as that file's own comment asked a later PR to do), `core/config.ts` (`gateway.routing`), `contract/schemas.ts`, `daemon/dispatcher.ts`, `daemon/server.ts`, `daemon/main.ts`. ## Three things worth a reviewer's attention **The fleet-wide one-lease rule keys on two different fields.** Admission is `requesterId`-keyed (§14) and runs inside one `SerializedDecision` together with the enqueue, so two concurrent requests for one requester cannot both pass before either enqueues. Ownership authorization for renew/release/exec is `ownerId`-keyed (§26). §4's proxy pattern means one principal may hold leases under many requester ids, so conflating them would be wrong in both directions. **The dispatch race.** Dispatch re-runs on every view change (§11), so a waiter whose `lease.request` is still in flight to one worker can be picked up again and sent to another. The `requesterId` admission check cannot catch this — it runs once, at admission, before either RPC. `#dispatchTargets` marks a waiter before the RPC and the loop skips marked waiters, mirroring the worker coordinator's own `#driving` guard. **Ownership round-trips (§27a).** The gateway forwards the lease's owner explicitly and the worker stores it, so a rebuilt index authorizes to the same principal it did before a gateway restart. Without it `requesterId` survives via the `gw:<instance id>:` prefix but `ownerId` does not, and `ownsLease` treats an unrecognised lease as authorized — it would fail open. Only an admin session may set the field; omitting it keeps the previous behaviour, so the change is additive. ## Testing `pnpm check` green: typecheck, e2e typecheck, lint, format, unit, and 56 e2e passed / 1 expected fail / 9 skipped. Nine behaviours were each verified by reintroducing the bug and confirming the test failed, rather than by inspection. Two of those tests were rewritten after that check showed they were passing for the wrong reason — the first dispatch-race and pass-over tests went through the fast admission path, which never touches the visible queue, so the race they claimed to exercise could not occur. ## Deviations and follow-ups - `WorkerDispatchTarget` gained `refresh()`. Without it there is no way to satisfy §11's "the gateway refreshes that worker's view" after a stale-view `NO_CAPACITY` short of waiting for the next event or the periodic tick. `WorkerLink` already had a compatible method, so this is a pure interface addition. - Non-admin `lease.list` reads through the index filtered by `ownerId` rather than scanning raw worker leases against a namespaced principal, which closes an ownership-collision gap once real fleet leases exist. - Left for #119: `WORKER_UNREACHABLE` retry and the "dispatched, then uplink lost" path (`#forwardToWorker` is the seam to wrap); `lease.release-all` currently throws naming the first unreachable worker after attempting the rest, because its output has no room for a partial result; drain lifecycle guarantees and the full reconnect-rebuild e2e. - **Open, needs a decision:** ADR §15 says an operator must keep a gateway's `lease.maxTtlMs` at or below every worker's, but nothing enforces or warns, and `WorkerView` carries no worker TTL cap to check against. Surfacing it needs a contract and `worker-link` change beyond this issue's scope. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01MA98m7ua7qvDFZjxFaww6Z --- _Generated by [Claude Code](https://claude.ai/code/session_01MA98m7ua7qvDFZjxFaww6Z)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 323a9dc commit 141ba1d

66 files changed

Lines changed: 6473 additions & 210 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

AGENTS.md

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,13 +10,17 @@ changes in this repo:
1010

1111
- [architecture.md](docs/agent-rules/architecture.md) — loosely coupled
1212
modules; platform-agnostic core; iOS and Android encapsulated in their own
13-
driver modules; event bus for observers only.
13+
driver modules; event bus for observers only; one place enforces a rule;
14+
bounded cross-process waits; every exit leaves one named state.
1415
- [events.md](docs/agent-rules/events.md) — event naming
1516
(`subject.past-tense-fact`), post-commit emission, payload contracts,
1617
keeping EVENTS.md in sync.
1718
- [safety.md](docs/agent-rules/safety.md) — registry-only destruction, never
1819
touch leased devices, no implicit downloads, ownership proven not inferred,
19-
root validation fails closed.
20+
root validation fails closed, wire input is a claim not a fact.
21+
- [testing.md](docs/agent-rules/testing.md) — a test's title is a claim its
22+
body must prove; every test must be able to fail for the right reason;
23+
untested code is code you can delete with a green suite.
2024

2125
So are the accepted records in [docs/adr/](docs/adr/). An ADR marked
2226
_Accepted — not yet implemented_ means the documentation already describes the

docs/ARCHITECTURE.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -177,10 +177,10 @@ agent / console ──token auth──> │ HTTP frontend + unix socket
177177
`driver.passthrough` answer `UNSUPPORTED_IN_GATEWAY_MODE` permanently: they
178178
act on one machine's devices, and stay per-worker. The lease lifecycle
179179
(`lease.request`/`renew`/`release`/`cancel`/`release-all`) and `device.exec`
180-
answer the same code until the fleet queue and routing land; reads —
181-
`lease.list`,
182-
`list.get`, `status.get`, `catalog.get`, `events.*` already answer for the
183-
whole fleet.
180+
are forwarded through the fleet's own queue and routing policy (§10-§19,
181+
`FleetLeaseCoordinator`) rather than answering `UNSUPPORTED_IN_GATEWAY_MODE`;
182+
reads — `lease.list`, `list.get`, `status.get`, `catalog.get`, `events.*`
183+
answer for the whole fleet the same way.
184184
- **The one piece of persisted gateway state** is the drained set
185185
(`workers.json`, owner-only): drain is an operator's decision about a
186186
machine, not a fact the machine reports, so it must survive both the

docs/CONFIGURATION.md

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ a warning. Inspect the effective, merged configuration at any time with
2424
| `gateway.routing` | **Gateway side.** Which routing policy places a queued request on a worker. `warm-then-free` is the only policy in v1: warm hit first, then the most free running capacity for the platform. See [Routing](ARCHITECTURE.md#routing). | `warm-then-free` |
2525
| `gateway.disconnectedRetentionMs` | **Gateway side.** How long a disconnected worker is kept (greyed, never dispatched to) before the gateway forgets it. The clock is held while the gateway still knows of gateway-issued leases on that worker, and that hold ends when the last of those leases passes its deadline. | `24 hours` |
2626
| `gateway.execTimeoutMs` | **Gateway side.** How long the gateway waits on a proxied `device.exec` before giving up. A backstop for a worker that never answers at all — deliberately longer than the worker's own `exec.timeoutMs`, which is authoritative because that side owns the process and can kill it, so an ordinary timeout surfaces as the worker's `EXEC_TIMEOUT` rather than racing this one. See [ADR 0005](adr/0005-gateway-and-worker-modes.md) §19e. | `11 minutes` |
27+
| `gateway.leaseRequestTimeoutMs` | **Gateway side.** How long the gateway waits on a forwarded `lease.request` before giving up on that worker for this request, answering `WORKER_UNREACHABLE`. Bounds the one uplink call that otherwise had no timeout of its own, so a wedged worker cannot park a request where neither a deadline nor `lease.cancel` could ever reach it again. Generous against a cold device provision-plus-boot; well below `gateway.execTimeoutMs`, since granting a lease should never take as long as a command run against the device afterward. | `5 minutes` |
2728
| `http.enabled` | Master switch for the network-facing HTTP API (see [HTTP-API.md](HTTP-API.md)). Off by default; the daemon binds nothing until this is `true`. A gateway is the fleet's contact point, so it must be `true` there — see [Modes](#modes-gateway-and-worker). | `false` |
2829
| `http.host` | Address the HTTP listener binds. `127.0.0.1` keeps it loopback-only; reaching it remotely is the operator's own tunnel (Tailscale, cloudflared, reverse proxy) — Simlock does no TLS termination in v1. | `127.0.0.1` |
2930
| `http.port` | Port the HTTP listener binds. Must be an integer `1`-`65535`. | `4700` |
@@ -33,6 +34,7 @@ a warning. Inspect the effective, merged configuration at any time with
3334
| `gateway.label` | **Worker side.** Display name for this worker in the gateway's views. Need not be unique, and is never used to route. | unset (views show the worker's id) |
3435
| `gateway.disconnectedRetentionMs` | **Gateway side.** How long a disconnected worker's view is kept before the gateway forgets it. The clock only applies once every lease on that view has passed its deadline. | `24 hours` |
3536
| `gateway.execTimeoutMs` | **Gateway side.** Backstop on a proxied `device.exec`, deliberately longer than the worker's own `exec.timeoutMs`, which is the authoritative one. | `11 minutes` |
37+
| `gateway.leaseRequestTimeoutMs` | **Gateway side.** Backstop on a forwarded `lease.request` -- expiry answers `WORKER_UNREACHABLE`, freeing the request for the queue's own deadline/cancel handling again. | `5 minutes` |
3638
| `diskPressure.freeBytesThreshold` | Free disk space below which Simlock treats the machine as under disk pressure. | `10 GiB` |
3739
| `eventBuffer.capacity` | Number of business events kept in the in-memory ring buffer (see `simlock events`). | `1000` |
3840
| `health.enabled` | Master switch for leased-device crash detection and recovery. | `true` |
@@ -61,8 +63,9 @@ must be non-negative numbers (milliseconds and bytes, respectively).
6163
`http.enabled` is a boolean, `http.host` a string, and `http.port` an
6264
integer in `1`-`65535`.
6365
`mode` is `worker` or `gateway`; `gateway.url`, `gateway.token` and
64-
`gateway.label` are strings, and `gateway.disconnectedRetentionMs` and
65-
`gateway.execTimeoutMs` positive numbers.
66+
`gateway.label` are strings, and `gateway.disconnectedRetentionMs`,
67+
`gateway.execTimeoutMs`, and `gateway.leaseRequestTimeoutMs` positive
68+
numbers.
6669
`ios.slim.enabled` is a boolean, `ios.slim.categories` an array of
6770
non-empty strings, and `ios.slim.bootTimeoutMs` a positive number.
6871
`mode` must be exactly `"worker"` or `"gateway"`. `gateway.url` must be an
@@ -75,7 +78,8 @@ warned about and ignored like every other worker key — a gateway is not
7578
misconfigured by leftovers from the config it was flipped out of.
7679
`gateway.label` is a non-empty string, `gateway.routing` one of the
7780
registered routing policies, and `exec.timeoutMs`, `gateway.execTimeoutMs`,
78-
and `gateway.disconnectedRetentionMs` positive numbers.
81+
`gateway.leaseRequestTimeoutMs`, and `gateway.disconnectedRetentionMs`
82+
positive numbers.
7983
**`mode: "gateway"` with `http.enabled: false` is rejected at load** (ADR 0005 §2), naming the key: a
8084
gateway is the fleet's contact point over HTTP, so one nothing can reach has
8185
no safe reading.
@@ -180,7 +184,7 @@ does not apply to it. It reads:
180184
| Key group | Why |
181185
|---|---|
182186
| `mode` | to be a gateway at all |
183-
| `gateway.routing`, `gateway.disconnectedRetentionMs`, `gateway.execTimeoutMs` | how to run the fleet |
187+
| `gateway.routing`, `gateway.disconnectedRetentionMs`, `gateway.execTimeoutMs`, `gateway.leaseRequestTimeoutMs` | how to run the fleet |
184188
| `http.*` | it is the fleet's contact point |
185189
| `lease.*` | `defaultTtlMs`/`maxTtlMs` bound what its own clients may ask for, before a request is dispatched — see below |
186190
| `log.*`, `eventBuffer.*` | logging and the event ring buffer, as anywhere |

0 commit comments

Comments
 (0)