Merge pull request #164 from JarvusInnovations/develop

Release: v1.0.5

Author: themightychris

.github/workflows/publish-npm.yml

@@ -13,16 +13,11 @@ jobs:
     steps:
       - uses: actions/checkout@v6
 
-      - uses: actions/setup-node@v4
+      - uses: actions/setup-node@v6
         with:
-          node-version: '22.x'
+          node-version: '24'
           registry-url: 'https://registry.npmjs.org'
-
-      - name: Install npm via corepack (bypasses broken bundled npm in current Node 22 toolcache)
-        run: |
-          corepack enable
-          corepack install --global npm@latest
-          npm --version
+          package-manager-cache: false  # never use caching in release builds
 
       - name: Set package.json version from tag
         run: |

docs/concepts.md

@@ -158,8 +158,8 @@ const daemon = await repo.startPushDaemon({
 });
 
 daemon.on('push',  ({ commit, durationMs }) => log.info({ commit, durationMs }));
-daemon.on('error', ({ commit, err, attempt }) => log.warn({ err, attempt }));
-daemon.on('retry', ({ commit, attempt, nextDelayMs }) => log.info({ attempt, nextDelayMs }));
+daemon.on('error', ({ commit, err, attempt, reason }) => log.warn({ err, attempt, reason }));
+daemon.on('retry', ({ commit, attempt, nextDelayMs, reason }) => log.info({ attempt, nextDelayMs, reason }));
 
 // ... later, at shutdown:
 await daemon.stop({ timeoutMs: 30_000 });

docs/recipes/production-push-daemon.md

@@ -51,12 +51,16 @@ daemon.on('push', ({ commit, durationMs }) => {
   log.info({ commit, durationMs }, 'pushed');
 });
 
-daemon.on('retry', ({ commit, attempt, nextDelayMs }) => {
-  log.info({ commit, attempt, nextDelayMs }, 'push retry scheduled');
+daemon.on('retry', ({ commit, attempt, nextDelayMs, reason }) => {
+  log.info({ commit, attempt, nextDelayMs, reason }, 'push retry scheduled');
 });
 
-daemon.on('error', ({ commit, err, attempt }) => {
-  log.warn({ commit, err: String(err), attempt }, 'push failed');
+daemon.on('error', ({ commit, err, attempt, reason }) => {
+  // `reason: 'non-fast-forward'` means the remote has work this daemon
+  // doesn't — page someone. The daemon will *not* retry it; pushing the same
+  // commit again would just re-fail. Future commits still trigger fresh
+  // attempts (each producing its own classified error).
+  log.warn({ commit, err: String(err), attempt, reason }, 'push failed');
 });
 
 daemon.on('stopped', () => {
@@ -71,7 +75,7 @@ const status = daemon.status();
 // {
 //   running: boolean,
 //   lastPushAt: ISO 8601 | null,
-//   lastError: { message, at, attempt } | null,
+//   lastError: { message, at, attempt, reason: 'non-fast-forward' | 'unknown' } | null,
 //   pendingCommits: number,
 //   currentBackoffMs: number | null,
 //   currentAttempt: number | null,
@@ -161,22 +165,40 @@ process.on('SIGINT', shutdown);
 1. Stops accepting new commits to the queue
 2. Waits for the in-flight push to complete or fail
 3. Drains the retry queue up to `timeoutMs` (default 30s)
-4. After the timeout, abandons remaining queued commits (they stay in the local repo and are picked up on the next daemon start, once #157 lands)
+4. After the timeout, abandons remaining queued commits — they stay in the local repo and are picked up on the next daemon start via the startup-backlog check
 5. Resolves once the daemon is fully idle
 
 For Kubernetes / Docker, set `terminationGracePeriodSeconds` to at least `timeoutMs` + a safety margin so the orchestrator doesn't SIGKILL mid-drain.
 
 ## Non-fast-forward rejections
 
-Currently the daemon retries every failure including non-fast-forward rejections. That's the v1.0 simplification — production-grade detection (stop retrying, escalate to a human) is tracked at [#156](https://github.com/JarvusInnovations/gitsheets/issues/156). For now, if your remote rejects a push as non-fast-forward, the daemon spins on it until you intervene. Watch the `error` event stream and surface persistent failures.
+When the remote contains work the daemon doesn't (a fast-forward isn't possible), the push fails and the daemon classifies it as terminal:
 
-The right intervention when this happens: stop your consumer, investigate the remote (something else committed to your branch), reconcile, restart.
+- Exactly one `error` event fires with `reason: 'non-fast-forward'`.
+- **No `retry` event follows.** The daemon never force-pushes; pushing the same commit again would just re-fail.
+- `daemon.status().lastError.reason` carries `'non-fast-forward'` so health checks / alerting can branch on it.
+- Subsequent `repo.transact` commits still trigger fresh push attempts. Each will also fail with `non-fast-forward` (and emit its own event) until the remote state is reconciled.
 
-## Multi-remote replication (deferred)
+The right intervention: stop the consumer, investigate the remote (something else committed to your branch), reconcile (rebase or merge), restart.
 
-A `Repository` can host one push daemon at a time, and post-commit notifications fire only on the `Repository` instance that ran the `transact` — so a second `Repository` opened against the same `gitDir` wouldn't see those commits and wouldn't push them. In-process multi-remote replication isn't a working pattern in v1.0.
+```typescript
+daemon.on('error', ({ reason, err }) => {
+  if (reason === 'non-fast-forward') {
+    // page on-call; do not auto-restart
+    alerts.send({ severity: 'critical', message: 'push rejected — remote diverged' });
+  } else {
+    log.warn({ err: String(err) }, 'transient push failure');
+  }
+});
+```
 
-This becomes workable once [#157 (push daemon startup-diff)](https://github.com/JarvusInnovations/gitsheets/issues/157) lands and a daemon catches up from the ref state rather than from in-process notifications. Until then, the right replication path is external — e.g., a `git push backup` triggered by an external scheduler or by a server-side hook on the primary remote.
+## Multi-remote replication (still external)
+
+A `Repository` can host one push daemon at a time, and post-commit notifications fire only on the `Repository` instance that ran the `transact` — so a second `Repository` opened against the same `gitDir` wouldn't see live commits via the in-process hook. The startup-backlog check runs *once* per daemon (at start), so a long-lived second daemon won't catch up on commits made after it started.
+
+In-process multi-remote replication isn't the right pattern. For a backup remote, drive it externally: an external scheduler triggering `git push backup main`, or a server-side hook on the primary that mirrors to the backup.
+
+What the startup-backlog *does* unlock: if your process restarts (intentional deploy, crash, OOM kill) with commits ahead of the remote, the new daemon pushes them on startup without needing an explicit `repo.transact` to nudge it.
 
 ## A complete production setup
 
@@ -202,8 +224,11 @@ if (process.env.PUSH_REMOTE) {
     maxRetries: Infinity,
   });
   pushDaemon.on('push',  ({ commit, durationMs }) => log.info({ commit, durationMs }, 'pushed'));
-  pushDaemon.on('error', ({ err, attempt })       => log.warn({ err: String(err), attempt }, 'push failed'));
-  pushDaemon.on('retry', ({ attempt, nextDelayMs }) => log.info({ attempt, nextDelayMs }, 'push retry'));
+  pushDaemon.on('error', ({ err, attempt, reason }) => {
+    log.warn({ err: String(err), attempt, reason }, 'push failed');
+    if (reason === 'non-fast-forward') alerts.page({ message: 'gitsheets remote diverged' });
+  });
+  pushDaemon.on('retry', ({ attempt, nextDelayMs, reason }) => log.info({ attempt, nextDelayMs, reason }, 'push retry'));
   log.info({ remote: process.env.PUSH_REMOTE }, 'push daemon started');
 }
 

docs/recipes/request-bound-transactions.md

@@ -87,8 +87,8 @@ import { repo } from './store.js';
 
 if (process.env.NODE_ENV === 'production') {
   const daemon = await repo.startPushDaemon({ remote: 'origin' });
-  daemon.on('error', ({ err, attempt }) =>
-    console.error('[push-daemon]', err, 'attempt', attempt),
+  daemon.on('error', ({ err, attempt, reason }) =>
+    console.error('[push-daemon]', err, 'attempt', attempt, 'reason', reason),
   );
 }
 ```

specs/api/conventions.md

@@ -29,7 +29,7 @@ import {
 Type-only exports (interfaces, type aliases) flow alongside the value exports
 above. Notable ones: `TransactionResult`, `TransactionOptions`, `Author`,
 `SheetConfig`, `UpsertResult`, `IndexKeyFn`, `DefineIndexOptions`,
-`QueryFilter`, `OpenStoreOptions`, `Store`, `StoreTx`, `InferRecord`,
+`QueryFilter`, `QueryOptions`, `OpenStoreOptions`, `Store`, `StoreTx`, `InferRecord`,
 `PushDaemonOptions`, `PushDaemonStatus`, `BackoffConfig`, `JSONSchema`,
 `StandardSchemaV1`, `ValidationIssue`, `RecordLike`.
 
@@ -67,17 +67,19 @@ Every error thrown by gitsheets extends `GitsheetsError` and carries a stable `c
 
 ## Cancellation
 
-Long-running iterations (`Sheet.query`) honor `AbortSignal` when one is passed in via the options. Without a signal, they run to completion.
+Long-running iterations (`Sheet.query`, `Sheet.queryFirst`, `Sheet.queryAll`) honor `AbortSignal` when one is passed in via the options. Without a signal, they run to completion.
 
 ```typescript
 const controller = new AbortController();
 for await (const record of sheet.query({}, { signal: controller.signal })) {
   // ...
-  if (someCondition) controller.abort();
+  if (someCondition) controller.abort();  // optional reason arg → signal.reason
 }
 ```
 
-`AbortError` is thrown after the next yield point. (Defer if implementation cost is high for v1.0; documented here as the convention for any iterator that adds cancellation later.)
+When the signal aborts, the next iteration (or the call itself, if aborted before invocation) throws `signal.reason` — which is whatever value was passed to `controller.abort(reason)`, or a `DOMException` with name `'AbortError'` if no reason was supplied. Callers can `try/catch` on the iteration and switch on the thrown value's shape.
+
+Any iterator added later in the public surface should follow the same convention (`opts.signal`, check before iteration, check at each yield).
 
 ## Async iteration over trees
 

specs/api/sheet.md

@@ -25,15 +25,15 @@ for await (const user of sheet.query({ accountLevel: 'staff' })) {
 - `filter` is a plain object. Each key on the filter is matched against the record's field by equality. Function-valued filter entries are called as `(recordValue, record) => boolean`. Nested objects descend recursively.
 - If the filter includes fields from the path template, gitsheets prunes the tree traversal to only matching subtrees (see [behaviors/path-templates.md](../behaviors/path-templates.md)).
 - Order of results: filesystem order within each tree level. Not guaranteed stable across implementation changes — sort in consumer code if order matters.
-- `opts.signal?: AbortSignal` — see [api/conventions.md](conventions.md#cancellation).
+- `opts.signal?: AbortSignal` — cancel a streaming query. The query checks the signal before iteration starts and again before each yield. If aborted, the next iteration throws `signal.reason` (a `DOMException` with name `'AbortError'` by default, or whatever value the consumer passed to `controller.abort(reason)`). See [api/conventions.md](conventions.md#cancellation).
 
-### `sheet.queryFirst(filter?)`
+### `sheet.queryFirst(filter?, opts?)`
 
-Returns `Promise<T | undefined>`. The first match, or `undefined`.
+Returns `Promise<T | undefined>`. The first match, or `undefined`. Honors the same `opts.signal` as `query`.
 
-### `sheet.queryAll(filter?)`
+### `sheet.queryAll(filter?, opts?)`
 
-Returns `Promise<T[]>`. All matches collected into an array. Convenience over `for await ... push`.
+Returns `Promise<T[]>`. All matches collected into an array. Convenience over `for await ... push`. Honors the same `opts.signal` as `query`.
 
 ### `sheet.pathForRecord(record)`
 

specs/behaviors/push-sync.md

@@ -36,10 +36,12 @@ If a consumer needs to incorporate changes from elsewhere (a manual edit, a sepa
 
 The underlying operation is `git push <remote> <branch>` with no `--force`. **Regular fast-forward only.**
 
-If the remote rejects the push (non-fast-forward, "remote contains work that you do not have"), the daemon does not retry — it emits an `error` event and surfaces a hard error in `daemon.status()`. The consumer needs to intervene; a never-force-push policy is non-negotiable.
+If the remote rejects the push (non-fast-forward, "remote contains work that you do not have"), the daemon does not retry — it emits an `error` event with `reason: 'non-fast-forward'` and surfaces a hard error in `daemon.status()` (also with `lastError.reason === 'non-fast-forward'`). The consumer needs to intervene; a never-force-push policy is non-negotiable.
 
 The daemon does *not* attempt to merge or rebase. The local commit history is the source of truth.
 
+Subsequent `repo.transact` commits will trigger fresh push attempts. If the remote is still ahead, those attempts also fail and emit their own `non-fast-forward` errors — each commit gets one classified error, never a retry storm.
+
 ## Backoff
 
 Default: exponential, base 1 second, multiplier 2, cap 1 hour.
@@ -72,16 +74,20 @@ Or accept the default with `backoff: 'exponential'`.
 
 ```typescript
 daemon.on('push',  ({ commit, durationMs }) => { ... });
-daemon.on('retry', ({ commit, attempt, nextDelayMs }) => { ... });
-daemon.on('error', ({ commit, err, attempt }) => { ... });
+daemon.on('retry', ({ commit, attempt, nextDelayMs, reason }) => { ... });
+daemon.on('error', ({ commit, err, attempt, reason }) => { ... });
 daemon.on('stopped', () => { ... });
 ```
 
 - `push` — successful push. `durationMs` is wall-clock for the push attempt.
-- `retry` — a failed attempt has been scheduled for retry. Fires *before* the delay.
-- `error` — a failure occurred. May or may not be retried; if retried, a `retry` event follows.
+- `retry` — a failed attempt has been scheduled for retry. Fires *before* the delay. `reason` mirrors the upstream `error`'s classification.
+- `error` — a failure occurred. `reason` is one of:
+  - `'non-fast-forward'` — terminal. No `retry` event follows. Status' `lastError.reason` carries the same value.
+  - `'unknown'` — anything else (network, auth, transient git errors). Retried per the configured backoff unless `maxRetries` is exhausted.
 - `stopped` — emitted after `daemon.stop()` completes draining.
 
+For startup-backlog errors (see "Startup backlog" below) the `commit` field is `null` and `attempt` is `0`; the failure surfaces through the same `error` event channel.
+
 Consumers building observability surfaces typically attach to `push`, `retry`, `error` for Prometheus / structured-logging fan-out.
 
 ## Status
@@ -91,7 +97,12 @@ daemon.status();
 // → {
 //   running: boolean,
 //   lastPushAt: ISO 8601 | null,
-//   lastError: { message: string, at: ISO 8601, attempt: number } | null,
+//   lastError: {
+//     message: string,
+//     at: ISO 8601,
+//     attempt: number,
+//     reason: 'non-fast-forward' | 'unknown',
+//   } | null,
 //   pendingCommits: number,
 //   currentBackoffMs: number | null,        // current delay, if a retry is scheduled
 //   currentAttempt: number | null,           // which attempt we're on for the next commit
@@ -136,6 +147,16 @@ Pushing a commit that's already on the remote is a no-op (`Everything up-to-date
 
 If the daemon restarts and finds commits in the local repo that aren't on the remote, it pushes them. The daemon doesn't track a "pending" state across restarts — it diffs local vs. remote on startup and queues anything ahead.
 
+## Startup backlog
+
+`repo.startPushDaemon` returns the handle synchronously to the caller, then performs the startup-backlog check on the next event-loop tick (so consumers can attach `error` / `push` listeners first):
+
+1. `git fetch <remote> <branch>` — best-effort, populates `refs/remotes/<remote>/<branch>`. If this fails (unreachable remote, auth) the daemon emits an `error` event with `commit: null`, `attempt: 0`, classified `reason`. The check continues — a previously-cached remote-tracking ref may still be usable.
+2. `git rev-list --count <remote>/<branch>..<branch>` — counts local commits the remote doesn't have. If the remote-tracking ref doesn't exist after the fetch (e.g., the branch was never on the remote), the check exits silently; the daemon stays alive for future `notifyCommit` events.
+3. If the count is non-zero, the daemon primes its pending counter and immediately drains — the queued commits flow through the same push pipeline as live ones.
+
+The startup check **never throws**. The consumer's `await repo.startPushDaemon(opts)` resolves with the handle even when the remote is unreachable; observability comes through `error` events and `daemon.status()`.
+
 ## Push frequency
 
 The daemon pushes once per commit by default. For high-frequency commit workloads, this can be wasteful — a `debounceMs` option may be added later to coalesce multiple commits into a single push (still all individual commits, just a single network round-trip). Out of scope for v1.0.

specs/deferred.md

@@ -107,21 +107,6 @@ When in doubt about whether an entry belongs in `deferred.md`, the litmus test i
 - **What:** `infer` generates a starter `[gitsheet.schema]` from existing records; `migrate-config` converts pre-v1.0 `[gitsheet.fields]` configs.
 - **Why deferred:** Validation-config tooling that supplements #130's library work. Library APIs exist; CLI surface follows.
 
-### `Sheet.query({ signal })` — AbortSignal — [#154](https://github.com/JarvusInnovations/gitsheets/issues/154)
-
-- **What:** Cancel a streaming query via AbortSignal.
-- **Why deferred:** Async generators naturally support `break`; the AbortSignal pattern is sugar for upstream HTTP integrations. Documented in `api/conventions.md`; not implemented in the v1.0 surface.
-
-### PushDaemon: detect non-fast-forward rejection and stop retrying — [#156](https://github.com/JarvusInnovations/gitsheets/issues/156)
-
-- **What:** Inspect git stderr; on a non-fast-forward push rejection, emit a single `error` event with a structured reason and stop retrying that batch (the spec at `behaviors/push-sync.md` requires this, but the v1.0 implementation retries every failure).
-- **Why deferred:** The substrate ships with the simpler retry-all semantics; differentiating error classes is a meaningful behavior change that wants its own review.
-
-### PushDaemon: startup diff to push pre-existing local commits — [#157](https://github.com/JarvusInnovations/gitsheets/issues/157)
-
-- **What:** On `startPushDaemon`, diff `<remote>/<branch>..<branch>` and queue any commits ahead of the remote, so a restarted daemon catches up. Spec at `behaviors/push-sync.md` requires this.
-- **Why deferred:** Substrate v1.0 only queues commits surfaced via `notifyCommit` after the daemon starts. The startup-diff path needs `git fetch` orchestration + careful error handling for an unreachable remote.
-
 ## Dropped (no plan)
 
 ### `backend/server.js` HTTP server

src/index.ts

@@ -20,6 +20,7 @@ export type {
   IndexKeyFn,
   DefineIndexOptions,
   QueryFilter,
+  QueryOptions,
 } from './sheet.js';
 
 export { mergePatch } from './patch.js';