Merge pull request #166 from JarvusInnovations/develop

Release: v1.1.0

Author: themightychris

docs/api.md

@@ -33,9 +33,11 @@ import type {
   OpenRepoOptions, OpenSheetOptions, OpenSheetsOptions,
   TransactionOptions, TransactionResult, TransactionHandler, Author,
   SheetConfig, SheetFieldConfig, SortRule, UpsertResult,
-  SheetConstructorOptions, IndexKeyFn, DefineIndexOptions, QueryFilter,
+  SheetConstructorOptions, IndexKeyFn, DefineIndexOptions, QueryFilter, QueryOptions,
+  DiffStatus, DiffOptions, DiffChange,
+  AttachmentBlobHandle, AttachmentEntry,
   OpenStoreOptions, Store, StoreTx, StoreTransactFn, ValidatorMap, InferRecord,
-  PushDaemonOptions, PushDaemonStatus, BackoffConfig,
+  PushDaemonOptions, PushDaemonStatus, PushFailureReason, BackoffConfig,
   JSONSchema, StandardSchemaV1, ValidationIssue, StandardSchemaResult,
   RecordLike, PathTemplateBlob, PathTemplateTree, PathTemplateQueryResult,
 } from 'gitsheets';
@@ -89,9 +91,9 @@ const store = await openStore(repo, {
 
 | Method | Returns |
 | --- | --- |
-| `sheet.query(filter?)` | `AsyncGenerator<T>` — iterator |
-| `sheet.queryFirst(filter?)` | `Promise<T \| undefined>` |
-| `sheet.queryAll(filter?)` | `Promise<T[]>` |
+| `sheet.query(filter?, opts?)` | `AsyncGenerator<T>` — iterator; `opts.signal` for AbortSignal cancellation |
+| `sheet.queryFirst(filter?, opts?)` | `Promise<T \| undefined>` — honors `opts.signal` |
+| `sheet.queryAll(filter?, opts?)` | `Promise<T[]>` — honors `opts.signal` |
 | `sheet.pathForRecord(record)` | `Promise<string>` — rendered path, no write |
 | `sheet.normalizeRecord(record)` | `Promise<T>` — canonical form, no write |
 
@@ -115,6 +117,15 @@ All write methods route through a transaction — permissive mode auto-opens one
 | `sheet.getAttachments(record)` | Map of name → BlobObject |
 | `sheet.setAttachment(record, name, blob)` | Add or replace |
 | `sheet.setAttachments(record, map)` | Bulk variant |
+| `sheet.deleteAttachment(record, name)` | Remove a single attachment; throws `NotFoundError` if missing |
+| `sheet.deleteAttachments(record)` | Remove all attachments; idempotent no-op when no attachment dir |
+| `sheet.attachments(record)` | `AsyncGenerator<AttachmentEntry>` yielding `{name, mimeType, blob}` with `.read()` / `.stream()` |
+
+### Diff
+
+| Method | Returns |
+| --- | --- |
+| `sheet.diffFrom(srcCommitHash?, opts?)` | `AsyncGenerator<DiffChange<T>>` — per-record changes since `srcCommitHash` (defaults to the empty tree). `opts.blobs` / `opts.records` / `opts.patches` attach hologit blob handles, parsed src/dst records, and RFC 6902 JSON Patches respectively. Throws `RefError` if `srcCommitHash` doesn't resolve. |
 
 ### Indexing
 
@@ -173,15 +184,17 @@ Returned from `repo.startPushDaemon(opts)`. EventEmitter with:
 | Event | Payload |
 | --- | --- |
 | `push` | `{ commit, durationMs }` |
-| `retry` | `{ commit, attempt, nextDelayMs }` |
-| `error` | `{ commit, err, attempt }` |
+| `retry` | `{ commit, attempt, nextDelayMs, reason }` |
+| `error` | `{ commit, err, attempt, reason }` — `reason: 'non-fast-forward' \| 'unknown'`. NFF is terminal (no retry). |
 | `stopped` | (none) |
 
 | Method | Purpose |
 | --- | --- |
-| `daemon.status()` | Snapshot: running, lastPushAt, lastError, pendingCommits, currentBackoffMs, currentAttempt |
+| `daemon.status()` | Snapshot: running, lastPushAt, lastError (with `reason`), pendingCommits, currentBackoffMs, currentAttempt |
 | `daemon.stop({ timeoutMs })` | Drain in-flight retries (graceful) |
 
+On `startPushDaemon` the daemon also runs a one-shot startup-backlog check (fetch + `rev-list --count <remote>/<branch>..<branch>`) and queues any commits ahead of the remote; this is how a restarted daemon catches up.
+
 → [`specs/behaviors/push-sync.md`](https://github.com/JarvusInnovations/gitsheets/blob/develop/specs/behaviors/push-sync.md) · [production recipe](recipes/production-push-daemon.md)
 
 ## Errors
@@ -234,4 +247,4 @@ Read [`specs/api/conventions.md`](https://github.com/JarvusInnovations/gitsheets
 
 ## Stability
 
-Everything documented in [`specs/`](https://github.com/JarvusInnovations/gitsheets/tree/develop/specs) (and re-exported from `gitsheets`) is stable from v1.0 forward. Internal modules under `dist/` that aren't re-exported are implementation details and may change in minor releases.
+Everything documented in [`specs/`](https://github.com/JarvusInnovations/gitsheets/tree/develop/specs) (and re-exported from `gitsheets`) is stable from v1.0 onward. Additions in minor versions are additive (new methods, new options). Internal modules under `dist/` that aren't re-exported are implementation details and may change in minor releases.

docs/cli.md

@@ -12,12 +12,20 @@ operation on a `Repository` (or `Sheet`/`Transaction`).
 | --- | --- | --- |
 | `--git-dir <path>` | discovered from cwd | `GIT_DIR` |
 | `--root <path>` | `/` | `GITSHEETS_ROOT` |
+| `--prefix <path>` | none | `GITSHEETS_PREFIX` |
 | `--ref <ref>` | `HEAD` | `GITSHEETS_REF` |
 | `--commit-to <ref>` | derived from `--ref` | — |
 | `--message <msg>` | auto-generated | — |
 | `--author-name <name>`, `--author-email <email>` | git config | — |
 | `--trailer <Key>=<value>` (repeatable) | none | — |
 
+`--prefix` scopes records to a sub-tree under each sheet's configured root —
+useful for multi-tenant deployments where one git repo holds many tenants
+under `<root>/<tenant>/...`. With `--prefix tenant-a`, a sheet whose config
+declares `root = 'users'` reads/writes records at `users/tenant-a/<path>.toml`.
+The sheet's `.gitsheets/<name>.toml` config file is unaffected — only the
+record data tree is scoped.
+
 ## Commands
 
 ### `gitsheets upsert <sheet> [input]`
@@ -28,14 +36,38 @@ Insert or update one or more records.
 gitsheets upsert users users.json
 gitsheets upsert users - < records.jsonl
 gitsheets upsert users '{"slug":"jane","email":"jane@x.org"}'
+gitsheets upsert users users.csv --format=csv
+gitsheets upsert users users.toml --format=toml
 ```
 
 `input` is inline JSON (single record `{...}` or array `[{...}, ...]`),
 a file path, or `-` for stdin.
 
-(In v1.0 `upsert` accepts no extra flags beyond the global tx flags.
-A `--patch` flag for RFC 7396 merge semantics is deferred to [#149](https://github.com/JarvusInnovations/gitsheets/issues/149);
-use the library `Sheet.patch(query, partial)` for now.)
+Flags:
+
+- `--format <json|toml|csv>` — input format. Default: inferred from extension (`*.toml` → TOML, `*.csv` → CSV), otherwise JSON.
+- `--encoding <enc>` — encoding for file/stdin input. Default `utf8`. Accepts any Node `BufferEncoding`.
+- `--delete-missing` — **DESTRUCTIVE**: full-replace mode. After upserting every input record, deletes any record in the sheet whose path isn't in the input set, in the same transaction. If validation fails on any input record, the whole transaction rolls back and the tree is unchanged.
+- `--attachment <name>=<source>` (repeatable) — attach a file alongside the (single) upserted record in the same transaction. `<source>` is a file path (relative to the input file's directory, or cwd when input is stdin) or `-` for stdin (only one `-` per command). Requires a single-record input set.
+
+```bash
+gitsheets upsert users '{"slug":"jane"}' \
+  --attachment avatar.jpg=./assets/jane.jpg \
+  --attachment bio.md=-
+```
+
+- `--patch` — treat each input record as an [RFC 7396 JSON Merge Patch](https://datatracker.ietf.org/doc/html/rfc7396). Fields present in the sheet's path template are split out as the query (which existing record to find); the remaining fields are merged into that record. `null` deletes a field. Cannot be combined with `--delete-missing` or `--attachment`.
+
+```bash
+# Update jane's email; delete her bio; add a team field.
+gitsheets upsert users '{"slug":"jane","email":"new@x.org","bio":null,"team":"eng"}' --patch
+```
+
+For sheets whose template uses expressions (e.g., `${{ slug.toLowerCase() }}`), the CLI does a best-effort identifier scan to find the query fields. If that doesn't yield the right split, use the library API `Sheet.patch(query, partial)` directly.
+
+TOML input supports three layouts: a `[[records]]` array of tables (recommended for multi-record files), a top-level table where every value is itself a table (each value becomes one record), or a single-record document.
+
+CSV input uses the first row as a header. Cell values stay as strings — type coercion belongs in the consumer schema, not in CSV parsing.
 
 Output (per record):
 
@@ -45,30 +77,55 @@ Output (per record):
 
 ### `gitsheets query <sheet>`
 
-Read records. Output is newline-delimited JSON.
+Read records. Output is newline-delimited JSON by default.
 
 ```bash
 gitsheets query users
 gitsheets query users --filter email=jane@x.org
 gitsheets query users --filter status=active --filter project_id=p1
 gitsheets query users --fields slug email --limit 100
+gitsheets query users --format=csv --fields slug email
+gitsheets query users --format=toml > users.toml
 ```
 
 Flags:
 
 - `--filter <field>=<value>` (repeatable) — equality filter
-- `--fields <name>...` — output column subset
+- `--fields <name>...` — output column subset (order preserved)
 - `--limit <n>`
+- `--format <json|csv|tsv|toml>` — output format. Default `json` (newline-delimited).
+- `--headers` / `--no-headers` — emit a header row for CSV/TSV. Default `true`.
+
+TOML output emits a `[[records]]` array of tables, round-trippable through `upsert --format=toml`.
 
 ### `gitsheets read <sheet> <path>`
 
 Read a single record by its rendered path.
 
 ```bash
 gitsheets read users jane
+gitsheets read users jane --format=toml
 ```
 
-Output: pretty-printed JSON.
+Flags:
+
+- `--format <json|toml|csv|tsv>` — output format. Default: pretty-printed JSON.
+
+### `gitsheets edit <sheet> <path>`
+
+Open a record in `$EDITOR` for interactive editing. On save:
+
+1. The CLI re-reads the tmpfile, parses it as TOML.
+2. If the file is byte-identical to what was written out, no commit is made (idempotent — no empty commits).
+3. Otherwise the parsed record is upserted in a transaction; validation, normalization, and path rendering all run.
+
+Editor resolution: `$VISUAL` → `$EDITOR` → `vi`. The editor is spawned through a shell, so `EDITOR="code --wait"` and similar wrapped commands work. Exit-code 0 from the editor means "save"; anything else aborts without commit.
+
+```bash
+gitsheets edit users jane
+```
+
+Validation failures abort with a clear message rather than re-opening the editor — re-edit the input via `gitsheets read users <path> --format=toml` plus `upsert` if you need a second pass.
 
 ### `gitsheets normalize <sheet>`
 
@@ -80,6 +137,55 @@ or after a manual edit that didn't preserve canonical key order.
 gitsheets normalize users
 ```
 
+### `gitsheets init <sheet>`
+
+Scaffold `.gitsheets/<sheet>.toml` with sensible defaults. Useful as a first-step before bulk-importing records.
+
+```bash
+gitsheets init users
+gitsheets init users --path='${{ slug }}'
+gitsheets init users --schema=./schemas/user.schema.json
+```
+
+Defaults:
+
+- `root = <sheet>` (the sheet name)
+- `path = '${{ id }}'`
+
+Flags:
+
+- `--path <tpl>` — override the path template.
+- `--schema <file>` — embed a JSON Schema file under `[gitsheet.schema]`. The schema is validated through the same loader the runtime uses; an invalid schema aborts before the transaction commits.
+- `--force` — overwrite an existing `.gitsheets/<sheet>.toml`. Without `--force`, init refuses to clobber.
+
+### `gitsheets infer <sheet>`
+
+Scan every record in the sheet and write a conservative starter `[gitsheet.schema]` block into `.gitsheets/<sheet>.toml`. The committed schema captures observed types per field, plus `minimum`/`maximum` hints for numeric fields and `items.type` for arrays. Fields present in every record are listed under `required`.
+
+```bash
+gitsheets infer users
+```
+
+Existing `root`, `path`, and any non-schema `fields.<name>.sort` rules are preserved. The result is a starting point — review and tighten (add `pattern`, `format`, etc.) before relying on it.
+
+### `gitsheets migrate-config <sheet>`
+
+Convert a pre-v1.0 `[gitsheet.fields]` config to a v1.0 `[gitsheet.schema]` config in one transaction.
+
+```bash
+gitsheets migrate-config orders
+```
+
+Migration table (matches [`specs/behaviors/validation.md`](https://github.com/JarvusInnovations/gitsheets/blob/develop/specs/behaviors/validation.md#migration-of-pre-v10-gitsheetfields-config)):
+
+| Pre-v1.0 field | v1.0 placement |
+| --- | --- |
+| `type: 'number' \| 'string' \| 'boolean'` | `[gitsheet.schema.properties.<name>].type` |
+| `enum: [...]` | `[gitsheet.schema.properties.<name>].enum` |
+| `default: <value>` | `[gitsheet.schema.properties.<name>].default` |
+| `sort: ...` | `[gitsheet.fields.<name>.sort]` (unchanged) |
+| `trueValues: [...]`, `falseValues: [...]` | Warning to stderr — move to CSV-ingest helper |
+
 ## Exit codes
 
 | Code | Meaning |
@@ -105,13 +211,3 @@ gitsheets: ValidationError: record failed JSON Schema validation
   code:   validation_failed
   issue:  email: must match format "email" (json-schema)
 ```
-
-## Deferred commands
-
-These are documented in [`specs/api/cli.md`](https://github.com/JarvusInnovations/gitsheets/blob/develop/specs/api/cli.md)
-but not shipped in the v1.0 substrate; they land in follow-up PRs:
-
-- `gitsheets edit <sheet> <path>` — open in `$EDITOR`
-- `gitsheets infer <sheet>` — generate a starter JSON Schema from records
-- `gitsheets migrate-config <sheet>` — convert legacy `[gitsheet.fields]`
-- `gitsheets init <sheet>` — scaffold a new sheet config (#139)

docs/migration-guide.md

@@ -27,7 +27,7 @@ If you've been running a pre-v1.0 internal install, here's what changes and how
 | `sheet.delete(query)` | `sheet.delete(query)` (same name) |
 | `sheet.query(filter)` | `sheet.query(filter)` async iterator (same name) |
 | `sheet.commit(message)` | `repo.transact({ message }, async tx => { await tx.sheet(...).upsert(...) })` |
-| `--patch-existing` (CLI, `deepmerge`) | `--patch` (CLI, RFC 7396) — deferred to [#149](https://github.com/JarvusInnovations/gitsheets/issues/149); use library `sheet.patch(query, partial)` for now |
+| `--patch-existing` (CLI, `deepmerge`) | `--patch` (CLI, RFC 7396 — semantic change: `null` deletes, arrays replace) |
 | HTTP `POST /sheets/:name` | Your own HTTP layer + `sheet.upsert(...)` |
 
 ## Permissive vs strict mode
@@ -147,11 +147,16 @@ Every gitsheets error extends `GitsheetsError` and carries a stable `code`. See
 
 ## CLI changes
 
-- `git sheet upsert` / `query` / `read` / `normalize` — same command shape, rebuilt against the new core (`edit` deferred to [#150](https://github.com/JarvusInnovations/gitsheets/issues/150))
-- `--patch-existing` → `--patch` (RFC 7396 semantics, deferred — use library `Sheet.patch` for now)
+- `git sheet upsert` / `query` / `read` / `normalize` — same command shape, rebuilt against the new core
+- `git sheet edit <sheet> <path>` — new in v1.1: `$EDITOR`-based round-trip on a record
+- `git sheet init <sheet>` — new in v1.1: scaffold a starter `.gitsheets/<sheet>.toml`
+- `git sheet infer <sheet>` / `migrate-config <sheet>` — new in v1.1: schema inference and pre-v1.0 fields-config migration
+- `--patch-existing` → `--patch` (RFC 7396 semantics: `null` deletes, arrays replace)
+- `--format` (`json|toml|csv` upsert; `json|csv|tsv|toml` query/read), `--encoding`, `--delete-missing`, `--attachment <name>=<source>` — all shipped in v1.1
+- `--prefix` (`GITSHEETS_PREFIX`) for multi-tenant sub-tree scoping
 - New global flags: `--message`, `--author-name`, `--author-email`, `--trailer Key=Value`, `--ref`, `--commit-to`
 - Exit codes are stable from v1.0 onward — see [CLI reference](cli.md#exit-codes)
-- `--format` (CSV / TOML), `--encoding`, `--delete-missing`, `--attachments` deferred to v1.x
+- `--working` (read/write the working tree state) remains deferred — tracked at [#165](https://github.com/JarvusInnovations/gitsheets/issues/165)
 
 ## Going forward
 

docs/recipes/migrating-config.md

@@ -2,7 +2,13 @@
 
 Pre-v1.0 gitsheets used a `[gitsheet.fields]` block for typed field configuration. v1.0 replaces it with a proper `[gitsheet.schema]` JSON Schema block. The mapping is lossless for `type` / `enum` / `default`; `sort` stays where it is (it's a different concept); `trueValues` / `falseValues` move to a CSV-ingest helper.
 
-> The `gitsheets migrate-config <sheet>` CLI command that automates this is tracked at [#151](https://github.com/JarvusInnovations/gitsheets/issues/151) — until that lands, follow this guide manually.
+The `gitsheets migrate-config <sheet>` CLI command (shipped in v1.1) automates this:
+
+```bash
+gitsheets migrate-config users --message='migrate users config to v1.0 schema'
+```
+
+This guide walks through what the command does step-by-step, useful when you want to understand the resulting config or apply migrations by hand.
 
 ## Mapping table
 
@@ -233,6 +239,6 @@ gitsheets: ValidationError: record failed JSON Schema validation
 ## See also
 
 - [Validation](../validation.md) — full pipeline
-- [Issue #151](https://github.com/JarvusInnovations/gitsheets/issues/151) — `gitsheets migrate-config` CLI (automated migration)
+- [`gitsheets migrate-config`](../cli.md#gitsheets-migrate-config-sheet) — the CLI command that automates this whole flow
 - [`specs/behaviors/validation.md`](https://github.com/JarvusInnovations/gitsheets/blob/develop/specs/behaviors/validation.md)
 - [`specs/behaviors/normalization.md`](https://github.com/JarvusInnovations/gitsheets/blob/develop/specs/behaviors/normalization.md)

docs/validation.md

@@ -149,7 +149,7 @@ interface ValidationIssue {
 | `enum: [...]` | `[gitsheet.schema.properties.<name>].enum` |
 | `default: <value>` | `[gitsheet.schema.properties.<name>].default` |
 | `sort: ...` | `[gitsheet.fields.<name>.sort]` (unchanged — different concept) |
-| `trueValues` / `falseValues` | Moves to a CSV-ingest helper (deferred) |
+| `trueValues` / `falseValues` | Out of scope for the validation layer — `gitsheets migrate-config` emits a warning; CSV-input boolean coercion stays a consumer concern |
 
 See the [migrating-config recipe](recipes/migrating-config.md) for a worked example.
 

package-lock.json

@@ -44,7 +44,10 @@
     "@iarna/toml": "^2.2.5",
     "ajv": "^8.20.0",
     "ajv-formats": "^3.0.1",
+    "csv-parse": "^6.2.1",
+    "csv-stringify": "^6.7.0",
     "hologit": "^0.49.1",
+    "rfc6902": "^5.2.0",
     "sort-keys": "^6.0.0",
     "yargs": "^18.0.0"
   },