Release: v2.3.0

docs/api.md

@@ -80,9 +80,12 @@ const store = await openStore(repo, {
 | `repo.openSheet<T>(name, opts?)` | Sheet handle; `opts.validator` (Standard Schema), `opts.root`, `opts.prefix` |
 | `repo.openSheets(opts?)` | All sheets keyed by name; `opts.root`, `opts.prefix` |
 | `repo.transact(opts, handler)` | Run a handler in a single-commit transaction |
+| `repo.withLock(fn)` | Run `fn` holding the write lock transact uses — coordinate non-transact git ops. Not reentrant (`lock_held`) |
 | `repo.requireExplicitTransactions()` | Switch to strict mode (one-way) |
 | `repo.startPushDaemon(opts)` | Start async push to a configured remote |
 | `repo.resolveRef(ref)` | Resolve a ref or commit hash; returns `string \| null` |
+| `repo.refresh()` | Rebind every live Sheet to the current HEAD tree — for out-of-band ref movement (a successful `transact` auto-refreshes) |
+| `repo.readBlobStream(ref, path)` | `Promise<Readable>` — stream a blob's bytes at `<ref>:<path>`, resolved at call time. Throws `RefError` / `NotFoundError` |
 
 `opts.prefix` scopes records to a sub-tree under each sheet's configured root — useful for multi-tenant deployments where one git repo holds many tenants under `<root>/<tenant>/...`. Mirrors the CLI `--prefix` flag (env `GITSHEETS_PREFIX`). The sheet's `.gitsheets/<name>.toml` config file is unaffected — only the record data tree is scoped.
 
@@ -100,6 +103,7 @@ const store = await openStore(repo, {
 | `sheet.queryFirst(filter?, opts?)` | `Promise<T \| undefined>` — honors `opts.signal`, `opts.withBody` |
 | `sheet.queryAll(filter?, opts?)` | `Promise<T[]>` — honors `opts.signal`, `opts.withBody` |
 | `sheet.loadBody(record)` | `Promise<T>` — hydrate a body-less record (content-typed sheets) |
+| `sheet.refresh()` | Rebind this sheet's read snapshot to the current HEAD tree (out-of-band movement; own commits auto-refresh) |
 | `sheet.pathForRecord(record)` | `Promise<string>` — rendered path, no write |
 | `sheet.normalizeRecord(record)` | `Promise<T>` — canonical form, no write |
 
@@ -122,10 +126,11 @@ All write methods route through a transaction — permissive mode auto-opens one
 
 | Method | Purpose |
 | --- | --- |
-| `sheet.getAttachment(record, name)` | One attachment's BlobObject or null |
-| `sheet.getAttachments(record)` | Map of name → BlobObject |
-| `sheet.setAttachment(record, name, blob)` | Add or replace |
-| `sheet.setAttachments(record, map)` | Bulk variant |
+| `sheet.getAttachment(record, name)` | One attachment's `AttachmentBlobHandle` or null |
+| `sheet.getAttachments(record)` | Map of name → `AttachmentBlobHandle` |
+| `sheet.getAttachmentStream(recordOrPath, name)` | `Promise<Readable \| null>` — stream an attachment's bytes without materializing the record |
+| `sheet.setAttachment(record, name, content)` | Add or replace — `content` may be raw bytes (`Buffer`/`Uint8Array`), a UTF-8 `string`, or a `BlobHandle` |
+| `sheet.setAttachments(record, map)` | Bulk variant; values accept the same types, mixed freely |
 | `sheet.deleteAttachment(record, name)` | Remove a single attachment; throws `NotFoundError` if missing |
 | `sheet.deleteAttachments(record)` | Remove all attachments; idempotent no-op when no attachment dir |
 | `sheet.attachments(record)` | `AsyncGenerator<AttachmentEntry>` yielding `{name, mimeType, blob}` with `.read()` / `.stream()` |
@@ -177,10 +182,13 @@ type Store<V extends ValidatorMap> = {
   readonly [K in keyof V]: Sheet<InferRecord<V[K]>>;
 } & {
   readonly transact: StoreTransactFn<V>;
+  readonly refresh: () => Promise<void>;
 };
 ```
 
-`store.<sheet>` for each sheet declared in `validators`. `store.transact(opts, async tx => ...)` mirrors `repo.transact` with `tx.<sheet>` aliases that thread validators through.
+`store.<sheet>` for each sheet declared in `validators`. `store.transact(opts, async tx => ...)` mirrors `repo.transact` with `tx.<sheet>` aliases that thread validators through. `store.refresh()` rebinds every sheet to the current HEAD tree (delegates to `repo.refresh()`).
+
+Reads through `store.<sheet>` follow the freshness model: a successful `store.transact` / `repo.transact` auto-refreshes, so post-commit reads reflect the committed state — see [`specs/behaviors/freshness.md`](https://github.com/JarvusInnovations/gitsheets/blob/develop/specs/behaviors/freshness.md).
 
 Sheets not in `validators` are accessible via `repo.openSheet(name)` for one-off un-typed access.
 
@@ -224,7 +232,7 @@ Subclasses:
 | --- | --- |
 | `ConfigError` | `config_missing`, `config_invalid` |
 | `ValidationError` | `validation_failed` (carries `issues: ValidationIssue[]`) |
-| `TransactionError` | `transaction_in_progress`, `transaction_required`, `parent_moved`, `commit_failed`, `push_daemon_running`, `transaction_closed` |
+| `TransactionError` | `transaction_in_progress`, `transaction_required`, `parent_moved`, `commit_failed`, `push_daemon_running`, `transaction_closed`, `lock_held` |
 | `IndexError` | `index_unique_conflict`, `index_not_defined` (carries `conflictingPaths`) |
 | `RefError` | `ref_not_found`, `not_an_ancestor` |
 | `PathTemplateError` | `path_render_failed`, `path_invalid_chars` |

docs/migration-guide.md

@@ -216,6 +216,107 @@ Patch release. Two fixes in service of snapshot-importer-style workflows.
 
 - `hologit` bumped from `^0.49.1` to `^0.50.2` to pick up `TreeObject.clearChildren()` (the O(1) primitive backing the new `Sheet#clear()`).
 
+## v1.x → v2.0 (the Rust core)
+
+v2.0 replaces the Node.js engine with the Rust `gitsheets-core` crate: TOML
+parse/serialize, normalization, validation, path templates, record CRUD, and
+the tree/blob/commit substrate all run natively, and the npm package becomes a
+thin marshalling shell over the `@gitsheets/core-napi` addon. The API surface
+is intentionally unchanged — but three behavior changes bite real consumers.
+All three surfaced during the first production 1.4.1 → 2.x upgrade; this
+section is what turns "16 mystery test failures" into "read the section, apply
+3 changes."
+
+### 1. The `hologit` dependency is gone
+
+2.x drops `hologit` entirely (tree/blob/commit ops run on `holo-tree`/gitoxide
+*inside* the core). Anything that reached into it breaks:
+
+- `repo.hologitRepo` no longer exists.
+- `BlobObject` (e.g. `BlobObject.write`) is no longer importable via gitsheets.
+
+The replacement for the common pattern — hashing binary content and attaching
+it to a record — is the built-in blob primitive plus the attachment API:
+
+```typescript
+// 1.x
+import { BlobObject } from 'hologit';
+
+const blob = await BlobObject.write(repo.hologitRepo, buffer);
+await sheet.setAttachments(record, { 'avatar.jpg': blob });
+```
+
+```typescript
+// 2.x
+const blob = await repo.writeBlob(buffer); // Promise<BlobHandle>
+await sheet.setAttachments(record, { 'avatar.jpg': blob });
+```
+
+`repo.writeBlob(buf)` hashes the bytes into the object database and returns a
+`BlobHandle` accepted everywhere a hologit `BlobObject` used to be
+(`setAttachment`, `setAttachments`). `getAttachment`/`getAttachments`/
+`sheet.attachments()` likewise return `BlobHandle`s with `.read()`/`.stream()`.
+
+### 2. `null`/`undefined`-valued fields
+
+TOML has no `null`, so a cleared optional field and a never-set field are the
+same on-disk state: an absent key. 1.x (`@iarna/toml`) enforced this by
+silently dropping null-valued keys at serialize time. **The initial 2.x
+releases instead threw on marshal** (`cannot marshal JS value of type
+Null/Undefined to a TOML value`) — breaking the standard consumer pattern of
+`.nullable().optional()` schemas with `?? null` normalization on write.
+
+**Fixed in this release**: the 1.x drop semantics are restored and now
+specced ([`specs/behaviors/normalization.md`](https://github.com/JarvusInnovations/gitsheets/blob/develop/specs/behaviors/normalization.md#null--undefined-handling)).
+A `null`/`undefined`-valued key is dropped, recursively (top-level fields,
+nested tables, and objects inside arrays), before validation and
+serialization — byte-identical to 1.x output, in every binding (Node and
+Python). If you shimmed this with a `stripNullish` helper at your write
+boundary, you can delete it.
+
+One deliberate edge-case divergence from 1.x: a `null`/`undefined` **array
+element** is an error naming the index (1.x silently dropped elements —
+`[1, null, 2]` → `[1, 2]` — which shifts sibling indices and silently changes
+data). Remove the element yourself if that's what you mean. A required field
+set to `null` fails JSON-Schema validation as *missing*, same as 1.x.
+
+### 3. Canonical bytes re-baseline (one-time)
+
+The canonical serializer is now the core's `gitsheets-core::canonical` (the
+Rust `toml` crate's formatting over a deep key sort), replacing `@iarna/toml`.
+A record that was already canonical under 1.x may re-serialize to different —
+but *value-identical* — bytes, once ([#196](https://github.com/JarvusInnovations/gitsheets/issues/196)).
+Three reformat classes, all proven data-lossless and idempotent over a
+~29.5k-record corpus ([PR #205](https://github.com/JarvusInnovations/gitsheets/pull/205)):
+
+1. **Integer digit-group underscores drop** — `legacyId = 31_618` →
+   `legacyId = 31618` (the dominant class).
+2. **String requote** — strings containing both `"` and `'` move from escaped
+   single-line form to readable triple-quoted `"""…"""` form.
+3. **Multiline trailing-quote layout** — a multiline string ending in `"`
+   loses `@iarna`'s line-continuation dance (same value, fewer lines).
+
+Until you re-baseline, the first 2.x write of a record whose 1.x bytes fall in
+one of these classes shows a spurious-looking (but value-neutral) diff. The
+recommended migration is a **one-time re-serialize commit** over each existing
+repo, which is idempotent (a second run produces zero diff — that's the
+adoption check):
+
+```bash
+# Re-normalize every *.toml record under a directory, in place.
+cargo run -p gitsheets-core --example normalize_tree -- path/to/records
+
+git add path/to/records
+git commit -m "chore: re-baseline records to the Rust canonical form"
+
+# Verify idempotence — a second pass must report 0 files re-normalized:
+cargo run -p gitsheets-core --example normalize_tree -- path/to/records
+```
+
+(Or use `git sheet normalize <sheet>` per sheet from the CLI.) See the
+[canonical-form re-baseline notes](https://github.com/JarvusInnovations/gitsheets/blob/develop/specs/behaviors/normalization.md#canonical-form-re-baseline-the-rust-serializer)
+for the full contract.
+
 ## Going forward
 
 Once migrated, the recipes are the fastest path to common patterns:

docs/validation.md

@@ -98,6 +98,8 @@ Internally the library calls `validator['~standard'].validate(record)`. The retu
 
 The Standard Schema layer is **optional**. Without it, only JSON Schema runs.
 
+Gitsheets' declared `StandardSchemaV1` types mirror the published Standard Schema v1 interface exactly, so a compliant validator (Zod v4, Valibot, ArkType, Effect Schema) assigns to `openSheet({ validator })` / `openStore({ validators })` directly — no `as` cast or wrapper needed.
+
 For full type inference across reads and writes, use [`openStore`](concepts.md#store) instead of `openSheet` — see the [typed-sheet-with-Zod recipe](recipes/typed-sheet-with-zod.md).
 
 ## Order of operations

packages/gitsheets/package.json

@@ -53,6 +53,7 @@
     "@types/yargs": "^17.0.35",
     "tmp": "^0.2.5",
     "typescript": "^6.0.3",
-    "vitest": "^4.1.6"
+    "vitest": "^4.1.6",
+    "zod": "^4.4.3"
   }
 }

packages/gitsheets/src/errors.ts

@@ -21,6 +21,7 @@ const STATUS_BY_CODE = {
   index_not_defined: 500,
   push_daemon_running: 409,
   transaction_closed: 409,
+  lock_held: 409,
   ref_not_found: 404,
   not_an_ancestor: 409,
   path_render_failed: 422,
@@ -77,7 +78,8 @@ export type TransactionErrorCode =
   | 'parent_moved'
   | 'commit_failed'
   | 'push_daemon_running'
-  | 'transaction_closed';
+  | 'transaction_closed'
+  | 'lock_held';
 
 export class TransactionError extends GitsheetsError {
   constructor(code: TransactionErrorCode, message: string, options?: GitsheetsErrorOptions) {

packages/gitsheets/src/index.ts

@@ -27,6 +27,7 @@ export type {
   DiffOptions,
   DiffChange,
   AttachmentBlobHandle,
+  AttachmentContent,
   AttachmentEntry,
 } from './sheet.js';
 
@@ -82,6 +83,8 @@ export { validateRecord } from './validation.js';
 export type {
   JSONSchema,
   StandardSchemaV1,
+  StandardSchemaProps,
+  StandardSchemaTypes,
   StandardSchemaIssue,
   StandardSchemaResult,
   StandardSchemaFailure,

packages/gitsheets/src/repo-blob-stream.test.ts

@@ -0,0 +1,138 @@
+// Streaming blob reads by key/path — repo.readBlobStream + sheet.getAttachmentStream.
+// See specs/behaviors/attachments.md#streaming-reads-by-keypath.
+
+import { mkdir, writeFile } from 'node:fs/promises';
+import { join } from 'node:path';
+import type { Readable } from 'node:stream';
+
+import { afterEach, describe, expect, it } from 'vitest';
+
+import { NotFoundError, RefError } from './errors.js';
+import { openRepo, type Repository } from './repository.js';
+import { testRepo, type TestRepoHandle } from './test-helpers/test-repo.js';
+
+const handles: TestRepoHandle[] = [];
+afterEach(async () => {
+  while (handles.length > 0) {
+    const h = handles.pop();
+    if (h) await h.cleanup();
+  }
+});
+
+const USERS = `[gitsheet]
+root = 'users'
+path = '\${{ slug }}'
+`;
+
+async function seededRepo(): Promise<{ fixture: TestRepoHandle; repo: Repository }> {
+  const fixture = await testRepo({ withInitialCommit: true });
+  handles.push(fixture);
+  await mkdir(join(fixture.path, '.gitsheets'), { recursive: true });
+  await writeFile(join(fixture.path, '.gitsheets', 'users.toml'), USERS);
+  await fixture.git('add', '.gitsheets/');
+  await fixture.git('commit', '-m', 'add users sheet');
+  const repo = await openRepo({ gitDir: fixture.gitDir });
+  return { fixture, repo };
+}
+
+async function collect(stream: Readable): Promise<Buffer> {
+  const chunks: Buffer[] = [];
+  for await (const chunk of stream) {
+    chunks.push(chunk as Buffer);
+  }
+  return Buffer.concat(chunks);
+}
+
+/** Binary bytes (not valid UTF-8) to prove byte fidelity end-to-end. */
+const BINARY = Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x00, 0xff, 0xfe, 0x0a, 0x1b]);
+
+async function commitAvatar(repo: Repository, bytes: Buffer): Promise<void> {
+  await repo.transact({ message: 'avatar' }, async (tx) => {
+    const sheet = tx.sheet('users');
+    await sheet.upsert({ slug: 'jane' });
+    const blob = await repo.writeBlob(bytes);
+    await sheet.setAttachment('jane', 'avatar.png', blob);
+  });
+}
+
+describe('repo.readBlobStream', () => {
+  it('streams byte-identical content for a committed attachment', async () => {
+    const { repo } = await seededRepo();
+    await commitAvatar(repo, BINARY);
+
+    const stream = await repo.readBlobStream('HEAD', 'users/jane/avatar.png');
+    expect(Buffer.compare(await collect(stream), BINARY)).toBe(0);
+  });
+
+  it('resolves the ref at call time — a later commit is immediately visible', async () => {
+    const { repo } = await seededRepo();
+    await commitAvatar(repo, Buffer.from('v1'));
+    await commitAvatar(repo, Buffer.from('v2'));
+
+    const stream = await repo.readBlobStream('HEAD', 'users/jane/avatar.png');
+    expect((await collect(stream)).toString()).toBe('v2');
+  });
+
+  it('throws NotFoundError(record_not_found) for a missing path', async () => {
+    const { repo } = await seededRepo();
+    const err = await repo.readBlobStream('HEAD', 'users/nope/avatar.png').catch((e: unknown) => e);
+    expect(err).toBeInstanceOf(NotFoundError);
+    expect((err as NotFoundError).code).toBe('record_not_found');
+  });
+
+  it('throws NotFoundError(record_not_found) for a directory path', async () => {
+    const { repo } = await seededRepo();
+    await commitAvatar(repo, BINARY);
+    const err = await repo.readBlobStream('HEAD', 'users/jane').catch((e: unknown) => e);
+    expect(err).toBeInstanceOf(NotFoundError);
+  });
+
+  it('throws RefError(ref_not_found) for a ref that does not resolve', async () => {
+    const { repo } = await seededRepo();
+    const err = await repo.readBlobStream('no-such-ref', 'users/jane/avatar.png').catch((e: unknown) => e);
+    expect(err).toBeInstanceOf(RefError);
+    expect((err as RefError).code).toBe('ref_not_found');
+  });
+});
+
+describe('sheet.getAttachmentStream', () => {
+  it('streams the attachment bytes by record path without materializing the record', async () => {
+    const { repo } = await seededRepo();
+    await commitAvatar(repo, BINARY);
+    const users = await repo.openSheet('users');
+
+    const stream = await users.getAttachmentStream('jane', 'avatar.png');
+    expect(stream).not.toBeNull();
+    expect(Buffer.compare(await collect(stream!), BINARY)).toBe(0);
+  });
+
+  it('accepts a record object too', async () => {
+    const { repo } = await seededRepo();
+    await commitAvatar(repo, BINARY);
+    const users = await repo.openSheet('users');
+
+    const stream = await users.getAttachmentStream({ slug: 'jane' }, 'avatar.png');
+    expect(Buffer.compare(await collect(stream!), BINARY)).toBe(0);
+  });
+
+  it('returns null when the attachment is absent', async () => {
+    const { repo } = await seededRepo();
+    await commitAvatar(repo, BINARY);
+    const users = await repo.openSheet('users');
+
+    expect(await users.getAttachmentStream('jane', 'missing.png')).toBeNull();
+  });
+
+  it('reads through the fresh snapshot after this repository commits (auto-refresh)', async () => {
+    const { repo } = await seededRepo();
+    const users = await repo.openSheet('users');
+
+    await commitAvatar(repo, Buffer.from('v1'));
+    let stream = await users.getAttachmentStream('jane', 'avatar.png');
+    expect((await collect(stream!)).toString()).toBe('v1');
+
+    await commitAvatar(repo, Buffer.from('v2'));
+    stream = await users.getAttachmentStream('jane', 'avatar.png');
+    expect((await collect(stream!)).toString()).toBe('v2');
+  });
+});