Skip to content

Commit e24fcf7

Browse files
Merge pull request #245 from JarvusInnovations/develop
Release: v2.3.0
2 parents 24d57cb + 1c642d0 commit e24fcf7

42 files changed

Lines changed: 2443 additions & 61 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

docs/api.md

Lines changed: 14 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -80,9 +80,12 @@ const store = await openStore(repo, {
8080
| `repo.openSheet<T>(name, opts?)` | Sheet handle; `opts.validator` (Standard Schema), `opts.root`, `opts.prefix` |
8181
| `repo.openSheets(opts?)` | All sheets keyed by name; `opts.root`, `opts.prefix` |
8282
| `repo.transact(opts, handler)` | Run a handler in a single-commit transaction |
83+
| `repo.withLock(fn)` | Run `fn` holding the write lock transact uses — coordinate non-transact git ops. Not reentrant (`lock_held`) |
8384
| `repo.requireExplicitTransactions()` | Switch to strict mode (one-way) |
8485
| `repo.startPushDaemon(opts)` | Start async push to a configured remote |
8586
| `repo.resolveRef(ref)` | Resolve a ref or commit hash; returns `string \| null` |
87+
| `repo.refresh()` | Rebind every live Sheet to the current HEAD tree — for out-of-band ref movement (a successful `transact` auto-refreshes) |
88+
| `repo.readBlobStream(ref, path)` | `Promise<Readable>` — stream a blob's bytes at `<ref>:<path>`, resolved at call time. Throws `RefError` / `NotFoundError` |
8689

8790
`opts.prefix` scopes records to a sub-tree under each sheet's configured root — useful for multi-tenant deployments where one git repo holds many tenants under `<root>/<tenant>/...`. Mirrors the CLI `--prefix` flag (env `GITSHEETS_PREFIX`). The sheet's `.gitsheets/<name>.toml` config file is unaffected — only the record data tree is scoped.
8891

@@ -100,6 +103,7 @@ const store = await openStore(repo, {
100103
| `sheet.queryFirst(filter?, opts?)` | `Promise<T \| undefined>` — honors `opts.signal`, `opts.withBody` |
101104
| `sheet.queryAll(filter?, opts?)` | `Promise<T[]>` — honors `opts.signal`, `opts.withBody` |
102105
| `sheet.loadBody(record)` | `Promise<T>` — hydrate a body-less record (content-typed sheets) |
106+
| `sheet.refresh()` | Rebind this sheet's read snapshot to the current HEAD tree (out-of-band movement; own commits auto-refresh) |
103107
| `sheet.pathForRecord(record)` | `Promise<string>` — rendered path, no write |
104108
| `sheet.normalizeRecord(record)` | `Promise<T>` — canonical form, no write |
105109

@@ -122,10 +126,11 @@ All write methods route through a transaction — permissive mode auto-opens one
122126

123127
| Method | Purpose |
124128
| --- | --- |
125-
| `sheet.getAttachment(record, name)` | One attachment's BlobObject or null |
126-
| `sheet.getAttachments(record)` | Map of name → BlobObject |
127-
| `sheet.setAttachment(record, name, blob)` | Add or replace |
128-
| `sheet.setAttachments(record, map)` | Bulk variant |
129+
| `sheet.getAttachment(record, name)` | One attachment's `AttachmentBlobHandle` or null |
130+
| `sheet.getAttachments(record)` | Map of name → `AttachmentBlobHandle` |
131+
| `sheet.getAttachmentStream(recordOrPath, name)` | `Promise<Readable \| null>` — stream an attachment's bytes without materializing the record |
132+
| `sheet.setAttachment(record, name, content)` | Add or replace — `content` may be raw bytes (`Buffer`/`Uint8Array`), a UTF-8 `string`, or a `BlobHandle` |
133+
| `sheet.setAttachments(record, map)` | Bulk variant; values accept the same types, mixed freely |
129134
| `sheet.deleteAttachment(record, name)` | Remove a single attachment; throws `NotFoundError` if missing |
130135
| `sheet.deleteAttachments(record)` | Remove all attachments; idempotent no-op when no attachment dir |
131136
| `sheet.attachments(record)` | `AsyncGenerator<AttachmentEntry>` yielding `{name, mimeType, blob}` with `.read()` / `.stream()` |
@@ -177,10 +182,13 @@ type Store<V extends ValidatorMap> = {
177182
readonly [K in keyof V]: Sheet<InferRecord<V[K]>>;
178183
} & {
179184
readonly transact: StoreTransactFn<V>;
185+
readonly refresh: () => Promise<void>;
180186
};
181187
```
182188

183-
`store.<sheet>` for each sheet declared in `validators`. `store.transact(opts, async tx => ...)` mirrors `repo.transact` with `tx.<sheet>` aliases that thread validators through.
189+
`store.<sheet>` for each sheet declared in `validators`. `store.transact(opts, async tx => ...)` mirrors `repo.transact` with `tx.<sheet>` aliases that thread validators through. `store.refresh()` rebinds every sheet to the current HEAD tree (delegates to `repo.refresh()`).
190+
191+
Reads through `store.<sheet>` follow the freshness model: a successful `store.transact` / `repo.transact` auto-refreshes, so post-commit reads reflect the committed state — see [`specs/behaviors/freshness.md`](https://github.com/JarvusInnovations/gitsheets/blob/develop/specs/behaviors/freshness.md).
184192

185193
Sheets not in `validators` are accessible via `repo.openSheet(name)` for one-off un-typed access.
186194

@@ -224,7 +232,7 @@ Subclasses:
224232
| --- | --- |
225233
| `ConfigError` | `config_missing`, `config_invalid` |
226234
| `ValidationError` | `validation_failed` (carries `issues: ValidationIssue[]`) |
227-
| `TransactionError` | `transaction_in_progress`, `transaction_required`, `parent_moved`, `commit_failed`, `push_daemon_running`, `transaction_closed` |
235+
| `TransactionError` | `transaction_in_progress`, `transaction_required`, `parent_moved`, `commit_failed`, `push_daemon_running`, `transaction_closed`, `lock_held` |
228236
| `IndexError` | `index_unique_conflict`, `index_not_defined` (carries `conflictingPaths`) |
229237
| `RefError` | `ref_not_found`, `not_an_ancestor` |
230238
| `PathTemplateError` | `path_render_failed`, `path_invalid_chars` |

docs/migration-guide.md

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -216,6 +216,107 @@ Patch release. Two fixes in service of snapshot-importer-style workflows.
216216

217217
- `hologit` bumped from `^0.49.1` to `^0.50.2` to pick up `TreeObject.clearChildren()` (the O(1) primitive backing the new `Sheet#clear()`).
218218

219+
## v1.x → v2.0 (the Rust core)
220+
221+
v2.0 replaces the Node.js engine with the Rust `gitsheets-core` crate: TOML
222+
parse/serialize, normalization, validation, path templates, record CRUD, and
223+
the tree/blob/commit substrate all run natively, and the npm package becomes a
224+
thin marshalling shell over the `@gitsheets/core-napi` addon. The API surface
225+
is intentionally unchanged — but three behavior changes bite real consumers.
226+
All three surfaced during the first production 1.4.1 → 2.x upgrade; this
227+
section is what turns "16 mystery test failures" into "read the section, apply
228+
3 changes."
229+
230+
### 1. The `hologit` dependency is gone
231+
232+
2.x drops `hologit` entirely (tree/blob/commit ops run on `holo-tree`/gitoxide
233+
*inside* the core). Anything that reached into it breaks:
234+
235+
- `repo.hologitRepo` no longer exists.
236+
- `BlobObject` (e.g. `BlobObject.write`) is no longer importable via gitsheets.
237+
238+
The replacement for the common pattern — hashing binary content and attaching
239+
it to a record — is the built-in blob primitive plus the attachment API:
240+
241+
```typescript
242+
// 1.x
243+
import { BlobObject } from 'hologit';
244+
245+
const blob = await BlobObject.write(repo.hologitRepo, buffer);
246+
await sheet.setAttachments(record, { 'avatar.jpg': blob });
247+
```
248+
249+
```typescript
250+
// 2.x
251+
const blob = await repo.writeBlob(buffer); // Promise<BlobHandle>
252+
await sheet.setAttachments(record, { 'avatar.jpg': blob });
253+
```
254+
255+
`repo.writeBlob(buf)` hashes the bytes into the object database and returns a
256+
`BlobHandle` accepted everywhere a hologit `BlobObject` used to be
257+
(`setAttachment`, `setAttachments`). `getAttachment`/`getAttachments`/
258+
`sheet.attachments()` likewise return `BlobHandle`s with `.read()`/`.stream()`.
259+
260+
### 2. `null`/`undefined`-valued fields
261+
262+
TOML has no `null`, so a cleared optional field and a never-set field are the
263+
same on-disk state: an absent key. 1.x (`@iarna/toml`) enforced this by
264+
silently dropping null-valued keys at serialize time. **The initial 2.x
265+
releases instead threw on marshal** (`cannot marshal JS value of type
266+
Null/Undefined to a TOML value`) — breaking the standard consumer pattern of
267+
`.nullable().optional()` schemas with `?? null` normalization on write.
268+
269+
**Fixed in this release**: the 1.x drop semantics are restored and now
270+
specced ([`specs/behaviors/normalization.md`](https://github.com/JarvusInnovations/gitsheets/blob/develop/specs/behaviors/normalization.md#null--undefined-handling)).
271+
A `null`/`undefined`-valued key is dropped, recursively (top-level fields,
272+
nested tables, and objects inside arrays), before validation and
273+
serialization — byte-identical to 1.x output, in every binding (Node and
274+
Python). If you shimmed this with a `stripNullish` helper at your write
275+
boundary, you can delete it.
276+
277+
One deliberate edge-case divergence from 1.x: a `null`/`undefined` **array
278+
element** is an error naming the index (1.x silently dropped elements —
279+
`[1, null, 2]``[1, 2]` — which shifts sibling indices and silently changes
280+
data). Remove the element yourself if that's what you mean. A required field
281+
set to `null` fails JSON-Schema validation as *missing*, same as 1.x.
282+
283+
### 3. Canonical bytes re-baseline (one-time)
284+
285+
The canonical serializer is now the core's `gitsheets-core::canonical` (the
286+
Rust `toml` crate's formatting over a deep key sort), replacing `@iarna/toml`.
287+
A record that was already canonical under 1.x may re-serialize to different —
288+
but *value-identical* — bytes, once ([#196](https://github.com/JarvusInnovations/gitsheets/issues/196)).
289+
Three reformat classes, all proven data-lossless and idempotent over a
290+
~29.5k-record corpus ([PR #205](https://github.com/JarvusInnovations/gitsheets/pull/205)):
291+
292+
1. **Integer digit-group underscores drop**`legacyId = 31_618`
293+
`legacyId = 31618` (the dominant class).
294+
2. **String requote** — strings containing both `"` and `'` move from escaped
295+
single-line form to readable triple-quoted `"""…"""` form.
296+
3. **Multiline trailing-quote layout** — a multiline string ending in `"`
297+
loses `@iarna`'s line-continuation dance (same value, fewer lines).
298+
299+
Until you re-baseline, the first 2.x write of a record whose 1.x bytes fall in
300+
one of these classes shows a spurious-looking (but value-neutral) diff. The
301+
recommended migration is a **one-time re-serialize commit** over each existing
302+
repo, which is idempotent (a second run produces zero diff — that's the
303+
adoption check):
304+
305+
```bash
306+
# Re-normalize every *.toml record under a directory, in place.
307+
cargo run -p gitsheets-core --example normalize_tree -- path/to/records
308+
309+
git add path/to/records
310+
git commit -m "chore: re-baseline records to the Rust canonical form"
311+
312+
# Verify idempotence — a second pass must report 0 files re-normalized:
313+
cargo run -p gitsheets-core --example normalize_tree -- path/to/records
314+
```
315+
316+
(Or use `git sheet normalize <sheet>` per sheet from the CLI.) See the
317+
[canonical-form re-baseline notes](https://github.com/JarvusInnovations/gitsheets/blob/develop/specs/behaviors/normalization.md#canonical-form-re-baseline-the-rust-serializer)
318+
for the full contract.
319+
219320
## Going forward
220321

221322
Once migrated, the recipes are the fastest path to common patterns:

docs/validation.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,8 @@ Internally the library calls `validator['~standard'].validate(record)`. The retu
9898

9999
The Standard Schema layer is **optional**. Without it, only JSON Schema runs.
100100

101+
Gitsheets' declared `StandardSchemaV1` types mirror the published Standard Schema v1 interface exactly, so a compliant validator (Zod v4, Valibot, ArkType, Effect Schema) assigns to `openSheet({ validator })` / `openStore({ validators })` directly — no `as` cast or wrapper needed.
102+
101103
For full type inference across reads and writes, use [`openStore`](concepts.md#store) instead of `openSheet` — see the [typed-sheet-with-Zod recipe](recipes/typed-sheet-with-zod.md).
102104

103105
## Order of operations

package-lock.json

Lines changed: 12 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

packages/gitsheets/package.json

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,7 @@
5353
"@types/yargs": "^17.0.35",
5454
"tmp": "^0.2.5",
5555
"typescript": "^6.0.3",
56-
"vitest": "^4.1.6"
56+
"vitest": "^4.1.6",
57+
"zod": "^4.4.3"
5758
}
5859
}

packages/gitsheets/src/errors.ts

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ const STATUS_BY_CODE = {
2121
index_not_defined: 500,
2222
push_daemon_running: 409,
2323
transaction_closed: 409,
24+
lock_held: 409,
2425
ref_not_found: 404,
2526
not_an_ancestor: 409,
2627
path_render_failed: 422,
@@ -77,7 +78,8 @@ export type TransactionErrorCode =
7778
| 'parent_moved'
7879
| 'commit_failed'
7980
| 'push_daemon_running'
80-
| 'transaction_closed';
81+
| 'transaction_closed'
82+
| 'lock_held';
8183

8284
export class TransactionError extends GitsheetsError {
8385
constructor(code: TransactionErrorCode, message: string, options?: GitsheetsErrorOptions) {

packages/gitsheets/src/index.ts

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ export type {
2727
DiffOptions,
2828
DiffChange,
2929
AttachmentBlobHandle,
30+
AttachmentContent,
3031
AttachmentEntry,
3132
} from './sheet.js';
3233

@@ -82,6 +83,8 @@ export { validateRecord } from './validation.js';
8283
export type {
8384
JSONSchema,
8485
StandardSchemaV1,
86+
StandardSchemaProps,
87+
StandardSchemaTypes,
8588
StandardSchemaIssue,
8689
StandardSchemaResult,
8790
StandardSchemaFailure,
Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
// Streaming blob reads by key/path — repo.readBlobStream + sheet.getAttachmentStream.
2+
// See specs/behaviors/attachments.md#streaming-reads-by-keypath.
3+
4+
import { mkdir, writeFile } from 'node:fs/promises';
5+
import { join } from 'node:path';
6+
import type { Readable } from 'node:stream';
7+
8+
import { afterEach, describe, expect, it } from 'vitest';
9+
10+
import { NotFoundError, RefError } from './errors.js';
11+
import { openRepo, type Repository } from './repository.js';
12+
import { testRepo, type TestRepoHandle } from './test-helpers/test-repo.js';
13+
14+
const handles: TestRepoHandle[] = [];
15+
afterEach(async () => {
16+
while (handles.length > 0) {
17+
const h = handles.pop();
18+
if (h) await h.cleanup();
19+
}
20+
});
21+
22+
const USERS = `[gitsheet]
23+
root = 'users'
24+
path = '\${{ slug }}'
25+
`;
26+
27+
async function seededRepo(): Promise<{ fixture: TestRepoHandle; repo: Repository }> {
28+
const fixture = await testRepo({ withInitialCommit: true });
29+
handles.push(fixture);
30+
await mkdir(join(fixture.path, '.gitsheets'), { recursive: true });
31+
await writeFile(join(fixture.path, '.gitsheets', 'users.toml'), USERS);
32+
await fixture.git('add', '.gitsheets/');
33+
await fixture.git('commit', '-m', 'add users sheet');
34+
const repo = await openRepo({ gitDir: fixture.gitDir });
35+
return { fixture, repo };
36+
}
37+
38+
async function collect(stream: Readable): Promise<Buffer> {
39+
const chunks: Buffer[] = [];
40+
for await (const chunk of stream) {
41+
chunks.push(chunk as Buffer);
42+
}
43+
return Buffer.concat(chunks);
44+
}
45+
46+
/** Binary bytes (not valid UTF-8) to prove byte fidelity end-to-end. */
47+
const BINARY = Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x00, 0xff, 0xfe, 0x0a, 0x1b]);
48+
49+
async function commitAvatar(repo: Repository, bytes: Buffer): Promise<void> {
50+
await repo.transact({ message: 'avatar' }, async (tx) => {
51+
const sheet = tx.sheet('users');
52+
await sheet.upsert({ slug: 'jane' });
53+
const blob = await repo.writeBlob(bytes);
54+
await sheet.setAttachment('jane', 'avatar.png', blob);
55+
});
56+
}
57+
58+
describe('repo.readBlobStream', () => {
59+
it('streams byte-identical content for a committed attachment', async () => {
60+
const { repo } = await seededRepo();
61+
await commitAvatar(repo, BINARY);
62+
63+
const stream = await repo.readBlobStream('HEAD', 'users/jane/avatar.png');
64+
expect(Buffer.compare(await collect(stream), BINARY)).toBe(0);
65+
});
66+
67+
it('resolves the ref at call time — a later commit is immediately visible', async () => {
68+
const { repo } = await seededRepo();
69+
await commitAvatar(repo, Buffer.from('v1'));
70+
await commitAvatar(repo, Buffer.from('v2'));
71+
72+
const stream = await repo.readBlobStream('HEAD', 'users/jane/avatar.png');
73+
expect((await collect(stream)).toString()).toBe('v2');
74+
});
75+
76+
it('throws NotFoundError(record_not_found) for a missing path', async () => {
77+
const { repo } = await seededRepo();
78+
const err = await repo.readBlobStream('HEAD', 'users/nope/avatar.png').catch((e: unknown) => e);
79+
expect(err).toBeInstanceOf(NotFoundError);
80+
expect((err as NotFoundError).code).toBe('record_not_found');
81+
});
82+
83+
it('throws NotFoundError(record_not_found) for a directory path', async () => {
84+
const { repo } = await seededRepo();
85+
await commitAvatar(repo, BINARY);
86+
const err = await repo.readBlobStream('HEAD', 'users/jane').catch((e: unknown) => e);
87+
expect(err).toBeInstanceOf(NotFoundError);
88+
});
89+
90+
it('throws RefError(ref_not_found) for a ref that does not resolve', async () => {
91+
const { repo } = await seededRepo();
92+
const err = await repo.readBlobStream('no-such-ref', 'users/jane/avatar.png').catch((e: unknown) => e);
93+
expect(err).toBeInstanceOf(RefError);
94+
expect((err as RefError).code).toBe('ref_not_found');
95+
});
96+
});
97+
98+
describe('sheet.getAttachmentStream', () => {
99+
it('streams the attachment bytes by record path without materializing the record', async () => {
100+
const { repo } = await seededRepo();
101+
await commitAvatar(repo, BINARY);
102+
const users = await repo.openSheet('users');
103+
104+
const stream = await users.getAttachmentStream('jane', 'avatar.png');
105+
expect(stream).not.toBeNull();
106+
expect(Buffer.compare(await collect(stream!), BINARY)).toBe(0);
107+
});
108+
109+
it('accepts a record object too', async () => {
110+
const { repo } = await seededRepo();
111+
await commitAvatar(repo, BINARY);
112+
const users = await repo.openSheet('users');
113+
114+
const stream = await users.getAttachmentStream({ slug: 'jane' }, 'avatar.png');
115+
expect(Buffer.compare(await collect(stream!), BINARY)).toBe(0);
116+
});
117+
118+
it('returns null when the attachment is absent', async () => {
119+
const { repo } = await seededRepo();
120+
await commitAvatar(repo, BINARY);
121+
const users = await repo.openSheet('users');
122+
123+
expect(await users.getAttachmentStream('jane', 'missing.png')).toBeNull();
124+
});
125+
126+
it('reads through the fresh snapshot after this repository commits (auto-refresh)', async () => {
127+
const { repo } = await seededRepo();
128+
const users = await repo.openSheet('users');
129+
130+
await commitAvatar(repo, Buffer.from('v1'));
131+
let stream = await users.getAttachmentStream('jane', 'avatar.png');
132+
expect((await collect(stream!)).toString()).toBe('v1');
133+
134+
await commitAvatar(repo, Buffer.from('v2'));
135+
stream = await users.getAttachmentStream('jane', 'avatar.png');
136+
expect((await collect(stream!)).toString()).toBe('v2');
137+
});
138+
});

0 commit comments

Comments
 (0)