This note is the authoritative description of qi's durable mutation and generation-rotation boundary. qi removes yin tombstones by rotating a whole JSON document into a new generation, and can make every accepted Store change pass through an application-owned checkpoint authority. Generation and persistence coordination are application policy in qi; yin remains unaware of document IDs, generations, peers, acknowledgments, persistence transactions, and forced bootstrap.
A synchronized document is identified by (document ID, generation). The document ID is stable. The generation is a positive, monotonically increasing uint64; it is a compatibility boundary, not a yin causal version.
Every synchronization artifact carries that identity:
- a cursor combines the identity with a yin
VersionVector; - a delta combines the identity with encoded yin delta bytes;
- a snapshot combines the identity with a durable yin snapshot;
- the persisted qi checkpoint is the snapshot envelope, so generation metadata and yin state are replaced as one unit.
The visible JSON projection remains distinct. It contains only live values. The payload inside a checkpoint remains a yin snapshot with causal metadata and tombstones. A delta remains a yin delta. qi does not reinterpret one artifact as another. Decoding a checkpoint or bootstrap snapshot always supplies the receiving Store's continuing runtime-local yin ReplicaID; neither checkpoint contents nor a sender identity can replace it or author the receiver's later writes.
The /qi/sync/2 request already carries Cursor.Identity.DocumentID; multi-document serving uses that field as its sole route key and makes no protocol-ID, request/response field, or JSON framing change. The transport bounds and decodes the frame, rejects a missing requester replica or document ID, and passes the authenticated remote libp2p peer plus the exact document ID to the resolver. It does not fall back to another Store.
The resolver owns document membership and availability, including application-defined peer/document authorization policy. Unknown, removed, denied, nil, and identity-mismatched resolutions all produce the same artifact-free unavailable response. A basic SourceRouter supplies concurrent registration and lookup but deliberately applies no authorization policy.
Store remains the artifact validation and acceptance boundary after routing. CatchUp compares the requested document and generation before choosing a delta or snapshot; ApplyCatchUp compares received identity before decoding a delta or calling yin ApplyDelta or Merge. Document mismatches return ErrDocumentMismatch; generation mismatches return ErrGenerationMismatch; a correctly routed artifact containing invalid yin bytes returns ErrMalformedSyncArtifact. This validation occurs before candidate creation or checkpoint persistence, so cross-document or cross-generation artifacts are never committed.
Each routed document therefore needs its own Store, immutable runtime-local yin replica identity, generation, and checkpoint authority. A shared libp2p host and resolver do not create a transaction or generation authority across Stores. The application must pair each durable Store with the correct independently loaded checkpoint and committer.
NewStore and the explicit in-memory constructors accept changes in memory and do not offer restart durability. OpenDurableStore instead takes initial checkpoint bytes and a CheckpointCommitter. The embedding application:
- loads either no bytes for a fresh document or the complete bytes from its authority;
- passes those bytes and the committer for that same authority to
OpenDurableStore; - gives that
Storeexclusive write authority for the checkpoint while it is open; and - reloads from the same authority when reopening after a restart.
The checkpoint is authoritative for its stored generation. The expected document ID must match, but a loaded checkpoint's generation supersedes the constructor's initial generation. Loading never supplies the runtime-local replica ID. Opening fresh state does not publish an initial checkpoint; the first accepted state-changing operation does.
The application owns the storage transaction, but qi owns the mutation acceptance sequence. A committer receives complete qi checkpoint bytes, not visible JSON, a bare yin snapshot, or a delta. It must atomically replace one logical authority and report whether publication happened. It must not mutate or retain the supplied byte slice. Store invokes it synchronously while holding its mutex, so it need not support concurrent or reentrant calls.
OpenStore is the filesystem convenience form of the same model when dataDir is non-empty. With an empty data directory it remains in-memory. The CLI's committer writes and syncs a temporary file, renames it over document.snapshot.json, and then syncs the containing directory.
The Store mutex serializes candidate creation, checkpoint encoding and publication, the in-memory swap, snapshots, cursor reads, synchronization, and rotation. For each durable local mutation, remote delta, bootstrap, or rotation, qi performs this sequence under that mutex:
- Validate input and document/generation identity before yin ingest or persistence.
- Clone the current yin state by a complete yin snapshot encode/decode round trip, preserving causal metadata and the Store's immutable local replica.
- Apply the operation only to that private candidate. A duplicate or stale causal no-op returns without calling the committer.
- Encode and validate a complete candidate qi checkpoint containing identity and yin snapshot.
- Call
CommitCheckpointwhile the old state remains visible throughStore. - If publication is reported, swap the Store identity and yin state to the candidate before unlocking.
This is deliberately commit-then-swap, rather than mutating live memory and attempting a later save. Callers can interpret outcomes as follows:
| Committer result | Store result | Accepted state |
|---|---|---|
(true, nil) |
success | Authority and memory contain the candidate. |
(false, err) |
ordinary wrapped error | Authority and memory retain the old state; the private candidate is discarded. |
(true, err) |
error wrapping ErrDurabilityUncertain and the cause |
Publication occurred and memory contains the candidate; durability across a crash is uncertain. |
(false, nil) |
contract error | Invalid committer response; memory remains unchanged. |
For methods returning (changed, error), pre-publication failure returns changed == false; known publication with durability uncertainty returns changed == true. Set and Delete communicate the same distinction through the error class. RotateCheckpoint returns the next identity with ErrDurabilityUncertain, but returns the zero identity after a pre-publication failure. Applications should not blindly retry a mutation after ErrDurabilityUncertain: the operation is already visible through the Store and the authority was published.
Checkpoint persistence and the visible projection are intentionally not one transaction. The checkpoint is the acceptance boundary. A projection write can be retried or regenerated from the Store without losing causal state.
Before publication, a crash or error leaves the old authoritative checkpoint, and reopening loads that old state. After publication, the checkpoint—not the process's last in-memory instruction—is authoritative. If a process crashes after commit but before the memory swap, reopening the published bytes reconstructs the candidate. If a committer reports post-publication uncertainty, the live Store still adopts the candidate so reads cannot contradict an authority known to have been replaced.
A custom authority must define how it resolves a crash during an uncertain durability window. On restart its loader must return one complete committed checkpoint selected by that authority's recovery rules, never a torn mixture. The filesystem committer's uncertainty window begins only after rename: while the process remains running, the renamed checkpoint is visible and authoritative; a directory-sync error is reported as ErrDurabilityUncertain. Reopening normally reads that renamed checkpoint, while durability across a system crash is exactly what the error could not prove.
For the CLI, startup precedence is checkpoint, then visible JSON, then {}. Whenever a checkpoint exists, startup loads it and rewrites the visible file from the recovered projection. Thus an interrupted or failed projection write does not require replaying a CRDT operation. A visible-file edit made while the CLI is stopped is not accepted over an existing checkpoint.
A suitable custom committer therefore must provide all of these properties:
- exclusive authority for one Store, with application loading paired to that exact authority;
- atomic replacement of complete opaque qi bytes, with no torn checkpoint observable;
- truthful
(published, error)classification at the logical publication point; - read-after-publication visibility consistent with a
published == trueresponse; - restart recovery that returns a complete old or new checkpoint according to documented crash semantics; and
- no hidden transaction that independently changes document identity, payload, or replica ownership.
qi does not coordinate a checkpoint with application database rows, multiple Stores, visible projections, or cursors. Applications needing those atomic relationships must implement them inside their own authority or use a larger transaction design; qi does not provide a journal or distributed transaction.
Only one coordinator may rotate a document generation:
- Active(g): ordinary local writes and remote applies are serialized by the Store mutex.
- Preparing(g+1): the coordinator holds that mutex, quiescing this Store, and obtains the live projection with
yin.MarshalJSONObject. - It creates fresh yin state with
yin.ParseJSONObjectusing the same stable local yinReplicaID. This deliberately creates no records for deleted keys and carries no old causal coverage. - It publishes one complete qi checkpoint containing
(document ID, g+1)and the new yin snapshot through the normal durable commit protocol. - Active(g+1): only after publication does the Store swap its in-memory identity and yin map. Cursors are generation-scoped, so old-generation causal coverage cannot be used in the new generation.
The local mutex is not distributed consensus. Before rotation, the application must select one coordinator and either quiesce all writers or accept an explicit cutoff after which unobserved old-generation writes are retired. Two peers must never independently publish different snapshots under the same next generation. This prototype does not implement membership, leases, consensus, or generation allocation.
A source receiving an older-generation cursor returns its current-generation snapshot, never a cross-generation delta. The receiver replaces its old yin state and then resumes ordinary delta catch-up in the new generation. A source behind a requester's generation reports a generation mismatch rather than guessing which state is authoritative.
Old snapshots and delayed old-generation deltas are rejected before yin is invoked. They are harmless replays. qi never translates an old delta into a local write in the new generation. An offline peer's unobserved old-generation edits are therefore discarded when it bootstraps. This is intentional peer-retirement policy and must be coordinated operationally before rotation.
A same-generation snapshot may be merged only after identity validation. This supports ordinary whole-state bootstrap without weakening the generation boundary. Duplicate same-generation deltas retain yin's idempotent no-op behavior and do not cause checkpoint publication.
No separately persisted cursor file exists today. If acknowledgment or peer metadata is later persisted, it must be committed with its DocumentIdentity; old-generation entries must be discarded rather than copied during rotation. Splitting current generation, snapshot, and cursors across independently replaced files would require a real transaction or recovery journal.
The durable flow uses the yin version pinned in go.mod: snapshot round trips clone candidate state, and its typed JSON delta codec ingests synchronization. yin correctly owns CRDT values, causal versions, snapshots, and deltas. It does not know which application document a map belongs to, which generation is current, whether an offline peer is retired, or what publication means for an application's storage system.
Putting commit classification or generation rotation in yin would couple a general CRDT library to qi's document envelope and storage policy without making the application transaction safer. qi therefore validates identities, serializes candidates, classifies publication, and forces generation bootstrap around canonical yin APIs. No yin change or database dependency is required.
Concrete API friction remains limited: qi must select the typed generic delta codec at each boundary; snapshot decode errors lack a snapshot-specific sentinel comparable to yin.ErrMalformedDelta, so qi wraps them in ErrMalformedSyncArtifact; and rebuilding from visible projection creates fresh writes, which is safe only because qi's generation identity prevents comparison with old state.