feat(sync): add local-first multi-machine artifact synchronization - #1239
Closed
wesm wants to merge 1 commit into
Closed
feat(sync): add local-first multi-machine artifact synchronization#1239wesm wants to merge 1 commit into
wesm wants to merge 1 commit into
Conversation
- docs: design thinner docbank artifact boundary - docs: keep agent plans outside repository - refactor(artifact): own repository traversal iterators - refactor(server): retain artifact iterators in HTTP cursors - refactor(artifact): remove opaque store cursors - refactor(artifact): make repository own local lifecycle - perf(artifact): export only queued session changes - feat(db): persist unfinished artifact imports - fix(db): validate canonical metadata retry names - perf(artifact): import only changed protocol work - refactor(artifact): share bounded transport exchange - refactor(artifact): remove obsolete store abstractions
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| targetCanonical string, | ||
| targetIdentity fs.FileInfo, | ||
| ) error { | ||
| currentTarget, err := os.Stat(targetCanonical) |
| func targetIdentityChainContains(target string, identity fs.FileInfo) bool { | ||
| current := target | ||
| for { | ||
| info, err := os.Stat(current) |
roborev: Combined Review (
|
Member
Author
|
I'm going to split this into smaller PRs. I'll be back |
wesm
added a commit
that referenced
this pull request
Jul 23, 2026
PR1 of the artifact-sync split (supersedes the closed #1239, extracted and pared from its reference branch with @maphew as co-author). This lands the foundation of `internal/artifact`: the frozen v1 wire format and the local content-addressed store, with no export/import pipeline, transports, server, db, or CLI integration yet. **What's here** - Wire format types (checkpoint, manifest, segment message, metadata event, raw source) with key-sorted canonical JSON encoding and golden-hash tests. The goldens are byte-identical to the reference implementation and are the frozen v1 format going forward; the `raw_source` contract (SHA-256 identity, size cap, `application/jsonl` media-type allowlist, sanitized relative paths) is frozen and validated here even though capture lands in PR3. - A reflection parity test that pins `manifestSession` to `db.Session`'s JSON-visible fields, so incidental session-struct growth cannot silently re-hash every manifest. - Zstd wire codec with pinned decoder window, encoded/decoded byte limits, and hash-verified reads (corruption-grade robustness; truncation and bit-rot detection). - `ArtifactStore` interface and the Docbank-backed implementation (`go.kenn.io/docbank`, new dependency, plus a kit upgrade), with a contract suite that runs against both SQLite drivers (mattn and modernc). - `Repository`: open-on-demand vault ownership under `$AGENTSVIEW_DATA_DIR/artifacts`. Nothing creates the vault for users who don't opt in; `serve` integration comes later. - `config.ValidateArtifactOriginID` with table tests. **Deliberately absent, arriving in later PRs** - Export ledger/pipeline and import (PR2), raw JSONL capture (PR3), folder transport + sync CLI (PR4), metadata/HLC machinery (PR5), GC/maintenance/quarantine listing and the pack scheduler (PR6). A few reference symbols whose only consumers live in those PRs were deferred with them (`contextArtifactReader`, decoded-limit consts, quarantine listing surface, compression tests); each returns verbatim with its consumer. `ErrArtifactUnsupported` is frozen store vocabulary whose returning consumer is PR6 maintenance. - `canonicalArtifactPath` is temporarily duplicated into `repository.go`; PR2's sync.go extraction must reconcile to a single copy. - `Quarantine` keeps its `reason` parameter (currently discarded): PR6's quarantine listing may persist it once docbank can store a reason. **Where to look** - `internal/artifact/wire.go` + `format_test.go` — the format freeze and golden hashes. - `internal/artifact/canonical_json.go` — deterministic encoder (byte-identical to the reference; it determines every content hash). - `internal/artifact/store_docbank.go` + `store_contract_test.go` — store semantics: immutable create, conflict detection, verified reads, quarantine/trash, two-driver matrix. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Wes McKinney <wesm@users.noreply.github.com>
wesm
added a commit
that referenced
this pull request
Jul 28, 2026
Second PR in the artifact-sync split (follows #1242, which froze the wire format and added the docbank-backed store). Extracted from the closed #1239 branch with review fixes applied during extraction. ## What this adds - **SQLite publication ledger** (`internal/db`): `artifact_export_queue`, `artifact_publications`, `artifact_publication_revisions`, `artifact_checkpoint_heads`/`_floors`, plus session triggers that enqueue owned-session changes. Triggers are origin-gated — they fire only once an artifact origin exists in `pg_sync_state` — so archives that never opt in carry no queue writes. Trigger DDL lives in Go and is installed after column migrations (drops run before), so no trigger references session columns while migrations run. - **Go enqueue hooks** for child-only mutations that don't touch trigger-covered session columns: batch message writes (queue generation sampled around the batch, enqueues exactly once when the triggers didn't fire), standalone usage-event replacement, and token-coverage backfill. Queue bootstrap is an explicit call at origin creation, not a migrate-time backfill. - **Origin lifecycle** (`internal/artifact`): `EnsureOrigin`/`AdoptOrigin`/`StoredOrigin`. Creating or adopting an origin bootstraps the export queue; a failed bootstrap rolls the origin back (deleting the key when there was none before, since the gates test key existence) so a retry re-runs population. Adopting over a different established origin force-requeues every owned session with a generation bump, because prior acknowledgements belong to the old origin. - **Checkpointed export pipeline** (`internal/artifact`): claims pending queue rows, publishes content-addressed session manifests and segments into the store, records publication revisions, and advances per-origin checkpoint heads with monotone sequence reservation. Stale claims (a writer advanced the generation mid-export) roll back atomically. Incremental export is bounded by the dirty batch; an unchanged archive costs a catalog identity check only. Full export streams all bodies, then re-checks the queue up to 32 settle rounds — hitting the bound returns the accumulated result with an error meaning "made progress, run again." - **Resync carriage**: `CopySyncStateFrom` carries queue, publication, revision, and checkpoint state across a full resync and re-dirties every copied queue row — the origin gate keeps triggers silent in the rebuild's temp DB, and a parser bump changes manifest hashes anyway, so the exporter must re-verify each session (unchanged content is cheap to skip, being content-addressed). ## Scope and limitations - Export-side only. Nothing wires the pipeline to the daemon or CLI yet; that lands with folder transport and import in a later PR. There is no import/read path here. - The ledger is local SQLite state. PostgreSQL push and shared query shapes are untouched. - The wire format is unchanged: no golden churn, format stays frozen at v1. ## Where to look - `internal/db/artifact_publication.go` — queue and ledger SQL, origin-gated enqueue - `internal/db/db.go` — trigger DDL split around column migrations, bootstrap/requeue - `internal/artifact/export.go` — publish → manifest → checkpoint ordering and claim lifecycle - `internal/artifact/origin.go` — origin lifecycle and rollback semantics - `internal/db/orphaned.go` — resync carriage and the pending-flag decision Co-authored-by: Wes McKinney <wesm@users.noreply.github.com>
cursor Bot
pushed a commit
to diazMelgarejo/periscope
that referenced
this pull request
Jul 29, 2026
Second PR in the artifact-sync split (follows kenn-io#1242, which froze the wire format and added the docbank-backed store). Extracted from the closed kenn-io#1239 branch with review fixes applied during extraction. - **SQLite publication ledger** (`internal/db`): `artifact_export_queue`, `artifact_publications`, `artifact_publication_revisions`, `artifact_checkpoint_heads`/`_floors`, plus session triggers that enqueue owned-session changes. Triggers are origin-gated — they fire only once an artifact origin exists in `pg_sync_state` — so archives that never opt in carry no queue writes. Trigger DDL lives in Go and is installed after column migrations (drops run before), so no trigger references session columns while migrations run. - **Go enqueue hooks** for child-only mutations that don't touch trigger-covered session columns: batch message writes (queue generation sampled around the batch, enqueues exactly once when the triggers didn't fire), standalone usage-event replacement, and token-coverage backfill. Queue bootstrap is an explicit call at origin creation, not a migrate-time backfill. - **Origin lifecycle** (`internal/artifact`): `EnsureOrigin`/`AdoptOrigin`/`StoredOrigin`. Creating or adopting an origin bootstraps the export queue; a failed bootstrap rolls the origin back (deleting the key when there was none before, since the gates test key existence) so a retry re-runs population. Adopting over a different established origin force-requeues every owned session with a generation bump, because prior acknowledgements belong to the old origin. - **Checkpointed export pipeline** (`internal/artifact`): claims pending queue rows, publishes content-addressed session manifests and segments into the store, records publication revisions, and advances per-origin checkpoint heads with monotone sequence reservation. Stale claims (a writer advanced the generation mid-export) roll back atomically. Incremental export is bounded by the dirty batch; an unchanged archive costs a catalog identity check only. Full export streams all bodies, then re-checks the queue up to 32 settle rounds — hitting the bound returns the accumulated result with an error meaning "made progress, run again." - **Resync carriage**: `CopySyncStateFrom` carries queue, publication, revision, and checkpoint state across a full resync and re-dirties every copied queue row — the origin gate keeps triggers silent in the rebuild's temp DB, and a parser bump changes manifest hashes anyway, so the exporter must re-verify each session (unchanged content is cheap to skip, being content-addressed). - Export-side only. Nothing wires the pipeline to the daemon or CLI yet; that lands with folder transport and import in a later PR. There is no import/read path here. - The ledger is local SQLite state. PostgreSQL push and shared query shapes are untouched. - The wire format is unchanged: no golden churn, format stays frozen at v1. - `internal/db/artifact_publication.go` — queue and ledger SQL, origin-gated enqueue - `internal/db/db.go` — trigger DDL split around column migrations, bootstrap/requeue - `internal/artifact/export.go` — publish → manifest → checkpoint ordering and claim lifecycle - `internal/artifact/origin.go` — origin lifecycle and rollback semantics - `internal/db/orphaned.go` — resync carriage and the pending-flag decision Co-authored-by: Wes McKinney <wesm@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This supersedes #1230. The replacement branch lives in kenn-io/agentsview so maintainers can update it directly while review proceeds.
AgentsView needs a way to exchange session archives and curation metadata across trusted machines without turning a shared service into the authority. This adds immutable, conflict-aware artifact synchronization through folders, HTTP peers, and S3-compatible storage while keeping SQLite authoritative for each local archive.
The runtime keeps its private artifact repository in Docbank, with canonical logical hashes, compressed loose and packed storage, bounded maintenance, exact checkpoint provenance, repair and quarantine handling, and an explicit fail-closed reset path. External wire paths remain compatible, and peer exchange is paged so background work stays bounded by changed artifacts rather than total archive size.
The Peers view and session conflict indicators expose convergence and metadata conflicts. Reviewers should focus on protocol semantics in internal/artifact, publication and provenance state in internal/db, server lifecycle and maintenance routes, and the synchronization watcher integration. Docbank is temporarily pinned to a public commit until the integration requirements settle and a release is cut.
Closes #692.
Closes #1034.
Closes #1035.