Add artifact publication ledger and export pipeline - #1251
Conversation
roborev: Combined Review (
|
roborev: Combined Review (
|
34b3440 to
dac339f
Compare
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
- fix artifact export lifecycle convergence and stale origin recovery - keep publication metadata and manifest versions consistent - document bounded export rejection invariants and implementation plan - bind publication claims to the active persisted origin - persist generation-scoped rejection diagnostics atomically - bound SQLite export preflight and nested hydration - route all export paths through bounded snapshot loading - isolate deterministic claim failures without starving FIFO work - cover rejection removal, mutation retry, resync, and adoption lifecycles - reject malformed canonical message metadata deterministically
8302e2e to
7c8425d
Compare
roborev: Combined Review (
|
A non-finite persisted session signal makes canonical manifest encoding fail deterministically for the same generation. Treat that data error as a rejection so one poisoned claim cannot block later FIFO exports.
roborev: Combined Review (
|
Checkpoint discovery must not retain raw identities for semantically equivalent but noncanonical JSON, because export reconstructs canonical bytes and would otherwise wedge on a permanent hash mismatch. Verify streamed canonical SHA and size before accepting bootstrap candidates.
roborev: Combined Review (
|
Checkpoint verification already drains unread bytes before authenticating EOF, so an early semantic decode failure cannot turn a malformed candidate into an incomplete-verification error. Make that contract explicit at candidate classification and protect it through the real-store bootstrap path.
roborev: Combined Review (
|
Bootstrap must distinguish an unsupported but valid checkpoint from malformed or corrupt data, otherwise a poisoned future-version marker can block every export. Configured origin adoption must likewise repair malformed persisted state instead of validating the value it is replacing.
roborev: Combined Review (
|
Recreated origins must republish live local sessions even when the prior state key disappears, otherwise clean ledger rows remain tied to a lost namespace. Manifest generation must also read raw name fields so agent-provided titles are not promoted into user-owned display names.
roborev: Combined Review (
|
An authenticated future checkpoint remains authoritative even when its canonical schema adds fields unknown to v1. Parse and hash unknown values generically while retaining the exact v1 field contract for current checkpoints, preventing this exporter from publishing a downgrade checkpoint.
roborev: Combined Review (
|
Future checkpoint authority must survive schema changes inside the sessions field, not only additional top-level fields. Authenticate that value generically and apply the current map and hash contract only after confirming v1, preventing downgrade publication over valid newer state.
roborev: Combined Review (
|
A database head can outlive the binary format that recorded it, so sequence state alone cannot authorize a downgraded exporter to publish a successor. Authenticate and decode the exact recorded checkpoint before every construction path while preserving the constant-work unchanged fast path, ensuring newer checkpoint authority remains blocking across retries.
roborev: Combined Review (
|
Checkpoint heads must remain a lower bound for future reservations even when legacy databases or reset stores lack a separate floor row. Advance the floor in the same transaction as head recording and derive it during startup and resync migration so recovery cannot restart below an authoritative head.
roborev: Combined Review (
|
Production sessions are tagged with the configured hostname, so treating only the legacy local marker as owned can silently omit every real session or delete its publication. Persist the runtime identity as archive state and use it across queueing, rebuild recovery, ownership listing, and export validation.\n\nChanging or first recording the identity re-dirties the active origin atomically, allowing existing archives to converge while retaining local as a compatibility alias.
roborev: Combined Review (
|
Second PR in the artifact-sync split (follows kenn-io#1242, which froze the wire format and added the docbank-backed store). Extracted from the closed kenn-io#1239 branch with review fixes applied during extraction. - **SQLite publication ledger** (`internal/db`): `artifact_export_queue`, `artifact_publications`, `artifact_publication_revisions`, `artifact_checkpoint_heads`/`_floors`, plus session triggers that enqueue owned-session changes. Triggers are origin-gated — they fire only once an artifact origin exists in `pg_sync_state` — so archives that never opt in carry no queue writes. Trigger DDL lives in Go and is installed after column migrations (drops run before), so no trigger references session columns while migrations run. - **Go enqueue hooks** for child-only mutations that don't touch trigger-covered session columns: batch message writes (queue generation sampled around the batch, enqueues exactly once when the triggers didn't fire), standalone usage-event replacement, and token-coverage backfill. Queue bootstrap is an explicit call at origin creation, not a migrate-time backfill. - **Origin lifecycle** (`internal/artifact`): `EnsureOrigin`/`AdoptOrigin`/`StoredOrigin`. Creating or adopting an origin bootstraps the export queue; a failed bootstrap rolls the origin back (deleting the key when there was none before, since the gates test key existence) so a retry re-runs population. Adopting over a different established origin force-requeues every owned session with a generation bump, because prior acknowledgements belong to the old origin. - **Checkpointed export pipeline** (`internal/artifact`): claims pending queue rows, publishes content-addressed session manifests and segments into the store, records publication revisions, and advances per-origin checkpoint heads with monotone sequence reservation. Stale claims (a writer advanced the generation mid-export) roll back atomically. Incremental export is bounded by the dirty batch; an unchanged archive costs a catalog identity check only. Full export streams all bodies, then re-checks the queue up to 32 settle rounds — hitting the bound returns the accumulated result with an error meaning "made progress, run again." - **Resync carriage**: `CopySyncStateFrom` carries queue, publication, revision, and checkpoint state across a full resync and re-dirties every copied queue row — the origin gate keeps triggers silent in the rebuild's temp DB, and a parser bump changes manifest hashes anyway, so the exporter must re-verify each session (unchanged content is cheap to skip, being content-addressed). - Export-side only. Nothing wires the pipeline to the daemon or CLI yet; that lands with folder transport and import in a later PR. There is no import/read path here. - The ledger is local SQLite state. PostgreSQL push and shared query shapes are untouched. - The wire format is unchanged: no golden churn, format stays frozen at v1. - `internal/db/artifact_publication.go` — queue and ledger SQL, origin-gated enqueue - `internal/db/db.go` — trigger DDL split around column migrations, bootstrap/requeue - `internal/artifact/export.go` — publish → manifest → checkpoint ordering and claim lifecycle - `internal/artifact/origin.go` — origin lifecycle and rollback semantics - `internal/db/orphaned.go` — resync carriage and the pending-flag decision Co-authored-by: Wes McKinney <wesm@users.noreply.github.com>
…b-f559 feat: upstream PR #23 stack onto purified (kenn-io#1274 DuckDB + kenn-io#1251 artifact + kenn-io#1284 Omnigent)
Two-parent merge retains all original SHAs from both lines: - parent1: origin/merged (Periscope fork Layer 2+3) - parent2: origin/cursor/agentsview-purified-onto-kenn-f559 (kenn-io replay + PR #26) Tree resolution (oramasys-method / ARCHITECTURE.md matryoshka): - Base tree: purified (upstream modernization + kenn-io#1274/kenn-io#1251/kenn-io#1284) - Overlay from merged: internal/summarize, internal/llm, context UI, install/release/sync scripts, jetbrains-plugin, ECC bundles, branding docs No synthetic upstream replay commits. Conflict resolution favors implementation plan layers; single merge commit minimizes artificial history. Synthetic pass commits preserved on cursor/agentsview-plus-periscope-synthetic-pass-f559.
Final waypoint of the staged upstream catch-up (specs/merge-upstream-waypoints.md). Merges the last 14 commits of u/main through b0b0553 (artifact-ledger reliability kenn-io#1251 follow-ups, duckdb push-watch fix, settings-page host layout, a kit-ui bump). No deep-rewrite pivots; a light cleanup pass. - Frontend: adopt upstream's newer kit-ui pin (97be355e); keep the fork's redacted-transcript CSS alongside upstream's new .settings-page-host; regenerate package-lock. - All prior fork features intact (snapshot publish, 1h cache pricing in microdollars, subagent rollup, private views, longest-prompts). The branch `cache-1h-and-projects` is now fully current with upstream. Green: go build/vet, 41 test packages (artifact + sync both pass this run), 2091 vitest tests, svelte-check, make build. (Environmental: the snapshot test needing a git `origin` remote.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Second PR in the artifact-sync split (follows #1242, which froze the wire format and added the docbank-backed store). Extracted from the closed #1239 branch with review fixes applied during extraction.
What this adds
internal/db):artifact_export_queue,artifact_publications,artifact_publication_revisions,artifact_checkpoint_heads/_floors, plus session triggers that enqueue owned-session changes. Triggers are origin-gated — they fire only once an artifact origin exists inpg_sync_state— so archives that never opt in carry no queue writes. Trigger DDL lives in Go and is installed after column migrations (drops run before), so no trigger references session columns while migrations run.internal/artifact):EnsureOrigin/AdoptOrigin/StoredOrigin. Creating or adopting an origin bootstraps the export queue; a failed bootstrap rolls the origin back (deleting the key when there was none before, since the gates test key existence) so a retry re-runs population. Adopting over a different established origin force-requeues every owned session with a generation bump, because prior acknowledgements belong to the old origin.internal/artifact): claims pending queue rows, publishes content-addressed session manifests and segments into the store, records publication revisions, and advances per-origin checkpoint heads with monotone sequence reservation. Stale claims (a writer advanced the generation mid-export) roll back atomically. Incremental export is bounded by the dirty batch; an unchanged archive costs a catalog identity check only. Full export streams all bodies, then re-checks the queue up to 32 settle rounds — hitting the bound returns the accumulated result with an error meaning "made progress, run again."CopySyncStateFromcarries queue, publication, revision, and checkpoint state across a full resync and re-dirties every copied queue row — the origin gate keeps triggers silent in the rebuild's temp DB, and a parser bump changes manifest hashes anyway, so the exporter must re-verify each session (unchanged content is cheap to skip, being content-addressed).Scope and limitations
Where to look
internal/db/artifact_publication.go— queue and ledger SQL, origin-gated enqueueinternal/db/db.go— trigger DDL split around column migrations, bootstrap/requeueinternal/artifact/export.go— publish → manifest → checkpoint ordering and claim lifecycleinternal/artifact/origin.go— origin lifecycle and rollback semanticsinternal/db/orphaned.go— resync carriage and the pending-flag decision