fix(opencode): bound shared-container sync to the changed session - #1291
fix(opencode): bound shared-container sync to the changed session#1291colinmollenhour wants to merge 15 commits into
Conversation
providerIncrementalContentChanged content-hashes a source's whole stored prefix as the last guard against a same-size, same-mtime, same-inode in-place rewrite. The verified-source gate absorbs that cost today: a signature is content-verified once to earn trust, and later passes over an unchanged source ride the trusted skip without reading bytes. No test covered that. Losing the trusted skip would turn every watcher-triggered pass into a full re-read of the whole Claude archive — invisible to correctness tests, visible only as daemon CPU on a large archive. Measured before adding the gate: the trust-earning pass reads once per session (5 and 50 for archives of those sizes) and every pass after it reads nothing. Route the freshness-gate caller through a computeFileHashPrefix var so the test can count reads, and assert the steady-state pass reads no content and does not scale with archive cardinality.
Every session in an OpenCode root lives in one physical opencode.db, but each session's fingerprint carried that container's file size. Any single session's write moved the container size and therefore changed the fingerprint of every other session in the root, dropping their freshness skip. One changed session re-parsed the entire root: measured on a 400 session container, a one-session change went from 400 skipped to 0 skipped and 48ms to 401ms, and it scales with container size. Replace the container size with a per-session composite mtime built from session.time_updated, project.time_updated, MAX(message.time_updated), and MAX(part.time_updated). Discovery, single-session source lookup, and the parse path all resolve mtime through one helper so a stored file_mtime is always comparable to the value the freshness gate checks it against. Containers whose schema lacks the child time_updated columns (older OpenCode, Kilo, MiMoCode, ICodeMate) keep the session-only mtime and the container-size fallback, so their behavior is unchanged. Verified against an isolated clone of a production container (13.5 GB, 5,981 sessions, 508k parts): 86% of parts carry time_updated different from time_created, so in-place child edits move the signal; 437 sessions have a child newer than their session row, so the session row alone is insufficient; no project's time_updated falls within 5s of its newest session, so folding project in tracks real worktree changes rather than session activity. Discovery costs ~0.5s there because part.data lives in SQLite overflow pages, so scanning (session_id, time_updated) never reads transcript bytes. Known gap, recorded in the format provenance doc: a write that moves SQLite's change counter while leaving every one of those timestamps untouched is not attributed to any session. OpenCode stamps time_updated on every row write, and a revert deletes the newest rows, which lowers the max and is still detected, so this is reachable only by external or manual database edits. The test fixture schema omitted time_updated on project, message, and part entirely, so it could not model shared-container freshness at all; align it with the production schema and have the mutation helpers maintain those columns the way OpenCode does.
There was a problem hiding this comment.
Pull request overview
This pull request fixes a performance regression in the OpenCode SQLite “shared container” format by changing the per-session freshness signal so that a single session write no longer invalidates every other session in the same opencode.db. This keeps background sync work bounded by the changed batch rather than scaling with total sessions in a container.
Changes:
- Replace shared-container size-based invalidation with a per-session composite mtime derived from session/project/message/part
time_updatedvalues. - Update OpenCode provider fingerprinting to omit container size when the composite mtime is available, while preserving legacy fallback behavior when the schema lacks required columns.
- Add/adjust integration and performance-invariant tests and fixtures to model production OpenCode schemas and pin the bounded-work invariants.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| internal/parser/opencode.go | Adds composite-mtime discovery/lookup helpers, cached schema probing, and stamps stored file_mtime from the same composite signal used for freshness gating. |
| internal/parser/opencode_provider.go | Threads composite-mtime knowledge through discovery/meta and drops container-size fingerprinting when the composite is supported. |
| internal/parser/opencode_provider_test.go | Updates fingerprint expectations to reflect composite-mtime behavior and size omission. |
| internal/sync/hash.go | Introduces an indirection seam for prefix hashing so perf invariant tests can count content reads. |
| internal/sync/engine.go | Routes freshness-gate prefix hashing through the indirection seam. |
| internal/sync/test_helpers_test.go | Makes OpenCode test fixtures mirror production schema (adds time_updated columns) and updates helpers to maintain them. |
| internal/sync/engine_integration_test.go | Adjusts integration assertions for composite-mtime semantics and project/worktree updates. |
| internal/sync/perf_invariant_test.go | Adds a new invariant test ensuring warm sync passes don’t rehash unchanged Claude archives. |
| internal/sync/opencode_container_perf_test.go | Adds a new cardinality-scaling regression test to pin per-session bounded work for shared OpenCode containers. |
| docs/internal/session-format-sources.md | Records the OpenCode SQLite change-detection evidence and known gaps in the format sources doc. |
Comments suppressed due to low confidence (1)
internal/sync/opencode_container_perf_test.go:67
- This map is currently tracking
stats.Synced, but it’s namedunchangedSkipsand the final assertion message talks about “work” growth. After renaming the map (above), update the assignment and final assertion to use the new name and describe what’s being compared.
unchangedSkips[n] = stats.Synced
})
}
assert.Equal(t, unchangedSkips[20], unchangedSkips[200],
"work for one changed session must not grow with container size")
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
roborev: Combined Review (
|
The single-session composite mtime reused the streaming form's grouped subqueries. A GROUP BY subquery is materialized over the whole container before the outer WHERE narrows to one session, so every per-session lookup scanned every message and part row in the container. The parse path calls that lookup once per session, making a full pass O(sessions x child rows). Give the single-session path its own correlated aggregates filtered by session_id so they ride the message/part session_id indexes. The result is identical; only the plan changes. On an isolated clone of a production container (13.5 GB, 5,981 sessions, 508k parts) one lookup goes from 319ms to 5ms, a full cold reindex from 13m52s to 1m20s, and the steady-state pass over every session from 11.4s to 1.03s. Pin the plan with a regression test that asserts the single-session query does not full-scan message or part; it fails against the grouped shape. Also address review feedback: rename the misleading unchangedSkips map in the container scaling test to rewritten, since it tracks rewritten sessions rather than skips, and scope the computeFileHashPrefix seam comment to freshness-gate callers, explaining why the incremental write-side hash refresh deliberately stays on the direct call.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
roborev: Combined Review (
|
Two correctness holes in the composite mtime, both able to leave archived sessions permanently stale. Deletions were invisible. The composite is a MAX over session, project, and child timestamps, so removing a message or part only moves it when the deleted row held the maximum. On an isolated clone of a production container 5,758 of 5,981 sessions (96%) carry a session or project timestamp at or above every child, so for almost every session a delete changed nothing and the removed content stayed archived indefinitely. An earlier commit message and the format provenance doc claimed a revert stays detectable because it lowers the max; that was wrong for those 96%, and both are corrected here. Carry a per-session digest of the watermark plus the child row counts in the fingerprint hash and declare FingerprintHashRequiredForFreshness, so a deleted child changes the fingerprint even when the watermark cannot. Containers without composite support emit an empty digest, which the gate treats as no constraint, so their behavior is unchanged. A concurrent write could also stamp stale content as current. Messages and parts are read through separate autocommit queries, and the watermark was read after them, so a write landing in between produced a torn transcript stamped with a watermark newer than the content read; every later sync then skipped it as fresh. Read the watermark first, so the stamp is never newer than the content and a concurrent change leaves the stored value behind the source for the next pass to pick up. Add a regression test that deletes a child while the session row holds a higher timestamp, which fails against the max-only signal. The digest costs nothing measurable: on the same container a cold reindex is 1m15s and the steady-state pass over all 5,981 sessions is 641ms.
This comment was marked as resolved.
This comment was marked as resolved.
roborev: Combined Review (
|
|
Apologies for the noise, I hope it's ok to continue fixing in this manner... I'm very impressed with Roborev which I see is one of your projects - very nice! |
Two more holes in the per-session freshness identity, both confirmed by tests that fail against the previous commit. Sources rebuilt by FindSource or reconciliation carry no discovery metadata, so the child digest was empty and the fingerprint hash was empty with it. An empty hash is treated as no constraint by the freshness gate, so a deletion-only change passed unnoticed on every non-discovery path. Return the digest from the single-session lookup and use it whenever the source did not carry one. The digest was also only (watermark, message count, part count), so replacing one child with another while preserving both counts and leaving every new timestamp below an already-higher session watermark produced an identical fingerprint and left stale content archived. Fold in the child time sums and the child id range: the sums catch an edit whose timestamp stays below the watermark, and the id range catches a same-count replacement that reuses the old timestamps. All of it reads from the child tables' main b-tree pages, so it still never touches the transcript text in overflow pages. On the production clone this costs a cold reindex 1m15s -> 1m20s and a steady-state pass over all 5,981 sessions 641ms -> 918ms.
All three findings verified as real. Two are fixed in 1839ab8; the third I am 1. Reconstructed sources carry no 2. Same-count replacement produces the same digest — confirmed, fixed. Cost on the production clone (13.5 GB, 5,981 sessions, 508k parts): cold reindex 3. Archive-wide child aggregation per watcher event — confirmed, not fixed. I have not fixed it because I do not think it can be, cheaply: a write to a On the test instrumentation point: the scaling test does assert Happy to take direction on (3) if maintainers want it closed in this PR rather |
roborev: Combined Review (
|
…used Two more findings, both confirmed by tests that fail against the previous commit. The digest folded session and project timestamps in only through the composite MAX, so a metadata update that landed below an already-higher child watermark moved neither the mtime nor the digest and a worktree rename could be skipped. Carry session.time_updated and project.time_updated as their own digest fields. openCodeSessionCompositeMtime also always ran the eight child COUNT/SUM/MIN/MAX subqueries, including for callers that discard the digest. One of those is OpenCodeSourceMtime, which backs the session watcher's 1.5s poll, so an open session view burned eight child-range scans per tick for a value it threw away. Add a watermark-only query and route the mtime-only callers (buildOpenCodeSession and openCodeSQLiteSessionMtime) at it, reserving the full aggregate for fingerprint generation. On the production clone the steady-state pass over all 5,981 sessions drops from 918ms to 715ms; the cold reindex is unchanged at ~1m20s.
Both findings verified and fixed in 585d9d3. 1. Digest misses metadata updates below the current watermark — confirmed, Worth noting for the record that this is narrower in practice than the other 2. Mtime-only polling runs the digest aggregates — confirmed, fixed. Added Measured on the production clone (13.5 GB, 5,981 sessions, 508k parts): the Both suites pass, |
roborev: Combined Review (
|
The digest reduced child state to counts, timestamp sums and min/max ids. None of those move when a non-boundary row is swapped for another carrying the same timestamp: the count holds, the sum holds, and the extrema hold because the replacement sorts between them. The session stayed classified as fresh with stale content archived. Digest every ordered (id, time_updated) pair instead, hashed to keep the stored value bounded. Counts and the watermark stay in the digest as cheap, readable discriminators, but the identity hash is what makes it collision-resistant. Add a regression test that swaps a middle part for a different id with an identical timestamp, preserving count, sum and extrema; it fails against the reduced aggregates. On the production clone this costs a cold reindex 1m20s -> 1m35s and a steady-state pass over all 5,981 sessions 715ms -> 1.197s.
…ver emit A legacy container reports composite=false and carries an empty child digest by design, but the fingerprint treated "empty" alone as "missing" and reopened the shared database for a per-session lookup. That put one extra connection and query on every session of every cold or changed-container pass, exactly the archive-scaled work this branch exists to remove. Only resolve a missing digest when composite freshness is expected. The watermark-only query test also asserted on query strings that were never executed, and its "full" variant paired the streaming counts expression with the single-session joins, so it was invalid SQL that still passed. Build it from the single-session expression and execute both against a real container, asserting they agree on the watermark.
The per-session lookups are correlated aggregates keyed on session_id, and SQLite does not index a foreign key automatically. Without an index each one degrades to a full child-table scan, and one of them backs the session watcher's 1.5s poll. Probe for a leftmost session_id index on message and part as part of composite support, so an unindexed container falls back to the session-only mtime instead of putting an archive scan on the poll path. Both test fixtures declared the child tables without any indexes while production OpenCode declares three, so the query-plan assertion was running against a schema that could not produce an indexed plan and proved nothing. Declare the production indexes in both fixtures. The assertion itself was also too weak: it rejected only "SCAN <table>", but SQLite reports SEARCH for some aggregate plans with no index in play. Require an explicit USING INDEX or USING COVERING INDEX for both child tables.
Pushed three commits ( Digest collisions on a middle-row replacement — fixed ( Legacy containers re-queried for a digest they never emit — fixed Missing indexes made an existing test vacuous — fixed ( Still open: archive-wide child aggregation per watcher event. Discovery For scale, the container gate still short-circuits discovery entirely when the |
roborev: Combined Review (
|
Confirmed, and I am not going to fix it in this PR without a maintainer The finding is correct. Discovery orders and aggregates every Why it is still open. A write to a shared SQLite container does not say On the two suggestions. Bounded batches are implementable: cap per-pass On the instrumentation. I have deliberately not added a test asserting that For context on the trade already banked: before this PR the same event re-parsed Happy to implement bounded batching if that trade is acceptable, or to leave |
A watcher event on a shared opencode.db previously listed every session through the grouped whole-container child scan, aggregating and group_concat-ing all message and part rows before any per-session freshness check, so a one-session write did work proportional to the entire archive (1.2s per event on a 13.5 GB, 5,981-session container). Changed-path classification now lists sessions through a bounded session-row watermark, MAX(session.time_updated, project.time_updated), touching no child tables, and marks those sources watermark-only. The engine skips a watermark-only source when its stored composite watermark already covers the carried value; only sessions whose watermark advances resolve the full composite and child digest through the existing indexed per-session lookup. Fingerprint adopts the looked-up composite alongside the digest so the stored watermark is never the cheap session-row value. The trade is explicit and bounded: a child write at or below the stored composite that leaves the session and project rows untouched is invisible to a watcher pass and is reconciled by the next full-discovery pass, whose carried digest still catches it. Full-sync discovery, cutoff passes, reconciliation, and tombstoning keep the complete digest listing; changed-path passes already never promote container trust, so the gate's promotion soundness is unchanged. Legacy containers without composite support keep the full listing and are never watermark-skipped. New instrumentation counts whole-container child scans and per-session child lookups, and the watcher perf test asserts a changed-path pass runs zero container scans with per-session lookups that do not grow between 20- and 200-session archives.
Following up on my previous comment: the author opted to fix this rather than What changed. Changed-path (watcher) classification no longer runs the Why this is sound. The stored Instrumentation. |
Fixed
Previously resolved (unchanged this round): both Copilot inline threads PR state: no open review items remain. |
roborev: Combined Review (
|
Two archive-scaling costs remained after the watermark-only watcher listing. First, a watcher event still materialized every session in the container as a source and a discovered file, then decided each skip with two point queries against the archive, so per-event allocations and queries scaled with the archive even though the child tables were no longer read. Changed-path classification now loads the container members' stored composite watermarks and data versions in one indexed range query (ListVirtualContainerMemberFreshness, riding idx_sessions_file_path) and drops covered watermark-only sources before they become discovered files. The comparison is per-session — a watermark that advances past its own stored composite is always kept, wherever other sessions' watermarks sit — so interleaved timestamps cannot hide a session-row change behind a newer sibling. Sessions with no stored row or a stale data version are kept unconditionally, and the filter fails open to the per-file gate on query errors. Second, periodic full passes aggregated every child identity even for a container whose captured state still matched the last fully verified pass, where the container gate then skipped every member: archive-sized scans and group_concat allocations for values nothing read. Discovery now receives a trust probe (ProviderConfig. SQLiteContainerUnchangedSinceTrust, keyed to the pass's pre-discovery captures) and lists trusted containers through the watermark form. Any write changes the container state and breaks trust, so the next full pass carries the complete child digest again — including for child-only edits below every watermark, which keeps the deferral contract intact. Reconciliation, tombstoning, and every other provider construction keep full-fidelity listings, and cutoff filtering reads the carried watermark instead of resolving per-session composites. The watcher perf test now pins that unchanged sessions are not even materialized and that sources processed per event do not grow with container size; a new idle-pass test pins zero child scans and zero per-session lookups on a trusted container, and that a subsequent child-only edit still reconciles through the digest.
Three findings on 1. 2. 3. |
Review of Fixed
Rebutted
PR state: no other open review items. |
roborev: Combined Review (
|
The deferral comments and doc entry drew the miss class too narrowly: they said a watcher pass cannot see a child write "at or below the stored composite", but any child-only write that leaves the session and project rows untouched is invisible to the session-row watermark, above the stored composite as well as below it. Correct the wording everywhere it appears and extend the deferral regression test with the above-composite variant: a child append with a fresh timestamp and an untouched session row defers on the watcher pass and reconciles on the next full-discovery pass. Also record, at the code sites themselves, the two deliberate design positions repeat reviews keep re-raising: the session-table scan in the watermark listing and the stored-member range query are O(session-count) over small fixed-width rows by design — irreducible without a watermark index that OpenCode's schema does not have and that is not agentsview's to add — and the child-only deferral is a documented contract with its reconcile path built in, not a detection gap awaiting a per-event child signal (which would be the archive-sized aggregation this path exists to avoid).
Both findings on 1. "Scans and materializes every session twice per event": rebutted. This 2. "Watermark filter can discard child-only updates": the trade is the |
Review of Rebutted
Fixed (wording + coverage)
For maintainers: the two rebutted items are now recorded at the code |
roborev: Combined Review (
|
…d metadata times The watermark filter compared the live session-row watermark against the stored composite MTimeNS, which is MAX(session, project, child times). On a session whose composite is dominated by a child timestamp, a later metadata update — a title, session, or project change stamped below that child maximum — advanced the session row without advancing past the composite, so the watcher pass skipped a change class it exists to catch and deferred it to the next full pass. Compare like-for-like instead. The persisted child digest already embeds the session and project times in their own right (they were folded in so the full pass could detect below-watermark metadata updates), so the stored session/project metadata watermark is recoverable from the fingerprint hash without any schema change: OpenCodeChildDigestMetadataWatermarkNS parses it back out, rejecting any hash shape this digest version did not write. Both gates — the batched pre-materialization filter and the per-file fallback — now compare the carried session-row watermark against that recovered value, falling back to the composite only for rows without a parseable digest (pre-digest fingerprints), which self-heals on the row's next reparse. The digest field layout is now load-bearing for this parse, so the format comment requires a prefix version bump on any layout change. The child-only deferral contract is unchanged: writes that touch neither the session nor project row still defer to the full pass. What changed is that session/project-row advances are now always candidates, wherever their own child timestamps sit. A watcher regression test covers the child-dominated composite case end to end, and a parser test pins the digest round-trip and the rejection of foreign hash shapes.
Confirmed and fixed in The defect. The watermark gates compared the live session-row watermark The fix is the suggested one, and it needed no schema change. The Tests. The requested watcher-path case is covered end to end: |
Review of Fixed
Unchanged: the child-only deferral contract and the bounded per-event PR state: no open review items. |
roborev: Combined Review (
|
… capture Streamed reconciliation built its providers without the trusted-container probe, so an idle ReconcileWatchRoots or ReconcileProviderRoots pass over a byte-unchanged container still aggregated every message and part row during DiscoverEach — background work scaling with archive size on the exact path that exists to be a cheap safety net. Reconciliation already captures container states before streaming, so the same predicate full discovery uses now rides ProviderConfig into the reconcile providers, and the streamed listing answers with the bounded watermark form for trusted containers. Their candidates all gate-skip before fingerprinting, and a container that changes mid-stream fails its recapture check and resolves full per-session fingerprints instead. That recapture check is now also enforced where it was previously only implied: watermarkOnlySQLiteSourceFresh skipped on the stored metadata watermark alone, so a trusted full discovery that listed watermark-only sources could still skip them after a concurrent write invalidated the pass's capture — hiding a child-only write beneath an unchanged metadata watermark for one extra pass. The skip now requires the current pass to hold a live, unfailed capture for the source's container (sqliteContainerPassCaptureValid); sources processed under an invalidated capture fall through to Fingerprint and resolve the full digest. An idle-reconcile regression test pins zero container child scans and zero per-session lookups over a trusted container, and that a child-only edit below every watermark still reconciles on the next pass.
Both findings on 1. Streamed reconciliation missed the trusted-container optimization: 2. Watermark skip could outlive an invalidated capture: confirmed, fixed |
Review of Fixed
PR state: no open review items. |
roborev: Combined Review (
|
Rebutted — the finding is factually incorrect on all three of its claims.
No change made. |
Review of The review otherwise assesses the synchronization work as sound, and no |
…-cutoff-io # Conflicts: # internal/sync/engine.go
roborev: Combined Review (
|
Squash of the reconciliation-scoping branch, including its base branch perf/bound-quick-sync-cutoff-io (kenn-io#1291) and review fixes. - fix(opencode): bound shared-container sync to the changed session - fix(opencode): keep single-session mtime lookup off the whole container - fix(opencode): detect deleted children and stop stamping torn reads - fix(opencode): carry the child digest on non-discovery source lookups - fix(opencode): carry metadata times in the digest and skip it when unused - fix(opencode): digest the complete ordered child identity - fix(opencode): do not re-query legacy containers for a digest they never emit - fix(opencode): require session_id indexes before composite freshness - perf(opencode): bound watcher-event container listing to the session row - perf(opencode): bound watcher materialization and idle full-pass scans - docs(opencode): state the full deferral contract and its bounded floor - fix(opencode): compare the watcher watermark like-for-like with stored metadata times - fix(opencode): bound reconcile discovery and guard watermark skips by capture - test(sync): pin steady-state Claude sync to zero source re-reads - fix(sync): bound reconciliation authority to the roots a pass was asked about - fix(sync): prove a scope in the spelling its traversal roots discover - test(db): bound ownership paging on the query rather than on what follows it - fix(sync): withhold deletion authority from a root that has no bounded proof - fix(db): keep a separator-terminated proof scope from matching no stored child - fix(parser): prove a reconciliation scope in the spelling its sources were stored under - fix(parser): prove a hermes profile in its container's configured spelling - fix(parser): resolve a hermes profile to its on-disk spelling, not the request's - fix(parser): keep an unowned profile child out of hermes traversal - fix(parser): resolve a hermes profile scope only for a profile the container owns - fix(parser): widen an unowned hermes profile child to its container instead of dropping it - fix(parser): widen an opencode container alias to its whole virtual membership - fix(parser): keep a covered request's descendant proof under its remaining ancestor - test(sync): stop NewProvider from writing shared test provider state - perf(sync): walk a shared reconciliation gateway once per pass - fix(sync): group traversal scopes without indexing an unproven slice - fix(parser): resolve a flat hermes root's descendants generically - fix(sync): revalidate the container capture before watermark-only filtering - perf(sync): bound watcher fast-path memory to one page plus the changed batch - fix(sync): advance the freshness pager past all-stale raw pages - fix(parser): widen multi-session container requests to their virtual membership
…1319) Part of the prep work for #1208, along with #1307 and #1318. The branch was squashed and includes #1291's changes; if #1291 merges on its own, this needs a rebase. ## The bug Asking reconciliation to check one directory could delete sessions in other directories. The requested path got widened to the whole configured root, and that widened claim was treated as proof when deciding what to tombstone. ## The fix Each provider now says exactly what a request covers: - Discovery may walk a wider directory, but the pass can only touch what was actually asked for. - Deleting anything requires proof the pass really looked there. - A pass that covered one whole container (a SQLite file, a Hermes archive) can clean up its members — after checking the session didn't just move somewhere else. - Blank or unrelated paths now do nothing instead of matching everything. Hermes, OpenCode, Zed, Shelley, Trae, Aider, VS Copilot, and Omnigent each get topology rules for their own layouts (shared databases, archives, profile folders). ## Also in here (from #1291) Watcher performance for shared SQLite databases: one session's write used to re-parse the whole database. Now each session has its own change signal, and a file event does work proportional to what changed, not to the archive size. ## Where to look - `internal/parser/provider.go` — the scope plan - `internal/sync/engine.go` — how the engine uses it - `internal/parser/hermes_provider.go`, `opencode_provider.go`, `multi_session_container.go` — per-provider rules Co-authored-by: Rod Boev <rodboev@users.noreply.github.com>
Problem
Every session in an OpenCode root lives in one physical
opencode.db, but eachsession's
SourceFingerprintcarried that shared container's file size(
openCodeFormatSourceSet.Fingerprint). The freshness gate inproviderSourceUnchangedInDBrequires both size and mtime to match, so anysingle session's write moved the container size and thereby changed the
fingerprint of every other session in the root, dropping their freshness skip.
The result: one changed session re-parses the entire root. On a large container
that means thousands of sessions re-read out of a multi-GB SQLite database every
time the watcher fires, which on an always-on daemon with a live OpenCode
session is effectively continuous.
dropUnchangedSharedSQLiteResultsdiscardsthe redundant results afterwards, so the writes stayed bounded and the problem
was invisible in correctness tests — only the reads, JSON decoding and transcript
building were wasted.
This is the same failure mode
providerFingerprintHashEstablishesFreshnessalready documents for Hermes ("any single-member change invalidates every
member's stat identity"). Hermes got a per-member escape hatch; the OpenCode
family never did, and it computes no fingerprint hash, so it had no fallback.
Fix
Replace the shared container's size with a per-session composite mtime:
Each component is per-session (or, for
project, correctly per-project — aworktree rename genuinely re-resolves cwd for every session in that project).
Discovery, single-session source lookup, and the parse path all resolve mtime
through one helper, so a stored
file_mtimeis always directly comparable tothe value the freshness gate checks it against.
The container is still
os.Stat-ed, but for existence only.Impact
Synthetic container, one session changed:
Before, the cost of one changed session scaled with the number of unchanged
sessions. After, unchanged sessions skip and the pass is bounded by the change.
Measured against an isolated clone of a production OpenCode container
(13.5 GB, 5,981 sessions, 104k messages, 508k parts):
The steady-state figure is the point. Before this change, any write to that
container dropped the freshness skip for all 5,981 sessions, so a pass that now
costs about a second instead re-parsed the entire container — every session read
back out of a 13.5 GB SQLite database, JSON-decoded, and rebuilt into a
transcript, only for
dropUnchangedSharedSQLiteResultsto discard all but theone that actually changed. With a live OpenCode session writing continuously,
that ran on every watcher batch.
Discovery itself, including the composite, resolves all 5,981 sessions in ~0.5s.
It is cheap because OpenCode keeps each part's
datain SQLite overflow pages,so scanning
(session_id, time_updated)never touches transcript bytes.Compatibility
Containers whose schema lacks the child
time_updatedcolumns (older OpenCode,Kilo, MiMoCode, ICodeMate) are detected by a cached
PRAGMAprobe and keep theprevious session-only mtime and the container-size fallback, so their
behaviour is unchanged. The legacy JSON storage-tree layout is untouched.
Because the composite differs from the previously stored value for most rows,
the first sync after upgrading re-parses the archive once to re-stamp
file_mtime, then settles into the steady-state pass above. On the 13.5 GBcontainer that one-time migration is the 1m35s reindex; it is self-healing and
does not repeat.
TL;DR on repeat automated-review findings
Recent review rounds keep re-raising two things that are settled design, not
defects:
OpenCode's schema has no watermark index and is not ours to alter, so any
sound candidate selection must read the session table once — a few
thousand small fixed-width rows, single-digit milliseconds. What this PR
removed is the archive-scaling part: child-table scans (hundreds of
thousands of rows) and per-event materialization, both now bounded by the
changed batch. A test asserting "rows scanned stays constant as sessions
grow" cannot honestly exist for this design.
test-pinned: a write that touches only child rows waits for the next
periodic full pass (minutes), whose digest catches it; live-viewed
sessions are covered by the 1.5s per-session poll. Detecting it per event
requires reading child rows — exactly the archive-sized work earlier
reviews required removing. The same trade cannot be simultaneously a
required optimization and a reported bug.
Watcher passes are bounded by the changed batch
The full digest listing aggregates every
messageandpartrow of acontainer, so running it on each watcher event made a one-session write do
work proportional to the archive (the 1.20s steady-state figure above).
Watcher changed-path passes now avoid it entirely: they list sessions through
a bounded session-row watermark —
MAX(session.time_updated, project.time_updated), touching no child tables — compare it per session andlike-for-like against the stored session/project metadata watermark
(recovered from the persisted child digest, which already embeds those times;
loaded in one indexed range query), and drop covered sessions before they are
even materialized into discovered files. Only the changed batch flows into
the sync pipeline and resolves the full composite and child digest through
the indexed per-session lookup. Like-for-like matters: the stored composite
can be dominated by a newer child timestamp, and comparing the session-row
watermark against it would hide a metadata update stamped below that child
maximum. A session or project row that advances past its own stored metadata
watermark is always a candidate, wherever other sessions' watermarks or its
own child timestamps sit.
Periodic full passes and streamed reconciliation passes get the same
treatment when there is nothing to do: a container whose captured state
still matches the last fully verified pass is listed through the watermark
form, because the container gate then skips every member before
fingerprinting and the child identity scan would be archive-sized work
nothing reads. Any write breaks that trust and the next pass carries the
complete digest again — and watermark-only skips additionally require the
pass's container capture to survive its recapture check, so a container
that changes mid-pass resolves full per-session digests instead of skipping
on a stale premise.
The trade is explicit: a child write at or below the stored composite that
leaves the session and project rows untouched is invisible to a watcher pass
and is reconciled by the next full-discovery pass over the now-untrusted
container, whose carried digest still catches it. Cutoff passes,
reconciliation, and tombstoning keep the complete digest listing;
changed-path passes already never promote container trust, so gate promotion
soundness is unchanged; legacy containers are never watermark-skipped.
Actively watched sessions do not rely on this path at all — their poll
resolves the composite per session. Instrumentation counts the two query
shapes, and regression tests assert a watcher event runs zero
whole-container child scans with sources processed and per-session lookups
that do not grow between 20- and 200-session archives, and that idle full
passes run zero child scans.
Deletions, and a correction
A MAX over timestamps cannot see a deletion. Removing a message or part only
moves the composite when the deleted row happened to hold the maximum, and on
the production clone 5,758 of 5,981 sessions (96%) carry a session or
project timestamp at or above every child — so for almost every session a delete
would have changed nothing and the removed content would have stayed archived
indefinitely.
An earlier revision of this description claimed a revert stays detectable
because it lowers the max. That was wrong for those 96%. The fingerprint hash
now carries a per-session digest of the watermark plus the child row counts, and
freshness compares it (
FingerprintHashRequiredForFreshness), so a deletechanges the fingerprint even when the watermark cannot. A regression test
deletes a child while the session row holds a higher timestamp; it fails against
the max-only signal.
The watermark is also now read before the message and part queries. Those are
separate autocommit reads, so a concurrent write landing between them previously
produced a torn transcript stamped with a watermark newer than the content read,
which every later sync would skip as fresh. Reading it first inverts the race:
the stamp is never newer than the content, so a concurrent change leaves the
stored value behind the source and the next pass re-syncs it.
Remaining gap: a write that leaves the watermark, the message count and the part
count all unchanged is not attributed to any session, which requires an in-place
edit that does not stamp
time_updated. It is recorded with its supportingevidence in
docs/internal/session-format-sources.md.Where to look
internal/parser/opencode.go— composite expression, cached schema probe,the single helper that discovery/lookup/parse all resolve mtime through, and
the bounded session-row watermark listing used by changed-path passes.
internal/parser/opencode_provider.go— the fingerprint no longer stamps theshared container's size when the composite is available; watermark-only
sources adopt the looked-up composite and digest together.
internal/sync/opencode_container_gate.go— the batched pre-materializationfilter for watermark-only changed-path sources, the per-file watermark skip,
and the discovery trust probe that lets idle full passes list trusted
containers without the child scan.
internal/sync/test_helpers_test.go— the fixture schema previously omittedtime_updatedonproject,messageandpartentirely, so it could notmodel shared-container freshness at all. That gap is why this regression was
not caught. The schema now matches production and the mutation helpers
maintain those columns the way OpenCode does.
Three existing assertions changed as a consequence, and reviewers should weigh
them: two asserted
file_mtimestays constant across a rewrite, which no longerholds now that the composite legitimately advances with the changed rows; the
third asserted a cwd-vetoed container produces zero skips, which is now one —
the persisted session skips on its own per-session freshness while the vetoed
session is still reprocessed. A trusted-container gate skip would produce two,
which is the promotion violation that test exists to catch, so the invariant it
guards is still pinned.
A cardinality-scaling regression test
(
TestOpenCodeSharedContainerChangeIsPerSessionBounded) pins the invariant andfails without the fix.