diff --git a/docs/internal/session-format-sources.md b/docs/internal/session-format-sources.md index f3a936c1b..6f7f2ee22 100644 --- a/docs/internal/session-format-sources.md +++ b/docs/internal/session-format-sources.md @@ -307,6 +307,82 @@ Grok section and remove the explicit registry exception in the coverage test. counts as a failure, matching the existing `exit status N` heuristic, and a timed-out command records `timeout: true` with no `exit` key, so it is not detected here. See #1256. +- **Change detection (SQLite layout):** every session in a root shares one + physical `opencode.db`, so the container's own size and mtime move whenever + any single session is written and cannot discriminate between sessions. + Agentsview instead builds a per-session composite from + `session.time_updated`, `project.time_updated`, `MAX(message.time_updated)`, + and `MAX(part.time_updated)` (`openCodeCompositeMtimeExpr`), and omits the + container size from the per-session fingerprint. Verified 2026-07-27 against + an isolated clone of a production container (13.5 GB, 5,981 sessions, 104k + messages, 508k parts): 432,779 of 508,400 parts (86%) carry + `time_updated != time_created`, so in-place child edits do move the signal; + 437 sessions have `MAX(part.time_updated) > session.time_updated`, so the + session row alone is insufficient; and no project's `time_updated` falls + within 5s of its newest session, so folding `project` in tracks genuine + worktree/metadata changes rather than ordinary session activity. The child + scans cost ~0.6s warm on that container because `part.data` lives in SQLite + overflow pages, so scanning `(session_id, time_updated)` does not read + transcript bytes. A MAX over timestamps cannot see a deletion: on that + container 5,758 of 5,981 sessions (96%) carry a session or project timestamp + at or above every child, so removing a message or part leaves the max + untouched. The fingerprint hash therefore carries a per-session digest of the + watermark plus the child row counts, and freshness compares it + (`FingerprintHashRequiredForFreshness`). An earlier revision of this entry + claimed a revert stays detectable because it lowers the max; that is wrong for + the 96% above, and the row counts are what actually cover deletions. Known + gap: a write that leaves the watermark, the message count and the part count + all unchanged is not attributed to any session, which requires an in-place + edit that does not stamp `time_updated`. Containers whose schema lacks the + child `time_updated` columns (older OpenCode, Kilo, MiMoCode, ICodeMate) fall + back to the session-only mtime plus the container size and emit an empty + digest, preserving prior behavior. Watcher events do not pay the child scan + at all: changed-path classification lists sessions through a bounded + session-row watermark (`MAX(session.time_updated, project.time_updated)`, + `ForEachOpenCodeSessionWatermarkMeta`, ordered by session id), compares it + per session and like-for-like against the stored session/project metadata + watermark recovered from the persisted child digest + (`OpenCodeChildDigestMetadataWatermarkNS`; rows without a parseable digest + fall back to the stored composite), merged in ascending virtual-path order + against a paged stored-freshness cursor + (`ListVirtualContainerMemberFreshnessPage` through + `storedMemberFreshnessPager` and `changedWatermarkSources`), and drops + covered sessions during the stream — only the changed batch is ever + materialized, peak memory per event is one stored page plus that batch, + and the surviving sources resolve the full composite and digest through + the indexed per-session lookup. The merge trusts stored authority only + while a container capture taken before the listing still matches a + recapture afterwards; a stale capture re-lists unfiltered and leaves the + decision to the per-file gates. The comparison must be like-for-like: the + stored composite can be dominated by a newer child timestamp, and + comparing the session-row watermark against it would hide a metadata + update (title, directory, worktree rename) whose stamp lands below that + child maximum. A session or project row that advances past its own stored + metadata watermark is always a candidate, wherever other sessions' + watermarks or its own child timestamps sit. Periodic full passes and + streamed reconciliation passes over a container whose captured state still + matches the last fully verified pass also list the watermark form + (`SQLiteContainerUnchangedSinceTrust`): every member gate-skips before + fingerprinting, so the child identity scan would be archive-sized work + nothing reads; any write breaks that trust and the next pass carries the + complete digest again. Watermark-only skips additionally require the + pass's container capture to still be valid + (`sqliteContainerPassCaptureValid`) — a container that changes between + listing and the recapture check resolves full per-session digests instead, + so a concurrent child-only write cannot hide beneath an unchanged metadata + watermark. The trade is explicit: any + child-only write that leaves the session and project rows untouched — + wherever its timestamps land relative to the stored composite — is + invisible to a watcher pass and is reconciled by the next full-discovery + pass over the now-untrusted container, whose digest still catches it; on + the production container above, 96% of sessions carry a session/project + timestamp at or above every child, and actively watched sessions bypass + this entirely via the per-session composite poll. Per-event work is + bounded by the changed batch plus one O(session-count) scan of small + fixed-width rows (the session table and the paged stored-member reads); + that floor is irreducible without a watermark index, which OpenCode's + schema does not have and which is not agentsview's to add — but only the + changed batch and one stored page are ever held in memory. - **Agentsview:** `internal/parser/opencode.go`, `internal/parser/opencode_provider.go`, and `internal/parser/opencode_storage_state.go`; legacy and database layouts are diff --git a/internal/db/sessions.go b/internal/db/sessions.go index aff3084e1..eb5069cd7 100644 --- a/internal/db/sessions.go +++ b/internal/db/sessions.go @@ -2231,6 +2231,141 @@ func (db *DB) GetFileInfoByPath( return s.Int64, m.Int64, true } +// VirtualContainerMemberFreshness is one stored virtual member's freshness +// signal: the newest stored file_mtime for its path, the minimum stored +// data version, and the newest row's fingerprint hash, mirroring +// GetFileInfoByPath, GetDataVersionByPath, and GetFileHashByPath. +type VirtualContainerMemberFreshness struct { + MTimeNS int64 + DataVersion int + Hash string +} + +// VirtualContainerMemberFreshnessRow pairs one virtual member path with its +// folded freshness signal. +type VirtualContainerMemberFreshnessRow struct { + Path string + VirtualContainerMemberFreshness +} + +// ListVirtualContainerMemberFreshnessPage returns the freshness signal for +// stored sessions whose file_path is a virtual member of the shared container +// at containerPath ("#"), excluding source-missing +// tombstones: at most limit member paths strictly after afterPath, in +// ascending path order, and whether the container's stored membership is +// exhausted. Changed-path classification merges a streamed watermark-only +// listing against these pages, so a one-session write flows one candidate +// into the sync pipeline while peak memory stays one page — never the +// container's full membership. +// +// Two queries per page keep each path's fold complete without materializing +// the container: a DISTINCT path page rides idx_sessions_file_path ('$' is +// the ASCII successor of '#', so the half-open range covers exactly the +// "#" prefix), and the row fetch is bounded to that page's +// [first, last] interval, so duplicate session rows for one path can never +// split across pages. Folded in Go rather than GROUP BY: the fold needs +// MAX(file_mtime), MIN(data_version), and the hash of the newest-mtime row, +// and SQLite's bare-column-from-the-extreme-row guarantee only holds with +// exactly one min/max aggregate in the query. +func (db *DB) ListVirtualContainerMemberFreshnessPage( + ctx context.Context, containerPath, afterPath string, limit int, +) ([]VirtualContainerMemberFreshnessRow, bool, error) { + if containerPath == "" || limit <= 0 { + return nil, true, nil + } + notMissing := " AND (deletion_cause IS NULL" + + " OR deletion_cause <> '" + deletionCauseSourceMissing + "')" + lower := containerPath + "#" + lowerOp := ">=" + if afterPath > lower { + lower = afterPath + lowerOp = ">" + } + pathRows, err := db.getReader().QueryContext(ctx, + "SELECT DISTINCT file_path FROM sessions"+ + " WHERE file_path "+lowerOp+" ? AND file_path < ? || '$'"+ + notMissing+ + " ORDER BY file_path LIMIT ?", + lower, containerPath, limit, + ) + if err != nil { + return nil, false, fmt.Errorf( + "listing container member paths %s: %w", containerPath, err, + ) + } + defer pathRows.Close() + var paths []string + for pathRows.Next() { + var path string + if err := pathRows.Scan(&path); err != nil { + return nil, false, fmt.Errorf( + "scanning container member path %s: %w", containerPath, err, + ) + } + paths = append(paths, path) + } + if err := pathRows.Err(); err != nil { + return nil, false, err + } + if len(paths) == 0 { + return nil, true, nil + } + done := len(paths) < limit + + rows, err := db.getReader().QueryContext(ctx, + "SELECT file_path, file_mtime, data_version, file_hash FROM sessions"+ + " WHERE file_path >= ? AND file_path <= ?"+notMissing, + paths[0], paths[len(paths)-1], + ) + if err != nil { + return nil, false, fmt.Errorf( + "listing container member freshness %s: %w", containerPath, err, + ) + } + defer rows.Close() + members := make(map[string]VirtualContainerMemberFreshness, len(paths)) + for rows.Next() { + var path string + var mtime, version sql.NullInt64 + var hash sql.NullString + if err := rows.Scan(&path, &mtime, &version, &hash); err != nil { + return nil, false, fmt.Errorf( + "scanning container member freshness %s: %w", + containerPath, err, + ) + } + row := VirtualContainerMemberFreshness{ + MTimeNS: mtime.Int64, + DataVersion: int(version.Int64), + Hash: hash.String, + } + member, seen := members[path] + if !seen { + members[path] = row + continue + } + if row.MTimeNS > member.MTimeNS { + member.MTimeNS = row.MTimeNS + member.Hash = row.Hash + } + if row.DataVersion < member.DataVersion { + member.DataVersion = row.DataVersion + } + members[path] = member + } + if err := rows.Err(); err != nil { + return nil, false, err + } + page := make([]VirtualContainerMemberFreshnessRow, 0, len(paths)) + for _, path := range paths { + page = append(page, VirtualContainerMemberFreshnessRow{ + Path: path, + VirtualContainerMemberFreshness: members[path], + }) + } + return page, done, nil +} + // GetProjectByPath returns the stored project for the newest // non-deleted session matching file_path. func (db *DB) GetProjectByPath(path string) (project string, ok bool) { @@ -2502,8 +2637,14 @@ func (db *DB) listActiveSessionSourceOwnershipScopeBatch( for _, scope := range scopes { root := scope.Path likeRoot := sqliteLikeEscape(root) + // A drive root or share root already ends in a separator, so the + // child prefix must not add a second one: stored children carry one. + childPrefix := likeRoot + string(filepath.Separator) + "%" + if strings.HasSuffix(root, string(filepath.Separator)) { + childPrefix = likeRoot + "%" + } rootClause := `(b.file_path = ? OR b.file_path LIKE ? ESCAPE '!')` - args = append(args, root, likeRoot+string(filepath.Separator)+"%") + args = append(args, root, childPrefix) if scope.IncludeVirtualMembers { // Mirror storedSourcePathHintInRoot: a virtual member is the // container plus '#' and a nonempty single segment. Nested stored @@ -2979,17 +3120,51 @@ func storedSourcePathHintInAnyRoot( return false } +// StoredSourcePathHintScopesContain reports whether one stored or discovered +// source path lies inside any of the bounded scopes, using the same +// platform-aware containment the SQLite queries in this file re-check. It is +// the single Go-side authority for path-to-scope membership: the SQL LIKE +// prefilters stay a superset of this predicate for ASCII paths, and every +// caller that pages or admits by scope must apply it rather than comparing +// prefixes itself. +func StoredSourcePathHintScopesContain( + path string, scopes []StoredSourcePathHintScope, +) bool { + return storedSourcePathHintInAnyRoot(path, scopes) +} + func storedSourcePathHintInRoot(path string, scope StoredSourcePathHintScope) bool { path = cleanStoredSourcePathHint(path) root := cleanStoredSourcePathHint(scope.Path) - if path == root || strings.HasPrefix(path, root+string(filepath.Separator)) { + if storedSourcePathSameOrDescendant(path, root) { return true } - suffix, ok := strings.CutPrefix(path, root+"#") - return ok && - scope.IncludeVirtualMembers && - suffix != "" && - !strings.ContainsAny(suffix, `/\`) + if !scope.IncludeVirtualMembers || len(path) < len(root)+2 || + path[len(root)] != '#' { + return false + } + // A virtual member is the container plus '#' and a nonempty single + // segment. The container prefix folds with the same platform semantics as + // the directory branch; the member segment rule stays byte-exact. + if rel, err := filepath.Rel(root, path[:len(root)]); err != nil || rel != "." { + return false + } + member := path[len(root)+1:] + return !strings.ContainsAny(member, `/\`) +} + +// storedSourcePathSameOrDescendant compares with filepath.Rel semantics so +// containment matches platform path equality: case-folded per element on +// Windows, byte-exact on Unix. This keeps the Go predicate aligned with the +// ASCII-case-insensitive SQL LIKE prefilter on Windows while staying exact on +// case-sensitive filesystems. +func storedSourcePathSameOrDescendant(path, root string) bool { + rel, err := filepath.Rel(root, path) + if err != nil { + return false + } + return rel == "." || rel != ".." && + !strings.HasPrefix(rel, ".."+string(filepath.Separator)) } func sqliteLikeEscape(value string) string { diff --git a/internal/db/stored_source_scope_test.go b/internal/db/stored_source_scope_test.go new file mode 100644 index 000000000..0a1cd9a89 --- /dev/null +++ b/internal/db/stored_source_scope_test.go @@ -0,0 +1,166 @@ +package db + +import ( + "fmt" + "path/filepath" + "runtime" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +// TestListActiveSessionSourceOwnershipScopesPagePagesChildrenOfASeparatorRoot +// covers the root spelling that already ends in a separator. A drive root or +// share root is its own parent, so appending a second separator to build the +// child prefix produces a pattern no stored child matches, and the pass pages +// zero rows and tombstones nothing while still crediting coverage. +func TestListActiveSessionSourceOwnershipScopesPagePagesChildrenOfASeparatorRoot( + t *testing.T, +) { + if runtime.GOOS != "windows" { + t.Skip("only a Windows drive root survives filepath.Clean with a trailing separator") + } + d := testDB(t) + // A drive root is the one spelling filepath.Clean cannot shorten, so it + // still carries its separator when the query is built. The paths are + // stored strings; no file has to exist for the predicate to be exercised. + const root = `C:\` + const child = `C:\session.jsonl` + seeds := []storedSourcePathSeed{ + {id: "child", agent: "claude", path: child}, + } + insertSessionsWithSourcePaths(t, d, seeds) + require.NoError(t, d.BaselineActiveSessionSourcePaths( + t.Context(), defaultMachine, sourcePathsFromSeeds(seeds), + )) + + page, err := d.ListActiveSessionSourceOwnershipScopesPage( + t.Context(), defaultMachine, "claude", + []StoredSourcePathHintScope{{Path: root}}, SessionSourceCursor{}, + ) + require.NoError(t, err) + require.Len(t, page, 1, + "a child of a separator-terminated root must page") + assert.Equal(t, "child", page[0].ID) +} + +// TestListActiveSessionSourceOwnershipScopesPageExcludesOutOfScopeRows bounds +// the query itself rather than what the engine does with its result. The +// engine skips an out-of-scope row before it ever stats the file, so a test +// that only records stat calls passes just as well against a widened query. +// The out-of-scope rows here sort between the in-scope rows and span a page +// boundary, so a query widened to the parent would surface them in the first +// page. +func TestListActiveSessionSourceOwnershipScopesPageExcludesOutOfScopeRows( + t *testing.T, +) { + d := testDB(t) + root := t.TempDir() + inScope := filepath.Join(root, "in") + outOfScope := filepath.Join(root, "in-sibling") + + var seeds []storedSourcePathSeed + wantIDs := make([]string, 0, WatchReconcileSourcePageSize+2) + for i := range WatchReconcileSourcePageSize + 2 { + id := fmt.Sprintf("in-%03d", i) + wantIDs = append(wantIDs, id) + seeds = append(seeds, storedSourcePathSeed{ + id: id, + agent: "claude", + path: filepath.Join(inScope, fmt.Sprintf("source-%03d.jsonl", i)), + }) + seeds = append(seeds, storedSourcePathSeed{ + id: fmt.Sprintf("out-%03d", i), + agent: "claude", + path: filepath.Join( + outOfScope, fmt.Sprintf("source-%03d.jsonl", i), + ), + }) + } + insertSessionsWithSourcePaths(t, d, seeds) + require.NoError(t, d.BaselineActiveSessionSourcePaths( + t.Context(), defaultMachine, sourcePathsFromSeeds(seeds), + )) + + scopes := []StoredSourcePathHintScope{{Path: inScope}} + var paged []SessionSourceOwnership + cursor := SessionSourceCursor{} + for range 8 { + page, err := d.ListActiveSessionSourceOwnershipScopesPage( + t.Context(), defaultMachine, "claude", scopes, cursor, + ) + require.NoError(t, err) + if len(page) == 0 { + break + } + paged = append(paged, page...) + cursor = page[len(page)-1].Cursor() + } + + gotIDs := make([]string, 0, len(paged)) + for _, ownership := range paged { + assert.True(t, + StoredSourcePathHintScopesContain(ownership.FilePath, scopes), + "paged row %s at %s is outside the requested scope", + ownership.ID, ownership.FilePath) + gotIDs = append(gotIDs, ownership.ID) + } + assert.Equal(t, wantIDs, gotIDs, + "every in-scope row pages exactly once and no sibling row joins it") +} + +func TestStoredSourcePathHintScopesContainDirectoryAndMemberBranches(t *testing.T) { + root := filepath.Join(t.TempDir(), "archive") + memberScope := []StoredSourcePathHintScope{{ + Path: filepath.Join(root, "state.db"), IncludeVirtualMembers: true, + }} + directoryScope := []StoredSourcePathHintScope{{Path: root}} + + assert.True(t, StoredSourcePathHintScopesContain(root, directoryScope)) + assert.True(t, StoredSourcePathHintScopesContain( + filepath.Join(root, "sessions", "a.jsonl"), directoryScope, + )) + assert.False(t, StoredSourcePathHintScopesContain( + filepath.Join(filepath.Dir(root), "sibling", "a.jsonl"), directoryScope, + )) + assert.False(t, StoredSourcePathHintScopesContain(root+"suffix", directoryScope), + "a sibling sharing the root as a string prefix is outside the scope") + + member := filepath.Join(root, "state.db") + "#session-1" + assert.True(t, StoredSourcePathHintScopesContain(member, memberScope)) + assert.False(t, StoredSourcePathHintScopesContain( + member, + []StoredSourcePathHintScope{{Path: filepath.Join(root, "state.db")}}, + ), "virtual members of the scope path itself require the provider's declaration") + assert.False(t, StoredSourcePathHintScopesContain( + filepath.Join(root, "state.db")+"#nested/session.json", memberScope, + ), "nested member segments belong to other sources") +} + +func TestStoredSourcePathHintScopesContainMatchesPlatformCase(t *testing.T) { + root := filepath.Join(t.TempDir(), "Archive") + variant := strings.ToUpper(root) + if variant == root { + t.Skip("path has no case variant") + } + scope := []StoredSourcePathHintScope{{Path: variant}} + memberScope := []StoredSourcePathHintScope{{ + Path: strings.ToUpper(filepath.Join(root, "state.db")), + IncludeVirtualMembers: true, + }} + inRoot := filepath.Join(root, "sessions", "a.jsonl") + member := filepath.Join(root, "state.db") + "#session-1" + + if runtime.GOOS == "windows" { + assert.True(t, StoredSourcePathHintScopesContain(inRoot, scope), + "Windows containment folds case like the filesystem") + assert.True(t, StoredSourcePathHintScopesContain(member, memberScope), + "the member container prefix folds like the directory branch") + } else { + assert.False(t, StoredSourcePathHintScopesContain(inRoot, scope), + "case-sensitive filesystems keep byte-exact containment") + assert.False(t, StoredSourcePathHintScopesContain(member, memberScope)) + } +} diff --git a/internal/db/virtual_member_freshness_test.go b/internal/db/virtual_member_freshness_test.go new file mode 100644 index 000000000..751061e64 --- /dev/null +++ b/internal/db/virtual_member_freshness_test.go @@ -0,0 +1,91 @@ +package db + +import ( + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func seedVirtualMemberRow( + t *testing.T, d *DB, id, path string, mtimeNS int64, version int, + hash string, +) { + t.Helper() + session := Session{ + ID: id, Project: "proj", Machine: defaultMachine, Agent: "opencode", + FilePath: &path, FileMtime: &mtimeNS, + } + if hash != "" { + session.FileHash = &hash + } + require.NoError(t, d.UpsertSession(session)) + require.NoError(t, d.SetSessionDataVersion(id, version)) +} + +// TestListVirtualContainerMemberFreshnessPagePagesCompleteFolds pins the +// paged freshness read: pages arrive in ascending path order honoring the +// limit, duplicate session rows for one member fold completely even when the +// page boundary lands on that member (the MAX-mtime row carries its hash +// while the MIN data version survives), source-missing tombstones and paths +// outside the container never surface, and the done flag reports exhaustion. +func TestListVirtualContainerMemberFreshnessPagePagesCompleteFolds( + t *testing.T, +) { + d := testDB(t) + const container = "/data/opencode.db" + // Member "a" is stored twice: the newer row carries the hash the fold + // must surface and the lower data version the fold must keep. + seedVirtualMemberRow(t, d, "a-old", container+"#a", 100, 5, "old-hash") + seedVirtualMemberRow(t, d, "a-new", container+"#a", 200, 2, "new-hash") + seedVirtualMemberRow(t, d, "b", container+"#b", 150, 5, "") + seedVirtualMemberRow(t, d, "c", container+"#c", 300, 5, "") + seedVirtualMemberRow(t, d, "outside", "/data/other.db#x", 400, 5, "") + _, err := d.getWriter().Exec( + `UPDATE sessions SET + deleted_at = strftime('%Y-%m-%dT%H:%M:%fZ','now'), + deletion_cause = 'source_missing' + WHERE id = 'b'`, + ) + require.NoError(t, err, "tombstone member b") + + page, done, err := d.ListVirtualContainerMemberFreshnessPage( + t.Context(), container, "", 1, + ) + require.NoError(t, err) + assert.False(t, done, "a full page must not report exhaustion") + require.Len(t, page, 1) + assert.Equal(t, container+"#a", page[0].Path) + assert.Equal(t, int64(200), page[0].MTimeNS, + "the fold surfaces the newest row's mtime") + assert.Equal(t, "new-hash", page[0].Hash, + "the hash rides the newest-mtime row") + assert.Equal(t, 2, page[0].DataVersion, + "the minimum stored data version survives the fold") + + page, done, err = d.ListVirtualContainerMemberFreshnessPage( + t.Context(), container, page[0].Path, 1, + ) + require.NoError(t, err) + assert.False(t, done) + require.Len(t, page, 1) + assert.Equal(t, container+"#c", page[0].Path, + "the tombstoned member must never surface") + + page, done, err = d.ListVirtualContainerMemberFreshnessPage( + t.Context(), container, page[0].Path, 1, + ) + require.NoError(t, err) + assert.True(t, done, "an empty page reports exhaustion") + assert.Empty(t, page) + + page, done, err = d.ListVirtualContainerMemberFreshnessPage( + t.Context(), container, "", 10, + ) + require.NoError(t, err) + assert.True(t, done, "a short page reports exhaustion") + require.Len(t, page, 2, + "one call over a large limit folds the whole membership") + assert.Equal(t, container+"#a", page[0].Path) + assert.Equal(t, container+"#c", page[1].Path) +} diff --git a/internal/parser/capabilities_sync_test.go b/internal/parser/capabilities_sync_test.go index 2fd6e17a8..1902e39c5 100644 --- a/internal/parser/capabilities_sync_test.go +++ b/internal/parser/capabilities_sync_test.go @@ -55,17 +55,26 @@ func TestProviderSyncSemanticsDeclarations(t *testing.T) { AgentShelley: { UnchangedResults: UnchangedResultMTimeAndHash, }, + // The OpenCode family shares one physical container per root, so + // freshness consults the per-session child digest: it is the only + // signal that sees a deleted message or part, which a MAX over + // timestamps cannot. Containers without composite support emit an + // empty hash, which the gate treats as no constraint. AgentOpenCode: { - UnchangedResults: UnchangedResultMTimeAndHash, + UnchangedResults: UnchangedResultMTimeAndHash, + FingerprintHashRequiredForFreshness: true, }, AgentKilo: { - UnchangedResults: UnchangedResultMTimeAndHash, + UnchangedResults: UnchangedResultMTimeAndHash, + FingerprintHashRequiredForFreshness: true, }, AgentMiMoCode: { - UnchangedResults: UnchangedResultMTimeAndHash, + UnchangedResults: UnchangedResultMTimeAndHash, + FingerprintHashRequiredForFreshness: true, }, AgentIcodemate: { - UnchangedResults: UnchangedResultMTimeAndHash, + UnchangedResults: UnchangedResultMTimeAndHash, + FingerprintHashRequiredForFreshness: true, }, AgentOmnigent: { FingerprintHashInCacheKey: true, diff --git a/internal/parser/hermes_provider.go b/internal/parser/hermes_provider.go index f51b53be6..a3223a576 100644 --- a/internal/parser/hermes_provider.go +++ b/internal/parser/hermes_provider.go @@ -13,6 +13,7 @@ import ( "log" "os" "path/filepath" + "slices" "sort" "strings" ) @@ -71,10 +72,19 @@ func (p *hermesProvider) SourcesForChangedPath( return p.sources.SourcesForChangedPath(ctx, req) } -func (p *hermesProvider) ReconciliationOwnershipScopes( - root string, -) []StoredSourceHintScope { - return p.sources.ReconciliationOwnershipScopes(root) +// ResolveReconciliationScopes overrides the generic directory topology with +// Hermes archive topology: state.db and sessions/ are two spellings of one +// archive, a profiles container fans out to per-profile archives, and any +// requested path inside an archive resolves to that whole archive's scope. +func (p *hermesProvider) ResolveReconciliationScopes( + _ context.Context, req ReconciliationScopeRequest, +) (ReconciliationScopePlan, error) { + if err := ValidateReconciliationScopeRoots( + AgentHermes, p.sources.roots, req.Roots, + ); err != nil { + return ReconciliationScopePlan{}, err + } + return p.sources.reconciliationScopePlan(req), nil } func (p *hermesProvider) SourceForReconciliation( @@ -333,9 +343,9 @@ func isHermesProfilesContainer(root string) bool { // per-profile archive roots. A missing container simply means no profiles // exist yet; any other ReadDir failure (permissions, transient I/O) is // returned so callers report the expansion as incomplete discovery instead -// of silently claiming zero profiles — ReconciliationOwnershipScopes still -// claims the whole container, so a swallowed failure would let the engine -// tombstone every stored hermes session under it as source_missing. +// of silently claiming zero profiles — the resolved reconciliation scope +// still proves the whole container, so a swallowed failure would let the +// engine tombstone every stored hermes session under it as source_missing. func hermesProfileArchiveRoots(profilesRoot string) ([]string, error) { entries, err := os.ReadDir(profilesRoot) if err != nil { @@ -662,31 +672,260 @@ func (s hermesSourceSet) WatchPlan(context.Context) (WatchPlan, error) { return WatchPlan{Roots: roots}, nil } -func (s hermesSourceSet) ReconciliationOwnershipScopes( - requestedRoot string, -) []StoredSourceHintScope { - requestedRoot = filepath.Clean(requestedRoot) - requestedMatchRoot := absoluteHermesPath(requestedRoot) - var scopes []StoredSourceHintScope - for _, configuredRoot := range s.roots { - if !hermesPathWithinOrSame( - absoluteHermesPath(configuredRoot), requestedMatchRoot, - ) { +// hermesReconciliationUnit is one atomic Hermes archive: the smallest scope a +// reconciliation request can resolve to. Every alias spelling of the archive +// resolves to the same identity and the same proof scopes, so no spelling can +// omit the state.db members or the sessions/ transcripts. +type hermesReconciliationUnit struct { + identity string + proofs []StoredSourceHintScope + stateDB string +} + +// hermesArchiveReconciliationUnit derives the archive identity and physical +// proof scopes for one archive root, mirroring the ownership scopes the +// provider historically declared: virtual members of a present state.db plus +// the transcript directory. A root that is not archive-shaped falls back to a +// plain directory unit. +func hermesArchiveReconciliationUnit(archiveRoot string) hermesReconciliationUnit { + stateDB, sessionsDir, ok := hermesArchiveRootPaths(archiveRoot) + if !ok { + clean := filepath.Clean(archiveRoot) + return hermesReconciliationUnit{ + identity: absoluteHermesPath(clean), + proofs: []StoredSourceHintScope{{Path: clean}}, + } + } + unit := hermesReconciliationUnit{stateDB: filepath.Clean(stateDB)} + unit.identity = absoluteHermesPath(filepath.Dir(unit.stateDB)) + if IsRegularFile(unit.stateDB) { + unit.proofs = append(unit.proofs, StoredSourceHintScope{ + Path: unit.stateDB, IncludeVirtualMembers: true, + }) + } + unit.proofs = append(unit.proofs, StoredSourceHintScope{ + Path: filepath.Clean(sessionsDir), + }) + return unit +} + +// hermesRequestDenotesArchive reports whether one requested root is inside or +// an alias spelling of the archive anchored at stateDB: the archive directory, +// the state.db file, the sessions/ directory, or any path beneath either. +func hermesRequestDenotesArchive(requested, stateDB string) bool { + if stateDB == "" { + return false + } + absStateDB := absoluteHermesPath(stateDB) + if qStateDB, _, ok := hermesArchiveRootPaths(requested); ok && + samePath(absoluteHermesPath(qStateDB), absStateDB) { + return true + } + if hermesPathWithinOrSame(requested, absStateDB) { + return true + } + sessionsDir := filepath.Join(filepath.Dir(absStateDB), "sessions") + return hermesPathWithinOrSame(requested, sessionsDir) +} + +// hermesProfileScopeRoot resolves a request inside a profiles container to one +// profile's archive root, spelled the way discovery stores it. Selection runs +// against the absolute container so any request spelling picks the right +// profile, but the returned root is always one the container enumerated, never +// anything the caller spelled. That enumeration joins the configured container +// with the on-disk directory name, which is the spelling discovery stores +// under, so proof matches and no request can mint a second identity for one +// profile by naming it differently. +// +// A request the container does not currently own returns false: a profile +// already deleted, a symlinked or non-directory child, which the enumeration +// deliberately skips, or a container that could not be read. The caller +// widens those to the container itself rather than reconstructing a root from +// the request. +func hermesProfileScopeRoot(profiles []string, requested string) (string, bool) { + for _, candidate := range profiles { + if hermesPathWithinOrSame(requested, absoluteHermesPath(candidate)) { + return candidate, true + } + } + return "", false +} + +func (s hermesSourceSet) reconciliationScopePlan( + req ReconciliationScopeRequest, +) ReconciliationScopePlan { + type hermesConfiguredScope struct { + root string // configured spelling handed back for traversal + match string // absolute cleaned form for containment checks + container bool + // profiles is the container's owned children in configured spelling, + // empty for a directly configured archive or an unreadable container. + profiles []string + unit hermesReconciliationUnit + } + var configured []hermesConfiguredScope + var required []string + for _, root := range s.roots { + if strings.TrimSpace(root) == "" { + continue + } + if strings.HasPrefix(strings.ToLower(root), "s3://") { + required = append(required, root) continue } - stateDB, sessionsDir, ok := hermesArchiveRootPaths(configuredRoot) + cfg := hermesConfiguredScope{ + root: root, + match: absoluteHermesPath(root), + } + if isHermesProfilesContainer(root) { + // The container is one coverage identity: profile membership is + // dynamic, so a pass covers it only by traversing the container + // itself, and discovery reports an unreadable container as + // incomplete, which withholds deletion authority. + cfg.container = true + cfg.unit = hermesReconciliationUnit{ + identity: cfg.match, + proofs: []StoredSourceHintScope{{Path: filepath.Clean(root)}}, + } + // Enumerated once per resolution rather than per request: a batch + // of paths under one container would otherwise re-read the same + // directory for each of them. + cfg.profiles, _ = hermesProfileArchiveRoots(filepath.Clean(root)) + } else { + cfg.unit = hermesArchiveReconciliationUnit(root) + } + configured = append(configured, cfg) + } + for _, cfg := range configured { + required = append(required, cfg.unit.identity) + } + plan := ReconciliationScopePlan{RequiredCoverageIdentities: required} + scopeIndex := make(map[string]int) + addScope := func(key, raw string, scope ReconciliationScope) { + index, ok := scopeIndex[key] if !ok { - scopes = append(scopes, StoredSourceHintScope{Path: configuredRoot}) + scope.RetryRoots = []string{raw} + scopeIndex[key] = len(plan.Scopes) + plan.Scopes = append(plan.Scopes, scope) + return + } + // One archive can resolve through several branches — its own + // configured root and a container's profile enumeration — in any + // order, depending on how overlapping roots are configured and + // requested. The colliding scopes describe the same unit, so merge + // every authority rather than keeping the first arrival's: dropping + // a later CoverageIdentities would leave a required identity + // permanently uncovered and withhold aggregate-member deletion from + // a pass that requested every configured root. + existing := &plan.Scopes[index] + if !slices.Contains(existing.RetryRoots, raw) { + existing.RetryRoots = append(existing.RetryRoots, raw) + } + for _, root := range scope.TraversalRoots { + if !slices.Contains(existing.TraversalRoots, root) { + existing.TraversalRoots = append(existing.TraversalRoots, root) + } + } + for _, proof := range scope.PhysicalProofScopes { + if !slices.Contains(existing.PhysicalProofScopes, proof) { + existing.PhysicalProofScopes = append( + existing.PhysicalProofScopes, proof, + ) + } + } + for _, identity := range scope.CoverageIdentities { + if !slices.Contains(existing.CoverageIdentities, identity) { + existing.CoverageIdentities = append( + existing.CoverageIdentities, identity, + ) + } + } + } + for _, raw := range req.Roots { + if strings.TrimSpace(raw) == "" || + strings.HasPrefix(strings.ToLower(raw), "s3://") { continue } - if IsRegularFile(stateDB) { - scopes = append(scopes, StoredSourceHintScope{ - Path: stateDB, IncludeVirtualMembers: true, - }) + requested := absoluteHermesPath(raw) + for _, cfg := range configured { + if hermesPathWithinOrSame(cfg.match, requested) { + // The request names the configured root or an ancestor + // covering it: the whole configured unit is one atomic scope. + addScope(cfg.unit.identity, raw, ReconciliationScope{ + TraversalRoots: []string{cfg.root}, + PhysicalProofScopes: cfg.unit.proofs, + CoverageIdentities: []string{cfg.unit.identity}, + }) + continue + } + if cfg.container { + if _, inside := hermesProfileRootForPath( + cfg.match, requested, + ); !inside { + continue + } + if profileRoot, ok := hermesProfileScopeRoot( + cfg.profiles, requested, + ); ok { + // A request inside one profile resolves to that profile's + // archive only. The container identity stays uncovered, so + // sibling profiles keep their sessions and their authority. + unit := hermesArchiveReconciliationUnit(profileRoot) + addScope(unit.identity, raw, ReconciliationScope{ + TraversalRoots: []string{profileRoot}, + PhysicalProofScopes: unit.proofs, + }) + continue + } + // The container does not own that child: it was deleted, or it + // is a symlink or a plain file the enumeration skips, or the + // container could not be read. Reconstructing a root from the + // request is what four earlier rounds each got wrong, and + // resolving nothing loses the removal: a deleted profile's + // sessions would then stay active until the daily archive + // audit happens to run. The container itself is the nearest + // scope spelled entirely from configuration, so the request + // widens to it, proving the container without covering it. + // Deletion still requires a stat per stored row, so live + // sibling profiles are retained, and the withheld coverage + // identity keeps aggregate-member removal out of reach. + addScope(cfg.unit.identity+"\x00unowned", raw, + ReconciliationScope{ + TraversalRoots: []string{cfg.root}, + PhysicalProofScopes: cfg.unit.proofs, + }) + continue + } + if hermesRequestDenotesArchive(requested, cfg.unit.stateDB) { + // Any alias spelling or interior path of an archive resolves + // to the identical archive scope, so no spelling can omit the + // state.db members or the sessions/ transcripts. + addScope(cfg.unit.identity, raw, ReconciliationScope{ + TraversalRoots: []string{cfg.root}, + PhysicalProofScopes: cfg.unit.proofs, + CoverageIdentities: []string{cfg.unit.identity}, + }) + continue + } + if cfg.unit.stateDB == "" && + hermesPathWithinOrSame(requested, cfg.match) { + // A plain transcript-directory root has no archive topology + // to widen into, so a requested descendant resolves + // generically: traverse the configured root, prove only the + // descendant in the configured spelling, claim no coverage. + // Resolving nothing instead would leave a deleted + // transcript's session active until the daily archive audit. + proof := descendantProofSpelling( + cfg.match, filepath.Clean(cfg.root), requested, + ) + addScope("descendant\x00"+proof, raw, ReconciliationScope{ + TraversalRoots: []string{cfg.root}, + PhysicalProofScopes: []StoredSourceHintScope{{Path: proof}}, + }) + } } - scopes = append(scopes, StoredSourceHintScope{Path: sessionsDir}) } - return scopes + return plan } func (s hermesSourceSet) SourceForReconciliation( diff --git a/internal/parser/hermes_provider_test.go b/internal/parser/hermes_provider_test.go index ea9794a0d..6eb50844b 100644 --- a/internal/parser/hermes_provider_test.go +++ b/internal/parser/hermes_provider_test.go @@ -443,8 +443,8 @@ func TestHermesStreamingTranscriptFailureContinuesLaterRoots(t *testing.T) { // the reconciliation tombstoning path: hermesProfileArchiveRoots used to // convert every os.ReadDir failure into an empty profile list, so a // transient permission or I/O failure on the profiles container made -// discovery look authoritatively empty while ReconciliationOwnershipScopes -// still claimed the whole container — and the engine tombstoned every +// discovery look authoritatively empty while the resolved reconciliation +// scope still proved the whole container — and the engine tombstoned every // stored hermes session under it as source_missing. Enumeration failures // must surface as DiscoveryIncompleteError so the engine retains // reconciliation markers and retries instead. diff --git a/internal/parser/multi_session_container.go b/internal/parser/multi_session_container.go index 2fbdfb575..635b5f66f 100644 --- a/internal/parser/multi_session_container.go +++ b/internal/parser/multi_session_container.go @@ -353,6 +353,34 @@ type multiSessionContainerSourceSet struct { cfg multiSessionConfig } +// ReconciliationContainer maps a requested reconciliation root to the +// physical container that atomically owns it: the container file itself, a +// member's virtual spelling, or a sidecar event path (SQLite WAL/SHM). The +// scope planner widens such a request to the container's whole virtual +// membership; the generic descendant proof would name only the bare path, +// which admits no "#" source and pages no member row — a +// successful no-op over the sessions the caller asked about. Classification +// runs with allowMissing so a deleted container still resolves and its +// members remain reclaimable. Requests arrive absolutized, so a relative +// configured root is also tried in its absolute spelling. +func (s multiSessionContainerSourceSet) ReconciliationContainer( + requested string, +) (string, bool) { + for _, root := range s.roots { + spellings := []string{root} + if abs := cleanReconciliationScopeRoot(root); abs != filepath.Clean(root) { + spellings = append(spellings, abs) + } + for _, spelling := range spellings { + if match, ok := s.cfg.classifyPath(spelling, requested, true); ok && + match.Container != "" { + return match.Container, true + } + } + } + return "", false +} + func (s multiSessionContainerSourceSet) Discover( ctx context.Context, ) ([]SourceRef, error) { diff --git a/internal/parser/opencode.go b/internal/parser/opencode.go index 4a416e467..d2d1dad72 100644 --- a/internal/parser/opencode.go +++ b/internal/parser/opencode.go @@ -5,13 +5,16 @@ import ( "crypto/sha256" "database/sql" "encoding/json" + "errors" "fmt" "os" "path/filepath" "regexp" "sort" + "strconv" "strings" "sync" + "sync/atomic" "time" "github.com/tidwall/gjson" @@ -25,6 +28,24 @@ type OpenCodeSessionMeta struct { SessionID string VirtualPath string FileMtime int64 + // CompositeMtime reports that FileMtime is the per-session composite + // (see openCodeCompositeMtimeExpr) rather than the session row's own + // time_updated. When true the fingerprint omits the shared container's + // size, because the composite already discriminates per session. + CompositeMtime bool + // ChildDigest is the per-session freshness identity carried into the + // fingerprint's Hash. It folds the composite watermark together with the + // child row counts, so a deletion that leaves the watermark untouched + // still changes it. Empty when the container has no composite support. + ChildDigest string + // WatermarkOnly reports that FileMtime is only the session-row watermark + // (session and project time_updated, no child tables) and ChildDigest is + // deliberately unresolved. Changed-path listings use this bounded form so + // a watcher event does not scan every child row in the container; the + // engine skips such a source when its stored composite watermark already + // covers the carried value, and resolves the full composite and digest + // through the indexed per-session lookup otherwise. + WatermarkOnly bool } // OpenCodeSQLiteSessionExists reports whether a session row with @@ -73,6 +94,128 @@ func ListOpenCodeSessionMeta( return metas, err } +// The two container freshness query shapes, counted so tests can pin that +// watcher-driven passes stay bounded by the changed batch: the grouped +// whole-container child scan must not run on a changed-path pass, and +// per-session child lookups must scale with the number of changed sessions, +// not the archive. +var ( + openCodeContainerChildScans atomic.Int64 + openCodeSessionChildLookups atomic.Int64 +) + +// OpenCodeContainerChildScans returns how many whole-container child-table +// identity scans (grouped message/part aggregation) have run. +func OpenCodeContainerChildScans() int64 { + return openCodeContainerChildScans.Load() +} + +// OpenCodeSessionChildLookups returns how many single-session indexed child +// digest lookups have run. +func OpenCodeSessionChildLookups() int64 { + return openCodeSessionChildLookups.Load() +} + +// ListOpenCodeSessionWatermarkMeta is the materialized form of +// ForEachOpenCodeSessionWatermarkMeta. +func ListOpenCodeSessionWatermarkMeta( + dbPath string, +) ([]OpenCodeSessionMeta, error) { + var metas []OpenCodeSessionMeta + err := ForEachOpenCodeSessionWatermarkMeta( + context.Background(), dbPath, + func(meta OpenCodeSessionMeta) error { + metas = append(metas, meta) + return nil + }, + ) + return metas, err +} + +// ForEachOpenCodeSessionWatermarkMeta streams session rows carrying only the +// session-row watermark: MAX(session.time_updated, project.time_updated), +// touching no child tables. A watcher event on a shared container must not +// pay a whole-archive child scan to find the one session that changed, so +// changed-path classification lists sessions through this bounded form and +// leaves the full composite and child digest to the indexed per-session +// lookup, which the engine runs only for sessions it cannot skip against +// their stored watermark. What this signal cannot see — a child-only write +// that leaves the session and project rows untouched, wherever its +// timestamps land relative to the stored composite — is a known, deliberate +// deferral reconciled by the next full-discovery pass, which still carries +// the complete child digest (see the change-detection entry in +// docs/internal/session-format-sources.md). Legacy containers without +// composite support keep the full listing; their conservative +// container-size fingerprint must not be bypassed by a watermark-only skip. +// +// This listing scans the session table once and that scan is O(session +// count) by design: OpenCode's schema indexes neither time_updated column, +// and the schema is not ours to alter, so any sound candidate selection +// must read every session row. The rows are few and fixed-width (one per +// session, no transcript bytes), which is what makes this the bounded form +// — the quantities that previously scaled with the archive were the child +// tables (two orders of magnitude more rows) and the per-event +// materialization downstream, and both are now bounded by the changed +// batch. +func ForEachOpenCodeSessionWatermarkMeta( + ctx context.Context, + dbPath string, + yield func(OpenCodeSessionMeta) error, +) error { + if _, err := os.Stat(dbPath); os.IsNotExist(err) { + return nil + } + + db, err := openOpenCodeDB(dbPath) + if err != nil { + return err + } + defer db.Close() + + composite, err := openCodeCompositeMtimeSupportedCached(db, dbPath) + if err != nil { + return err + } + // Ordered by id so the virtual paths ("#") stream in ascending + // byte order: the changed-path merge walks a paged stored-freshness + // cursor in step with this stream, and both sides must share one order. + query := "SELECT s.id, s.time_updated FROM session s ORDER BY s.id" + if composite { + query = "SELECT s.id, " + openCodeSessionRowWatermarkExpr + + " FROM session s" + openCodeSessionCompositeMtimeJoins + + " ORDER BY s.id" + } + + rows, err := db.QueryContext(ctx, query) + if err != nil { + return fmt.Errorf( + "listing opencode session watermarks: %w", err, + ) + } + defer rows.Close() + + for rows.Next() { + var id string + var watermark int64 + if err := rows.Scan(&id, &watermark); err != nil { + return fmt.Errorf( + "scanning opencode session watermark: %w", err, + ) + } + observeStreamingDiscoveryBuffer(ctx, 1) + if err := yield(OpenCodeSessionMeta{ + SessionID: id, + VirtualPath: dbPath + "#" + id, + FileMtime: watermark * 1_000_000, + CompositeMtime: composite, + WatermarkOnly: composite, + }); err != nil { + return err + } + } + return rows.Err() +} + // ForEachOpenCodeSessionMeta streams lightweight session rows directly from // SQLite. The callback runs while the read-only query is open and receives one // row at a time; callers must not retain database-owned values. @@ -91,9 +234,20 @@ func ForEachOpenCodeSessionMeta( } defer db.Close() - rows, err := db.QueryContext(ctx, - "SELECT id, time_updated FROM session", - ) + composite, err := openCodeCompositeMtimeSupportedCached(db, dbPath) + if err != nil { + return err + } + query := "SELECT s.id, s.time_updated, s.time_updated, 0, 0, 0, '', '' " + + "FROM session s" + if composite { + openCodeContainerChildScans.Add(1) + query = "SELECT s.id, " + openCodeCompositeMtimeExpr + ", " + + openCodeCompositeCountsExpr + + " FROM session s" + openCodeCompositeMtimeJoins + } + + rows, err := db.QueryContext(ctx, query) if err != nil { return fmt.Errorf( "listing opencode sessions: %w", err, @@ -103,9 +257,11 @@ func ForEachOpenCodeSessionMeta( for rows.Next() { var id string - var timeUpdated int64 + var agg openCodeChildAggregate if err := rows.Scan( - &id, &timeUpdated, + &id, &agg.watermark, &agg.sessionTime, &agg.projectTime, + &agg.messages, &agg.parts, + &agg.messageIdent, &agg.partIdent, ); err != nil { return fmt.Errorf( "scanning opencode session meta: %w", err, @@ -113,9 +269,11 @@ func ForEachOpenCodeSessionMeta( } observeStreamingDiscoveryBuffer(ctx, 1) if err := yield(OpenCodeSessionMeta{ - SessionID: id, - VirtualPath: dbPath + "#" + id, - FileMtime: timeUpdated * 1_000_000, + SessionID: id, + VirtualPath: dbPath + "#" + id, + FileMtime: agg.watermark * 1_000_000, + CompositeMtime: composite, + ChildDigest: agg.digest(composite), }); err != nil { return err } @@ -123,6 +281,153 @@ func ForEachOpenCodeSessionMeta( return rows.Err() } +// openCodeSessionCompositeMtime returns one session's composite change signal +// in milliseconds, and whether the container schema supports it. Discovery, +// single-session source lookup, and the parse path all resolve mtime through +// this so a session's stored file_mtime always equals the value the freshness +// gate compares it against. +func openCodeSessionCompositeMtime( + db *sql.DB, dbPath, sessionID string, +) (int64, string, bool, error) { + composite, err := openCodeCompositeMtimeSupportedCached(db, dbPath) + if err != nil { + return 0, "", false, err + } + query := "SELECT s.time_updated, s.time_updated, 0, 0, 0, '', '' " + + "FROM session s WHERE s.id = ?" + if composite { + openCodeSessionChildLookups.Add(1) + query = "SELECT " + openCodeSessionCompositeMtimeExpr + ", " + + openCodeSessionCompositeCountsExpr + + " FROM session s" + openCodeSessionCompositeMtimeJoins + + " WHERE s.id = ?" + } + var agg openCodeChildAggregate + if err := db.QueryRow(query, sessionID).Scan( + &agg.watermark, &agg.sessionTime, &agg.projectTime, + &agg.messages, &agg.parts, + &agg.messageIdent, &agg.partIdent, + ); err != nil { + if errors.Is(err, sql.ErrNoRows) { + return 0, "", composite, nil + } + return 0, "", composite, fmt.Errorf( + "loading opencode session mtime %s#%s: %w", + dbPath, sessionID, err, + ) + } + return agg.watermark, agg.digest(composite), composite, nil +} + +// openCodeSessionWatermark resolves only the composite watermark, skipping the +// eight child COUNT/SUM/MIN/MAX subqueries that make up the digest. Callers +// that stamp or compare an mtime do not need the digest, and one of them +// (OpenCodeSourceMtime) backs the session watcher's 1.5s poll, so computing a +// digest there would burn eight child-range scans per tick per watched session +// for a value the caller discards. +func openCodeSessionWatermark( + db *sql.DB, dbPath, sessionID string, +) (int64, bool, error) { + composite, err := openCodeCompositeMtimeSupportedCached(db, dbPath) + if err != nil { + return 0, false, err + } + query := "SELECT s.time_updated FROM session s WHERE s.id = ?" + if composite { + query = "SELECT " + openCodeSessionCompositeMtimeExpr + + " FROM session s" + openCodeSessionCompositeMtimeJoins + + " WHERE s.id = ?" + } + var watermark int64 + if err := db.QueryRow(query, sessionID).Scan(&watermark); err != nil { + if errors.Is(err, sql.ErrNoRows) { + return 0, composite, nil + } + return 0, composite, fmt.Errorf( + "loading opencode session mtime %s#%s: %w", + dbPath, sessionID, err, + ) + } + return watermark, composite, nil +} + +// openCodeChildAggregate is the cheap per-session identity read alongside the +// watermark. Each component covers a change the others cannot: +// +// - watermark: an edit or insert that advances a timestamp +// - session/project times: a metadata update, including a worktree rename, +// that lands below an already-higher child watermark +// - counts: a deletion, which never moves a MAX +// - child identity: every ordered (id, time_updated) pair, hashed. Reduced +// aggregates are not collision-resistant — swapping a non-boundary row for +// one carrying the same timestamp preserves counts, sums and min/max ids +// alike, so only the complete set separates them. +// +// All of it lives in the child tables' main b-tree pages, so computing it does +// not read the transcript text held in overflow pages. +type openCodeChildAggregate struct { + watermark int64 + sessionTime int64 + projectTime int64 + messages int64 + parts int64 + messageIdent string + partIdent string +} + +// The field layout is load-bearing beyond equality comparison: +// OpenCodeChildDigestMetadataWatermarkNS recovers the session/project times +// from a stored digest by position. Any layout change must bump the prefix +// version so stale digests fail that parse (and the equality gate) instead +// of yielding wrong fields. +func (a openCodeChildAggregate) digest(composite bool) string { + if !composite { + return "" + } + sum := sha256.Sum256([]byte(a.messageIdent + "\x00" + a.partIdent)) + return fmt.Sprintf( + "%s%d:%d:%d:%d:%d:%x", + openCodeChildDigestPrefix, + a.watermark, a.sessionTime, a.projectTime, + a.messages, a.parts, + sum[:16], + ) +} + +const openCodeChildDigestPrefix = "opencode-child:v1:" + +// OpenCodeChildDigestMetadataWatermarkNS recovers the session/project +// metadata watermark (nanoseconds) embedded in a stored child digest. The +// watermark filter compares the live session-row watermark like-for-like +// against this value: the stored composite MTimeNS may be dominated by a +// newer child timestamp, and comparing the session-row watermark against +// that composite would hide a metadata update (title, directory, worktree) +// whose stamp lands below it. Returns false for any other hash shape — +// legacy fingerprints, storage fingerprints, future digest versions — which +// callers treat as "compare against the composite instead", the +// conservative pre-digest behavior. +func OpenCodeChildDigestMetadataWatermarkNS(hash string) (int64, bool) { + rest, ok := strings.CutPrefix(hash, openCodeChildDigestPrefix) + if !ok { + return 0, false + } + fields := strings.Split(rest, ":") + if len(fields) != 6 { + return 0, false + } + // Validate every numeric field, not just the two consumed: a digest with + // any malformed component is not a digest this version wrote, and the + // caller's composite fallback is the safe answer for it. + for _, field := range fields[:5] { + if _, err := strconv.ParseInt(field, 10, 64); err != nil { + return 0, false + } + } + sessionTime, _ := strconv.ParseInt(fields[1], 10, 64) + projectTime, _ := strconv.ParseInt(fields[2], 10, 64) + return max(sessionTime, projectTime) * 1_000_000, true +} + // parseOpenCodeDBSession parses a single session by ID from the // OpenCode SQLite database. The OpenCode-format provider owns this // path; Kilo and MiMoCode reuse it and relabel the result. @@ -378,9 +683,15 @@ type openCodeSessionRow struct { timeUpdated int64 } +// openCodeSessionSchemaCacheEntry memoizes both schema probes for one +// container. Each probe has its own "resolved" flag so populating one never +// makes the other report a false negative from its zero value. type openCodeSessionSchemaCacheEntry struct { - state SQLiteContainerState - hasDirectory bool + state SQLiteContainerState + hasDirectory bool + directoryOnce bool + hasComposite bool + compositeOnce bool } // openCodeSessionSchemaCache memoizes whether session.directory exists for @@ -402,7 +713,7 @@ func openCodeSessionHasDirectoryCached( openCodeSessionSchemaCacheMu.Lock() entry, hit := openCodeSessionSchemaCache[dbPath] openCodeSessionSchemaCacheMu.Unlock() - if hit && entry.state == state { + if hit && entry.state == state && entry.directoryOnce { return entry.hasDirectory, nil } hasDirectory, err := openCodeSessionTableHasDirectory(db) @@ -410,14 +721,208 @@ func openCodeSessionHasDirectoryCached( return false, err } openCodeSessionSchemaCacheMu.Lock() - openCodeSessionSchemaCache[dbPath] = openCodeSessionSchemaCacheEntry{ - state: state, - hasDirectory: hasDirectory, - } + prev := openCodeSessionSchemaCache[dbPath] + if prev.state != state { + prev = openCodeSessionSchemaCacheEntry{} + } + prev.state = state + prev.hasDirectory = hasDirectory + prev.directoryOnce = true + openCodeSessionSchemaCache[dbPath] = prev openCodeSessionSchemaCacheMu.Unlock() return hasDirectory, nil } +// openCodeCompositeMtimeExpr is the per-session change signal for a +// SQLite-backed OpenCode container. Every session in a root shares one +// physical opencode.db, so the container file's own size and mtime move +// whenever any single session is written and cannot discriminate between +// sessions. These four columns can: +// +// - session.time_updated — the session row itself +// - project.time_updated — the owning project (worktree renames re-resolve +// every session in that project, which is the correct scope; verified on a +// production container that this does not track ordinary session activity) +// - max(message.time_updated) / max(part.time_updated) — child content, +// including in-place edits that leave time_created untouched +// +// The child scans read only small columns; OpenCode keeps each part's `data` +// in SQLite overflow pages, so this does not read transcript bytes. +// The streaming form groups the child tables once for the whole container, so +// listing every session costs a single pass over each child table. +const openCodeCompositeMtimeExpr = `MAX(s.time_updated, + COALESCE(pr.time_updated, 0), + COALESCE(m.mx, 0), + COALESCE(p.mx, 0))` + +const openCodeCompositeMtimeJoins = ` + LEFT JOIN project pr ON pr.id = s.project_id + LEFT JOIN ( + SELECT session_id, MAX(time_updated) mx, COUNT(*) n, + group_concat(id || ':' || time_updated) ident + FROM (SELECT session_id, id, time_updated FROM message + ORDER BY session_id, id) + GROUP BY session_id + ) m ON m.session_id = s.id + LEFT JOIN ( + SELECT session_id, MAX(time_updated) mx, COUNT(*) n, + group_concat(id || ':' || time_updated) ident + FROM (SELECT session_id, id, time_updated FROM part + ORDER BY session_id, id) + GROUP BY session_id + ) p ON p.session_id = s.id` + +// openCodeCompositeCountsExpr yields the child row counts that make the +// signal deletion-sensitive. A MAX over timestamps cannot see a delete: on a +// real container the session or project row usually already holds the higher +// value, so removing a message or part leaves the max untouched and the +// session would look fresh with the deleted content still archived. +const openCodeCompositeCountsExpr = `s.time_updated, + COALESCE(pr.time_updated, 0), + COALESCE(m.n, 0), COALESCE(p.n, 0), + COALESCE(m.ident, ''), COALESCE(p.ident, '')` + +const openCodeSessionCompositeCountsExpr = ` + s.time_updated, + COALESCE(pr.time_updated, 0), + (SELECT COUNT(*) FROM message WHERE session_id = s.id), + (SELECT COUNT(*) FROM part WHERE session_id = s.id), + (SELECT COALESCE(group_concat(id || ':' || time_updated), '') + FROM (SELECT id, time_updated FROM message + WHERE session_id = s.id ORDER BY id)), + (SELECT COALESCE(group_concat(id || ':' || time_updated), '') + FROM (SELECT id, time_updated FROM part + WHERE session_id = s.id ORDER BY id))` + +// The single-session form must NOT reuse the grouped subqueries above: a +// GROUP BY subquery is materialized over the whole container before the outer +// WHERE narrows to one session, so every per-session lookup would scan every +// message and part in the container. Correlated aggregates filtered by +// session_id ride the message/part session_id indexes instead, which is the +// difference between an index seek and an archive-wide scan on every call. +const openCodeSessionCompositeMtimeExpr = `MAX(s.time_updated, + COALESCE(pr.time_updated, 0), + COALESCE(( + SELECT MAX(time_updated) FROM message WHERE session_id = s.id + ), 0), + COALESCE(( + SELECT MAX(time_updated) FROM part WHERE session_id = s.id + ), 0))` + +const openCodeSessionCompositeMtimeJoins = ` + LEFT JOIN project pr ON pr.id = s.project_id` + +// openCodeSessionRowWatermarkExpr is the bounded change signal used by +// watermark-only changed-path listings: the session and project rows alone, +// no child aggregation. The session table holds one small row per session, so +// listing every session through this costs a scan of the session and project +// tables only — bounded by session count, never by message/part volume. +const openCodeSessionRowWatermarkExpr = `MAX(s.time_updated, + COALESCE(pr.time_updated, 0))` + +// openCodeCompositeMtimeSupportedCached reports whether this container's schema +// carries every column openCodeCompositeMtimeExpr needs. Older OpenCode-family +// containers (Kilo, MiMoCode, ICodeMate, legacy OpenCode) omit the child +// time_updated columns; those keep the previous session-only mtime and the +// container-stat fallback in Fingerprint. +func openCodeCompositeMtimeSupportedCached( + db *sql.DB, dbPath string, +) (bool, error) { + state, ok := StatSQLiteContainerState(dbPath) + if !ok { + return openCodeSupportsCompositeMtime(db) + } + openCodeSessionSchemaCacheMu.Lock() + entry, hit := openCodeSessionSchemaCache[dbPath] + openCodeSessionSchemaCacheMu.Unlock() + if hit && entry.state == state && entry.compositeOnce { + return entry.hasComposite, nil + } + supported, err := openCodeSupportsCompositeMtime(db) + if err != nil { + return false, err + } + openCodeSessionSchemaCacheMu.Lock() + prev := openCodeSessionSchemaCache[dbPath] + if prev.state != state { + prev = openCodeSessionSchemaCacheEntry{state: state} + } + prev.state = state + prev.hasComposite = supported + prev.compositeOnce = true + openCodeSessionSchemaCache[dbPath] = prev + openCodeSessionSchemaCacheMu.Unlock() + return supported, nil +} + +func openCodeSupportsCompositeMtime(db *sql.DB) (bool, error) { + for _, probe := range []struct{ table, column string }{ + {"message", "time_updated"}, + {"part", "time_updated"}, + {"project", "time_updated"}, + } { + has, err := openCodeTableHasColumn(db, probe.table, probe.column) + if err != nil || !has { + return false, err + } + } + // The per-session lookups are correlated aggregates keyed on session_id. + // SQLite does not index a foreign key automatically, so without a + // session_id index each one degrades to a full child-table scan — and one + // of these backs the session watcher's 1.5s poll. Fall back to the + // session-only mtime rather than put an archive scan on that path. + for _, table := range []string{"message", "part"} { + indexed, err := openCodeTableIndexesColumn(db, table, "session_id") + if err != nil || !indexed { + return false, err + } + } + return true, nil +} + +// openCodeTableIndexesColumn reports whether table has an index whose leftmost +// column is column, which is what makes a WHERE column = ? lookup a seek. +func openCodeTableIndexesColumn( + db *sql.DB, table, column string, +) (bool, error) { + rows, err := db.Query( + `SELECT 1 FROM pragma_index_list(?) il + JOIN pragma_index_info(il.name) ii + WHERE ii.seqno = 0 AND ii.name = ?`, + table, column, + ) + if err != nil { + return false, fmt.Errorf( + "listing opencode %s indexes: %w", table, err, + ) + } + defer rows.Close() + if rows.Next() { + return true, rows.Err() + } + return false, rows.Err() +} + +// openCodeTableHasColumn reports whether table carries column. An unknown +// table yields no PRAGMA rows and reports false rather than erroring, so a +// container missing an optional table degrades to the legacy signal. +func openCodeTableHasColumn( + db *sql.DB, table, column string, +) (bool, error) { + rows, err := db.Query(`SELECT 1 FROM pragma_table_info(?) WHERE name = ?`, + table, column) + if err != nil { + return false, fmt.Errorf( + "listing opencode %s table info: %w", table, err, + ) + } + defer rows.Close() + if rows.Next() { + return true, rows.Err() + } + return false, rows.Err() +} + func openCodeSessionTableHasDirectory(db *sql.DB) (bool, error) { rows, err := db.Query(`PRAGMA table_info(session)`) if err != nil { @@ -622,6 +1127,27 @@ func buildOpenCodeSession( s openCodeSessionRow, cwd, projectWorktree, dbPath, machine string, ) (*ParsedSession, []ParsedMessage, error) { + // Capture the watermark BEFORE reading children. Messages and parts are + // read through separate autocommit queries, so a concurrent write landing + // between them would otherwise be stamped with a watermark newer than the + // content actually read, and every later sync would skip the session as + // fresh — permanently archiving a torn transcript. Reading the watermark + // first inverts the race: the stamp is never newer than the content, so a + // concurrent change leaves the stored value behind the source and the next + // pass re-syncs it. + // + // Stamp the same composite the fingerprint reports, so the stored + // file_mtime is directly comparable to it. Falling back to the session + // row's own time_updated keeps legacy containers on their prior value. + fileMtime := s.timeUpdated + if composite, _, err := openCodeSessionWatermark( + db, dbPath, s.id, + ); err != nil { + return nil, nil, err + } else if composite != 0 { + fileMtime = composite + } + msgs, err := loadOpenCodeMessages(db, s.id) if err != nil { return nil, nil, fmt.Errorf( @@ -641,7 +1167,7 @@ func buildOpenCodeSession( cwd, projectWorktree, dbPath+"#"+s.id, - s.timeUpdated*1_000_000, + fileMtime*1_000_000, machine, msgs, parts, @@ -1403,38 +1929,64 @@ func openCodeStorageFingerprintHash(raw string) string { return fmt.Sprintf("%x", sum) } -func openCodeSQLiteSessionMtime( +// openCodeSQLiteSessionMtimeComposite is openCodeSQLiteSessionMtime with the +// schema-support flag the fingerprint needs to decide whether the shared +// container's size still has to act as a fallback change signal. +func openCodeSQLiteSessionMtimeComposite( dbPath, sessionID string, -) (int64, error) { +) (int64, string, bool, error) { if _, err := os.Stat(dbPath); err != nil { if os.IsNotExist(err) { - return 0, nil + return 0, "", false, nil } - return 0, fmt.Errorf( + return 0, "", false, fmt.Errorf( "stat opencode db %s: %w", dbPath, err, ) } db, err := openOpenCodeDB(dbPath) if err != nil { - return 0, err + return 0, "", false, err } defer db.Close() - row := db.QueryRow( - "SELECT time_updated FROM session WHERE id = ?", - sessionID, + timeUpdated, digest, composite, err := openCodeSessionCompositeMtime( + db, dbPath, sessionID, ) - var timeUpdated int64 - if err := row.Scan(&timeUpdated); err != nil { - if err == sql.ErrNoRows { + if err != nil { + return 0, "", false, err + } + if timeUpdated == 0 { + return 0, digest, composite, nil + } + return timeUpdated * 1_000_000, digest, composite, nil +} + +func openCodeSQLiteSessionMtime( + dbPath, sessionID string, +) (int64, error) { + if _, err := os.Stat(dbPath); err != nil { + if os.IsNotExist(err) { return 0, nil } return 0, fmt.Errorf( - "loading opencode session mtime %s#%s: %w", - dbPath, sessionID, err, + "stat opencode db %s: %w", dbPath, err, ) } + + db, err := openOpenCodeDB(dbPath) + if err != nil { + return 0, err + } + defer db.Close() + + timeUpdated, _, err := openCodeSessionWatermark(db, dbPath, sessionID) + if err != nil { + return 0, err + } + if timeUpdated == 0 { + return 0, nil + } return timeUpdated * 1_000_000, nil } diff --git a/internal/parser/opencode_provider.go b/internal/parser/opencode_provider.go index 7b0fdbd1e..4414cb854 100644 --- a/internal/parser/opencode_provider.go +++ b/internal/parser/opencode_provider.go @@ -70,7 +70,9 @@ func (f openCodeFormatProviderFactory) NewProvider(cfg ProviderConfig) Provider Caps: openCodeFormatProviderCapabilities(), Config: cfg, }, - sources: newOpenCodeFormatSourceSet(cfg.Roots, f.spec), + sources: newOpenCodeFormatSourceSet( + cfg.Roots, f.spec, cfg.SQLiteContainerUnchangedSinceTrust, + ), } } @@ -104,6 +106,24 @@ func (p *openCodeFormatProvider) SourceForReconciliation( return p.sources.SourceForReconciliation(ctx, path, project) } +// ResolveReconciliationScopes widens a request naming the family database, a +// WAL or SHM sidecar, or one virtual member to the container itself. The +// container's membership is atomic: a proof of the bare database path admits +// no member row, and a proof of one member would let a completed pass promote +// container-state trust over siblings it never verified. +func (p *openCodeFormatProvider) ResolveReconciliationScopes( + _ context.Context, req ReconciliationScopeRequest, +) (ReconciliationScopePlan, error) { + if err := ValidateReconciliationScopeRoots( + p.Def.Type, p.Config.Roots, req.Roots, + ); err != nil { + return ReconciliationScopePlan{}, err + } + return containerAwareReconciliationScopePlan( + p.Config.Roots, req.Roots, p.sources.reconciliationContainer, + ), nil +} + func (p *openCodeFormatProvider) FindSource( ctx context.Context, req FindSourceRequest, @@ -175,55 +195,71 @@ func (p *openCodeFormatProvider) Parse( // the OpenCode storage and SQLite readers, then relabel the result onto // their own agent and ID prefix. type openCodeProviderSpec struct { - agent AgentType - format openCodeFormat - dbName string - listSQLite func(string) ([]OpenCodeSessionMeta, error) - streamSQLite func(context.Context, string, func(OpenCodeSessionMeta) error) error - sourceMtime func(string) (int64, error) - relabel func(*ParsedSession) + agent AgentType + format openCodeFormat + dbName string + listSQLite func(string) ([]OpenCodeSessionMeta, error) + // listSQLiteWatermark is the bounded changed-path form of listSQLite: it + // carries only the session-row watermark and no child digest, so a + // watcher event on the shared container never scans the child tables. + listSQLiteWatermark func(string) ([]OpenCodeSessionMeta, error) + streamSQLite func(context.Context, string, func(OpenCodeSessionMeta) error) error + // streamSQLiteWatermark is the bounded trusted-container form of + // streamSQLite, used by streamed reconciliation discovery for containers + // the engine's container gate will skip wholesale. + streamSQLiteWatermark func(context.Context, string, func(OpenCodeSessionMeta) error) error + sourceMtime func(string) (int64, error) + relabel func(*ParsedSession) } func openCodeProviderSpecForAgent(agent AgentType) openCodeProviderSpec { switch agent { case AgentOpenCode: return openCodeProviderSpec{ - agent: AgentOpenCode, - format: openCodeFmt, - dbName: openCodeFmt.dbName, - listSQLite: ListOpenCodeSessionMeta, - streamSQLite: ForEachOpenCodeSessionMeta, - sourceMtime: OpenCodeSourceMtime, + agent: AgentOpenCode, + format: openCodeFmt, + dbName: openCodeFmt.dbName, + listSQLite: ListOpenCodeSessionMeta, + listSQLiteWatermark: ListOpenCodeSessionWatermarkMeta, + streamSQLite: ForEachOpenCodeSessionMeta, + streamSQLiteWatermark: ForEachOpenCodeSessionWatermarkMeta, + sourceMtime: OpenCodeSourceMtime, } case AgentKilo: return openCodeProviderSpec{ - agent: AgentKilo, - format: kiloFmt, - dbName: kiloFmt.dbName, - listSQLite: ListKiloSessionMeta, - streamSQLite: streamOpenCodeSessionMetaAs(KiloSQLiteVirtualPath), - sourceMtime: KiloSourceMtime, - relabel: relabelOpenCodeSessionAsKilo, + agent: AgentKilo, + format: kiloFmt, + dbName: kiloFmt.dbName, + listSQLite: ListKiloSessionMeta, + listSQLiteWatermark: listOpenCodeSessionWatermarkMetaAs(KiloSQLiteVirtualPath), + streamSQLite: streamOpenCodeSessionMetaAs(KiloSQLiteVirtualPath), + streamSQLiteWatermark: streamOpenCodeSessionWatermarkMetaAs(KiloSQLiteVirtualPath), + sourceMtime: KiloSourceMtime, + relabel: relabelOpenCodeSessionAsKilo, } case AgentMiMoCode: return openCodeProviderSpec{ - agent: AgentMiMoCode, - format: mimoFmt, - dbName: mimoFmt.dbName, - listSQLite: ListMiMoCodeSessionMeta, - streamSQLite: streamOpenCodeSessionMetaAs(MiMoCodeSQLiteVirtualPath), - sourceMtime: MiMoCodeSourceMtime, - relabel: relabelOpenCodeSessionAsMiMoCode, + agent: AgentMiMoCode, + format: mimoFmt, + dbName: mimoFmt.dbName, + listSQLite: ListMiMoCodeSessionMeta, + listSQLiteWatermark: listOpenCodeSessionWatermarkMetaAs(MiMoCodeSQLiteVirtualPath), + streamSQLite: streamOpenCodeSessionMetaAs(MiMoCodeSQLiteVirtualPath), + streamSQLiteWatermark: streamOpenCodeSessionWatermarkMetaAs(MiMoCodeSQLiteVirtualPath), + sourceMtime: MiMoCodeSourceMtime, + relabel: relabelOpenCodeSessionAsMiMoCode, } case AgentIcodemate: return openCodeProviderSpec{ - agent: AgentIcodemate, - format: icodemateFmt, - dbName: icodemateFmt.dbName, - listSQLite: ListIcodemateSessionMeta, - streamSQLite: streamOpenCodeSessionMetaAs(IcodemateSQLiteVirtualPath), - sourceMtime: IcodemateSourceMtime, - relabel: relabelOpenCodeSessionAsIcodemate, + agent: AgentIcodemate, + format: icodemateFmt, + dbName: icodemateFmt.dbName, + listSQLite: ListIcodemateSessionMeta, + listSQLiteWatermark: listOpenCodeSessionWatermarkMetaAs(IcodemateSQLiteVirtualPath), + streamSQLite: streamOpenCodeSessionMetaAs(IcodemateSQLiteVirtualPath), + streamSQLiteWatermark: streamOpenCodeSessionWatermarkMetaAs(IcodemateSQLiteVirtualPath), + sourceMtime: IcodemateSourceMtime, + relabel: relabelOpenCodeSessionAsIcodemate, } default: return openCodeProviderSpec{} @@ -243,6 +279,37 @@ func streamOpenCodeSessionMetaAs( } } +func streamOpenCodeSessionWatermarkMetaAs( + virtualPath func(string, string) string, +) func(context.Context, string, func(OpenCodeSessionMeta) error) error { + return func( + ctx context.Context, dbPath string, yield func(OpenCodeSessionMeta) error, + ) error { + return ForEachOpenCodeSessionWatermarkMeta( + ctx, dbPath, + func(meta OpenCodeSessionMeta) error { + meta.VirtualPath = virtualPath(dbPath, meta.SessionID) + return yield(meta) + }, + ) + } +} + +func listOpenCodeSessionWatermarkMetaAs( + virtualPath func(string, string) string, +) func(string) ([]OpenCodeSessionMeta, error) { + return func(dbPath string) ([]OpenCodeSessionMeta, error) { + metas, err := ListOpenCodeSessionWatermarkMeta(dbPath) + if err != nil { + return nil, err + } + for i := range metas { + metas[i].VirtualPath = virtualPath(dbPath, metas[i].SessionID) + } + return metas, nil + } +} + // resolve detects the OpenCode storage backend for a root. func (spec openCodeProviderSpec) resolve(root string) OpenCodeSource { return resolveOpenCodeFormatSource(spec.format, root) @@ -312,25 +379,47 @@ func (spec openCodeProviderSpec) parseSQLite( type openCodeFormatSource struct { Root string Path string - // MTimeNS carries the session's time_updated (already listed during + // MTimeNS carries the session's change signal (already listed during // SQLite discovery, scaled to nanoseconds) so Fingerprint does not // reopen the shared DB once per session. Zero means unknown and makes // Fingerprint fall back to querying the DB. MTimeNS int64 + // CompositeMTime reports that MTimeNS is the per-session composite + // (session, project, and child message/part time_updated) rather than + // the session row's own time_updated. It gates dropping the shared + // container's size from the fingerprint. + CompositeMTime bool + // ChildDigest carries the deletion-sensitive per-session identity into + // Fingerprint.Hash. + ChildDigest string + // WatermarkOnly marks MTimeNS as only the session-row watermark from a + // bounded changed-path listing (see OpenCodeSessionMeta.WatermarkOnly). + // The engine may skip such a source against its stored composite + // watermark without resolving the child digest. + WatermarkOnly bool } type openCodeFormatSourceSet struct { roots []string spec openCodeProviderSpec + // containerTrusted, when non-nil, reports that a shared container is + // byte-identical to the last fully verified pass (see + // ProviderConfig.SQLiteContainerUnchangedSinceTrust). Discover answers + // with the bounded watermark-only listing for such containers: the + // engine's container gate skips every member before fingerprinting, so + // the full child digest would be archive-sized work nothing reads. + containerTrusted func(dbPath string) bool } func newOpenCodeFormatSourceSet( roots []string, spec openCodeProviderSpec, + containerTrusted func(dbPath string) bool, ) openCodeFormatSourceSet { return openCodeFormatSourceSet{ - roots: cleanJSONLRoots(roots), - spec: spec, + roots: cleanJSONLRoots(roots), + spec: spec, + containerTrusted: containerTrusted, } } @@ -357,7 +446,10 @@ func (s openCodeFormatSourceSet) Discover(ctx context.Context) ([]SourceRef, err if src.DBPath == "" || !IsRegularFile(src.DBPath) { continue } - dbSources, err := s.sqliteSources(ctx, root, src.DBPath, storageIDs) + trusted := s.containerTrusted != nil && s.containerTrusted(src.DBPath) + dbSources, err := s.sqliteSources( + ctx, root, src.DBPath, storageIDs, trusted, + ) if err != nil { if ctx.Err() != nil { return nil, err @@ -458,7 +550,16 @@ func (s openCodeFormatSourceSet) discoverRootEach( } var callbackErr error var membershipErr error - err := s.spec.streamSQLite(ctx, src.DBPath, func(meta OpenCodeSessionMeta) error { + // A container the engine's gate will skip wholesale streams the bounded + // watermark listing: computing every session's child digest for a pass + // that verifies nothing would be archive-sized work nothing reads (see + // ProviderConfig.SQLiteContainerUnchangedSinceTrust). + stream := s.spec.streamSQLite + if s.containerTrusted != nil && s.containerTrusted(src.DBPath) && + s.spec.streamSQLiteWatermark != nil { + stream = s.spec.streamSQLiteWatermark + } + err := stream(ctx, src.DBPath, func(meta OpenCodeSessionMeta) error { if storageIDs != nil { _, exists, err := storageIDs.get(ctx, meta.SessionID) if err != nil { @@ -574,6 +675,32 @@ func (s openCodeFormatSourceSet) WatchPlan(context.Context) (WatchPlan, error) { return WatchPlan{Roots: roots}, nil } +// reconciliationContainer maps a requested path to the SQLite container that +// atomically owns it, without statting: a deleted database must still resolve +// so its members remain reclaimable through the container proof. A virtual +// spelling splits at the raw separator instead of parseVirtual, whose exact +// basename check would let a Windows case variant of the database name skip +// widening and admit a single member; the alias comparison below already +// carries the platform's case rule. +func (s openCodeFormatSourceSet) reconciliationContainer( + requested string, +) (string, bool) { + physical := requested + if idx := strings.LastIndex(requested, "#"); idx > 0 && idx < len(requested)-1 { + physical = requested[:idx] + } + for _, root := range s.roots { + db := cleanReconciliationScopeRoot(filepath.Join(root, s.spec.dbName)) + for _, alias := range []string{db, db + "-wal", db + "-shm"} { + if reconciliationScopeSamePath(alias, requested) || + reconciliationScopeSamePath(alias, physical) { + return db, true + } + } + } + return "", false +} + func (s openCodeFormatSourceSet) SourcesForChangedPath( ctx context.Context, req ChangedPathRequest, @@ -603,7 +730,7 @@ func (s openCodeFormatSourceSet) SourcesForChangedPath( } for _, root := range s.roots { sources, ok, err := s.sourcesForChangedPathInRoot( - ctx, root, req.Path, pathExists, + ctx, root, req, pathExists, ) if err != nil || ok { return sources, err @@ -697,6 +824,19 @@ func (s openCodeFormatSourceSet) FindSource( return SourceRef{}, false, nil } +// sourceMtimeWithComposite resolves a source's change signal when discovery did +// not carry one (FindSource lookups, storage sessions), reporting whether the +// value is the per-session composite. +func (s openCodeFormatSourceSet) sourceMtimeWithComposite( + path string, +) (int64, string, bool, error) { + if dbPath, sessionID, ok := s.spec.parseVirtual(path); ok { + return openCodeSQLiteSessionMtimeComposite(dbPath, sessionID) + } + mtime, err := s.spec.sourceMtime(path) + return mtime, "", false, err +} + func (s openCodeFormatSourceSet) Fingerprint( ctx context.Context, source SourceRef, @@ -709,12 +849,37 @@ func (s openCodeFormatSourceSet) Fingerprint( return SourceFingerprint{}, fmt.Errorf("%s source path unavailable", s.spec.agent) } mtime := sourceCarriedMTimeNS(source) - if mtime == 0 { - var err error - mtime, err = s.spec.sourceMtime(path) + composite := sourceCarriedCompositeMTime(source) + digest := sourceCarriedChildDigest(source) + // Only re-open the container when a digest is actually expected. A legacy + // container reports composite=false and carries an empty digest by design, + // so treating "empty" alone as "missing" would reopen and re-query the + // shared database once per session on every cold or changed-container pass. + if mtime == 0 || (composite && digest == "") { + // Sources rebuilt by FindSource or reconciliation carry no discovery + // metadata, and watermark-only changed-path sources carry a + // deliberately unresolved digest. Without this the hash would be + // empty, and an empty hash is treated as no constraint by the + // freshness gate — so a deletion-only change would pass unnoticed on + // every non-discovery path. + lookupMtime, lookupDigest, lookupComposite, err := + s.sourceMtimeWithComposite(path) if err != nil { return SourceFingerprint{}, err } + // Adopt the looked-up watermark alongside the digest: a + // watermark-only source carries the session-row watermark, which can + // sit below the composite the digest folds in. The stored MTimeNS + // must always be the composite, or the next full-discovery pass + // would see a mismatched watermark and re-parse an unchanged session. + if lookupMtime != 0 { + mtime, composite = lookupMtime, lookupComposite + } else if mtime == 0 { + composite = lookupComposite + } + if digest == "" { + digest = lookupDigest + } } fingerprint := SourceFingerprint{ Key: firstNonEmptyJSONLString(source.FingerprintKey, source.Key, path), @@ -725,7 +890,25 @@ func (s openCodeFormatSourceSet) Fingerprint( if err != nil { return SourceFingerprint{}, fmt.Errorf("stat %s: %w", dbPath, err) } - fingerprint.Size = info.Size() + // The watermark alone cannot see a deleted child, because the session + // or project row usually already holds the higher timestamp. The + // digest folds in the child row counts so a delete changes the + // fingerprint; FingerprintHashRequiredForFreshness makes the gate + // compare it against the stored value. + fingerprint.Hash = digest + // Every session in this root shares one physical container, so the + // container's size moves whenever any single session is written. + // Stamping it onto a per-session fingerprint made one session's + // append change the fingerprint of every other session in the + // container, dropping their freshness skip and re-parsing the whole + // root for one changed session. When MTimeNS is the per-session + // composite it already discriminates per session (including in-place + // child edits and project worktree renames), so the container stat + // is existence-only. Legacy containers whose schema cannot produce + // the composite keep the size as their conservative fallback. + if !composite { + fingerprint.Size = info.Size() + } return fingerprint, nil } info, err := os.Stat(path) @@ -749,6 +932,49 @@ func (s openCodeFormatSourceSet) Fingerprint( // sourceCarriedMTimeNS returns the discovery-listed session mtime carried on // a SQLite-backed source, or zero when the source was built without one // (storage sessions, FindSource lookups). +func sourceCarriedChildDigest(source SourceRef) string { + switch src := source.Opaque.(type) { + case openCodeFormatSource: + return src.ChildDigest + case *openCodeFormatSource: + if src != nil { + return src.ChildDigest + } + } + return "" +} + +// SourceWatermarkOnlyMTimeNS returns the carried session-row watermark for a +// shared-container source listed by a watermark-only changed-path scan, and +// whether the source is such a listing. Full-discovery sources carry the +// composite watermark and child digest instead and report false, as do +// legacy containers without composite support. +func SourceWatermarkOnlyMTimeNS(source SourceRef) (int64, bool) { + switch src := source.Opaque.(type) { + case openCodeFormatSource: + if src.WatermarkOnly { + return src.MTimeNS, true + } + case *openCodeFormatSource: + if src != nil && src.WatermarkOnly { + return src.MTimeNS, true + } + } + return 0, false +} + +func sourceCarriedCompositeMTime(source SourceRef) bool { + switch src := source.Opaque.(type) { + case openCodeFormatSource: + return src.CompositeMTime + case *openCodeFormatSource: + if src != nil { + return src.CompositeMTime + } + } + return false +} + func sourceCarriedMTimeNS(source SourceRef) int64 { switch src := source.Opaque.(type) { case openCodeFormatSource: @@ -790,11 +1016,16 @@ func (s openCodeFormatSourceSet) sqliteSources( root string, dbPath string, storageIDs map[string]struct{}, + watermarkOnly bool, ) ([]SourceRef, error) { if err := ctx.Err(); err != nil { return nil, err } - metas, err := s.spec.listSQLite(dbPath) + lister := s.spec.listSQLite + if watermarkOnly && s.spec.listSQLiteWatermark != nil { + lister = s.spec.listSQLiteWatermark + } + metas, err := lister(dbPath) if err != nil { return nil, err } @@ -817,6 +1048,101 @@ func (s openCodeFormatSourceSet) sqliteSources( return sources, nil } +// changedWatermarkSources answers a shared-container change event with only +// the members whose carried session-row watermark is not already covered by +// the caller's stored freshness. The watermark listing streams in ascending +// virtual-path order and the stored side arrives through a paged cursor in +// the same order, so peak memory is one stored page plus the changed batch — +// never the container's full membership. A pager failure fails open: the +// remaining stream is kept unfiltered and the caller's per-file gate decides. +// Legacy rows without composite support (WatermarkOnly false) are always +// kept; their conservative container-size fingerprint must not be bypassed. +func (s openCodeFormatSourceSet) changedWatermarkSources( + ctx context.Context, + root string, + dbPath string, + storageIDs map[string]struct{}, + freshness StoredMemberFreshnessPager, +) ([]SourceRef, error) { + cursor := storedMemberFreshnessCursor{pager: freshness} + var sources []SourceRef + err := s.spec.streamSQLiteWatermark(ctx, dbPath, func(meta OpenCodeSessionMeta) error { + if _, exists := storageIDs[meta.SessionID]; exists { + return nil + } + if meta.WatermarkOnly && !cursor.failed { + covered, err := cursor.covers(ctx, meta.VirtualPath, meta.FileMtime) + if err != nil { + if ctx.Err() != nil { + return ctx.Err() + } + cursor.failed = true + } else if covered { + return nil + } + } + source, ok := s.sqliteSourceRefFromMeta(root, meta) + if !ok { + return nil + } + sources = append(sources, source) + return nil + }) + if err != nil { + return nil, err + } + return sources, nil +} + +const storedMemberFreshnessPageSize = 512 + +// storedMemberFreshnessCursor advances through the caller's paged stored +// freshness in step with an ascending virtual-path stream, retaining one page +// at a time. +type storedMemberFreshnessCursor struct { + pager StoredMemberFreshnessPager + rows []StoredMemberFreshness + index int + after string + done bool + failed bool +} + +// covers reports whether the stored side vouches for path at watermarkNS. +// Paths must arrive in ascending order across calls. +func (c *storedMemberFreshnessCursor) covers( + ctx context.Context, path string, watermarkNS int64, +) (bool, error) { + for { + for c.index < len(c.rows) { + row := c.rows[c.index] + if row.Path < path { + c.index++ + continue + } + if row.Path > path { + return false, nil + } + return watermarkNS <= row.CoveredThroughNS, nil + } + if c.done { + return false, nil + } + rows, done, err := c.pager(ctx, c.after, storedMemberFreshnessPageSize) + if err != nil { + return false, err + } + c.rows, c.index, c.done = rows, 0, done + if len(rows) > 0 { + c.after = rows[len(rows)-1].Path + } else if !done { + // A pager reporting neither rows nor completion cannot make + // progress; treat the stored side as exhausted. + c.done = true + } + } +} + // sqliteSourceRefFromMeta builds a SourceRef for a session row already listed // from the SQLite DB at root. It validates the virtual path parses and that its // DB lives under root, but unlike sourceRef it skips the per-row @@ -839,6 +1165,9 @@ func (s openCodeFormatSourceSet) sqliteSourceRefFromMeta( ref := s.newSourceRef(root, path, "") if src, ok := ref.Opaque.(openCodeFormatSource); ok { src.MTimeNS = meta.FileMtime + src.CompositeMTime = meta.CompositeMtime + src.ChildDigest = meta.ChildDigest + src.WatermarkOnly = meta.WatermarkOnly ref.Opaque = src } return ref, true @@ -847,9 +1176,10 @@ func (s openCodeFormatSourceSet) sqliteSourceRefFromMeta( func (s openCodeFormatSourceSet) sourcesForChangedPathInRoot( ctx context.Context, root string, - path string, + req ChangedPathRequest, pathExists bool, ) ([]SourceRef, bool, error) { + path := req.Path rel, ok := relUnder(root, path) if !ok { return nil, false, nil @@ -883,7 +1213,17 @@ func (s openCodeFormatSourceSet) sourcesForChangedPathInRoot( if s.spec.resolve(root).Mode == OpenCodeSourceStorage { storageIDs = s.spec.storageIDs(root) } - sources, err := s.sqliteSources(ctx, root, dbPath, storageIDs) + if req.AllowWatermarkOnlySources && + req.StoredMemberFreshnessPage != nil && + s.spec.streamSQLiteWatermark != nil { + sources, err := s.changedWatermarkSources( + ctx, root, dbPath, storageIDs, req.StoredMemberFreshnessPage, + ) + return sources, true, err + } + sources, err := s.sqliteSources( + ctx, root, dbPath, storageIDs, req.AllowWatermarkOnlySources, + ) return sources, true, err } @@ -1128,6 +1468,11 @@ func openCodeFormatProviderCapabilities() Capabilities { }, Sync: ProviderSyncSemantics{ UnchangedResults: UnchangedResultMTimeAndHash, + // The per-session digest is the only signal that sees a deleted + // child, so freshness must consult it. Containers without + // composite support produce an empty hash, which the gate treats + // as no constraint, preserving their previous behavior. + FingerprintHashRequiredForFreshness: true, }, } } diff --git a/internal/parser/opencode_provider_test.go b/internal/parser/opencode_provider_test.go index 1adef6b05..bce507f44 100644 --- a/internal/parser/opencode_provider_test.go +++ b/internal/parser/opencode_provider_test.go @@ -148,7 +148,7 @@ func TestOpenCodeStreamingPartialSQLiteFailureContinuesLaterRoots(t *testing.T) } } sources := newOpenCodeFormatSourceSet( - []string{partialRoot, healthyRoot}, spec, + []string{partialRoot, healthyRoot}, spec, nil, ) var paths []string @@ -736,9 +736,12 @@ func TestOpenCodeProviderSQLiteFingerprintUsesDiscoveryMeta(t *testing.T) { "fingerprint must not reopen the SQLite DB for a discovered source") assert.Equal(t, OpenCodeSQLiteVirtualPath(dbPath, "ses_meta"), fp.Key) assert.Equal(t, int64(1700000010000000000), fp.MTimeNS, - "fingerprint mtime must be the discovered time_updated in ns") - assert.Equal(t, int64(len(garbage)), fp.Size, - "fingerprint size stays the shared container file size") + "fingerprint mtime must be the discovered composite in ns") + assert.Zero(t, fp.Size, + "a per-session fingerprint must not carry the shared container's "+ + "size: every session in the root shares one opencode.db, so any "+ + "one session's write would change every other session's "+ + "fingerprint and drop its freshness skip") } func TestOpenCodeProviderHybridDiscoveryFiltersSQLiteDuplicate(t *testing.T) { @@ -1098,5 +1101,260 @@ func newTestDBAt( copyOpenCodeSchemaTemplate(t, dbPath) db, err := sql.Open("sqlite3", dbPath) require.NoError(t, err, "open test db") + // Close before TempDir cleanup: Windows cannot delete a database file + // that still has an open handle. Close is idempotent, so tests that + // close the writer themselves are unaffected. + t.Cleanup(func() { _ = db.Close() }) return dbPath, &OpenCodeSeeder{db: db, t: t}, db } + +// TestOpenCodeSingleSessionMtimeDoesNotScanContainer pins the query shape of +// the single-session composite lookup. Reusing the streaming form's grouped +// subqueries here materializes an aggregate over every message and part in the +// container before the outer WHERE narrows to one session, so each per-session +// lookup would scan the whole archive. Assert the plan touches the child tables +// through their session_id indexes rather than a full scan. +func TestOpenCodeSingleSessionMtimeDoesNotScanContainer(t *testing.T) { + root := t.TempDir() + _, seeder, db := newTestDBAt(t, filepath.Join(root, "opencode.db")) + seeder.AddProject("prj_1", "/home/user/code/app") + seeder.AddSession( + "ses_a", "prj_1", "", "A", 1700000000000, 1700000010000, + ) + t.Cleanup(func() { _ = db.Close() }) + + query := "SELECT " + openCodeSessionCompositeMtimeExpr + + " FROM session s" + openCodeSessionCompositeMtimeJoins + + " WHERE s.id = ?" + rows, err := db.Query("EXPLAIN QUERY PLAN "+query, "ses_a") + require.NoError(t, err) + defer rows.Close() + + var plan strings.Builder + for rows.Next() { + var id, parent, notUsed int + var detail string + require.NoError(t, rows.Scan(&id, &parent, ¬Used, &detail)) + plan.WriteString(detail) + plan.WriteString("\n") + } + require.NoError(t, rows.Err()) + + got := plan.String() + for _, table := range []string{"message", "part"} { + assert.NotContains(t, got, "SCAN "+table, + "single-session composite mtime must not full-scan %s; plan:\n%s", + table, got) + // SEARCH alone is not proof of a seek: SQLite reports SEARCH for some + // aggregate plans without an index, so require the index explicitly. + assert.Regexp(t, + `(?s)(SEARCH|SCAN) `+table+`[^\n]*USING (COVERING )?INDEX`, + got, + "single-session composite mtime must reach %s through an index; "+ + "plan:\n%s", table, got) + } +} + +// TestOpenCodeWatermarkOnlyQuerySkipsDigestScans pins that the mtime-only path +// does not compute the digest aggregates. OpenCodeSourceMtime backs the session +// watcher's 1.5s poll, so pulling the eight child COUNT/SUM/MIN/MAX subqueries +// in there would burn child-range scans per tick for a discarded value. +func TestOpenCodeWatermarkOnlyQuerySkipsDigestScans(t *testing.T) { + watermarkOnly := "SELECT " + openCodeSessionCompositeMtimeExpr + + " FROM session s" + openCodeSessionCompositeMtimeJoins + + " WHERE s.id = ?" + full := "SELECT " + openCodeSessionCompositeMtimeExpr + ", " + + openCodeSessionCompositeCountsExpr + + " FROM session s" + openCodeSessionCompositeMtimeJoins + + " WHERE s.id = ?" + + assert.NotContains(t, watermarkOnly, "COUNT(", + "the mtime-only query must not compute child counts") + assert.NotContains(t, watermarkOnly, "group_concat(", + "the mtime-only query must not build child identities") + assert.Contains(t, full, "COUNT(", + "the fingerprint query must still compute the digest aggregates") + + // Both must be executable, not merely string-shaped: assert against a real + // container so a query that only looks right still fails here. + root := t.TempDir() + _, seeder, db := newTestDBAt(t, filepath.Join(root, "opencode.db")) + seeder.AddProject("prj_1", "/home/user/code/app") + seeder.AddSession( + "ses_a", "prj_1", "", "A", 1700000000000, 1700000010000, + ) + t.Cleanup(func() { _ = db.Close() }) + + var watermark int64 + require.NoError(t, + db.QueryRow(watermarkOnly, "ses_a").Scan(&watermark), + "the mtime-only query must execute") + assert.Equal(t, int64(1700000010000), watermark) + + var ( + w, st, pt, mn, pn int64 + mIdent, pIdent string + ) + require.NoError(t, + db.QueryRow(full, "ses_a").Scan( + &w, &st, &pt, &mn, &pn, &mIdent, &pIdent, + ), + "the fingerprint query must execute") + assert.Equal(t, watermark, w, + "both queries must agree on the watermark") +} + +// TestOpenCodeChangedPathWatermarkMergeEmitsOnlyUncovered pins the bounded +// changed-path listing: with a stored-freshness pager supplied, the provider +// merges the streamed watermark listing against paged stored authority and +// emits only members whose watermark advanced. The pager here returns one +// row per page, so the assertion that every member still resolves correctly +// also proves the merge consumes the stored side incrementally instead of +// materializing the container's membership. +func TestOpenCodeChangedPathWatermarkMergeEmitsOnlyUncovered(t *testing.T) { + dbPath, seeder, _ := newTestDB(t) + seeder.AddProject("proj", "/home/user/app") + const base = int64(1779012000000) + seeder.AddSession("ses-a", "proj", "", "a", base, base) + seeder.AddSession("ses-b", "proj", "", "b", base, base+1000) + seeder.AddSession("ses-c", "proj", "", "c", base, base) + + provider, ok := NewProvider(AgentOpenCode, ProviderConfig{ + Roots: []string{filepath.Dir(dbPath)}, Machine: "local", + }) + require.True(t, ok) + + // Stored authority covers ses-a fully and ses-b only through an older + // watermark; ses-c has no stored row and must be kept. + stored := []StoredMemberFreshness{ + {Path: dbPath + "#ses-a", CoveredThroughNS: base * 1_000_000}, + {Path: dbPath + "#ses-b", CoveredThroughNS: base * 1_000_000}, + } + pagerCalls := 0 + pager := func( + _ context.Context, after string, limit int, + ) ([]StoredMemberFreshness, bool, error) { + pagerCalls++ + require.Positive(t, limit, "pages must be bounded") + for _, row := range stored { + if row.Path > after { + return []StoredMemberFreshness{row}, false, nil + } + } + return nil, true, nil + } + + sources, err := provider.SourcesForChangedPath( + t.Context(), ChangedPathRequest{ + Path: dbPath, WatchRoot: filepath.Dir(dbPath), + AllowWatermarkOnlySources: true, + StoredMemberFreshnessPage: pager, + }, + ) + require.NoError(t, err) + var paths []string + for _, source := range sources { + paths = append(paths, source.DisplayPath) + } + assert.ElementsMatch(t, + []string{dbPath + "#ses-b", dbPath + "#ses-c"}, paths, + "only the advanced member and the unknown member are emitted") + assert.GreaterOrEqual(t, pagerCalls, 2, + "the stored side is consumed page by page") +} + +// TestOpenCodeChangedPathWatermarkMergeFailsOpenOnPagerError pins the +// fail-open contract: a stored-side failure keeps every remaining source so +// the caller's per-file gates decide, matching the pre-merge behavior of a +// failed freshness query. +func TestOpenCodeChangedPathWatermarkMergeFailsOpenOnPagerError(t *testing.T) { + dbPath, seeder, _ := newTestDB(t) + seeder.AddProject("proj", "/home/user/app") + const base = int64(1779012000000) + seeder.AddSession("ses-a", "proj", "", "a", base, base) + seeder.AddSession("ses-b", "proj", "", "b", base, base) + + provider, ok := NewProvider(AgentOpenCode, ProviderConfig{ + Roots: []string{filepath.Dir(dbPath)}, Machine: "local", + }) + require.True(t, ok) + + pager := func( + context.Context, string, int, + ) ([]StoredMemberFreshness, bool, error) { + return nil, false, errors.New("stored freshness unavailable") + } + sources, err := provider.SourcesForChangedPath( + t.Context(), ChangedPathRequest{ + Path: dbPath, WatchRoot: filepath.Dir(dbPath), + AllowWatermarkOnlySources: true, + StoredMemberFreshnessPage: pager, + }, + ) + require.NoError(t, err) + assert.Len(t, sources, 2, + "a pager failure must keep every source for the per-file gates") +} + +// TestOpenCodeChangedPathWatermarkMergeMaterializesOnlyChangedBatch is the +// cardinality-scaling regression for the watcher fast path: a container with +// many stored-covered sessions and one changed session must emit exactly the +// changed batch. Before the merge, the listing materialized every session as +// a SourceRef and the caller loaded every stored member into a map, so each +// database event allocated O(total sessions); the emitted-length assertion +// here fails against any regression to that shape, and the small/large +// comparison pins that per-event output does not scale with the archive. +func TestOpenCodeChangedPathWatermarkMergeMaterializesOnlyChangedBatch( + t *testing.T, +) { + emittedForContainerOf := func(sessions int) int { + dbPath, seeder, _ := newTestDB(t) + seeder.AddProject("proj", "/home/user/app") + const base = int64(1779012000000) + var stored []StoredMemberFreshness + for i := range sessions { + id := fmt.Sprintf("ses-%06d", i) + seeder.AddSession(id, "proj", "", id, base, base) + stored = append(stored, StoredMemberFreshness{ + Path: dbPath + "#" + id, CoveredThroughNS: base * 1_000_000, + }) + } + // One session advances past its stored coverage. + changed := fmt.Sprintf("ses-%06d", sessions/2) + _, err := seeder.db.ExecContext(t.Context(), + "UPDATE session SET time_updated = ? WHERE id = ?", + base+1000, changed, + ) + require.NoError(t, err, "advance changed session") + + provider, ok := NewProvider(AgentOpenCode, ProviderConfig{ + Roots: []string{filepath.Dir(dbPath)}, Machine: "local", + }) + require.True(t, ok) + pager := func( + _ context.Context, after string, limit int, + ) ([]StoredMemberFreshness, bool, error) { + start := 0 + for start < len(stored) && stored[start].Path <= after { + start++ + } + end := min(start+limit, len(stored)) + return stored[start:end], end == len(stored), nil + } + sources, err := provider.SourcesForChangedPath( + t.Context(), ChangedPathRequest{ + Path: dbPath, WatchRoot: filepath.Dir(dbPath), + AllowWatermarkOnlySources: true, + StoredMemberFreshnessPage: pager, + }, + ) + require.NoError(t, err) + return len(sources) + } + + small := emittedForContainerOf(8) + large := emittedForContainerOf(1200) + assert.Equal(t, 1, small) + assert.Equal(t, small, large, + "the emitted batch must not scale with container size") +} diff --git a/internal/parser/opencode_test.go b/internal/parser/opencode_test.go index 157cebe7f..ce31f9523 100644 --- a/internal/parser/opencode_test.go +++ b/internal/parser/opencode_test.go @@ -76,6 +76,14 @@ CREATE TABLE part ( data TEXT NOT NULL, FOREIGN KEY (message_id) REFERENCES message(id) ); + +-- SQLite does not index a foreign key automatically. Production OpenCode +-- declares these, and the per-session freshness lookups depend on them, so the +-- fixture must carry them or plan assertions prove nothing. +CREATE INDEX message_session_time_created_id_idx + ON message (session_id, time_created, id); +CREATE INDEX part_session_idx ON part (session_id); +CREATE INDEX part_message_id_id_idx ON part (message_id, id); ` func assertEq[T comparable](t *testing.T, name string, got, want T) { @@ -160,6 +168,10 @@ func newTestDB(t *testing.T) (string, *OpenCodeSeeder, *sql.DB) { copyOpenCodeSchemaTemplate(t, dbPath) db, err := sql.Open("sqlite3", dbPath) require.NoError(t, err, "open test db") + // Close before TempDir cleanup: Windows cannot delete a database file + // that still has an open handle. Close is idempotent, so tests that + // close the writer themselves are unaffected. + t.Cleanup(func() { _ = db.Close() }) seeder := &OpenCodeSeeder{db: db, t: t} return dbPath, seeder, db @@ -1501,6 +1513,108 @@ func TestListOpenCodeSessionMeta_NonexistentDB(t *testing.T) { assertEq(t, "metas len", len(metas), 0) } +// TestListOpenCodeSessionWatermarkMeta pins the bounded changed-path listing: +// on a composite-capable container it carries only the session-row watermark +// (session and project time_updated, never child times) with no digest, so +// listing every session touches no message or part rows. +func TestListOpenCodeSessionWatermarkMeta(t *testing.T) { + dbPath, seeder, db := newTestDB(t) + defer db.Close() + + seeder.AddProject("prj_1", "/home/user/code/app") + seeder.AddSession( + "ses_wm", "prj_1", "", "Watermark", 1700000000000, 1700000060000, + ) + seeder.AddMessage( + "msg_1", "ses_wm", 1700000000000, 1700099999000, `{"role":"user"}`, + ) + seeder.AddPart( + "prt_1", "msg_1", "ses_wm", 1700000000000, 1700099999000, + `{"type":"text","text":"hi"}`, + ) + // Project row above the session row: the watermark is MAX(session, + // project). Child rows sit above both and must NOT be reflected. + _, err := db.Exec( + "UPDATE project SET time_updated = ? WHERE id = ?", + 1700000070000, "prj_1", + ) + require.NoError(t, err, "raise project time") + + metas, err := ListOpenCodeSessionWatermarkMeta(dbPath) + require.NoError(t, err, "ListOpenCodeSessionWatermarkMeta") + require.Len(t, metas, 1) + + m := metas[0] + assert.Equal(t, "ses_wm", m.SessionID) + assert.Equal(t, dbPath+"#ses_wm", m.VirtualPath) + assert.True(t, m.WatermarkOnly, "composite container must list watermark-only") + assert.True(t, m.CompositeMtime) + assert.Empty(t, m.ChildDigest, "watermark listing must not resolve the child digest") + assert.Equal(t, int64(1700000070000)*1_000_000, m.FileMtime, + "watermark must be MAX(session, project) and exclude child times") +} + +// TestOpenCodeChildDigestMetadataWatermarkNS pins the digest round-trip the +// watcher's like-for-like comparison depends on: the session/project times a +// digest embeds must come back out as the metadata watermark, and every +// other hash shape must be rejected so callers fall back to the composite. +func TestOpenCodeChildDigestMetadataWatermarkNS(t *testing.T) { + agg := openCodeChildAggregate{ + watermark: 1700000099000, + sessionTime: 1700000060000, + projectTime: 1700000070000, + messages: 2, + parts: 5, + messageIdent: "m1:1", + partIdent: "p1:1", + } + got, ok := OpenCodeChildDigestMetadataWatermarkNS(agg.digest(true)) + require.True(t, ok, "digest must round-trip its metadata watermark") + assert.Equal(t, int64(1700000070000)*1_000_000, got, + "metadata watermark must be MAX(session, project), not the composite") + + for _, hash := range []string{ + "", + agg.digest(false), + openCodeStorageFingerprintPrefix + "abcdef", + "opencode-child:v2:1:2:3:4:5:aabb", + "opencode-child:v1:1:2:3", + "opencode-child:v1:x:2:3:4:5:aabb", + "opencode-child:v1:1:x:3:4:5:aabb", + "opencode-child:v1:1:2:x:4:5:aabb", + } { + _, ok := OpenCodeChildDigestMetadataWatermarkNS(hash) + assert.False(t, ok, "hash %q must be rejected", hash) + } +} + +// TestListOpenCodeSessionWatermarkMeta_LegacySchema pins that containers +// without composite support keep the full listing's shape: session-only +// mtime, no composite, and no watermark-only marker, so the engine never +// watermark-skips a session whose only change signal is the container size. +func TestListOpenCodeSessionWatermarkMeta_LegacySchema(t *testing.T) { + dbPath, seeder, db := newLegacyOpenCodeTestDB(t) + defer db.Close() + + seeder.AddProject("prj_legacy", "/home/user/code/legacy-app") + seeder.AddSession( + "ses_legacy", "prj_legacy", "", "Legacy", 1700000000000, 1700000060000, + ) + + metas, err := ListOpenCodeSessionWatermarkMeta(dbPath) + require.NoError(t, err, "ListOpenCodeSessionWatermarkMeta legacy") + require.Len(t, metas, 1) + + full, err := ListOpenCodeSessionMeta(dbPath) + require.NoError(t, err, "ListOpenCodeSessionMeta legacy") + require.Len(t, full, 1) + + assert.False(t, metas[0].WatermarkOnly, + "legacy containers must not be marked watermark-only") + assert.Equal(t, full[0], metas[0], + "legacy watermark listing must match the full listing") +} + // TestParseOpenCodeDB_TokenUsage verifies that an assistant // message with modelID and tokens populates ParsedMessage.Model // and TokenUsage in the agentsview-native key shape, and that diff --git a/internal/parser/provider.go b/internal/parser/provider.go index 2d8ad17f3..ca8a6f741 100644 --- a/internal/parser/provider.go +++ b/internal/parser/provider.go @@ -3,6 +3,10 @@ package parser import ( "context" "errors" + "fmt" + "path/filepath" + "slices" + "strings" "time" ) @@ -55,6 +59,17 @@ type ProviderConfig struct { // path (Aider) use it to seed those IDs from the canonical remote path // rather than the changing temp path. Most providers ignore it. PathRewriter func(string) string + // SQLiteContainerUnchangedSinceTrust reports that the shared SQLite + // container at dbPath is byte-identical to the last pass that verified + // every one of its sessions, as captured before this discovery began. + // Providers that fan such a container out to per-session sources may + // answer discovery for it with the bounded watermark-only listing: the + // caller's container gate will skip every member before fingerprinting, + // so computing the full child digest would be archive-sized work for + // values nothing reads. Nil (the default, and every non-discovery + // construction) means no container is trusted and listings stay + // full-fidelity. + SQLiteContainerUnchangedSinceTrust func(dbPath string) bool } // Clone returns an independent config snapshot. @@ -93,6 +108,53 @@ type Provider interface { context.Context, IncrementalRequest, ) (IncrementalOutcome, IncrementalStatus, error) + + // ResolveReconciliationScopes maps one reconciliation request onto the + // provider's configured topology. The provider that owns the format is the + // only authority for alias, container, and gateway relationships between + // roots; callers consume the returned plan verbatim and never recompute + // ancestor/descendant, filename, or alias relationships themselves. A + // returned error means the topology could not be resolved and the request + // must fail closed: no coverage credit and no deletion authority. + ResolveReconciliationScopes( + context.Context, ReconciliationScopeRequest, + ) (ReconciliationScopePlan, error) +} + +// ReconciliationScopeRequest carries the reconciliation roots a caller asked +// about, exactly as supplied. Callers must not pre-expand or normalize them; +// the provider owns the mapping onto its configured topology. +type ReconciliationScopeRequest struct { + Roots []string +} + +// ReconciliationScope is one provider-resolved unit of reconciliation +// authority. Its four fields are consumed independently: TraversalRoots are +// the roots discovery must walk (a provider may require a wider gateway than +// the request, such as a Claude projects directory for a requested project +// child); PhysicalProofScopes bound which discovered sources and stored +// ownership rows the pass may claim; CoverageIdentities are credited against +// the plan's RequiredCoverageIdentities only once the scope's admitted stream +// completes; RetryRoots are the caller's own requested roots, so a failed +// scope retries at the width the caller asked for rather than the traversal +// width. +type ReconciliationScope struct { + TraversalRoots []string + PhysicalProofScopes []StoredSourceHintScope + CoverageIdentities []string + RetryRoots []string +} + +// ReconciliationScopePlan is the complete resolution of one request against +// one provider's configured topology. An empty plan means the request touches +// nothing the provider owns and the pass is a bounded no-op for it. +// RequiredCoverageIdentities enumerate every coverage identity the provider's +// full configured scope carries; a pass holds full-coverage deletion +// authority only when the completed scopes' CoverageIdentities include all of +// them. +type ReconciliationScopePlan struct { + Scopes []ReconciliationScope + RequiredCoverageIdentities []string } // ReconciliationSourceResolver rebuilds the exact source emitted by streaming @@ -164,15 +226,6 @@ type StoredSourceHintScopeProvider interface { StoredSourceHintScopes(ChangedPathRequest) []StoredSourceHintScope } -// ReconciliationOwnershipScopeProvider maps one logical configured root to -// the bounded stored-source scopes it physically owns. Providers whose stored -// identities are virtual members of a sibling container use this to keep -// deletion ownership paging within that container without broadening the scan -// to the provider's full archive. -type ReconciliationOwnershipScopeProvider interface { - ReconciliationOwnershipScopes(root string) []StoredSourceHintScope -} - // ProviderBase is embedded by concrete providers to make optional source // methods callable with zero-value no-op behavior. type ProviderBase struct { @@ -232,6 +285,288 @@ func (b ProviderBase) unsupported(feature string) error { } } +// ResolveReconciliationScopes supplies the generic directory topology every +// provider inherits: each configured root is one atomic coverage unit, a +// requested descendant traverses from its deepest configured ancestor while +// proving only the descendant, and an ancestor request splits into the +// configured roots it covers without claiming unrelated paths beneath it. +func (b ProviderBase) ResolveReconciliationScopes( + _ context.Context, req ReconciliationScopeRequest, +) (ReconciliationScopePlan, error) { + if err := ValidateReconciliationScopeRoots( + b.Def.Type, b.Config.Roots, req.Roots, + ); err != nil { + return ReconciliationScopePlan{}, err + } + return directoryReconciliationScopePlan(b.Config.Roots, req.Roots), nil +} + +// isRemoteReconciliationScopeRoot mirrors the sync engine's remote-root +// filter: remote object roots are owned by the remote sync path and resolve +// no local reconciliation scope. +func isRemoteReconciliationScopeRoot(root string) bool { + return strings.HasPrefix(strings.ToLower(root), "s3://") +} + +// ValidateReconciliationScopeRoots rejects a local root that cannot become a +// bounded proof scope. A scope proves itself by matching stored source paths +// against a path prefix, and discovery writes those paths by joining the +// configured spelling with each relative source path. A root that cleans to +// "." has no spelling to join: discovery emits bare filenames, so no prefix +// bounds them and the stored-hint normalizer discards the scope, which would +// credit coverage over zero rows. Every other spelling, relative or absolute, +// yields a usable prefix. Remote roots are exempt; a local pass never covers +// an object store. +func ValidateReconciliationScopeRoots( + agent AgentType, configuredRoots, requestedRoots []string, +) error { + for _, group := range [][]string{configuredRoots, requestedRoots} { + for _, root := range group { + if strings.TrimSpace(root) == "" || + isRemoteReconciliationScopeRoot(root) || + filepath.Clean(root) != "." { + continue + } + return fmt.Errorf( + "%s reconciliation root %q resolves to the working "+ + "directory: sources discovered under it are stored "+ + "without a directory prefix, so no reconciliation scope "+ + "can prove what it covers", + agent, root, + ) + } + } + return nil +} + +// cleanReconciliationScopeRoot normalizes one local root for comparison and +// coverage identity. Blank roots must be discarded before this runs: cleaning +// "" resolves to the process working directory, which can encompass configured +// roots. This is deliberately not the spelling a scope proves itself with; see +// reconciliationProofSpelling. +func cleanReconciliationScopeRoot(root string) string { + cleaned := filepath.Clean(root) + abs, err := filepath.Abs(cleaned) + if err != nil { + return cleaned + } + return abs +} + +// reconciliationProofSpelling is the spelling a scope proves itself with: +// the configured one, cleaned. Comparison and coverage identity absolutize so +// a request naming the same directory differently still matches, but proof +// matches stored source paths as a prefix, and those were written by joining +// the configured spelling. Absolutizing proof would leave a relative root's +// stored paths unreachable, and would rewrite a rooted-but-driveless root such +// as "/sessions" onto the current Windows drive, which discovery never used. +func reconciliationProofSpelling(root string) string { + return filepath.Clean(root) +} + +// reconciliationScopeSamePath and reconciliationScopeWithinOrSame compare with +// filepath.Rel semantics so equality matches the platform: case-folded per +// element on Windows, byte-exact elsewhere. +func reconciliationScopeSamePath(a, b string) bool { + rel, err := filepath.Rel(a, b) + return err == nil && rel == "." +} + +func reconciliationScopeWithinOrSame(path, root string) bool { + rel, err := filepath.Rel(root, path) + if err != nil { + return false + } + return rel == "." || rel != ".." && + !strings.HasPrefix(rel, ".."+string(filepath.Separator)) +} + +// descendantProofSpelling re-expresses a requested descendant in the spelling +// its gateway's stored sources carry. Requests arrive absolute because they +// come from watch and polling roots, but a source discovered under a relative +// configured root was stored relative, so an absolute descendant prefix would +// match none of them. +func descendantProofSpelling(gatewayClean, gatewayProof, descendant string) string { + rel, err := filepath.Rel(gatewayClean, descendant) + if err != nil { + return descendant + } + if rel == "." { + return gatewayProof + } + return filepath.Join(gatewayProof, rel) +} + +// reconciliationContainerTopology reports the multi-member container that +// atomically owns a requested path: the container file itself, a WAL or SHM +// sidecar, or a virtual member spelling. The planner widens such a request to +// the whole container so no scope can prove a strict subset of a container's +// members: a partial proof would either admit nothing (a bare container path +// matches no member row) or let a completed pass promote container-state +// trust over members it never verified. Classification must not stat, so a +// deleted container still resolves and its members stay reclaimable. +type reconciliationContainerTopology func(requested string) (container string, ok bool) + +func directoryReconciliationScopePlan( + configuredRoots, requestedRoots []string, +) ReconciliationScopePlan { + return containerAwareReconciliationScopePlan( + configuredRoots, requestedRoots, nil, + ) +} + +func containerAwareReconciliationScopePlan( + configuredRoots, requestedRoots []string, + containers reconciliationContainerTopology, +) ReconciliationScopePlan { + type configuredScopeRoot struct { + // display is the configured spelling handed back for traversal so + // discovery sees exactly the roots it was configured with. + display string + // clean is the absolute form used for comparison and coverage + // identity; proof is the configured form stored sources carry. + clean string + proof string + } + var configured []configuredScopeRoot + var required []string + for _, root := range configuredRoots { + if strings.TrimSpace(root) == "" { + continue + } + if isRemoteReconciliationScopeRoot(root) { + // A remote root is never coverable by a local pass; keeping its + // identity required makes full-coverage authority unreachable. + required = append(required, root) + continue + } + clean := cleanReconciliationScopeRoot(root) + if slices.ContainsFunc(configured, func(existing configuredScopeRoot) bool { + return reconciliationScopeSamePath(existing.clean, clean) + }) { + continue + } + configured = append(configured, configuredScopeRoot{ + display: root, clean: clean, + proof: reconciliationProofSpelling(root), + }) + required = append(required, clean) + } + plan := ReconciliationScopePlan{RequiredCoverageIdentities: required} + fullIndex := make(map[string]int, len(configured)) + appendRetry := func(index int, raw string) { + scope := &plan.Scopes[index] + if !slices.Contains(scope.RetryRoots, raw) { + scope.RetryRoots = append(scope.RetryRoots, raw) + } + } + type descendantScope struct { + gateway configuredScopeRoot + proof string + virtual bool + retry []string + } + var descendants []descendantScope + for _, raw := range requestedRoots { + if strings.TrimSpace(raw) == "" || isRemoteReconciliationScopeRoot(raw) { + continue + } + requested := cleanReconciliationScopeRoot(raw) + virtual := false + if containers != nil { + if container, ok := containers(requested); ok { + requested = cleanReconciliationScopeRoot(container) + virtual = true + } + } + exact := false + for _, cfg := range configured { + if !reconciliationScopeWithinOrSame(cfg.clean, requested) { + continue + } + // The request names the configured root exactly or an ancestor + // covering it: the whole configured root is one atomic scope. + exact = exact || reconciliationScopeSamePath(cfg.clean, requested) + if index, ok := fullIndex[cfg.clean]; ok { + appendRetry(index, raw) + continue + } + fullIndex[cfg.clean] = len(plan.Scopes) + plan.Scopes = append(plan.Scopes, ReconciliationScope{ + TraversalRoots: []string{cfg.display}, + PhysicalProofScopes: []StoredSourceHintScope{{Path: cfg.proof}}, + CoverageIdentities: []string{cfg.clean}, + RetryRoots: []string{raw}, + }) + } + if exact { + continue + } + // A requested descendant traverses from its deepest configured + // ancestor (providers may treat the configured root as a layout + // gateway) while proving only the descendant, so coverage stays + // incomplete and no sibling is claimed. Covering some other + // configured root as an ancestor above does not discharge this: + // with roots /a and /a/b/c, a request for /a/b claims /a/b/c fully + // and still proves /a/b under /a, or removals under /a/b would + // silently stop reconciling. Only an exact configured-root match + // makes a descendant scope redundant. + var gateway *configuredScopeRoot + for i, cfg := range configured { + if reconciliationScopeSamePath(requested, cfg.clean) || + !reconciliationScopeWithinOrSame(requested, cfg.clean) { + continue + } + if gateway == nil || reconciliationScopeWithinOrSame( + cfg.clean, gateway.clean, + ) { + gateway = &configured[i] + } + } + if gateway == nil { + continue + } + merged := false + for i := range descendants { + if reconciliationScopeSamePath(descendants[i].proof, requested) { + if !slices.Contains(descendants[i].retry, raw) { + descendants[i].retry = append(descendants[i].retry, raw) + } + descendants[i].virtual = descendants[i].virtual || virtual + merged = true + break + } + } + if !merged { + descendants = append(descendants, descendantScope{ + gateway: *gateway, proof: requested, virtual: virtual, + retry: []string{raw}, + }) + } + } + for _, descendant := range descendants { + gateway := descendant.gateway + if index, ok := fullIndex[gateway.clean]; ok { + // The same request set already claims the full gateway; the + // descendant is subsumed and only contributes retry identities. + for _, raw := range descendant.retry { + appendRetry(index, raw) + } + continue + } + plan.Scopes = append(plan.Scopes, ReconciliationScope{ + TraversalRoots: []string{gateway.display}, + PhysicalProofScopes: []StoredSourceHintScope{{ + Path: descendantProofSpelling(gateway.clean, gateway.proof, + descendant.proof), + IncludeVirtualMembers: descendant.virtual, + }}, + RetryRoots: descendant.retry, + }) + } + return plan +} + // SourceRef is the engine-visible handle for provider-owned source data. It is // the only source identity the engine should carry between discovery, changed // path classification, lookup, fingerprinting, parsing, skip-cache checks, and @@ -405,8 +740,45 @@ type ChangedPathRequest struct { // still validate ownership against the changed path/watch root before // emitting them. StoredSourcePaths []string + // AllowWatermarkOnlySources tells providers that fan a shared container + // out to per-session virtual sources that the caller's downstream + // freshness gate can cheaply skip watermark-only sources, so the provider + // may answer with a bounded session-row listing instead of computing + // every session's full child digest. Callers that consume the returned + // sources directly (reconciliation, tombstoning) must leave this unset to + // keep full-fidelity fingerprint metadata. + AllowWatermarkOnlySources bool + // StoredMemberFreshnessPage, set only alongside AllowWatermarkOnlySources, + // pages the caller's stored freshness for the changed shared container in + // ascending virtual-path order. A provider answering with the bounded + // watermark listing merges against it during the stream and omits members + // whose carried watermark is already covered, so a one-session write + // emits one source without materializing the container's full membership. + // Nil means the caller holds no stored authority and every source must be + // returned. Callers gate this on a container capture taken before the + // listing and re-request unfiltered when that capture goes stale. + StoredMemberFreshnessPage StoredMemberFreshnessPager +} + +// StoredMemberFreshness is one stored virtual member's coverage authority: a +// listed source at Path whose carried watermark is at or below +// CoveredThroughNS is provably unchanged and may be omitted from a +// changed-path listing. Rows the caller cannot vouch for (a stale data +// version, an unparseable stored identity) are simply not emitted, which +// keeps their sources listed. +type StoredMemberFreshness struct { + Path string + CoveredThroughNS int64 } +// StoredMemberFreshnessPager returns stored freshness rows strictly after +// afterPath in ascending path order, at most limit of them, and whether the +// container's stored membership is exhausted. Implementations must be safe to +// call repeatedly within one listing. +type StoredMemberFreshnessPager func( + ctx context.Context, afterPath string, limit int, +) ([]StoredMemberFreshness, bool, error) + // FindSourceRequest contains lookup inputs and persisted source hints for // provider-owned source resolution. RawSessionID and FullSessionID identify the // requested logical session. StoredFilePath and FingerprintKey are advisory diff --git a/internal/parser/reconciliation_scope_test.go b/internal/parser/reconciliation_scope_test.go new file mode 100644 index 000000000..d59ea865b --- /dev/null +++ b/internal/parser/reconciliation_scope_test.go @@ -0,0 +1,777 @@ +package parser + +import ( + "os" + "path/filepath" + "runtime" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func resolveDirectoryPlan( + t *testing.T, configured, requested []string, +) ReconciliationScopePlan { + t.Helper() + base := ProviderBase{Config: ProviderConfig{Roots: configured}} + plan, err := base.ResolveReconciliationScopes( + t.Context(), ReconciliationScopeRequest{Roots: requested}, + ) + require.NoError(t, err) + return plan +} + +func TestDirectoryPlanExactConfiguredRootIsOneAtomicScope(t *testing.T) { + root := t.TempDir() + plan := resolveDirectoryPlan(t, []string{root}, []string{root}) + + require.Len(t, plan.Scopes, 1) + scope := plan.Scopes[0] + assert.Equal(t, []string{root}, scope.TraversalRoots) + assert.Equal(t, + []StoredSourceHintScope{{Path: filepath.Clean(root)}}, + scope.PhysicalProofScopes) + assert.Equal(t, []string{filepath.Clean(root)}, scope.CoverageIdentities) + assert.Equal(t, []string{root}, scope.RetryRoots) + assert.Equal(t, []string{filepath.Clean(root)}, + plan.RequiredCoverageIdentities) +} + +func TestDirectoryPlanDescendantTraversesGatewayAndProvesDescendant(t *testing.T) { + root := t.TempDir() + descendant := filepath.Join(root, "project") + plan := resolveDirectoryPlan(t, []string{root}, []string{descendant}) + + require.Len(t, plan.Scopes, 1) + scope := plan.Scopes[0] + assert.Equal(t, []string{root}, scope.TraversalRoots, + "traversal must come from the configured gateway") + assert.Equal(t, + []StoredSourceHintScope{{Path: descendant}}, + scope.PhysicalProofScopes, + "proof must stay bounded to the requested descendant") + assert.Empty(t, scope.CoverageIdentities, + "a descendant request cannot cover the configured root") + assert.Equal(t, []string{descendant}, scope.RetryRoots) +} + +func TestDirectoryPlanDescendantUsesDeepestConfiguredAncestor(t *testing.T) { + outer := t.TempDir() + inner := filepath.Join(outer, "inner") + requested := filepath.Join(inner, "leaf") + plan := resolveDirectoryPlan(t, []string{outer, inner}, []string{requested}) + + require.Len(t, plan.Scopes, 1) + assert.Equal(t, []string{inner}, plan.Scopes[0].TraversalRoots, + "the deepest configured ancestor is the traversal gateway") +} + +func TestDirectoryPlanAncestorClaimKeepsSiblingDescendantProof(t *testing.T) { + outer := t.TempDir() + requested := filepath.Join(outer, "mid") + inner := filepath.Join(requested, "leaf") + plan := resolveDirectoryPlan(t, []string{outer, inner}, []string{requested}) + + require.Len(t, plan.Scopes, 2, + "covering the inner root must not drop the descendant proof under the outer") + full := plan.Scopes[0] + assert.Equal(t, []string{inner}, full.CoverageIdentities, + "the covered configured root is claimed fully") + descendant := plan.Scopes[1] + assert.Equal(t, []string{outer}, descendant.TraversalRoots) + assert.Equal(t, + []StoredSourceHintScope{{Path: requested}}, + descendant.PhysicalProofScopes, + "the request still proves itself under its configured ancestor") + assert.Empty(t, descendant.CoverageIdentities) +} + +func TestDirectoryPlanAncestorSplitsIntoCoveredConfiguredRoots(t *testing.T) { + parent := t.TempDir() + first := filepath.Join(parent, "first") + second := filepath.Join(parent, "second") + plan := resolveDirectoryPlan(t, []string{first, second}, []string{parent}) + + require.Len(t, plan.Scopes, 2) + var identities []string + for _, scope := range plan.Scopes { + identities = append(identities, scope.CoverageIdentities...) + assert.Equal(t, []string{parent}, scope.RetryRoots, + "retry identities are the caller's own roots") + require.Len(t, scope.PhysicalProofScopes, 1) + assert.NotEqual(t, parent, scope.PhysicalProofScopes[0].Path, + "an ancestor request must not claim unrelated paths beneath itself") + } + assert.ElementsMatch(t, []string{first, second}, identities) +} + +func TestDirectoryPlanUnrelatedBlankAndRemoteRootsResolveNothing(t *testing.T) { + root := t.TempDir() + unrelated := t.TempDir() + plan := resolveDirectoryPlan(t, []string{root}, []string{ + "", " ", "s3://bucket/prefix", unrelated, + }) + + assert.Empty(t, plan.Scopes) + assert.Equal(t, []string{filepath.Clean(root)}, + plan.RequiredCoverageIdentities) +} + +func TestDirectoryPlanBlankRootNeverResolvesToWorkingDirectory(t *testing.T) { + cwd, err := os.Getwd() + require.NoError(t, err) + // The working directory is a configured root, so a blank request that + // leaked through filepath.Abs("") would claim it. + plan := resolveDirectoryPlan(t, []string{cwd}, []string{""}) + assert.Empty(t, plan.Scopes, + "a blank root must be discarded before normalization") +} + +func TestDirectoryPlanRemoteConfiguredRootKeepsCoverageUnreachable(t *testing.T) { + root := t.TempDir() + remote := "s3://bucket/sessions" + plan := resolveDirectoryPlan(t, []string{root, remote}, []string{root}) + + require.Len(t, plan.Scopes, 1) + assert.ElementsMatch(t, + []string{filepath.Clean(root), remote}, + plan.RequiredCoverageIdentities, + "a remote configured root is never coverable by a local pass") +} + +func TestDirectoryPlanDeduplicatesPathEquivalentRoots(t *testing.T) { + root := t.TempDir() + spelledDot := root + string(filepath.Separator) + "." + plan := resolveDirectoryPlan( + t, []string{root, spelledDot}, []string{root, spelledDot}, + ) + + require.Len(t, plan.Scopes, 1) + assert.Equal(t, []string{filepath.Clean(root)}, + plan.RequiredCoverageIdentities) + assert.ElementsMatch(t, []string{root, spelledDot}, + plan.Scopes[0].RetryRoots, + "merged spellings keep every caller retry identity") +} + +func TestDirectoryPlanFullScopeSubsumesRequestedDescendant(t *testing.T) { + root := t.TempDir() + descendant := filepath.Join(root, "project") + plan := resolveDirectoryPlan(t, []string{root}, []string{root, descendant}) + + require.Len(t, plan.Scopes, 1) + assert.Equal(t, []string{filepath.Clean(root)}, + plan.Scopes[0].CoverageIdentities) + assert.ElementsMatch(t, []string{root, descendant}, + plan.Scopes[0].RetryRoots) +} + +func TestDirectoryPlanEmptyConfigurationIsEmptyPlan(t *testing.T) { + plan := resolveDirectoryPlan(t, nil, []string{t.TempDir()}) + assert.Empty(t, plan.Scopes) + assert.Empty(t, plan.RequiredCoverageIdentities) +} + +// writeHermesPlanArchive lays out one archive directory with a state.db file +// and a sessions directory; plan resolution only stats the layout, so the +// state.db content is irrelevant. +func writeHermesPlanArchive(t *testing.T, dir string) (stateDB, sessionsDir string) { + t.Helper() + require.NoError(t, os.MkdirAll(dir, 0o755)) + stateDB = filepath.Join(dir, "state.db") + sessionsDir = filepath.Join(dir, "sessions") + require.NoError(t, os.WriteFile(stateDB, []byte("db"), 0o644)) + require.NoError(t, os.MkdirAll(sessionsDir, 0o755)) + return stateDB, sessionsDir +} + +func resolveHermesPlan( + t *testing.T, configured, requested []string, +) ReconciliationScopePlan { + t.Helper() + provider, ok := NewProvider(AgentHermes, ProviderConfig{Roots: configured}) + require.True(t, ok) + plan, err := provider.ResolveReconciliationScopes( + t.Context(), ReconciliationScopeRequest{Roots: requested}, + ) + require.NoError(t, err) + return plan +} + +func TestHermesReconciliationPlanAliasSpellingsResolveIdentically(t *testing.T) { + archive := filepath.Join(t.TempDir(), "archive") + stateDB, sessionsDir := writeHermesPlanArchive(t, archive) + + base := resolveHermesPlan(t, []string{archive}, []string{archive}) + viaState := resolveHermesPlan(t, []string{archive}, []string{stateDB}) + viaSessions := resolveHermesPlan(t, []string{archive}, []string{sessionsDir}) + + require.Len(t, base.Scopes, 1) + for _, plan := range []ReconciliationScopePlan{viaState, viaSessions} { + require.Len(t, plan.Scopes, 1) + assert.Equal(t, base.Scopes[0].TraversalRoots, + plan.Scopes[0].TraversalRoots) + assert.Equal(t, base.Scopes[0].PhysicalProofScopes, + plan.Scopes[0].PhysicalProofScopes, + "no spelling may omit state.db members or sessions/ transcripts") + assert.Equal(t, base.Scopes[0].CoverageIdentities, + plan.Scopes[0].CoverageIdentities) + assert.Equal(t, base.RequiredCoverageIdentities, + plan.RequiredCoverageIdentities) + } + assert.Equal(t, []string{stateDB}, viaState.Scopes[0].RetryRoots) + assert.Equal(t, []string{sessionsDir}, viaSessions.Scopes[0].RetryRoots) + + proofs := base.Scopes[0].PhysicalProofScopes + require.Len(t, proofs, 2) + assert.Equal(t, StoredSourceHintScope{ + Path: stateDB, IncludeVirtualMembers: true, + }, proofs[0]) + assert.Equal(t, StoredSourceHintScope{Path: sessionsDir}, proofs[1]) +} + +func TestHermesReconciliationPlanFlatRootDescendantProvesItself(t *testing.T) { + // No state.db and no sessions/ subdirectory: transcripts live directly + // under the configured root, so the unit has no archive topology. + root := t.TempDir() + requested := filepath.Join(root, "gone.jsonl") + + plan := resolveHermesPlan(t, []string{root}, []string{requested}) + + require.Len(t, plan.Scopes, 1, + "a flat root must resolve a descendant generically, not to nothing") + scope := plan.Scopes[0] + assert.Equal(t, []string{root}, scope.TraversalRoots) + assert.Equal(t, + []StoredSourceHintScope{{Path: requested}}, + scope.PhysicalProofScopes, + "proof stays bounded to the requested transcript") + assert.Empty(t, scope.CoverageIdentities, + "a descendant request cannot cover the configured root") + assert.Equal(t, []string{requested}, scope.RetryRoots) +} + +func TestHermesReconciliationPlanProfileRequestIsolatesSiblings(t *testing.T) { + home := t.TempDir() + container := filepath.Join(home, ".hermes", "profiles") + profileA := filepath.Join(container, "alpha") + profileB := filepath.Join(container, "beta") + _, sessionsA := writeHermesPlanArchive(t, profileA) + writeHermesPlanArchive(t, profileB) + + plan := resolveHermesPlan(t, []string{container}, []string{ + filepath.Join(sessionsA, "one.jsonl"), + }) + + require.Len(t, plan.Scopes, 1) + scope := plan.Scopes[0] + assert.Equal(t, []string{profileA}, scope.TraversalRoots) + for _, proof := range scope.PhysicalProofScopes { + assert.True(t, + hermesPathWithinOrSame(absoluteHermesPath(proof.Path), profileA), + "proof %q must stay inside the requested profile", proof.Path) + } + assert.Empty(t, scope.CoverageIdentities, + "a single profile cannot cover the container identity") + assert.Equal(t, []string{absoluteHermesPath(container)}, + plan.RequiredCoverageIdentities) +} + +// TestHermesReconciliationPlanOverlappingRootsMergeScopeAuthority pins the +// scope-collision merge: with a profiles container and one of its profiles +// both configured, a request can reach the profile's archive through the +// container's profile branch (no coverage) before its own configured-root +// branch (coverage), depending on ordering. The colliding scopes describe +// one unit, so the merge must keep every authority — dropping the later +// coverage identity would leave a required identity permanently uncovered +// and withhold aggregate-member deletion from a pass that requested every +// configured root. +func TestHermesReconciliationPlanOverlappingRootsMergeScopeAuthority( + t *testing.T, +) { + home := t.TempDir() + container := filepath.Join(home, ".hermes", "profiles") + profile := filepath.Join(container, "alpha") + writeHermesPlanArchive(t, profile) + + // Container configured before the profile, requested in the reverse + // order: the profile request resolves through the container first. + plan := resolveHermesPlan(t, + []string{container, profile}, + []string{profile, container}, + ) + + covered := make(map[string]bool) + for _, scope := range plan.Scopes { + for _, identity := range scope.CoverageIdentities { + covered[identity] = true + } + } + require.NotEmpty(t, plan.RequiredCoverageIdentities) + for _, required := range plan.RequiredCoverageIdentities { + assert.True(t, covered[required], + "requesting every configured root must cover %q", required) + } +} + +func TestHermesReconciliationPlanContainerRequestCoversContainer(t *testing.T) { + home := t.TempDir() + container := filepath.Join(home, ".hermes", "profiles") + writeHermesPlanArchive(t, filepath.Join(container, "alpha")) + + plan := resolveHermesPlan(t, []string{container}, []string{container}) + + require.Len(t, plan.Scopes, 1) + scope := plan.Scopes[0] + assert.Equal(t, []string{container}, scope.TraversalRoots) + assert.Equal(t, + []StoredSourceHintScope{{Path: filepath.Clean(container)}}, + scope.PhysicalProofScopes) + assert.Equal(t, []string{absoluteHermesPath(container)}, + scope.CoverageIdentities) +} + +func TestHermesReconciliationPlanRejectsUnrelatedSiblingArchive(t *testing.T) { + parent := t.TempDir() + archive := filepath.Join(parent, "archive") + sibling := filepath.Join(parent, "sibling") + writeHermesPlanArchive(t, archive) + writeHermesPlanArchive(t, sibling) + + plan := resolveHermesPlan(t, []string{archive}, []string{sibling}) + + assert.Empty(t, plan.Scopes, + "a sibling archive outside the configured topology resolves nothing") +} + +// TestDirectoryPlanProvesARelativeRootInItsConfiguredSpelling pins the split +// between the two spellings a configured root carries. Comparison and coverage +// identity absolutize so a request naming the same directory differently still +// matches, but a proof scope is matched as a prefix against stored source +// paths, and discovery wrote those by joining the configured spelling. +// Absolutizing proof leaves every stored source unreachable: the pass pages +// zero ownership rows and still credits coverage. +func TestDirectoryPlanProvesARelativeRootInItsConfiguredSpelling(t *testing.T) { + workingDir := t.TempDir() + t.Chdir(workingDir) + absolute := filepath.Join(workingDir, "archive") + require.NoError(t, os.MkdirAll(absolute, 0o755)) + + plan := resolveDirectoryPlan(t, []string{"archive"}, []string{absolute}) + + require.Len(t, plan.Scopes, 1) + scope := plan.Scopes[0] + assert.Equal(t, + []StoredSourceHintScope{{Path: "archive"}}, + scope.PhysicalProofScopes, + "proof must carry the spelling stored sources were written under") + assert.Equal(t, []string{"archive"}, scope.TraversalRoots) + assert.Equal(t, []string{absolute}, scope.CoverageIdentities, + "coverage identity absolutizes so an absolute request matches it") +} + +// TestDirectoryPlanProvesADescendantInItsGatewaySpelling extends the same +// split to a descendant request. Requests arrive absolute because they come +// from watch and polling roots, so the descendant has to be re-expressed under +// the gateway's configured spelling before it can prove anything. +func TestDirectoryPlanProvesADescendantInItsGatewaySpelling(t *testing.T) { + workingDir := t.TempDir() + t.Chdir(workingDir) + descendant := filepath.Join(workingDir, "archive", "project") + require.NoError(t, os.MkdirAll(descendant, 0o755)) + + plan := resolveDirectoryPlan(t, []string{"archive"}, []string{descendant}) + + require.Len(t, plan.Scopes, 1) + scope := plan.Scopes[0] + assert.Equal(t, + []StoredSourceHintScope{{Path: filepath.Join("archive", "project")}}, + scope.PhysicalProofScopes) + assert.Equal(t, []string{"archive"}, scope.TraversalRoots) + assert.Empty(t, scope.CoverageIdentities) +} + +// TestReconciliationScopeRootsRejectTheWorkingDirectory covers the one +// spelling that cannot be bounded at all: discovery under a root that cleans +// to "." emits bare filenames, so no prefix matches them and the stored-hint +// normalizer discards the scope. Failing closed keeps a pass from reporting +// success over zero proven rows, and keeps the provider reachable instead of +// resolving an empty plan the engine would skip silently. +func TestReconciliationScopeRootsRejectTheWorkingDirectory(t *testing.T) { + for _, root := range []string{".", filepath.Join("foo", "..")} { + t.Run(root, func(t *testing.T) { + base := ProviderBase{ + Def: AgentDef{Type: AgentClaude}, + Config: ProviderConfig{Roots: []string{root}}, + } + _, err := base.ResolveReconciliationScopes( + t.Context(), ReconciliationScopeRequest{Roots: []string{root}}, + ) + require.Error(t, err, + "an unprovable root must fail closed, not resolve no scope") + assert.Contains(t, err.Error(), "working directory") + }) + } +} + +// TestHermesReconciliationPlanRejectsTheWorkingDirectory holds the Hermes +// override to the same contract; its archive proofs are built by cleaning the +// configured root, so it degenerates identically. +func TestHermesReconciliationPlanRejectsTheWorkingDirectory(t *testing.T) { + provider, ok := NewProvider(AgentHermes, ProviderConfig{Roots: []string{"."}}) + require.True(t, ok) + + _, err := provider.ResolveReconciliationScopes( + t.Context(), ReconciliationScopeRequest{Roots: []string{"."}}, + ) + require.Error(t, err) + assert.Contains(t, err.Error(), "working directory") +} + +// TestHermesReconciliationPlanProvesAProfileInItsContainerSpelling extends the +// generic two-spelling rule to the Hermes profiles fan-out. A request inside a +// profile arrives absolute, and matching it against the container has to be +// absolute too, but expandProfilesContainer builds each profile root by +// joining the configured container spelling, so a relative container's sources +// are stored relative. Proving such a profile with an absolute scope pages no +// ownership rows and leaves a removed member active. +func TestHermesReconciliationPlanProvesAProfileInItsContainerSpelling( + t *testing.T, +) { + workingDir := t.TempDir() + t.Chdir(workingDir) + container := filepath.Join(".hermes", "profiles") + archive := filepath.Join(container, "alpha") + writeHermesPlanArchive(t, archive) + absoluteArchive := filepath.Join(workingDir, archive) + + plan := resolveHermesPlan(t, + []string{container}, []string{absoluteArchive}) + + require.Len(t, plan.Scopes, 1) + scope := plan.Scopes[0] + assert.Equal(t, []string{archive}, scope.TraversalRoots, + "traversal must stay in the configured container spelling") + assert.Equal(t, []StoredSourceHintScope{ + {Path: filepath.Join(archive, "state.db"), IncludeVirtualMembers: true}, + {Path: filepath.Join(archive, "sessions")}, + }, scope.PhysicalProofScopes, + "proof must carry the spelling stored sources were written under") + assert.Empty(t, scope.CoverageIdentities, + "one profile cannot cover the container") +} + +// TestHermesReconciliationPlanCanonicalizesProfileCase keeps a request's own +// casing out of the scope it resolves. On Windows "alpha" and "ALPHA" select +// the same physical profile, so carrying the requested spelling into traversal, +// proof, and identity would mint a second coverage identity for one archive and +// let reconciliation persist the request-cased path. +func TestHermesReconciliationPlanCanonicalizesProfileCase(t *testing.T) { + if runtime.GOOS != "windows" { + t.Skip("case-insensitive path selection is Windows behavior") + } + home := t.TempDir() + container := filepath.Join(home, ".hermes", "profiles") + archive := filepath.Join(container, "alpha") + writeHermesPlanArchive(t, archive) + shouted := filepath.Join(container, "ALPHA") + + plan := resolveHermesPlan(t, []string{container}, []string{archive, shouted}) + + require.Len(t, plan.Scopes, 1, + "two spellings of one profile must resolve one scope") + scope := plan.Scopes[0] + assert.Equal(t, []string{archive}, scope.TraversalRoots, + "traversal must use the on-disk profile spelling") + assert.Equal(t, []StoredSourceHintScope{ + {Path: filepath.Join(archive, "state.db"), IncludeVirtualMembers: true}, + {Path: filepath.Join(archive, "sessions")}, + }, scope.PhysicalProofScopes) + assert.Equal(t, []string{archive, shouted}, scope.RetryRoots, + "both request spellings must retry against the one scope") +} + +// TestHermesReconciliationPlanWidensAnUnownedChildToItsContainer covers every +// request the container cannot resolve to an owned profile. Reconstructing a +// root from the request is what four earlier rounds each got wrong, since it +// puts the caller's casing into proof and lets traversal follow a symlink out +// of the container, and resolving nothing loses the removal entirely. The +// container is the nearest scope spelled entirely from configuration, and it +// proves without covering, so a deleted profile's sources page and a live +// sibling's are stat'd and kept. +func TestHermesReconciliationPlanWidensAnUnownedChildToItsContainer(t *testing.T) { + home := t.TempDir() + container := filepath.Join(home, ".hermes", "profiles") + require.NoError(t, os.MkdirAll(container, 0o755)) + outside := filepath.Join(home, "elsewhere") + writeHermesPlanArchive(t, outside) + require.NoError(t, os.WriteFile( + filepath.Join(container, "notes.txt"), []byte("x"), 0o644)) + + requests := map[string]string{ + "removed": filepath.Join(container, "gone"), + "not a directory": filepath.Join(container, "notes.txt"), + } + link := filepath.Join(container, "alpha") + if err := os.Symlink(outside, link); err == nil { + requests["symlink"] = link + } else { + t.Logf("symlink creation unavailable, case skipped: %v", err) + } + + for name, requested := range requests { + t.Run(name, func(t *testing.T) { + plan := resolveHermesPlan(t, []string{container}, []string{requested}) + + require.Len(t, plan.Scopes, 1) + scope := plan.Scopes[0] + assert.Equal(t, []string{container}, scope.TraversalRoots, + "traversal must be the configured container, never the request") + assert.Equal(t, + []StoredSourceHintScope{{Path: container}}, + scope.PhysicalProofScopes) + assert.Empty(t, scope.CoverageIdentities, + "a child request cannot cover the container") + assert.Equal(t, []string{requested}, scope.RetryRoots) + }) + } +} + +// TestHermesReconciliationPlanKeepsContainerCoverageIndependentOfAnUnownedChild +// pins the two container scopes apart. They share an identity, so keying them +// together would let whichever request arrived first decide whether the batch +// credits coverage. +func TestHermesReconciliationPlanKeepsContainerCoverageIndependentOfAnUnownedChild( + t *testing.T, +) { + home := t.TempDir() + container := filepath.Join(home, ".hermes", "profiles") + require.NoError(t, os.MkdirAll(container, 0o755)) + removed := filepath.Join(container, "gone") + + plan := resolveHermesPlan(t, + []string{container}, []string{removed, container}) + + require.Len(t, plan.Scopes, 2) + var covering int + for _, scope := range plan.Scopes { + if len(scope.CoverageIdentities) > 0 { + covering++ + assert.Equal(t, []string{container}, scope.RetryRoots) + } + } + assert.Equal(t, 1, covering, + "the container request still credits coverage regardless of order") +} + +func resolveOpenCodePlan( + t *testing.T, agent AgentType, configured, requested []string, +) ReconciliationScopePlan { + t.Helper() + provider, ok := NewProvider(agent, ProviderConfig{Roots: configured}) + require.True(t, ok) + plan, err := provider.ResolveReconciliationScopes( + t.Context(), ReconciliationScopeRequest{Roots: requested}, + ) + require.NoError(t, err) + return plan +} + +// TestOpenCodePlanWidensAContainerAliasToItsVirtualMembers pins the atomic +// container: any spelling that lands on the database proves the container's +// whole membership. A proof of the bare database path admits no member row, +// and a proof of one member would let a completed pass promote +// container-state trust over siblings it never verified. +func TestOpenCodePlanWidensAContainerAliasToItsVirtualMembers(t *testing.T) { + root := t.TempDir() + dbPath := filepath.Join(root, "opencode.db") + for name, requested := range map[string]string{ + "database": dbPath, + "wal sidecar": dbPath + "-wal", + "shm sidecar": dbPath + "-shm", + "virtual member": OpenCodeSQLiteVirtualPath(dbPath, "ses-1"), + } { + t.Run(name, func(t *testing.T) { + plan := resolveOpenCodePlan( + t, AgentOpenCode, []string{root}, []string{requested}, + ) + require.Len(t, plan.Scopes, 1) + scope := plan.Scopes[0] + assert.Equal(t, []string{root}, scope.TraversalRoots, + "traversal must come from the configured gateway") + assert.Equal(t, + []StoredSourceHintScope{{ + Path: dbPath, IncludeVirtualMembers: true, + }}, + scope.PhysicalProofScopes, + "every container alias proves the container's whole membership") + assert.Empty(t, scope.CoverageIdentities, + "a container request cannot cover the configured root") + assert.Equal(t, []string{requested}, scope.RetryRoots) + }) + } +} + +// TestOpenCodePlanWidensACaseVariantContainerAliasOnWindows pins the widening +// against respelled aliases: stored-path admission is case-insensitive on +// Windows, so a case variant that escaped widening would prove exactly one +// member and reopen the partial-membership trust hole. +func TestOpenCodePlanWidensACaseVariantContainerAliasOnWindows(t *testing.T) { + if runtime.GOOS != "windows" { + t.Skip("case-insensitive path selection is Windows behavior") + } + root := t.TempDir() + dbPath := filepath.Join(root, "opencode.db") + variant := filepath.Join(root, "OPENCODE.DB") + for name, requested := range map[string]string{ + "database": variant, + "wal sidecar": variant + "-wal", + "virtual member": OpenCodeSQLiteVirtualPath(variant, "ses-1"), + } { + t.Run(name, func(t *testing.T) { + plan := resolveOpenCodePlan( + t, AgentOpenCode, []string{root}, []string{requested}, + ) + require.Len(t, plan.Scopes, 1) + assert.Equal(t, + []StoredSourceHintScope{{ + Path: dbPath, IncludeVirtualMembers: true, + }}, + plan.Scopes[0].PhysicalProofScopes, + "a respelled alias still proves the whole membership") + assert.Equal(t, []string{requested}, plan.Scopes[0].RetryRoots) + }) + } +} + +func TestOpenCodePlanCoalescesContainerAliasesIntoOneScope(t *testing.T) { + root := t.TempDir() + dbPath := filepath.Join(root, "opencode.db") + aliases := []string{ + OpenCodeSQLiteVirtualPath(dbPath, "ses-a"), + dbPath + "-wal", + OpenCodeSQLiteVirtualPath(dbPath, "ses-b"), + dbPath, + } + plan := resolveOpenCodePlan(t, AgentOpenCode, []string{root}, aliases) + + require.Len(t, plan.Scopes, 1, + "aliases of one container are one atomic scope") + assert.ElementsMatch(t, aliases, plan.Scopes[0].RetryRoots, + "every caller spelling stays a retry identity") +} + +func TestOpenCodePlanKeepsAStorageDescendantProofExact(t *testing.T) { + root := t.TempDir() + storage := filepath.Join(root, "storage", "session") + plan := resolveOpenCodePlan( + t, AgentOpenCode, []string{root}, []string{storage}, + ) + + require.Len(t, plan.Scopes, 1) + assert.Equal(t, + []StoredSourceHintScope{{Path: storage}}, + plan.Scopes[0].PhysicalProofScopes, + "a plain directory descendant owns no virtual members") +} + +func TestOpenCodeFamilyPlansWidenTheirOwnContainers(t *testing.T) { + for _, agent := range []AgentType{ + AgentKilo, AgentMiMoCode, AgentIcodemate, + } { + t.Run(string(agent), func(t *testing.T) { + root := t.TempDir() + dbPath := filepath.Join( + root, openCodeProviderSpecForAgent(agent).dbName, + ) + plan := resolveOpenCodePlan( + t, agent, []string{root}, + []string{OpenCodeSQLiteVirtualPath(dbPath, "ses-1")}, + ) + require.Len(t, plan.Scopes, 1) + assert.Equal(t, + []StoredSourceHintScope{{ + Path: dbPath, IncludeVirtualMembers: true, + }}, + plan.Scopes[0].PhysicalProofScopes) + }) + } +} + +// TestOpenCodePlanProvesAContainerInItsConfiguredSpelling extends the +// two-spelling split to the widened container: comparison absolutizes so the +// absolute request finds the relative root's database, and proof re-expresses +// the container under the configured spelling stored sources carry. +func TestOpenCodePlanProvesAContainerInItsConfiguredSpelling(t *testing.T) { + workingDir := t.TempDir() + t.Chdir(workingDir) + require.NoError(t, os.MkdirAll(filepath.Join(workingDir, "ocroot"), 0o755)) + absoluteDB := filepath.Join(workingDir, "ocroot", "opencode.db") + + plan := resolveOpenCodePlan( + t, AgentOpenCode, []string{"ocroot"}, []string{absoluteDB}, + ) + + require.Len(t, plan.Scopes, 1) + assert.Equal(t, + []StoredSourceHintScope{{ + Path: filepath.Join("ocroot", "opencode.db"), + IncludeVirtualMembers: true, + }}, + plan.Scopes[0].PhysicalProofScopes, + "proof must carry the spelling stored sources were written under") + assert.Equal(t, []string{"ocroot"}, plan.Scopes[0].TraversalRoots) +} + +// TestMultiSessionPlanWidensContainerAliasesToVirtualMembership pins the +// container topology for the shared multi-session base (Zed here, standing in +// for Shelley, Trae, Aider, Visual Studio Copilot, and Omnigent): a request +// naming the physical container, one virtual member, or a WAL sidecar widens +// to the container's whole virtual membership. The generic descendant proof +// would name only the bare path, admitting no "#" source +// and paging no member row. The container deliberately does not exist on +// disk: a deleted container must still resolve so its members stay +// reclaimable. +func TestMultiSessionPlanWidensContainerAliasesToVirtualMembership( + t *testing.T, +) { + root := t.TempDir() + container := filepath.Join(root, "threads", "threads.db") + provider, ok := NewProvider(AgentZed, ProviderConfig{Roots: []string{root}}) + require.True(t, ok) + + for _, requested := range []string{ + container, + container + "#thread-1", + container + "-wal", + } { + plan, err := provider.ResolveReconciliationScopes( + t.Context(), ReconciliationScopeRequest{Roots: []string{requested}}, + ) + require.NoError(t, err, requested) + require.Len(t, plan.Scopes, 1, requested) + scope := plan.Scopes[0] + assert.Equal(t, []string{root}, scope.TraversalRoots, requested) + assert.Equal(t, + []StoredSourceHintScope{{ + Path: container, IncludeVirtualMembers: true, + }}, + scope.PhysicalProofScopes, + "request %q must prove the container's whole membership", requested) + assert.Empty(t, scope.CoverageIdentities, + "a container request cannot cover the configured root") + } + + unrelated, err := provider.ResolveReconciliationScopes( + t.Context(), ReconciliationScopeRequest{ + Roots: []string{filepath.Join(root, "settings.json")}, + }, + ) + require.NoError(t, err) + require.Len(t, unrelated.Scopes, 1) + assert.Equal(t, + []StoredSourceHintScope{{Path: filepath.Join(root, "settings.json")}}, + unrelated.Scopes[0].PhysicalProofScopes, + "a non-container descendant keeps the exact generic proof") +} diff --git a/internal/parser/source_set.go b/internal/parser/source_set.go index 6879936df..84870bc1a 100644 --- a/internal/parser/source_set.go +++ b/internal/parser/source_set.go @@ -98,6 +98,36 @@ func (p *SourceSetProvider) SourcesForChangedPath( return p.sources.SourcesForChangedPath(ctx, req) } +// reconciliationContainerTopologyProvider is implemented by source sets whose +// members are virtual children of a physical container. The provider-level +// scope resolver widens a request naming the container, a sidecar, or one +// member to the container's whole membership; without it a container-path +// request would prove only the bare path, which admits no member source and +// pages no member row — a successful no-op over sessions the caller asked +// about. +type reconciliationContainerTopologyProvider interface { + ReconciliationContainer(requested string) (string, bool) +} + +// ResolveReconciliationScopes applies the source set's container topology +// when it declares one, and otherwise inherits the generic directory plan. +func (p *SourceSetProvider) ResolveReconciliationScopes( + ctx context.Context, req ReconciliationScopeRequest, +) (ReconciliationScopePlan, error) { + topology, ok := p.sources.(reconciliationContainerTopologyProvider) + if !ok { + return p.ProviderBase.ResolveReconciliationScopes(ctx, req) + } + if err := ValidateReconciliationScopeRoots( + p.Def.Type, p.Config.Roots, req.Roots, + ); err != nil { + return ReconciliationScopePlan{}, err + } + return containerAwareReconciliationScopePlan( + p.Config.Roots, req.Roots, topology.ReconciliationContainer, + ), nil +} + func (p *SourceSetProvider) StoredSourceHintScopes( req ChangedPathRequest, ) []StoredSourceHintScope { diff --git a/internal/sync/engine.go b/internal/sync/engine.go index d281c6ee4..c245594cf 100644 --- a/internal/sync/engine.go +++ b/internal/sync/engine.go @@ -1103,11 +1103,35 @@ func (e *Engine) classifyProviderChangedPath( !changedPathWithinAnyRoot(path, watchRoots) { continue } + // Capture the shared container's state before any watermark-only + // listing below: the provider-side freshness merge may trust the + // caller's stored authority only while the container provably has + // not changed across the listing window, and the pass-level capture + // guard does not exist yet at classification time. + watermarkContainer := openCodeContainerPathForChangedPathEvent( + agentType, roots, path, + ) + var watermarkPreState parser.SQLiteContainerState + watermarkPreStateOK := false + if watermarkContainer != "" { + watermarkPreState, watermarkPreStateOK = + statSQLiteContainerState(watermarkContainer) + } for _, watchRoot := range watchRoots { request := parser.ChangedPathRequest{ Path: path, EventKind: eventKind, WatchRoot: watchRoot, + // Shared-container providers merge the bounded session-row + // listing against the paged stored freshness below and emit + // only members whose watermark advanced, instead of a + // whole-container child digest scan. + AllowWatermarkOnlySources: true, + } + if watermarkContainer != "" && watermarkPreStateOK && + !e.forceParse && e.pathRewriter == nil { + request.StoredMemberFreshnessPage = + e.storedMemberFreshnessPager(watermarkContainer) } if provider.Capabilities().Source.StoredSourceHints == parser.CapabilitySupported { if resolver, ok := provider.(parser.StoredSourceHintScopeProvider); ok { @@ -1136,6 +1160,18 @@ func (e *Engine) classifyProviderChangedPath( ctx, request, ) + if err == nil && request.StoredMemberFreshnessPage != nil { + if post, ok := statSQLiteContainerState(watermarkContainer); !ok || + post != watermarkPreState { + // The container changed while the merged listing ran: a + // commit inside that window can advance a session past + // its listed watermark, so the merge may have dropped a + // changed member. Re-list without stored authority and + // let the per-file gates decide. + request.StoredMemberFreshnessPage = nil + sources, err = provider.SourcesForChangedPath(ctx, request) + } + } if err != nil { if !errors.Is(err, parser.ErrUnsupportedProviderFeature) { classificationErr = errors.Join( @@ -3275,10 +3311,10 @@ func (e *Engine) reconcileWatchRoots( } // ReconcileProviderRoots runs the bounded scheduled pass for one provider. It -// bypasses the cross-provider expansion in logicalRootsForWatchRoots so a -// shallow-watched agent never enumerates or tombstones another agent's sessions -// under an overlapping root. Providers whose scheduled discovery is -// non-authoritative leave deletion proof to the archive audit. +// resolves scopes against that provider's topology only, so a shallow-watched +// agent never enumerates or tombstones another agent's sessions under an +// overlapping root. Providers whose scheduled discovery is non-authoritative +// leave deletion proof to the archive audit. func (e *Engine) ReconcileProviderRoots( ctx context.Context, agent parser.AgentType, roots []string, ) error { @@ -3396,14 +3432,15 @@ func (e *Engine) reconcileScopedWatchRoots( func (e *Engine) reconcileScopedWatchRootsLocked( ctx context.Context, agent parser.AgentType, roots []string, full, force bool, ) (SyncStats, int, passEpilogueEligibility, error) { - var logicalRoots []string - var excludedRemoteRoots int - if agent == "" { - logicalRoots, excludedRemoteRoots = e.localReconciliationRoots(roots, full) - } else { - logicalRoots, excludedRemoteRoots = e.agentReconciliationRoots(agent, roots) - } - if !full && len(roots) > 0 && len(logicalRoots) == 0 && excludedRemoteRoots > 0 { + fullCoverage := full || (agent == "" && len(roots) == 0) + plans, excludedRemoteRoots := e.resolveReconciliationPlans( + ctx, agent, roots, full, fullCoverage, + ) + if !fullCoverage && !reconciliationPlansNeedPass(plans) { + // No provider resolved any scope for the request: every root was + // blank, remote, or unrelated to every configured topology. Complete + // as a bounded no-op before any spool allocation, preserving the + // remote-root accounting. e.setLastReconciliationResult(ReconciliationResult{ Complete: true, Metrics: ReconciliationMetrics{ExcludedRemoteRoots: excludedRemoteRoots}, @@ -3411,7 +3448,7 @@ func (e *Engine) reconcileScopedWatchRootsLocked( return SyncStats{}, 0, passEpilogueEligibility{}, nil } stats, metrics, tombstoned, eligibility, err := e.reconcileWatchRootsStreamedLocked( - ctx, agent, logicalRoots, full, force, + ctx, plans, fullCoverage, force, ) metrics.ExcludedRemoteRoots = excludedRemoteRoots complete := err == nil && ctx.Err() == nil && !stats.Aborted && @@ -3444,12 +3481,133 @@ func (e *Engine) ReconciliationRootsForAgent(agent string) []string { return append([]string(nil), e.agentDirs[parser.AgentType(agent)]...) } +// providerReconciliationPlan pairs one authoritative provider with the scope +// plan it resolved for the caller's request. A resolution failure is carried +// rather than returned so sibling providers still run, and the failed +// provider reports the caller's own roots for retry. +type providerReconciliationPlan struct { + agent parser.AgentType + plan parser.ReconciliationScopePlan + requestRoots []string + err error +} + +// resolveReconciliationPlans asks each in-scope authoritative provider to map +// the request onto its own topology. Providers own traversal, proof, +// coverage, and retry authority; the engine only selects which providers to +// ask and preserves the historical remote-root accounting. +func (e *Engine) resolveReconciliationPlans( + ctx context.Context, + agentFilter parser.AgentType, + roots []string, + full, fullCoverage bool, +) ([]providerReconciliationPlan, int) { + agents := make([]parser.AgentType, 0, len(e.providerFactories)) + for agent := range e.providerFactories { + agents = append(agents, agent) + } + slices.SortFunc(agents, func(a, b parser.AgentType) int { + return strings.Compare(string(a), string(b)) + }) + var plans []providerReconciliationPlan + for _, agent := range agents { + if e.providerMigrationModes[agent] != parser.ProviderMigrationProviderAuthoritative { + continue + } + if agentFilter != "" && agent != agentFilter { + continue + } + if len(e.agentDirs[agent]) == 0 { + continue + } + factory := e.providerFactories[agent] + if factory == nil { + continue + } + requestRoots := roots + if fullCoverage { + // A full authoritative request covers each provider's complete + // configured scope; a partial request lets each provider resolve + // the same exact caller roots. + requestRoots = e.agentDirs[agent] + } + filtered := make([]string, 0, len(requestRoots)) + for _, root := range requestRoots { + if strings.TrimSpace(root) == "" || isRemoteReconciliationRoot(root) { + continue + } + filtered = append(filtered, root) + } + if len(filtered) == 0 { + continue + } + provider := factory.NewProvider(parser.ProviderConfig{ + Roots: e.agentDirs[agent], Machine: e.machine, + PathRewriter: e.pathRewriter, + }) + plan, err := provider.ResolveReconciliationScopes( + ctx, parser.ReconciliationScopeRequest{Roots: filtered}, + ) + plans = append(plans, providerReconciliationPlan{ + agent: agent, plan: plan, requestRoots: filtered, err: err, + }) + } + return plans, e.excludedRemoteReconciliationRoots(agentFilter, roots, full) +} + +// excludedRemoteReconciliationRoots preserves the historical ExcludedRemoteRoots +// semantics: agent-scoped requests count every remote occurrence, unscoped +// partial requests count unique remote roots, and a full recovery counts +// unique remote configured roots. +func (e *Engine) excludedRemoteReconciliationRoots( + agentFilter parser.AgentType, roots []string, full bool, +) int { + if agentFilter != "" { + remote := 0 + for _, root := range roots { + if isRemoteReconciliationRoot(root) { + remote++ + } + } + return remote + } + requested := roots + if full { + requested = nil + for _, dirs := range e.agentDirs { + requested = append(requested, dirs...) + } + } + remote := make(map[string]struct{}) + for _, root := range requested { + if isRemoteReconciliationRoot(root) { + remote[root] = struct{}{} + } + } + return len(remote) +} + +// reconciliationPlansNeedPass reports whether any provider resolved a scope +// or failed to resolve; either requires the streamed pass to run so work is +// performed or the failure is accounted with retry roots. +func reconciliationPlansNeedPass(plans []providerReconciliationPlan) bool { + for _, plan := range plans { + if plan.err != nil || len(plan.plan.Scopes) > 0 { + return true + } + } + return false +} + // reconcileWatchRootsStreamedLocked runs one streamed reconciliation pass. // The caller must hold syncMu: reconcileScopedWatchRoots takes it per pass, // and ReconcileProviderRootsGrouped holds it across every group plus the // shared epilogue so no other pass can interleave with a pending epilogue. +// It accepts resolved plans rather than raw roots so no caller can bypass +// provider scope resolution with a string slice. func (e *Engine) reconcileWatchRootsStreamedLocked( - ctx context.Context, agent parser.AgentType, roots []string, full, force bool, + ctx context.Context, plans []providerReconciliationPlan, + fullCoverage, force bool, ) ( stats SyncStats, metrics ReconciliationMetrics, tombstoned int, eligibility passEpilogueEligibility, retErr error, @@ -3498,16 +3656,10 @@ func (e *Engine) reconcileWatchRootsStreamedLocked( } }() - scope := newRootSyncScope(roots) - if full { - scope = nil - } else if scope != nil { - scope.agent = agent - } - preContainerStates := e.captureSQLiteContainerStatesForAgent(agent, roots) + preContainerStates := e.capturePlannedSQLiteContainerStates(plans, fullCoverage) providers, completedScopes, failedRoots, failures, discoveryErr, err := e.streamReconciliationCandidates( - ctx, scope, spool, + ctx, plans, spool, preContainerStates, ) stats.providerFailures = failures if err != nil { @@ -3528,12 +3680,12 @@ func (e *Engine) reconcileWatchRootsStreamedLocked( defer func() { e.finishSQLiteContainerPass( retErr != nil || stats.Aborted || stats.Failed > 0 || stats.providerFailures > 0, - scope == nil, + fullCoverage, ) }() var verifiedPass uint64 - if scope == nil && e.pathRewriter == nil { + if fullCoverage && e.pathRewriter == nil { verifiedPass = e.beginVerifiedSourcePass() defer func() { e.finishVerifiedSourcePass( @@ -3687,8 +3839,8 @@ func (e *Engine) reconcileWatchRootsStreamedLocked( } if retErr == nil && ctx.Err() == nil && !stats.Aborted && stats.Failed == 0 && stats.providerFailures == 0 { - tombstoned, retErr = e.tombstoneMissingWatchSourcesForAgentLocked( - ctx, roots, agent, spool, + tombstoned, retErr = e.tombstoneMissingWatchSourceScopesLocked( + ctx, completedScopes, spool, ) } else if canTombstoneCompletedScopes && ctx.Err() == nil && !stats.Aborted && stats.Failed == 0 { @@ -3718,27 +3870,23 @@ func (e *Engine) tombstoneCompletedReconciliationScopesLocked( spool reconciliationSpoolStore, incomplete *incompleteReconciliationError, ) (deleted int, retErr error) { - for _, scope := range incomplete.completed { - count, err := e.tombstoneMissingWatchSourcesForAgentLocked( - ctx, scope.roots, scope.agent, spool, - ) - deleted += count - if err == nil { - continue - } - retryRoots := append([]string(nil), incomplete.roots...) - for _, completed := range incomplete.completed { - retryRoots = append(retryRoots, completed.roots...) - } - slices.Sort(retryRoots) - retryRoots = slices.Compact(retryRoots) - return deleted, &incompleteReconciliationError{ - failures: incomplete.failures, - roots: retryRoots, - cause: errors.Join(incomplete, err), - } + deleted, err := e.tombstoneMissingWatchSourceScopesLocked( + ctx, incomplete.completed, spool, + ) + if err == nil { + return deleted, nil + } + retryRoots := append([]string(nil), incomplete.roots...) + for _, completed := range incomplete.completed { + retryRoots = append(retryRoots, completed.roots...) + } + slices.Sort(retryRoots) + retryRoots = slices.Compact(retryRoots) + return deleted, &incompleteReconciliationError{ + failures: incomplete.failures, + roots: retryRoots, + cause: errors.Join(incomplete, err), } - return deleted, nil } func (e *Engine) baselineReconciliationCandidates( @@ -3784,8 +3932,9 @@ func eligibleReconciliationBaselines( func (e *Engine) streamReconciliationCandidates( ctx context.Context, - scope *rootSyncScope, + plans []providerReconciliationPlan, spool reconciliationSpoolStore, + preContainerStates map[string]parser.SQLiteContainerState, ) ( map[parser.AgentType]parser.Provider, []reconciliationProviderScope, @@ -3799,85 +3948,154 @@ func (e *Engine) streamReconciliationCandidates( var failedRoots []string var failures int var discoveryErr error - agents := make([]parser.AgentType, 0, len(e.providerFactories)) - for agent := range e.providerFactories { - agents = append(agents, agent) - } - slices.SortFunc(agents, func(a, b parser.AgentType) int { - return strings.Compare(string(a), string(b)) - }) - for _, agent := range agents { - if e.providerMigrationModes[agent] != parser.ProviderMigrationProviderAuthoritative { - continue - } - if !scope.matchesAgent(agent) { + // Trusted containers stream the bounded watermark listing here for the + // same reason full discovery lists them that way: every candidate they + // spool will gate-skip, so the child digest would be archive-sized work + // nothing reads. The predicate is keyed to this pass's pre-discovery + // captures; a container that changes mid-stream fails its recapture + // check and its candidates resolve full fingerprints instead. + containerTrusted := e.sqliteContainerTrustedForDiscovery(preContainerStates) + for _, plan := range plans { + agent := plan.agent + if plan.err != nil { + failures++ + failedRoots = append(failedRoots, plan.requestRoots...) + log.Printf("%s provider reconciliation scopes: %v", agent, plan.err) + discoveryErr = errors.Join(discoveryErr, fmt.Errorf( + "%s provider reconciliation scopes: %w", agent, plan.err, + )) continue } - roots := slices.DeleteFunc(append([]string(nil), e.agentDirs[agent]...), func(root string) bool { - return isRemoteReconciliationRoot(root) || !scope.includes(root) - }) - if len(roots) == 0 { + if len(plan.plan.Scopes) == 0 { continue } factory := e.providerFactories[agent] if factory == nil { continue } + // traversalRoots is the ordered union across the plan's scopes: the + // rehydration provider and candidate preference ranking see the same + // root list one whole-provider discovery would have used. + var traversalRoots []string + var planRetryRoots []string + for _, scope := range plan.plan.Scopes { + for _, root := range scope.TraversalRoots { + if !slices.Contains(traversalRoots, root) { + traversalRoots = append(traversalRoots, root) + } + } + planRetryRoots = append(planRetryRoots, scope.RetryRoots...) + } provider := factory.NewProvider(parser.ProviderConfig{ - Roots: roots, Machine: e.machine, PathRewriter: e.pathRewriter, + Roots: traversalRoots, Machine: e.machine, PathRewriter: e.pathRewriter, + SQLiteContainerUnchangedSinceTrust: containerTrusted, }) providers[agent] = provider if provider.Capabilities().Source.StreamingDiscovery != parser.CapabilitySupported { failures++ - failedRoots = append(failedRoots, roots...) + failedRoots = append(failedRoots, planRetryRoots...) log.Printf("%s provider discovery: streaming discovery unsupported", agent) continue } - discoverer, ok := provider.(parser.StreamingDiscoverer) - if !ok { - failures++ - failedRoots = append(failedRoots, roots...) - continue - } watchRoots, err := parser.ResolveWatchRoots(ctx, provider) if err != nil { failures++ - failedRoots = append(failedRoots, roots...) + failedRoots = append(failedRoots, planRetryRoots...) discoveryErr = errors.Join(discoveryErr, fmt.Errorf( "%s provider watch roots: %w", agent, err, )) continue } - var spoolErr error - err = discoverer.DiscoverEach(ctx, func(source parser.SourceRef) error { - candidate, ok := e.reconciliationCandidate(provider, source, roots, watchRoots) + for _, group := range reconciliationTraversalGroups(plan.plan.Scopes) { + var groupRetryRoots []string + for _, scope := range group.scopes { + groupRetryRoots = append(groupRetryRoots, scope.RetryRoots...) + } + scopeProvider := factory.NewProvider(parser.ProviderConfig{ + Roots: group.roots, Machine: e.machine, + PathRewriter: e.pathRewriter, + SQLiteContainerUnchangedSinceTrust: containerTrusted, + }) + discoverer, ok := scopeProvider.(parser.StreamingDiscoverer) if !ok { - return nil + failures++ + failedRoots = append(failedRoots, groupRetryRoots...) + continue } - spoolErr = spool.Add(ctx, candidate) - return spoolErr - }) - if err != nil { - if spoolErr != nil { - return providers, completedScopes, - failedRoots, failures, discoveryErr, spoolErr + groupProofs := make([][]db.StoredSourcePathHintScope, len(group.scopes)) + for i, scope := range group.scopes { + groupProofs[i] = storedSourceDBHintScopes(scope.PhysicalProofScopes) } - if ctx.Err() != nil { - return providers, completedScopes, - failedRoots, failures, discoveryErr, ctx.Err() + var spoolErr error + err = discoverer.DiscoverEach(ctx, func(source parser.SourceRef) error { + candidate, ok := e.reconciliationCandidate( + provider, source, traversalRoots, watchRoots, + ) + if !ok { + return nil + } + admitted := false + for _, proofScopes := range groupProofs { + if db.StoredSourcePathHintScopesContain(candidate.Path, proofScopes) { + admitted = true + break + } + } + if !admitted { + if reconciliationPathWithinTraversal( + candidate.Path, group.roots, + ) { + // An unrequested sibling inside the traversal gateway: + // dropping before the spool is baseline-safe because + // ReplaceActiveSessionSourceBaselines diffs only + // candidates present in the page. + return nil + } + // Outside both traversal and every proof is a provider + // contract violation; fail the group closed so its + // sessions are preserved and its own retry roots are + // returned. + return fmt.Errorf( + "source %s outside traversal and proof scope", + candidate.Path, + ) + } + spoolErr = spool.Add(ctx, candidate) + return spoolErr + }) + if err != nil { + if spoolErr != nil { + return providers, completedScopes, + failedRoots, failures, discoveryErr, spoolErr + } + if ctx.Err() != nil { + return providers, completedScopes, + failedRoots, failures, discoveryErr, ctx.Err() + } + log.Printf("%s provider streaming discovery: %v", agent, err) + failures++ + failedRoots = append(failedRoots, groupRetryRoots...) + discoveryErr = errors.Join(discoveryErr, fmt.Errorf( + "%s provider streaming discovery: %w", agent, err, + )) + continue + } + for _, scope := range group.scopes { + completedScopes = append(completedScopes, reconciliationProviderScope{ + agent: agent, + roots: append([]string(nil), scope.RetryRoots...), + proofScopes: append( + []parser.StoredSourceHintScope(nil), scope.PhysicalProofScopes..., + ), + coverageIdentities: append( + []string(nil), scope.CoverageIdentities..., + ), + requiredCoverageIdentities: append( + []string(nil), plan.plan.RequiredCoverageIdentities..., + ), + }) } - log.Printf("%s provider streaming discovery: %v", agent, err) - failures++ - failedRoots = append(failedRoots, roots...) - discoveryErr = errors.Join(discoveryErr, fmt.Errorf( - "%s provider streaming discovery: %w", agent, err, - )) - continue } - completedScopes = append(completedScopes, reconciliationProviderScope{ - agent: agent, - roots: append([]string(nil), roots...), - }) } slices.Sort(failedRoots) failedRoots = slices.Compact(failedRoots) @@ -3885,9 +4103,67 @@ func (e *Engine) streamReconciliationCandidates( failedRoots, failures, discoveryErr, nil } +// reconciliationTraversalGroup is one shared discovery walk: every scope in +// the group declares the identical traversal-root set, so a single stream +// serves them all, with each scope keeping its own proof and coverage +// authority. +type reconciliationTraversalGroup struct { + roots []string + scopes []parser.ReconciliationScope +} + +// reconciliationTraversalGroups clusters one plan's scopes by their exact +// traversal-root sets. A request naming N descendants under one configured +// gateway resolves N scopes that all traverse that gateway; without grouping +// the pass would walk the gateway N times to admit each proof, and the walk +// is the archive-scale part. A group failure fails every member scope, which +// matches the shared traversal: the same walk would have failed each of them +// individually. +func reconciliationTraversalGroups( + scopes []parser.ReconciliationScope, +) []reconciliationTraversalGroup { + var groups []reconciliationTraversalGroup + for _, scope := range scopes { + matched := false + for i := range groups { + if slices.Equal(groups[i].roots, scope.TraversalRoots) { + groups[i].scopes = append(groups[i].scopes, scope) + matched = true + break + } + } + if !matched { + groups = append(groups, reconciliationTraversalGroup{ + roots: scope.TraversalRoots, + scopes: []parser.ReconciliationScope{scope}, + }) + } + } + return groups +} + +// reconciliationPathWithinTraversal reports whether a discovered candidate +// lies inside one of the scope's traversal gateways, resolving provider +// virtual member syntax to the physical container first. +func reconciliationPathWithinTraversal(path string, traversalRoots []string) bool { + cleaned := cleanRootPath(validatedProviderSourceStatPath(path)) + for _, root := range traversalRoots { + if samePathOrDescendant(cleaned, cleanRootPath(root)) { + return true + } + } + return false +} + +// reconciliationProviderScope is the completed-scope record: the caller's own +// retry roots plus the provider-issued proof, coverage, and required-coverage +// authorities that tombstoning consumes. type reconciliationProviderScope struct { - agent parser.AgentType - roots []string + agent parser.AgentType + roots []string + proofScopes []parser.StoredSourceHintScope + coverageIdentities []string + requiredCoverageIdentities []string } type incompleteReconciliationError struct { @@ -4143,13 +4419,57 @@ func sameReconciliationSourcePath(left, right string) bool { canonicalReconciliationSourceIdentity(right) } +// reconciliationMemberRelocated reports whether the provider still resolves +// the session anywhere in its full configured scope. A scoped pass proves a +// member gone from its own container, but the same logical member may have +// moved to another configured root the pass never streamed; a deletion +// claimed here would outlive the move until that root happens to sync. The +// lookup passes only the session identity — a stored-path probe could +// resolve the stale spelling without verifying the row exists. +func reconciliationMemberRelocated( + ctx context.Context, provider parser.Provider, fullSessionID string, +) (bool, error) { + if provider == nil { + // Without a provider the move cannot be ruled out; report the member + // as possibly relocated so deletion is withheld. + return true, nil + } + _, found, err := provider.FindSource(ctx, parser.FindSourceRequest{ + FullSessionID: fullSessionID, + }) + if err != nil { + return false, fmt.Errorf( + "resolve possibly relocated member %s: %w", fullSessionID, err, + ) + } + return found, nil +} + +// reconciliationProofCoversContainerMembership reports whether one completed +// scope's proof claims the whole virtual membership of the container at +// physicalPath. Such a scope streamed and admitted exactly that container's +// members, so a member row absent from the spool is provably gone from the +// container even though the pass covered no configured root in full. +func reconciliationProofCoversContainerMembership( + proofs []parser.StoredSourceHintScope, physicalPath string, +) bool { + for _, proof := range proofs { + if proof.IncludeVirtualMembers && + sameReconciliationSourcePath(proof.Path, physicalPath) { + return true + } + } + return false +} + // aggregateOwnedMemberGone reports whether an ownership row whose stored // FilePath is a still-present multi-member container has lost its member. // Container discovery records the container path itself for every member, // so a removed member never trips the missing-path stat; compare the row // against the streamed pass's membership instead. The check requires a -// spool and full provider-root coverage — absence from a partial stream -// proves nothing — and providers opt in via +// spool and member authority — full provider-root coverage, or a completed +// scope whose proof spans the row's whole container membership; absence +// from any narrower stream proves nothing — and providers opt in via // parser.ReconciliationAggregateMemberResolver. func aggregateOwnedMemberGone( ctx context.Context, @@ -4157,9 +4477,9 @@ func aggregateOwnedMemberGone( provider parser.Provider, agent parser.AgentType, ownership db.SessionSourceOwnership, - allProviderRootsCovered bool, + memberAuthority bool, ) (bool, error) { - if spool == nil || provider == nil || !allProviderRootsCovered { + if spool == nil || provider == nil || !memberAuthority { return false, nil } resolver, ok := provider.(parser.ReconciliationAggregateMemberResolver) @@ -4226,20 +4546,68 @@ func mergeReconciliationSyncStats(dst *SyncStats, src SyncStats) { dst.Aborted = dst.Aborted || src.Aborted } +// tombstoneMissingWatchSourcesLocked adapts a raw changed-path root list to +// the typed scope authority by resolving it through each provider's own +// topology, exactly as a reconciliation pass would. func (e *Engine) tombstoneMissingWatchSourcesLocked( ctx context.Context, roots []string, spool reconciliationSpoolStore, ) (deleted int, retErr error) { - return e.tombstoneMissingWatchSourcesForAgentLocked( - ctx, roots, "", spool, - ) + plans, _ := e.resolveReconciliationPlans(ctx, "", roots, false, false) + var scopes []reconciliationProviderScope + for _, plan := range plans { + if plan.err != nil { + return 0, fmt.Errorf( + "%s provider reconciliation scopes: %w", plan.agent, plan.err, + ) + } + for _, scope := range plan.plan.Scopes { + scopes = append(scopes, reconciliationProviderScope{ + agent: plan.agent, + roots: scope.RetryRoots, + proofScopes: scope.PhysicalProofScopes, + coverageIdentities: scope.CoverageIdentities, + requiredCoverageIdentities: plan.plan.RequiredCoverageIdentities, + }) + } + } + return e.tombstoneMissingWatchSourceScopesLocked(ctx, scopes, spool) +} + +// reconciliationCoverageComplete reports whether the scopes completed for one +// provider cover every coverage identity its full configured scope requires. +// This is the provider-issued replacement for recomputing root geometry: a +// remote or unresolved configured root keeps its required identity uncovered, +// so full-coverage deletion authority stays unreachable. +func reconciliationCoverageComplete(scopes []reconciliationProviderScope) bool { + covered := make(map[string]struct{}) + var required []string + for _, scope := range scopes { + if len(scope.requiredCoverageIdentities) > 0 { + required = scope.requiredCoverageIdentities + } + for _, identity := range scope.coverageIdentities { + covered[identity] = struct{}{} + } + } + if len(required) == 0 { + return false + } + for _, identity := range required { + if _, ok := covered[identity]; !ok { + return false + } + } + return true } -func (e *Engine) tombstoneMissingWatchSourcesForAgentLocked( +// tombstoneMissingWatchSourceScopesLocked pages ownership rows inside each +// completed scope's provider-issued physical proof and tombstones rows whose +// sources are provably gone. +func (e *Engine) tombstoneMissingWatchSourceScopesLocked( ctx context.Context, - roots []string, - agentFilter parser.AgentType, + scopes []reconciliationProviderScope, spool reconciliationSpoolStore, ) (deleted int, retErr error) { if e.pathRewriter != nil { @@ -4248,33 +4616,31 @@ func (e *Engine) tombstoneMissingWatchSourcesForAgentLocked( // provider lookup cannot authoritatively prove source loss. return 0, nil } - for agent, dirs := range e.agentDirs { - if agentFilter != "" && agent != agentFilter { - continue + var agents []parser.AgentType + scopesByAgent := make(map[parser.AgentType][]reconciliationProviderScope) + for _, scope := range scopes { + if _, ok := scopesByAgent[scope.agent]; !ok { + agents = append(agents, scope.agent) } + scopesByAgent[scope.agent] = append(scopesByAgent[scope.agent], scope) + } + for _, agent := range agents { + agentScopes := scopesByAgent[agent] var provider parser.Provider var replacementIndex reconciliationSpoolStore ownsReplacementIndex := false - allProviderRootsCovered := reconciliationCoversConfiguredRoots(roots, dirs) + allProviderRootsCovered := reconciliationCoverageComplete(agentScopes) if factory := e.providerFactories[agent]; factory != nil { provider = factory.NewProvider(parser.ProviderConfig{ Roots: e.agentDirs[agent], Machine: e.machine, PathRewriter: e.pathRewriter, }) } - for _, root := range roots { - if !slices.ContainsFunc(dirs, func(dir string) bool { - return samePathOrDescendant(cleanRootPath(dir), cleanRootPath(root)) || - samePathOrDescendant(cleanRootPath(root), cleanRootPath(dir)) - }) { + for _, scope := range agentScopes { + ownershipScopes := storedSourceDBHintScopes(scope.proofScopes) + if len(ownershipScopes) == 0 { continue } - ownershipScopes := []parser.StoredSourceHintScope{{Path: root}} - if resolver, ok := provider.(parser.ReconciliationOwnershipScopeProvider); ok { - if resolved := resolver.ReconciliationOwnershipScopes(root); len(resolved) > 0 { - ownershipScopes = resolved - } - } var cursor db.SessionSourceCursor for { if err := ctx.Err(); err != nil { @@ -4282,7 +4648,7 @@ func (e *Engine) tombstoneMissingWatchSourcesForAgentLocked( } page, err := e.db.ListActiveSessionSourceOwnershipScopesPage( ctx, e.machine, string(agent), - storedSourceDBHintScopes(ownershipScopes), cursor, + ownershipScopes, cursor, ) if err != nil { return deleted, fmt.Errorf( @@ -4294,6 +4660,15 @@ func (e *Engine) tombstoneMissingWatchSourcesForAgentLocked( } missingByPath := make(map[string]bool, len(page)) for _, ownership := range page { + if !db.StoredSourcePathHintScopesContain( + ownership.FilePath, ownershipScopes, + ) { + // The SQL LIKE prefilter is ASCII-case-insensitive + // while this predicate is platform-exact; retain any + // row the pass holds no proof over and leave the + // keyset cursor untouched. + continue + } statPath := ownership.FilePath persistentMemberContainerExists := false missing, ok := missingByPath[statPath] @@ -4303,9 +4678,17 @@ func (e *Engine) tombstoneMissingWatchSourcesForAgentLocked( missingByPath[statPath] = missing } if !missing { + // An aggregate row stores its container path, so + // full-root coverage OR a scope proof spanning that + // container's whole membership grants member-absence + // authority: either way the completed stream + // enumerated every member the row could resolve to. gone, checkErr := aggregateOwnedMemberGone( ctx, spool, provider, agent, ownership, - allProviderRootsCovered, + allProviderRootsCovered || + reconciliationProofCoversContainerMembership( + scope.proofScopes, ownership.FilePath, + ), ) if checkErr != nil { return deleted, checkErr @@ -4313,6 +4696,21 @@ func (e *Engine) tombstoneMissingWatchSourcesForAgentLocked( if !gone { continue } + if !allProviderRootsCovered { + // Membership authority came from the scope's own + // container proof; a same-ID copy under another + // configured root would make this a move, not a + // deletion. + relocated, err := reconciliationMemberRelocated( + ctx, provider, ownership.ID, + ) + if err != nil { + return deleted, err + } + if relocated { + continue + } + } // The container still exists but the streamed pass no // longer yields this member; tombstone directly — the // guards below all assume a vanished stored path. @@ -4353,8 +4751,12 @@ func (e *Engine) tombstoneMissingWatchSourcesForAgentLocked( } replacementIndex = spool } else { + // Deliberate bypass of the pass's narrowed proof: + // the index must span the provider's full + // configured scope so a replacement beyond the + // narrowed pass stays resolvable. replacementIndex, err = e.buildReconciliationReplacementIndex( - ctx, provider, dirs, + ctx, provider, e.agentDirs[agent], ) if err != nil { return deleted, fmt.Errorf( @@ -4397,9 +4799,16 @@ func (e *Engine) tombstoneMissingWatchSourcesForAgentLocked( statPath, ownership.ID, ) if valid { - if !allProviderRootsCovered { + if !allProviderRootsCovered && + !reconciliationProofCoversContainerMembership( + scope.proofScopes, physicalPath, + ) { // A scoped pass cannot prove that this member does not - // exist in a persistent container under another root. + // exist in a persistent container under another root — + // unless the completed scope's proof is this container's + // whole virtual membership, in which case the admitted + // stream enumerated exactly this container and spool + // absence is authoritative for rows bound to it. continue } if _, statErr := e.lstatSource(physicalPath); statErr != nil { @@ -4456,12 +4865,19 @@ func (e *Engine) tombstoneMissingWatchSourcesForAgentLocked( ) } if found { - _, _, virtual := parser.ParseVirtualSourcePath( + container, _, virtual := parser.ParseVirtualSourcePath( providerDiscoveredPath(source), ) - if virtual && !allProviderRootsCovered { + if virtual && !allProviderRootsCovered && + !reconciliationProofCoversContainerMembership( + scope.proofScopes, container, + ) { // A scoped pass cannot prove that the same logical - // member did not move to another configured root. + // member did not move to another configured root — + // unless the completed scope's proof spans this + // container's whole virtual membership; the + // relocation guard before the tombstone below + // then rules out a copy under another root. continue } if spool == nil || !virtual { @@ -4501,6 +4917,27 @@ func (e *Engine) tombstoneMissingWatchSourcesForAgentLocked( continue } } + if !allProviderRootsCovered { + // A scoped pass never streamed the other configured + // roots, so before tombstoning a virtual member — + // whose home container may itself be gone — ask the + // provider across its full configured scope whether + // the session still resolves anywhere: a same-ID copy + // under another root is a move, not a deletion. + if _, _, virtual := parser.ParseVirtualSourcePath( + ownership.FilePath, + ); virtual { + relocated, err := reconciliationMemberRelocated( + ctx, provider, ownership.ID, + ) + if err != nil { + return deleted, err + } + if relocated { + continue + } + } + } changed, err := e.tombstoneSessionSourceOwnership( ctx, ownership.Machine, ownership.Agent, ownership.ID, ownership.FilePath, @@ -4547,20 +4984,6 @@ func (e *Engine) tombstoneSessionSourceOwnership( return true, nil } -func reconciliationCoversConfiguredRoots(requested, configured []string) bool { - for _, configuredRoot := range configured { - if isRemoteReconciliationRoot(configuredRoot) || - !slices.ContainsFunc(requested, func(requestedRoot string) bool { - return samePathOrDescendant( - cleanRootPath(configuredRoot), cleanRootPath(requestedRoot), - ) - }) { - return false - } - } - return true -} - func (e *Engine) buildReconciliationReplacementIndex( ctx context.Context, provider parser.Provider, configuredRoots []string, ) (result reconciliationSpoolStore, retErr error) { @@ -4605,105 +5028,6 @@ func (e *Engine) lstatSource(path string) (os.FileInfo, error) { return os.Lstat(path) } -func (e *Engine) logicalRootsForWatchRoots(roots []string) []string { - var logical []string - for _, root := range roots { - cleanedRoot := cleanRootPath(root) - matched := false - for _, dirs := range e.agentDirs { - for _, dir := range dirs { - cleanedDir := cleanRootPath(dir) - if !samePathOrDescendant(cleanedRoot, cleanedDir) && - !samePathOrDescendant(cleanedDir, cleanedRoot) { - continue - } - if !slices.Contains(logical, cleanedDir) { - logical = append(logical, cleanedDir) - } - matched = true - } - } - if !matched && !slices.Contains(logical, cleanedRoot) { - logical = append(logical, cleanedRoot) - } - } - return logical -} - -// logicalRootsForAgentWatchRoots resolves the given roots against one agent's -// configured dirs only. Unlike logicalRootsForWatchRoots it never crosses into -// another provider's dirs, so an overlapping ancestor root cannot drag other -// providers into a scoped reconciliation. -func (e *Engine) logicalRootsForAgentWatchRoots( - agent parser.AgentType, roots []string, -) []string { - dirs := e.agentDirs[agent] - var logical []string - for _, root := range roots { - cleanedRoot := cleanRootPath(root) - matched := false - for _, dir := range dirs { - cleanedDir := cleanRootPath(dir) - if !samePathOrDescendant(cleanedRoot, cleanedDir) && - !samePathOrDescendant(cleanedDir, cleanedRoot) { - continue - } - if !slices.Contains(logical, cleanedDir) { - logical = append(logical, cleanedDir) - } - matched = true - } - if !matched && !slices.Contains(logical, cleanedRoot) { - logical = append(logical, cleanedRoot) - } - } - return logical -} - -// agentReconciliationRoots restricts the requested roots to a single agent's -// configured dirs and excludes remote object roots, mirroring -// localReconciliationRoots but without the cross-provider expansion. -func (e *Engine) agentReconciliationRoots( - agent parser.AgentType, roots []string, -) ([]string, int) { - local := make([]string, 0, len(roots)) - remote := 0 - for _, root := range roots { - if isRemoteReconciliationRoot(root) { - remote++ - continue - } - local = append(local, root) - } - return e.logicalRootsForAgentWatchRoots(agent, local), remote -} - -// localReconciliationRoots expands a full watcher recovery to every configured -// local root before tombstoning. Remote object roots are owned by the remote -// sync path: local reconciliation neither enumerates nor tombstones them, and -// reports the exclusion explicitly in its result metrics. -func (e *Engine) localReconciliationRoots( - roots []string, full bool, -) ([]string, int) { - requested := append([]string(nil), roots...) - if full { - requested = requested[:0] - for _, dirs := range e.agentDirs { - requested = append(requested, dirs...) - } - } - local := make([]string, 0, len(requested)) - remote := make(map[string]struct{}) - for _, root := range requested { - if isRemoteReconciliationRoot(root) { - remote[root] = struct{}{} - continue - } - local = append(local, root) - } - return e.logicalRootsForWatchRoots(local), len(remote) -} - func isRemoteReconciliationRoot(root string) bool { return strings.HasPrefix(strings.ToLower(root), "s3://") } @@ -4886,7 +5210,9 @@ func (e *Engine) syncAllLocked( var all []parser.DiscoveredFile counts := make(map[parser.AgentType]int) - providerFound, providerFailures := e.discoverProviderSources(ctx, scope) + providerFound, providerFailures := e.discoverProviderSources( + ctx, scope, preContainerStates, + ) for _, file := range providerFound { counts[file.Agent]++ } @@ -5147,9 +5473,11 @@ const slowProviderDiscoveryThreshold = 100 * time.Millisecond func (e *Engine) discoverProviderSources( ctx context.Context, scope *rootSyncScope, + preContainerStates map[string]parser.SQLiteContainerState, ) ([]parser.DiscoveredFile, int) { var files []parser.DiscoveredFile var failures int + containerTrusted := e.sqliteContainerTrustedForDiscovery(preContainerStates) agents := make([]parser.AgentType, 0, len(e.providerFactories)) for agent := range e.providerFactories { @@ -5185,8 +5513,9 @@ func (e *Engine) discoverProviderSources( continue } provider := factory.NewProvider(parser.ProviderConfig{ - Roots: filteredRoots, - Machine: e.machine, + Roots: filteredRoots, + Machine: e.machine, + SQLiteContainerUnchangedSinceTrust: containerTrusted, }) // Shared-database providers are streamed source-by-source by their // dedicated sync phase. Calling Discover here would build an archive-sized @@ -5671,6 +6000,25 @@ func (e *Engine) discoveredFileEffectiveMtime( } return mtime, nil } + // Watermark-only shared-container sources carry their session-row + // watermark from discovery. Consulting the provider Fingerprint instead + // would resolve the full composite with one indexed child lookup per + // session, scaling cutoff filtering with the container instead of the + // changed batch — and these sources are only listed for containers that + // provably have not changed since their last verified pass, where the + // carried watermark and the composite are equally stale. That proof is + // the pass's container capture: a container that changed between the + // listing and the recapture check may have advanced a session past its + // carried watermark, so the stale value cannot decide the cutoff — fall + // through and resolve the live composite instead. + if file.ProviderSource != nil { + if wm, ok := parser.SourceWatermarkOnlyMTimeNS(*file.ProviderSource); ok { + if dbPath, _, ok := sqliteContainerSourceForFile(file); ok && + e.sqliteContainerPassCaptureValid(dbPath) { + return wm, nil + } + } + } // Provider-authoritative sources resolve freshness through the provider // Fingerprint so composite provider-owned source state participates in // incremental-sync cutoff checks. @@ -7402,6 +7750,19 @@ func (e *Engine) processProviderFile( }, true } + // Watermark-only shared-container sources (changed-path classification) + // carry just the session-row watermark. When it does not advance past + // the stored composite watermark, the session and project rows provably + // did not change, so skip before Fingerprint pays the per-session child + // lookup; a child-only edit this cannot see is reconciled by the next + // full-discovery pass, whose digest comparison still catches it. + if freshMtime, fresh := e.watermarkOnlySQLiteSourceFresh(source, file); fresh { + return processResult{ + skip: true, + mtime: freshMtime, + }, true + } + fingerprint, err := provider.Fingerprint(ctx, source) if err != nil { if file.ForceParse && @@ -8853,7 +9214,7 @@ func (e *Engine) providerIncrementalContentChanged( if !ok || storedHash == "" { return false, false } - curHash, err := ComputeFileHashPrefix(hashPath, info.Size()) + curHash, err := computeFileHashPrefix(hashPath, info.Size()) if err != nil { return false, false } diff --git a/internal/sync/engine_integration_test.go b/internal/sync/engine_integration_test.go index 7a0e711cb..d82c9e6a6 100644 --- a/internal/sync/engine_integration_test.go +++ b/internal/sync/engine_integration_test.go @@ -695,11 +695,16 @@ func TestSyncEngineOpenCodeSQLiteSameMtimeContentChangeUsesFingerprint( "local_modified_at before rewrite") time.Sleep(20 * time.Millisecond) + // The session row's own time_updated deliberately stays at + // 1779012030000. Production OpenCode stamps time_updated on every child + // row it writes, so the replacement children carry a newer one; that is + // the per-session signal the composite mtime reads, and it must catch a + // content change the session row alone cannot show. oc.replaceTextContent( t, "same-mtime-sqlite", "changed prompt with same session mtime", "changed answer with same session mtime", - 1779012000000, + 1779012600000, ) stats = env.engine.SyncAll(context.Background(), nil) @@ -716,8 +721,11 @@ func TestSyncEngineOpenCodeSQLiteSameMtimeContentChangeUsesFingerprint( require.NotNil(t, after.FileHash, "file_hash after rewrite") require.NotNil(t, after.LocalModifiedAt, "local_modified_at after rewrite") - assert.Equal(t, *before.FileMtime, *after.FileMtime, - "same-mtime rewrite keeps the OpenCode SQLite session mtime") + assert.Greater(t, *after.FileMtime, *before.FileMtime, + "child content newer than the session row must advance the stored "+ + "composite mtime: that per-session signal is what detects the "+ + "change without the shared container's stat invalidating every "+ + "other session in the same opencode.db") assert.NotEqual(t, *before.FileHash, *after.FileHash, "changed SQLite child content must change the storage fingerprint") assert.Greater(t, *after.LocalModifiedAt, *before.LocalModifiedAt, @@ -800,7 +808,7 @@ func TestSyncEngineOpenCodeSQLiteStatIdenticalContentChangeStillReemits( oc.replaceTextContent( t, "stat-twin", "replaced prompt", "replaced answer", - 1779012000000, + 1779012600000, ) after, err := os.Stat(dbPath) require.NoError(t, err, "stat opencode.db after rewrite") @@ -909,8 +917,15 @@ func TestSyncEngineOpenCodeSQLiteCwdFilteredContainerStaysUntrusted( stats = engine.SyncAll(context.Background(), nil) require.False(t, stats.Aborted, "second sync aborted: %+v", stats) - assert.Equal(t, 0, stats.Skipped, - "a container with cwd-vetoed sessions must not be gate-skipped") + // Exactly one skip: the persisted allowed session rides its own + // per-session freshness check. The vetoed session was never written, so + // it has no stored row to be fresh against and must be processed again. + // A trusted-container gate skip would cover both sessions and make this + // 2, which is the promotion violation this test exists to catch. + assert.Equal(t, 1, stats.Skipped, + "a container with cwd-vetoed sessions must not be gate-skipped: "+ + "only the persisted session may skip, and only on its own "+ + "per-session freshness") kept, err := database.GetSessionFull( context.Background(), "opencode:keep-session", @@ -1189,9 +1204,8 @@ func TestSyncEngineOpenCodeSQLiteSameMtimeMetadataChangeUsesFingerprint( assert.Equal(t, "original_app", before.Project) time.Sleep(20 * time.Millisecond) - oc.mustExec(t, "update project worktree", - "UPDATE project SET worktree = ? WHERE id = ?", - "/home/user/code/renamed-app", "proj", + oc.updateProjectWorktree( + t, "proj", "/home/user/code/renamed-app", 1779015630000, ) stats = env.engine.SyncAll(context.Background(), nil) @@ -1206,8 +1220,11 @@ func TestSyncEngineOpenCodeSQLiteSameMtimeMetadataChangeUsesFingerprint( require.NotNil(t, after.FileHash, "file_hash after rewrite") require.NotNil(t, after.LocalModifiedAt, "local_modified_at after rewrite") - assert.Equal(t, *before.FileMtime, *after.FileMtime, - "metadata-only rewrite keeps the OpenCode SQLite session mtime") + assert.Greater(t, *after.FileMtime, *before.FileMtime, + "a project worktree rename must advance the session's composite "+ + "mtime: project.time_updated is part of the per-session change "+ + "signal, which is what re-resolves cwd without the shared "+ + "container's stat invalidating every unrelated session") assert.NotEqual(t, *before.FileHash, *after.FileHash, "changed SQLite metadata must change the storage fingerprint") assert.Greater(t, *after.LocalModifiedAt, *before.LocalModifiedAt, diff --git a/internal/sync/engine_test.go b/internal/sync/engine_test.go index 0874c8049..cd0ae1190 100644 --- a/internal/sync/engine_test.go +++ b/internal/sync/engine_test.go @@ -986,12 +986,55 @@ func (factory manyStreamingFactory) Capabilities() parser.Capabilities { return factory.provider.Capabilities() } -func (factory manyStreamingFactory) NewProvider(parser.ProviderConfig) parser.Provider { - return factory.provider +// perCallScopeProviderBase builds the ProviderBase a factory hands to its +// per-call wrapper: the shared Def and Caps with this call's config. Shared +// test providers must never have Config written by NewProvider — sync workers +// construct providers concurrently, so the write races with the +// value-receiver reads of the embedded base (Definition, Capabilities). +func perCallScopeProviderBase( + shared parser.ProviderBase, cfg parser.ProviderConfig, +) parser.ProviderBase { + return parser.ProviderBase{ + Def: shared.Def, Caps: shared.Caps, Config: cfg.Clone(), + } } -func (factory directStreamingFactory) NewProvider(parser.ProviderConfig) parser.Provider { - return factory.provider +// manyStreamingScopedProvider overlays per-call reconciliation scope +// resolution on the shared provider; behavior and counters stay shared. +type manyStreamingScopedProvider struct { + *manyStreamingProvider + scopes parser.ProviderBase +} + +func (p manyStreamingScopedProvider) ResolveReconciliationScopes( + ctx context.Context, req parser.ReconciliationScopeRequest, +) (parser.ReconciliationScopePlan, error) { + return p.scopes.ResolveReconciliationScopes(ctx, req) +} + +func (factory manyStreamingFactory) NewProvider(cfg parser.ProviderConfig) parser.Provider { + return manyStreamingScopedProvider{ + manyStreamingProvider: factory.provider, + scopes: perCallScopeProviderBase(factory.provider.ProviderBase, cfg), + } +} + +type directStreamingScopedProvider struct { + *directStreamingProvider + scopes parser.ProviderBase +} + +func (p directStreamingScopedProvider) ResolveReconciliationScopes( + ctx context.Context, req parser.ReconciliationScopeRequest, +) (parser.ReconciliationScopePlan, error) { + return p.scopes.ResolveReconciliationScopes(ctx, req) +} + +func (factory directStreamingFactory) NewProvider(cfg parser.ProviderConfig) parser.Provider { + return directStreamingScopedProvider{ + directStreamingProvider: factory.provider, + scopes: perCallScopeProviderBase(factory.provider.ProviderBase, cfg), + } } type baselineDBBackedProvider struct { @@ -1820,8 +1863,22 @@ func (f failingDBBackedFactory) Capabilities() parser.Capabilities { return f.provider.Capabilities() } -func (f failingDBBackedFactory) NewProvider(parser.ProviderConfig) parser.Provider { - return f.provider +type failingDBBackedScopedProvider struct { + *failingDBBackedProvider + scopes parser.ProviderBase +} + +func (p failingDBBackedScopedProvider) ResolveReconciliationScopes( + ctx context.Context, req parser.ReconciliationScopeRequest, +) (parser.ReconciliationScopePlan, error) { + return p.scopes.ResolveReconciliationScopes(ctx, req) +} + +func (f failingDBBackedFactory) NewProvider(cfg parser.ProviderConfig) parser.Provider { + return failingDBBackedScopedProvider{ + failingDBBackedProvider: f.provider, + scopes: perCallScopeProviderBase(f.provider.ProviderBase, cfg), + } } func TestReconcileWatchRootsFailsWhenDBBackedDiscoveryFails(t *testing.T) { @@ -2353,13 +2410,14 @@ func TestTombstoneMissingWatchSourcesScopesSharedPathByAgent(t *testing.T) { {Agent: "codex", FilePath: shared}, }, )) - engine := &Engine{ - db: database, machine: "local", - agentDirs: map[parser.AgentType][]string{ + engine := NewEngine(database, EngineConfig{ + AgentDirs: map[parser.AgentType][]string{ parser.AgentClaude: {root}, parser.AgentCodex: {root}, }, - } + Machine: "local", + }) + t.Cleanup(engine.Close) deleted, err := tombstoneMissingWatchSourcesUnderSyncLock( t.Context(), engine, []string{root}, @@ -2462,12 +2520,13 @@ func TestTombstoneMissingWatchSourcesPaginatesLargeArchive(t *testing.T) { require.NoError(t, database.BaselineActiveSessionSourcePaths( t.Context(), "local", sources, )) - engine := &Engine{ - db: database, machine: "local", - agentDirs: map[parser.AgentType][]string{ + engine := NewEngine(database, EngineConfig{ + AgentDirs: map[parser.AgentType][]string{ parser.AgentClaude: {root}, }, - } + Machine: "local", + }) + t.Cleanup(engine.Close) deleted, err := tombstoneMissingWatchSourcesUnderSyncLock( t.Context(), engine, []string{root}, @@ -2501,15 +2560,19 @@ func TestTombstoneMissingWatchSourcesDoesNotRediscoverEachOwnership(t *testing.T Type: parser.AgentCowork, IDPrefix: "cowork:", FileBased: true, }, }} - engine := &Engine{ - db: database, machine: "local", - agentDirs: map[parser.AgentType][]string{ + engine := NewEngine(database, EngineConfig{ + AgentDirs: map[parser.AgentType][]string{ parser.AgentCowork: {root}, }, - providerFactories: providerFactoryMap([]parser.ProviderFactory{ + Machine: "local", + ProviderFactories: []parser.ProviderFactory{ lookupSourceFactory{provider: provider}, - }), - } + }, + ProviderMigrationModes: map[parser.AgentType]parser.ProviderMigrationMode{ + parser.AgentCowork: parser.ProviderMigrationProviderAuthoritative, + }, + }) + t.Cleanup(engine.Close) deleted, err := tombstoneMissingWatchSourcesUnderSyncLock( t.Context(), engine, []string{root}, @@ -2546,18 +2609,19 @@ func TestTombstoneMissingWatchSourcesDoesNotInferUnvalidatedVirtualPaths(t *test t.Context(), "local", sources, )) var statCalls atomic.Int32 - engine := &Engine{ - db: database, machine: "local", - agentDirs: map[parser.AgentType][]string{ + engine := NewEngine(database, EngineConfig{ + AgentDirs: map[parser.AgentType][]string{ parser.AgentClaude: {root}, }, - lstat: func(path string) (os.FileInfo, error) { - statCalls.Add(1) - _, expected := wantPaths[path] - assert.True(t, expected, "only the exact stored path may be checked") - delete(wantPaths, path) - return nil, os.ErrNotExist - }, + Machine: "local", + }) + t.Cleanup(engine.Close) + engine.lstat = func(path string) (os.FileInfo, error) { + statCalls.Add(1) + _, expected := wantPaths[path] + assert.True(t, expected, "only the exact stored path may be checked") + delete(wantPaths, path) + return nil, os.ErrNotExist } deleted, err := tombstoneMissingWatchSourcesUnderSyncLock( @@ -7212,8 +7276,22 @@ func (f lookupSourceFactory) Capabilities() parser.Capabilities { return f.provider.Capabilities() } -func (f lookupSourceFactory) NewProvider(parser.ProviderConfig) parser.Provider { - return f.provider +type lookupSourceScopedProvider struct { + *lookupSourceProvider + scopes parser.ProviderBase +} + +func (p lookupSourceScopedProvider) ResolveReconciliationScopes( + ctx context.Context, req parser.ReconciliationScopeRequest, +) (parser.ReconciliationScopePlan, error) { + return p.scopes.ResolveReconciliationScopes(ctx, req) +} + +func (f lookupSourceFactory) NewProvider(cfg parser.ProviderConfig) parser.Provider { + return lookupSourceScopedProvider{ + lookupSourceProvider: f.provider, + scopes: perCallScopeProviderBase(f.provider.ProviderBase, cfg), + } } type lookupSourceProvider struct { diff --git a/internal/sync/grouped_reconcile_test.go b/internal/sync/grouped_reconcile_test.go index ad9a6c11f..5279779d0 100644 --- a/internal/sync/grouped_reconcile_test.go +++ b/internal/sync/grouped_reconcile_test.go @@ -249,18 +249,31 @@ func TestGroupedReconcileContainerProbesDoNotScaleWithProviderGroups(t *testing. }) } - t.Run("unscoped group still captures containers", func(t *testing.T) { + t.Run("unscoped group captures only planned containers", func(t *testing.T) { root := filepath.Join(t.TempDir(), "claude-0") writeGroupedClaudeFixture(t, root, "session-0") engine := newContainerEngine(t, []string{root}) probes := countContainerProbes(t) + // An unscoped partial pass resolves each provider's own topology; a + // Claude-only root yields no OpenCode scopes, so no container can be + // discovered and none may be probed. require.NoError(t, engine.ReconcileProviderRootsGrouped(t.Context(), []ProviderRootsGroup{{Agent: "", Roots: []string{root}}}, )) - + assert.Zero(t, probes.Load(), + "an unscoped pass whose plans stream no shared container must not probe one") + + // A full unscoped recovery covers every provider's configured scope + // and still captures every configured container. The fixture + // container is deliberately not a valid database, so the pass itself + // reports OpenCode discovery as failed; only the probe count is + // pinned here. + _ = engine.ReconcileProviderRootsGrouped(t.Context(), + []ProviderRootsGroup{{Agent: "", Roots: nil}}, + ) assert.Positive(t, probes.Load(), - "the unscoped pass must still capture configured containers") + "a full unscoped pass must still capture configured containers") }) } diff --git a/internal/sync/hash.go b/internal/sync/hash.go index 425f7854b..d90ff14ad 100644 --- a/internal/sync/hash.go +++ b/internal/sync/hash.go @@ -30,6 +30,18 @@ func ComputeFileHash(path string) (string, error) { return hash, nil } +// computeFileHashPrefix indirects ComputeFileHashPrefix so cardinality-scaling +// tests can count the source-content reads a sync pass performs *to decide +// freshness*. Only providerIncrementalContentChanged goes through it today. +// +// The incremental-append path also hashes (to refresh a stored fingerprint +// after consuming new bytes) and deliberately calls ComputeFileHashPrefix +// directly: that read is write-side work proportional to data that genuinely +// changed, not a per-session freshness probe, so counting it would blur the +// invariant TestWarmFullSyncDoesNotRehashClaudeArchive pins. Route a new +// freshness gate through this var; leave write-side hashing on the direct call. +var computeFileHashPrefix = ComputeFileHashPrefix + // ComputeFileHashPrefix returns the SHA-256 hex digest of the first size bytes // of the file at path. It returns an error if the file is shorter than size. func ComputeFileHashPrefix(path string, size int64) (string, error) { diff --git a/internal/sync/hermes_archive_test.go b/internal/sync/hermes_archive_test.go index 16b5bec55..bdf25cabc 100644 --- a/internal/sync/hermes_archive_test.go +++ b/internal/sync/hermes_archive_test.go @@ -508,6 +508,58 @@ func TestReconcileHermesDefaultSessionsRootTombstonesRemovedStateMember(t *testi "authoritative reconciliation must tombstone a removed state.db member") } +// TestReconcileHermesScopedArchiveTombstonesRemovedStateMember pins +// aggregate-member authority for a pass scoped to one complete archive: the +// scope's proof spans the archive's whole state.db membership, so a removed +// member is reclaimed even though a second configured root stays uncovered +// and full provider-root coverage is out of reach. +func TestReconcileHermesScopedArchiveTombstonesRemovedStateMember( + t *testing.T, +) { + archiveRoot := t.TempDir() + stateDB := writeHermesArchiveStateDB(t, archiveRoot) + sessionsDir := filepath.Join(archiveRoot, "sessions") + require.NoError(t, os.MkdirAll(sessionsDir, 0o755)) + otherRoot := t.TempDir() + writeHermesTranscriptFile(t, otherRoot, "keeper") + + database := dbtest.OpenTestDB(t) + engine := NewEngine(database, EngineConfig{ + AgentDirs: map[parser.AgentType][]string{ + parser.AgentHermes: {sessionsDir, otherRoot}, + }, + Machine: "local", + }) + t.Cleanup(engine.Close) + require.NoError(t, engine.ReconcileWatchRootsAfterLostEvents( + t.Context(), []string{sessionsDir, otherRoot}, false, + )) + stored, err := database.GetSession(t.Context(), "hermes:child") + require.NoError(t, err) + require.NotNil(t, stored, "initial reconciliation must store the state member") + + conn, err := sql.Open("sqlite3", stateDB) + require.NoError(t, err) + _, err = conn.ExecContext(t.Context(), "DELETE FROM sessions WHERE id = 'child'") + require.NoError(t, err) + require.NoError(t, conn.Close()) + + // The pass is scoped to the archive only; the other configured root is + // deliberately not requested, so full provider-root coverage is absent. + require.NoError(t, engine.ReconcileProviderRoots( + t.Context(), parser.AgentHermes, []string{sessionsDir}, + )) + + stored, err = database.GetSession(t.Context(), "hermes:child") + require.NoError(t, err) + assert.Nil(t, stored, + "an archive-scoped pass proves its own membership and reclaims the member") + survivor, err := database.GetSession(t.Context(), "hermes:keeper") + require.NoError(t, err) + assert.NotNil(t, survivor, + "the unrequested root's sessions are untouched") +} + // Streamed discovery must open state.db a bounded number of times per pass: // once to stream members and once for the transcript membership check — not // once per transcript file, which scales reconciliation work with archive @@ -1001,3 +1053,98 @@ func writeHermesArchiveStateDB(t *testing.T, root string) string { require.NoError(t, conn.Close()) return stateDB } + +// TestReconcileHermesRemovedProfileTombstonesItsTranscripts pins what a +// removal event under a profiles container reclaims. The profile directory is +// gone by the time the pass resolves, so no owned profile matches and the +// request widens to the container. That scope proves the container without +// covering it, which is enough to stat the removed profile's stored +// transcripts and tombstone them while a live sibling's are retained. +func TestReconcileHermesRemovedProfileTombstonesItsTranscripts(t *testing.T) { + container := filepath.Join(t.TempDir(), ".hermes", "profiles") + writeHermesProfileTranscript(t, container, "research", "gone") + writeHermesProfileTranscript(t, container, "writing", "kept") + + database := dbtest.OpenTestDB(t) + engine := NewEngine(database, EngineConfig{ + AgentDirs: map[parser.AgentType][]string{ + parser.AgentHermes: {container}, + }, + Machine: "local", + }) + t.Cleanup(engine.Close) + require.Equal(t, 2, engine.SyncAll(t.Context(), nil).Synced) + + removed := filepath.Join(container, "research") + require.NoError(t, os.RemoveAll(removed)) + require.NoError(t, engine.ReconcileWatchRootsAfterLostEvents( + t.Context(), []string{removed}, false, + )) + + stored, err := database.GetSession(t.Context(), "hermes:gone") + require.NoError(t, err) + assert.Nil(t, stored, + "a removal event must reclaim the deleted profile's transcripts") + survivor, err := database.GetSession(t.Context(), "hermes:kept") + require.NoError(t, err) + assert.NotNil(t, survivor, "a live sibling profile keeps its sessions") +} + +// TestReconcileHermesFlatRootDescendantTombstonesRemovedTranscript pins +// descendant resolution for a plain transcript-directory root: with no +// state.db and no sessions/ subdirectory there is no archive topology, so a +// requested transcript resolves generically — traverse the configured root, +// prove only the requested file — and a deleted transcript is reclaimed +// while its sibling keeps its session. +func TestReconcileHermesFlatRootDescendantTombstonesRemovedTranscript( + t *testing.T, +) { + root := t.TempDir() + writeHermesTranscriptFile(t, root, "gone") + writeHermesTranscriptFile(t, root, "kept") + + database := dbtest.OpenTestDB(t) + engine := NewEngine(database, EngineConfig{ + AgentDirs: map[parser.AgentType][]string{ + parser.AgentHermes: {root}, + }, + Machine: "local", + }) + t.Cleanup(engine.Close) + require.Equal(t, 2, engine.SyncAll(t.Context(), nil).Synced) + + removed := filepath.Join(root, "gone.jsonl") + require.NoError(t, os.Remove(removed)) + require.NoError(t, engine.ReconcileProviderRoots( + t.Context(), parser.AgentHermes, []string{removed}, + )) + + stored, err := database.GetSession(t.Context(), "hermes:gone") + require.NoError(t, err) + assert.Nil(t, stored, + "a removed flat-root transcript must be reclaimed by its own request") + survivor, err := database.GetSession(t.Context(), "hermes:kept") + require.NoError(t, err) + assert.NotNil(t, survivor, + "a sibling transcript outside the proof keeps its session") +} + +func writeHermesProfileTranscript( + t *testing.T, container, profile, sessionID string, +) { + t.Helper() + writeHermesTranscriptFile( + t, filepath.Join(container, profile, "sessions"), sessionID, + ) +} + +func writeHermesTranscriptFile(t *testing.T, dir, sessionID string) { + t.Helper() + require.NoError(t, os.MkdirAll(dir, 0o755)) + require.NoError(t, os.WriteFile( + filepath.Join(dir, sessionID+".jsonl"), []byte( + `{"role":"session_meta","platform":"cli","timestamp":"2026-05-14T10:00:00Z"}`+"\n"+ + `{"role":"user","content":"hello","timestamp":"2026-05-14T10:01:00Z"}`+"\n"+ + `{"role":"assistant","content":"Done.","timestamp":"2026-05-14T10:02:00Z"}`+"\n", + ), 0o600)) +} diff --git a/internal/sync/opencode_container_gate.go b/internal/sync/opencode_container_gate.go index 2adc7d7b4..d53578e25 100644 --- a/internal/sync/opencode_container_gate.go +++ b/internal/sync/opencode_container_gate.go @@ -3,6 +3,7 @@ package sync import ( + "context" "maps" "path/filepath" "slices" @@ -122,25 +123,38 @@ func (e *Engine) captureSQLiteContainerStates( return states } -// captureSQLiteContainerStatesForAgent scopes the pre-discovery capture to -// what an agent-scoped pass can discover. A pass scoped to a provider outside -// the OpenCode SQLite family streams no shared containers, so probing every -// configured container there would repeat once per provider group in a -// grouped poll; an in-family scope probes only its own containers that -// overlap the pass's reconciliation roots, keeping capture work bounded by -// the batch rather than the agent's full configuration. The unscoped pass -// (empty agent) still captures every configured container. -func (e *Engine) captureSQLiteContainerStatesForAgent( - agent parser.AgentType, roots []string, +// capturePlannedSQLiteContainerStates scopes the pre-discovery capture to +// what the resolved reconciliation plans can discover. A pass whose plans +// name no OpenCode-family provider streams no shared containers, so probing +// every configured container there would repeat once per provider group in a +// grouped poll; an in-family plan probes only its own containers that +// overlap its scopes' traversal roots, keeping capture work bounded by the +// batch rather than the agent's full configuration. A full-coverage pass +// still captures every configured container. +func (e *Engine) capturePlannedSQLiteContainerStates( + plans []providerReconciliationPlan, fullCoverage bool, ) map[string]parser.SQLiteContainerState { - if agent == "" { + if fullCoverage { return e.captureSQLiteContainerStates(nil) } - if e.forceParse || !slices.Contains(openCodeFamilySQLiteAgents, agent) { + if e.forceParse { return nil } states := make(map[string]parser.SQLiteContainerState) - e.captureAgentSQLiteContainerStates(agent, roots, states) + for _, plan := range plans { + if plan.err != nil || + !slices.Contains(openCodeFamilySQLiteAgents, plan.agent) { + continue + } + var roots []string + for _, scope := range plan.plan.Scopes { + roots = append(roots, scope.TraversalRoots...) + } + if len(roots) == 0 { + continue + } + e.captureAgentSQLiteContainerStates(plan.agent, roots, states) + } return states } @@ -197,6 +211,148 @@ func addSQLiteContainerState( states[dbPath] = state } +// openCodeContainerPathForChangedPathEvent maps a changed-path event to the +// shared SQLite container it names for one OpenCode-family agent, or "" +// when the agent has no container or the event is not a container write. +func openCodeContainerPathForChangedPathEvent( + agent parser.AgentType, + roots []string, + path string, +) string { + if openCodeFormatDBName(agent) == "" { + return "" + } + for _, dir := range roots { + if dir == "" || strings.HasPrefix(dir, "s3://") { + continue + } + if container := openCodeContainerPathForEvent(agent, dir, path); container != "" { + return container + } + } + return "" +} + +// storedMemberFreshnessPager pages stored freshness for one shared container +// in ascending virtual-path order, translating each folded row into the +// coverage authority the provider's changed-path merge consumes: a listed +// member whose carried session-row watermark is at or below the row's +// covered-through watermark is provably unchanged and is omitted from the +// listing, so a one-session write flows one candidate into the sync pipeline +// while peak memory stays one page — never the container's full membership. +// +// The covered-through watermark is the session/project metadata watermark +// recovered from the stored child digest (storedSessionRowWatermarkNS), +// keeping the comparison per-session and like-for-like: a session or project +// row that advances past its own stored metadata watermark is always kept, +// wherever other sessions' watermarks or its own child timestamps sit. Rows +// behind the current data version are not emitted at all — a version rewrite +// must keep the source — and sessions with no stored row are kept by the +// merge's absent-row rule. +// +// Known, deliberate deferral (not a detection gap to "fix" here): a +// child-only write that leaves the session and project rows untouched is +// invisible to the session-row watermark wherever its timestamps land — +// above or below the stored composite alike. Detecting it per event would +// require reading child rows, which is exactly the archive-sized work this +// path exists to avoid. Such writes reconcile on the next full-discovery +// pass, whose digest still catches them (the write itself broke container +// trust, so that pass carries the full digest); actively watched sessions +// bypass this path entirely via the per-session composite poll. The +// contract is documented in docs/internal/session-format-sources.md and +// pinned by TestOpenCodeWatcherPassDefersChildOnlyEditToFullDiscovery. +func (e *Engine) storedMemberFreshnessPager( + container string, +) parser.StoredMemberFreshnessPager { + current := db.CurrentDataVersion() + return func( + ctx context.Context, afterPath string, limit int, + ) ([]parser.StoredMemberFreshness, bool, error) { + var rows []parser.StoredMemberFreshness + // Withheld rows shrink the emitted page below the raw page, so keep + // reading raw pages — advancing by the raw cursor, not the emitted + // one — until something is vouchable or the container is exhausted. + // Returning an empty page with done=false instead would read as + // exhaustion to the merge cursor, silently un-covering every stored + // member past the first all-stale page. + for { + page, done, err := e.db.ListVirtualContainerMemberFreshnessPage( + ctx, container, afterPath, limit, + ) + if err != nil { + return nil, false, err + } + for _, row := range page { + if row.DataVersion < current { + continue + } + rows = append(rows, parser.StoredMemberFreshness{ + Path: row.Path, + CoveredThroughNS: storedSessionRowWatermarkNS( + row.VirtualContainerMemberFreshness, + ), + }) + } + if done || len(rows) > 0 { + return rows, done, nil + } + if len(page) == 0 { + // The page contract returns done for an empty page; treat a + // violation as exhaustion rather than spinning on it. + return rows, true, nil + } + afterPath = page[len(page)-1].Path + } + } +} + +// storedSessionRowWatermarkNS resolves the stored value a carried session-row +// watermark is compared against, like-for-like: the session/project metadata +// watermark recovered from the stored child digest. Comparing against the +// stored composite MTimeNS instead would over-skip — a composite dominated by +// a newer child timestamp would hide a metadata update (title, directory, +// worktree rename) whose stamp lands below it. Rows without a parseable +// digest (pre-digest fingerprints, future digest versions) fall back to the +// composite, the conservative pre-digest behavior that self-heals on the +// row's next reparse. +func storedSessionRowWatermarkNS( + member db.VirtualContainerMemberFreshness, +) int64 { + if metadata, ok := parser.OpenCodeChildDigestMetadataWatermarkNS( + member.Hash, + ); ok { + return metadata + } + return member.MTimeNS +} + +// sqliteContainerTrustedForDiscovery returns discovery's trust probe: it +// reports containers whose pre-discovery capture matches the last fully +// verified state, meaning every member will gate-skip before fingerprinting +// and the full child digest would be computed for nothing. The probe is +// keyed to the pass's own pre-discovery captures so a container that +// changes between capture and listing can never look trusted with a newer +// session set (the gate separately fails such containers for the pass). +// Nil when nothing was captured or every parse is forced. +func (e *Engine) sqliteContainerTrustedForDiscovery( + preStates map[string]parser.SQLiteContainerState, +) func(string) bool { + if len(preStates) == 0 || e.forceParse { + return nil + } + return func(dbPath string) bool { + dbPath = filepath.Clean(dbPath) + state, ok := preStates[dbPath] + if !ok { + return false + } + e.containerMu.Lock() + trusted, ok := e.trustedSQLiteContainers[dbPath] + e.containerMu.Unlock() + return ok && trusted.state == state + } +} + func openCodeContainerPathForEvent( agent parser.AgentType, root string, @@ -336,6 +492,85 @@ func (e *Engine) sqliteContainerSourceFresh(file parser.DiscoveredFile) bool { e.db.GetSessionFilePath(fullID) == e.effectiveSourcePath(file.Path) } +// watermarkOnlySQLiteSourceFresh reports whether a shared-container session +// whose source carries only the session-row watermark is already covered by +// its stored session/project metadata watermark, compared like-for-like: +// the stored value is recovered from the persisted child digest, falling +// back to the stored composite MTimeNS for rows without a parseable digest. +// A session-row watermark at or below the stored metadata watermark proves +// the session and project rows did not advance, so the parse is skipped +// without resolving the child digest. What the watermark cannot see — any +// child-only write that leaves the session and project rows untouched — is +// deliberately deferred to the next full-discovery pass, whose carried +// digest still catches it (see storedMemberFreshnessPager for the full +// contract). That keeps per-event work bounded by the changed batch instead +// of the archive. +func (e *Engine) watermarkOnlySQLiteSourceFresh( + source parser.SourceRef, + file parser.DiscoveredFile, +) (int64, bool) { + if e.forceParse || file.ForceParse { + return 0, false + } + watermark, ok := parser.SourceWatermarkOnlyMTimeNS(source) + if !ok { + return 0, false + } + // The skip is only sound while the pass's container capture is valid. A + // trusted full discovery lists watermark-only sources; if the container + // changes between that listing and the pass's recapture check, the + // capture is invalidated and a concurrent child-only write may hide + // beneath an unchanged metadata watermark — those sources must fall + // through to Fingerprint and resolve the full digest instead. + if dbPath, _, ok := sqliteContainerSourceForFile(file); !ok || + !e.sqliteContainerPassCaptureValid(dbPath) { + return 0, false + } + lookupPath := providerDiscoveredPath(source) + if lookupPath == "" { + return 0, false + } + if e.pathRewriter != nil { + lookupPath = e.pathRewriter(lookupPath) + } + _, storedMtime, found := e.db.GetFileInfoByPath(lookupPath) + if !found { + return 0, false + } + limit := storedMtime + if hash, ok := e.db.GetFileHashByPath(lookupPath); ok { + if metadata, parsed := parser.OpenCodeChildDigestMetadataWatermarkNS( + hash, + ); parsed { + limit = metadata + } + } + if limit < watermark { + return 0, false + } + if e.db.GetDataVersionByPath(lookupPath) < db.CurrentDataVersion() { + return 0, false + } + return storedMtime, true +} + +// sqliteContainerPassCaptureValid reports whether the current pass still +// holds a live capture for the container: one was taken before discovery, +// the post-discovery recapture matched it, and no processing failure has +// poisoned the container since. Watermark-only skips require this — an +// invalidated capture means the container changed while the pass was +// listing it, and the watermark cannot see what that change touched. +func (e *Engine) sqliteContainerPassCaptureValid(dbPath string) bool { + e.containerMu.Lock() + defer e.containerMu.Unlock() + pass := e.containerPass + if pass == nil || pass.failed[dbPath] { + return false + } + _, ok := pass.captured[dbPath] + return ok +} + // noteSQLiteContainerResult records a processed file's outcome for // promotion bookkeeping. Skips count as completions: a skipped session was // either gate-skipped against an already-trusted state or individually diff --git a/internal/sync/opencode_container_gate_internal_test.go b/internal/sync/opencode_container_gate_internal_test.go index 1a8a4f6ef..756c168f3 100644 --- a/internal/sync/opencode_container_gate_internal_test.go +++ b/internal/sync/opencode_container_gate_internal_test.go @@ -25,6 +25,280 @@ func newContainerTestDB(t *testing.T) (string, *sql.DB) { return path, conn } +// newCompositeContainerTestDB creates an OpenCode container whose schema +// carries the composite change-signal columns and session_id indexes, so +// watermark-only listings are supported. +func newCompositeContainerTestDB(t *testing.T) (string, *sql.DB) { + t.Helper() + path := filepath.Join(t.TempDir(), "opencode.db") + conn, err := sql.Open("sqlite3", path) + require.NoError(t, err, "open container db") + t.Cleanup(func() { _ = conn.Close() }) + _, err = conn.Exec(` + CREATE TABLE project ( + id TEXT PRIMARY KEY, + worktree TEXT NOT NULL, + time_updated INTEGER NOT NULL DEFAULT 0 + ); + CREATE TABLE session ( + id TEXT PRIMARY KEY, + project_id TEXT NOT NULL, + time_created INTEGER NOT NULL, + time_updated INTEGER NOT NULL + ); + CREATE TABLE message ( + id TEXT PRIMARY KEY, + session_id TEXT NOT NULL, + data TEXT NOT NULL, + time_created INTEGER NOT NULL, + time_updated INTEGER NOT NULL DEFAULT 0 + ); + CREATE TABLE part ( + id TEXT PRIMARY KEY, + session_id TEXT NOT NULL, + message_id TEXT NOT NULL, + data TEXT NOT NULL, + time_created INTEGER NOT NULL, + time_updated INTEGER NOT NULL DEFAULT 0 + ); + CREATE INDEX message_session_idx ON message (session_id); + CREATE INDEX part_session_idx ON part (session_id); + `) + require.NoError(t, err, "create composite schema") + return path, conn +} + +// seedCoveredVirtualMember stores one virtual member whose stored freshness +// fully covers watermarkMS, stamped with the current data version as a +// completed parse would be (UpsertSession seeds data_version 0 by design). +func seedCoveredVirtualMember( + t *testing.T, database *db.DB, sessionID, virtualPath string, + watermarkMS int64, +) { + t.Helper() + storedMtime := watermarkMS * 1_000_000 + require.NoError(t, database.UpsertSession(db.Session{ + ID: sessionID, Agent: "opencode", Project: "project", + Machine: "local", FilePath: &virtualPath, FileMtime: &storedMtime, + })) + require.NoError(t, database.SetSessionDataVersion( + sessionID, db.CurrentDataVersion(), + )) +} + +// TestStoredMemberFreshnessPagerEmitsOnlyVouchableRows pins the pager's +// translation of stored rows into coverage authority: rows behind the +// current data version are omitted entirely so their sources stay listed, +// a stored child digest yields its embedded session/project metadata +// watermark, and a plain fingerprint falls back to the stored composite. +func TestStoredMemberFreshnessPagerEmitsOnlyVouchableRows(t *testing.T) { + database := openTestDB(t) + const container = "/data/opencode.db" + seedCoveredVirtualMember(t, database, "opencode:a", container+"#a", 100) + + digest := "opencode-child:v1:900:20:30:1:2:abcd" + digestPath := container + "#b" + digestMtime := int64(900) * 1_000_000 + require.NoError(t, database.UpsertSession(db.Session{ + ID: "opencode:b", Agent: "opencode", Project: "project", + Machine: "local", FilePath: &digestPath, FileMtime: &digestMtime, + FileHash: &digest, + })) + require.NoError(t, database.SetSessionDataVersion( + "opencode:b", db.CurrentDataVersion(), + )) + + stalePath := container + "#c" + staleMtime := int64(100) * 1_000_000 + require.NoError(t, database.UpsertSession(db.Session{ + ID: "opencode:c", Agent: "opencode", Project: "project", + Machine: "local", FilePath: &stalePath, FileMtime: &staleMtime, + })) + + e := &Engine{db: database, machine: "local"} + rows, done, err := e.storedMemberFreshnessPager(container)( + t.Context(), "", 10, + ) + require.NoError(t, err) + assert.True(t, done) + require.Len(t, rows, 2, + "the stale-version row must not be emitted at all") + assert.Equal(t, container+"#a", rows[0].Path) + assert.Equal(t, int64(100)*1_000_000, rows[0].CoveredThroughNS, + "a plain fingerprint falls back to the stored composite") + assert.Equal(t, container+"#b", rows[1].Path) + assert.Equal(t, int64(30)*1_000_000, rows[1].CoveredThroughNS, + "a child digest yields its embedded metadata watermark") +} + +// TestStoredMemberFreshnessPagerAdvancesPastAllStalePages pins the pager's +// raw-cursor advance: version-stale rows are withheld from the emitted page, +// and when a whole raw page is stale the pager must keep reading from the +// raw cursor instead of returning an empty not-done page — the merge cursor +// reads that as exhaustion, which would silently un-cover every stored +// member past the first all-stale page and let one event's work scale with +// the remainder of the archive. +func TestStoredMemberFreshnessPagerAdvancesPastAllStalePages(t *testing.T) { + database := openTestDB(t) + const container = "/data/opencode.db" + // Two stale-version members sort before the covered current-version + // member, so a limit-2 first page is entirely withheld. + for _, id := range []string{"a", "b"} { + path := container + "#" + id + mtime := int64(100) * 1_000_000 + require.NoError(t, database.UpsertSession(db.Session{ + ID: "opencode:" + id, Agent: "opencode", Project: "project", + Machine: "local", FilePath: &path, FileMtime: &mtime, + })) + } + seedCoveredVirtualMember(t, database, "opencode:c", container+"#c", 500) + + e := &Engine{db: database, machine: "local"} + rows, done, err := e.storedMemberFreshnessPager(container)( + t.Context(), "", 2, + ) + require.NoError(t, err) + require.Len(t, rows, 1, + "the pager must advance past the all-stale page to the vouchable row") + assert.Equal(t, container+"#c", rows[0].Path) + assert.Equal(t, int64(500)*1_000_000, rows[0].CoveredThroughNS) + assert.True(t, done) +} + +// TestClassifyChangedPathWatermarkMergeRelistsOnStaleCapture pins the +// classification-time capture guard around the merged listing: while the +// container provably has not changed across the listing window, covered +// members are dropped during the stream and a fully covered container +// classifies to nothing; when every recapture differs from the pre-listing +// capture, the merge cannot be trusted and classification re-lists without +// stored authority, keeping every member for the per-file gates. +func TestClassifyChangedPathWatermarkMergeRelistsOnStaleCapture(t *testing.T) { + dbPath, conn := newCompositeContainerTestDB(t) + const base = int64(1779012000000) + for _, id := range []string{"ses-1", "ses-2"} { + _, err := conn.Exec( + "INSERT INTO session (id, project_id, time_created, time_updated)"+ + " VALUES (?, 'proj', ?, ?)", + id, base, base, + ) + require.NoError(t, err, "insert session row") + } + + database := openTestDB(t) + engine := NewEngine(database, EngineConfig{ + AgentDirs: map[parser.AgentType][]string{ + parser.AgentOpenCode: {filepath.Dir(dbPath)}, + }, + Machine: "local", + }) + t.Cleanup(engine.Close) + seedCoveredVirtualMember(t, database, "opencode:ses-1", dbPath+"#ses-1", base) + seedCoveredVirtualMember(t, database, "opencode:ses-2", dbPath+"#ses-2", base) + + files, err := engine.classifyProviderChangedPath(t.Context(), dbPath) + require.NoError(t, err) + assert.Empty(t, files, + "a fully covered container classifies to nothing under a live capture") + + // A capture that never repeats: the post-listing revalidation always + // mismatches, so the merged listing must be discarded and re-listed + // without stored authority. + orig := statSQLiteContainerState + t.Cleanup(func() { statSQLiteContainerState = orig }) + var drift int64 + statSQLiteContainerState = func( + path string, + ) (parser.SQLiteContainerState, bool) { + state, ok := orig(path) + drift++ + state.DBSize += drift + return state, ok + } + + files, err = engine.classifyProviderChangedPath(t.Context(), dbPath) + require.NoError(t, err) + assert.Len(t, files, 2, + "a stale capture must keep every member for the per-file gates") +} + +// TestDiscoveredFileWatermarkCutoffRequiresLiveCapture pins cutoff +// filtering's trust in carried session-row watermarks: the carried value may +// decide the incremental cutoff only while the pass's container capture is +// live. A child-only commit landing during discovery leaves the session-row +// watermark behind the live composite; if the stale carried value were +// trusted after the recapture invalidated the pass, the file would fall +// below the cutoff and be dropped before full fingerprinting ever saw the +// update. Without a live capture the effective mtime must resolve the live +// composite instead. +func TestDiscoveredFileWatermarkCutoffRequiresLiveCapture(t *testing.T) { + dbPath, conn := newCompositeContainerTestDB(t) + const sessionRow = int64(1779012000000) + const childWrite = int64(1779012500000) + _, err := conn.Exec( + "INSERT INTO session (id, project_id, time_created, time_updated)"+ + " VALUES ('ses-1', 'proj', ?, ?)", + sessionRow, sessionRow, + ) + require.NoError(t, err, "insert session row") + _, err = conn.Exec( + "INSERT INTO message (id, session_id, data, time_created, time_updated)"+ + " VALUES ('msg-1', 'ses-1', '{}', ?, ?)", + childWrite, childWrite, + ) + require.NoError(t, err, "insert message row") + + root := filepath.Dir(dbPath) + provider, ok := parser.NewProvider( + parser.AgentOpenCode, + parser.ProviderConfig{Roots: []string{root}, Machine: "local"}, + ) + require.True(t, ok) + sources, err := provider.SourcesForChangedPath( + t.Context(), parser.ChangedPathRequest{ + Path: dbPath, WatchRoot: root, AllowWatermarkOnlySources: true, + }, + ) + require.NoError(t, err) + require.Len(t, sources, 1) + carried, watermarkOnly := parser.SourceWatermarkOnlyMTimeNS(sources[0]) + require.True(t, watermarkOnly) + require.Equal(t, sessionRow*1_000_000, carried, + "the carried watermark must be the session row alone") + + engine := NewEngine(openTestDB(t), EngineConfig{ + AgentDirs: map[parser.AgentType][]string{ + parser.AgentOpenCode: {root}, + }, + Machine: "local", + }) + t.Cleanup(engine.Close) + file := parser.DiscoveredFile{ + Agent: parser.AgentOpenCode, + Path: sources[0].DisplayPath, + ProviderSource: &sources[0], + ProviderProcess: true, + } + + // No live capture: the stale carried watermark cannot decide the + // cutoff, so the live composite (dominated by the child write) decides. + mtime, err := engine.discoveredFileEffectiveMtime(t.Context(), file) + require.NoError(t, err) + assert.Equal(t, childWrite*1_000_000, mtime, + "without a live capture the effective mtime is the live composite") + + // With a live, matching capture the carried watermark is trusted. + pre, ok := statSQLiteContainerState(dbPath) + require.True(t, ok) + engine.beginSQLiteContainerPass( + []parser.DiscoveredFile{file}, + map[string]parser.SQLiteContainerState{dbPath: pre}, + ) + mtime, err = engine.discoveredFileEffectiveMtime(t.Context(), file) + require.NoError(t, err) + assert.Equal(t, carried, mtime, + "a live capture lets the carried watermark decide the cutoff") +} + // TestSQLiteContainerPassPromotesOnlyPreDiscoveryCaptures pins the gate's // ordering invariant: the state promoted to trusted must have been captured // BEFORE discovery listed the container's sessions. Discovery reads the diff --git a/internal/sync/opencode_container_perf_test.go b/internal/sync/opencode_container_perf_test.go new file mode 100644 index 000000000..261dfd67f --- /dev/null +++ b/internal/sync/opencode_container_perf_test.go @@ -0,0 +1,571 @@ +package sync_test + +import ( + "context" + "fmt" + "testing" + + "go.kenn.io/agentsview/internal/parser" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +// TestOpenCodeSharedContainerChangeIsPerSessionBounded pins the "background +// sync work is bounded by the changed batch, not total archive size" rule for +// shared SQLite containers. +// +// Every session in an OpenCode root lives in one physical opencode.db. Stamping +// that container's size onto each session's fingerprint made any single +// session's write change every other session's fingerprint, so one changed +// session re-parsed the whole root — on a production container that is +// thousands of sessions re-read out of a multi-GB database every time the +// watcher fires. The per-session composite mtime (session, project, and child +// message/part time_updated) replaces it, so a one-session change must leave +// every other session skipped regardless of how many there are. +func TestOpenCodeSharedContainerChangeIsPerSessionBounded(t *testing.T) { + if testing.Short() { + t.Skip("skipping integration test") + } + + rewritten := make(map[int]int) + for _, n := range []int{20, 200} { + t.Run(fmt.Sprintf("sessions_%d", n), func(t *testing.T) { + env := setupSingleAgentTestEnv(t, parser.AgentOpenCode) + oc := createOpenCodeDB(t, env.opencodeDir) + oc.addProject(t, "proj", "/home/user/code/app") + for i := range n { + seedOpenCodeSQLiteTextSession( + t, oc, "proj", fmt.Sprintf("ses%05d", i), + 1779012000000, 1779012030000, + "prompt", "answer", + ) + } + require.Equal(t, n, + env.engine.SyncAll(context.Background(), nil).Synced) + + // Change exactly one session. This also grows the shared + // container file, which is precisely the signal that used to + // invalidate every other session in it. + oc.updateSessionTime(t, "ses00000", 1779015630000) + oc.replaceTextContent( + t, "ses00000", "changed prompt", "changed answer", + 1779015600000, + ) + + stats := env.engine.SyncAll(context.Background(), nil) + require.False(t, stats.Aborted, "sync aborted: %+v", stats) + assert.Equal(t, 1, stats.Synced, + "only the changed session may be rewritten") + assert.Equal(t, n-1, stats.Skipped, + "every unchanged session in the shared container must skip") + rewritten[n] = stats.Synced + }) + } + + assert.Equal(t, rewritten[20], rewritten[200], + "sessions rewritten for one changed session must not grow with "+ + "container size") +} + +// TestOpenCodeWatcherEventIsWatermarkBounded pins the same rule for the +// watcher's changed-path pass, one level deeper: a one-session write must +// not read the container's child tables at all, and must not even +// materialize the unchanged sessions. Changed-path classification lists +// candidates through the bounded session-row watermark (no message/part +// aggregation) filtered by the container's newest stored watermark, so the +// sources processed and the child rows examined per event both scale with +// the changed batch and not with the archive. +func TestOpenCodeWatcherEventIsWatermarkBounded(t *testing.T) { + if testing.Short() { + t.Skip("skipping integration test") + } + + lookups := make(map[int]int64) + processed := make(map[int]int) + for _, n := range []int{20, 200} { + t.Run(fmt.Sprintf("sessions_%d", n), func(t *testing.T) { + env := setupSingleAgentTestEnv(t, parser.AgentOpenCode) + oc := createOpenCodeDB(t, env.opencodeDir) + oc.addProject(t, "proj", "/home/user/code/app") + for i := range n { + seedOpenCodeSQLiteTextSession( + t, oc, "proj", fmt.Sprintf("ses%05d", i), + 1779012000000, 1779012030000, + "prompt", "answer", + ) + } + require.Equal(t, n, + env.engine.SyncAll(context.Background(), nil).Synced) + + oc.updateSessionTime(t, "ses00000", 1779015630000) + oc.replaceTextContent( + t, "ses00000", "changed prompt", "changed answer", + 1779015600000, + ) + + scansBefore := parser.OpenCodeContainerChildScans() + lookupsBefore := parser.OpenCodeSessionChildLookups() + require.NoError(t, env.engine.SyncPathsContext( + context.Background(), []string{oc.path}, + )) + stats := env.engine.LastSyncStats() + assert.Equal(t, 1, stats.Synced, + "only the changed session may be rewritten") + assert.Zero(t, stats.Skipped, + "unchanged sessions must not even be materialized as sources") + assert.Zero(t, + parser.OpenCodeContainerChildScans()-scansBefore, + "a watcher event must not aggregate the whole container's "+ + "child tables") + lookups[n] = parser.OpenCodeSessionChildLookups() - lookupsBefore + processed[n] = stats.Synced + stats.Skipped + stats.Failed + assertMessageContent( + t, env.db, "opencode:ses00000", + "changed prompt", "changed answer", + ) + }) + } + + assert.Equal(t, lookups[20], lookups[200], + "per-session child lookups for one changed session must not grow "+ + "with container size") + assert.Equal(t, processed[20], processed[200], + "sources processed for one changed session must not grow with "+ + "container size") +} + +// TestOpenCodeWatcherPassDefersChildOnlyEditToFullDiscovery documents the +// staleness contract the watermark-only watcher pass trades on: a child-only +// write that leaves the session and project rows untouched is invisible to +// the session-row watermark — wherever its timestamps land relative to the +// stored composite — and stays archived as-is until the next full-discovery +// pass, whose child digest still reconciles it. Both variants are pinned +// here: a replacement below the stored composite and an append above it. +// Actively watched sessions do not rely on this path; the per-session +// watcher poll resolves the composite directly. +func TestOpenCodeWatcherPassDefersChildOnlyEditToFullDiscovery(t *testing.T) { + env := setupSingleAgentTestEnv(t, parser.AgentOpenCode) + oc := createOpenCodeDB(t, env.opencodeDir) + oc.addProject(t, "proj", "/home/user/code/app") + // Session row far ahead of every child, so the replacement below stays + // under the stored composite. + seedOpenCodeSQLiteTextSession( + t, oc, "proj", "below-mark", + 1779012000000, 1779099999000, + "original prompt", "original answer", + ) + require.Equal(t, 1, env.engine.SyncAll(context.Background(), nil).Synced) + + // Child-only replacement: same counts, new rows and content, timestamps + // below the session row's watermark, session and project rows untouched. + oc.replaceTextContent( + t, "below-mark", "swapped prompt", "swapped answer", 1779012500000, + ) + + scansBefore := parser.OpenCodeContainerChildScans() + lookupsBefore := parser.OpenCodeSessionChildLookups() + require.NoError(t, env.engine.SyncPathsContext( + context.Background(), []string{oc.path}, + )) + assert.Zero(t, parser.OpenCodeContainerChildScans()-scansBefore, + "the watcher pass must not scan child tables for a child-only edit") + assert.Zero(t, parser.OpenCodeSessionChildLookups()-lookupsBefore, + "a child-only edit below the watermark yields no candidates") + assertMessageContent( + t, env.db, "opencode:below-mark", + "original prompt", "original answer", + ) + + fullStats := env.engine.SyncAll(context.Background(), nil) + assert.Equal(t, 1, fullStats.Synced, + "full discovery must reconcile the deferred child-only edit") + assertMessageContent( + t, env.db, "opencode:below-mark", + "swapped prompt", "swapped answer", + ) + + // Same deferral when the child write lands ABOVE the stored composite: + // a new message appended with a fresh timestamp while the session row + // stays untouched still cannot move the session-row watermark. + oc.addMessage( + t, "below-mark-msg-late", "below-mark", "assistant", 1779200000000, + ) + oc.addTextPart( + t, "below-mark-part-late", "below-mark", "below-mark-msg-late", + "late answer", 1779200000000, + ) + + scansBefore = parser.OpenCodeContainerChildScans() + lookupsBefore = parser.OpenCodeSessionChildLookups() + require.NoError(t, env.engine.SyncPathsContext( + context.Background(), []string{oc.path}, + )) + assert.Zero(t, parser.OpenCodeContainerChildScans()-scansBefore, + "the watcher pass must not scan child tables for an above-composite "+ + "child append") + assert.Zero(t, parser.OpenCodeSessionChildLookups()-lookupsBefore, + "an above-composite child-only append yields no candidates") + assertMessageContent( + t, env.db, "opencode:below-mark", + "swapped prompt", "swapped answer", + ) + + fullStats = env.engine.SyncAll(context.Background(), nil) + assert.Equal(t, 1, fullStats.Synced, + "full discovery must reconcile the deferred above-composite append") + assertMessageContent( + t, env.db, "opencode:below-mark", + "swapped prompt", "swapped answer", "late answer", + ) +} + +// TestOpenCodeFullPassSkipsAfterWatcherPassParse pins that a session parsed +// through the watermark-only watcher pass stores the full composite +// watermark and digest, not the cheap session-row watermark it was +// discovered with. The children deliberately end above the session row so +// the two values differ; if the cheap watermark leaked into the stored +// fingerprint, the next full pass would see a mismatch and re-parse an +// unchanged session with a fresh child lookup. +func TestOpenCodeFullPassSkipsAfterWatcherPassParse(t *testing.T) { + env := setupSingleAgentTestEnv(t, parser.AgentOpenCode) + oc := createOpenCodeDB(t, env.opencodeDir) + oc.addProject(t, "proj", "/home/user/code/app") + for i := range 3 { + seedOpenCodeSQLiteTextSession( + t, oc, "proj", fmt.Sprintf("ses%05d", i), + 1779012000000, 1779012030000, + "prompt", "answer", + ) + } + require.Equal(t, 3, env.engine.SyncAll(context.Background(), nil).Synced) + + oc.updateSessionTime(t, "ses00000", 1779015630000) + oc.replaceTextContent( + t, "ses00000", "changed prompt", "changed answer", 1779015600000, + ) + oc.mustExec(t, "raise children above the session row", + "UPDATE part SET time_updated = ? WHERE session_id = ?", + 1779099999000, "ses00000") + + require.NoError(t, env.engine.SyncPathsContext( + context.Background(), []string{oc.path}, + )) + require.Equal(t, 1, env.engine.LastSyncStats().Synced) + + lookupsBefore := parser.OpenCodeSessionChildLookups() + stats := env.engine.SyncAll(context.Background(), nil) + assert.Zero(t, stats.Synced, + "the full pass must not rewrite sessions the watcher pass stored") + assert.Equal(t, 3, stats.Skipped) + assert.Zero(t, parser.OpenCodeSessionChildLookups()-lookupsBefore, + "full-pass skips must not pay per-session child lookups") +} + +// TestOpenCodeWatcherCatchesMetadataUpdateUnderChildDominatedComposite pins +// the like-for-like watermark comparison. The stored composite is a MAX over +// session, project, and child times, so when a child timestamp dominates it, +// a later metadata update (title, session/project time) can advance the +// session row while staying below the composite. Comparing the session-row +// watermark against the composite would wrongly skip that session on the +// watcher pass; comparing against the stored session/project metadata +// watermark recovered from the persisted digest catches it — still without +// touching the container's child tables. +func TestOpenCodeWatcherCatchesMetadataUpdateUnderChildDominatedComposite( + t *testing.T, +) { + env := setupSingleAgentTestEnv(t, parser.AgentOpenCode) + oc := createOpenCodeDB(t, env.opencodeDir) + oc.addProject(t, "proj", "/home/user/code/app") + seedOpenCodeSQLiteTextSession( + t, oc, "proj", "meta-mark", + 1779012000000, 1779012030000, + "prompt", "answer", + ) + // Children exceed both the previous and the soon-to-advance metadata + // timestamps, so the stored composite is child-dominated. + oc.mustExec(t, "raise children above all metadata times", + "UPDATE part SET time_updated = ? WHERE session_id = ?", + 1779099999000, "meta-mark") + require.Equal(t, 1, env.engine.SyncAll(context.Background(), nil).Synced) + + // Metadata advances past its own stored value but stays below the + // child-dominated composite. + oc.mustExec(t, "retitle session below the composite", + "UPDATE session SET title = ?, time_updated = ? WHERE id = ?", + "renamed by watcher", 1779012040000, "meta-mark") + + scansBefore := parser.OpenCodeContainerChildScans() + require.NoError(t, env.engine.SyncPathsContext( + context.Background(), []string{oc.path}, + )) + stats := env.engine.LastSyncStats() + assert.Equal(t, 1, stats.Synced, + "a metadata update below the child-dominated composite must "+ + "re-parse on the watcher pass") + assert.Zero(t, parser.OpenCodeContainerChildScans()-scansBefore, + "the watcher pass must still not scan the container's child tables") + + // OpenCode's LLM-generated title lands in first_message. + var firstMessage string + require.NoError(t, env.db.Reader().QueryRow( + "SELECT first_message FROM sessions WHERE id = ?", + "opencode:meta-mark", + ).Scan(&firstMessage)) + assert.Equal(t, "renamed by watcher", firstMessage, + "the watcher pass must archive the metadata update") +} + +// TestOpenCodeIdleReconcilePassSkipsContainerChildScan pins the same +// trusted-container bound on the streamed reconciliation path: an idle +// ReconcileWatchRoots pass over a trusted, untouched container must not +// aggregate the child tables (its candidates all gate-skip), while any +// write breaks trust and the next reconcile carries the full digest again — +// including for a child-only edit below every watermark. +func TestOpenCodeIdleReconcilePassSkipsContainerChildScan(t *testing.T) { + env := setupSingleAgentTestEnv(t, parser.AgentOpenCode) + oc := createOpenCodeDB(t, env.opencodeDir) + oc.addProject(t, "proj", "/home/user/code/app") + for i := range 5 { + seedOpenCodeSQLiteTextSession( + t, oc, "proj", fmt.Sprintf("ses%05d", i), + 1779012000000, 1779099999000, + "prompt", "answer", + ) + } + require.Equal(t, 5, env.engine.SyncAll(context.Background(), nil).Synced) + + scansBefore := parser.OpenCodeContainerChildScans() + lookupsBefore := parser.OpenCodeSessionChildLookups() + require.NoError(t, env.engine.ReconcileWatchRoots( + context.Background(), []string{env.opencodeDir}, false, + )) + assert.Zero(t, parser.OpenCodeContainerChildScans()-scansBefore, + "an idle reconcile pass must not aggregate the container's child "+ + "tables") + assert.Zero(t, parser.OpenCodeSessionChildLookups()-lookupsBefore, + "an idle reconcile pass must not pay per-session child lookups") + assertMessageContent( + t, env.db, "opencode:ses00000", "prompt", "answer", + ) + + // A child-only replacement below every watermark breaks trust via the + // container state, and the next reconcile carries the digest again. + oc.replaceTextContent( + t, "ses00000", "swapped prompt", "swapped answer", 1779012500000, + ) + require.NoError(t, env.engine.ReconcileWatchRoots( + context.Background(), []string{env.opencodeDir}, false, + )) + assertMessageContent( + t, env.db, "opencode:ses00000", + "swapped prompt", "swapped answer", + ) +} + +// TestOpenCodeIdleFullPassSkipsContainerChildScan pins that a periodic full +// pass over a trusted, untouched container does not aggregate the child +// tables at all: the container gate will skip every member before +// fingerprinting, so discovery lists the bounded watermark form instead of +// computing archive-sized child identities nothing reads. Any write breaks +// container trust, and the next full pass carries the complete digest again +// — including for child-only edits below every watermark. +func TestOpenCodeIdleFullPassSkipsContainerChildScan(t *testing.T) { + env := setupSingleAgentTestEnv(t, parser.AgentOpenCode) + oc := createOpenCodeDB(t, env.opencodeDir) + oc.addProject(t, "proj", "/home/user/code/app") + for i := range 5 { + // Session rows far ahead of every child, so the later child-only + // replacement stays below the stored composite. + seedOpenCodeSQLiteTextSession( + t, oc, "proj", fmt.Sprintf("ses%05d", i), + 1779012000000, 1779099999000, + "prompt", "answer", + ) + } + require.Equal(t, 5, env.engine.SyncAll(context.Background(), nil).Synced) + + scansBefore := parser.OpenCodeContainerChildScans() + lookupsBefore := parser.OpenCodeSessionChildLookups() + stats := env.engine.SyncAll(context.Background(), nil) + assert.Zero(t, stats.Synced) + assert.Equal(t, 5, stats.Skipped, + "every session of a trusted container must gate-skip") + assert.Zero(t, parser.OpenCodeContainerChildScans()-scansBefore, + "an idle full pass must not aggregate the container's child tables") + assert.Zero(t, parser.OpenCodeSessionChildLookups()-lookupsBefore, + "an idle full pass must not pay per-session child lookups") + + // A child-only replacement below every watermark breaks trust via the + // container state, and the next full pass carries the digest again. + oc.replaceTextContent( + t, "ses00000", "swapped prompt", "swapped answer", 1779012500000, + ) + stats = env.engine.SyncAll(context.Background(), nil) + assert.Equal(t, 1, stats.Synced, + "a write breaks container trust and full discovery reconciles it") + assertMessageContent( + t, env.db, "opencode:ses00000", + "swapped prompt", "swapped answer", + ) +} + +// TestOpenCodeDeletedChildIsDetected pins deletion sensitivity. The composite +// mtime is a MAX over session/project/child timestamps, so when the session or +// project row already holds the higher value — the common case on a real +// container — deleting a message or part does not move the max at all. Without +// a deletion-sensitive component the session looks fresh and the removed +// content stays archived indefinitely. +func TestOpenCodeDeletedChildIsDetected(t *testing.T) { + env := setupSingleAgentTestEnv(t, parser.AgentOpenCode) + oc := createOpenCodeDB(t, env.opencodeDir) + oc.addProject(t, "proj", "/home/user/code/app") + // Session row timestamp is deliberately far ahead of every child, so a + // deleted child cannot lower the composite. + seedOpenCodeSQLiteTextSession( + t, oc, "proj", "del-session", + 1779012000000, 1779099999000, + "keep prompt", "drop answer", + ) + + stats := env.engine.SyncAll(context.Background(), nil) + require.False(t, stats.Aborted) + require.Equal(t, 1, stats.Synced) + assertMessageContent( + t, env.db, "opencode:del-session", "keep prompt", "drop answer", + ) + + // Remove the assistant message and its parts, leaving session and project + // timestamps untouched. + oc.mustExec(t, "delete assistant parts", + "DELETE FROM part WHERE session_id = ? AND message_id LIKE ?", + "del-session", "%assistant%") + oc.mustExec(t, "delete assistant message", + "DELETE FROM message WHERE session_id = ? AND id LIKE ?", + "del-session", "%assistant%") + + stats = env.engine.SyncAll(context.Background(), nil) + require.False(t, stats.Aborted) + assert.Equal(t, 1, stats.Synced, + "a deleted child must not be hidden behind an unchanged composite max") +} + +// TestOpenCodeDeletedChildDetectedViaReconciliation covers the same deletion +// hole on the reconciliation path. Sources rebuilt by FindSource rather than +// carried from discovery metadata have no child digest, so the fingerprint hash +// is empty and the freshness gate treats it as no constraint. +func TestOpenCodeDeletedChildDetectedViaReconciliation(t *testing.T) { + env := setupSingleAgentTestEnv(t, parser.AgentOpenCode) + oc := createOpenCodeDB(t, env.opencodeDir) + oc.addProject(t, "proj", "/home/user/code/app") + seedOpenCodeSQLiteTextSession( + t, oc, "proj", "recon-del", + 1779012000000, 1779099999000, + "keep prompt", "drop answer", + ) + require.Equal(t, 1, env.engine.SyncAll(context.Background(), nil).Synced) + + oc.mustExec(t, "delete assistant parts", + "DELETE FROM part WHERE session_id = ? AND message_id LIKE ?", + "recon-del", "%assistant%") + oc.mustExec(t, "delete assistant message", + "DELETE FROM message WHERE session_id = ? AND id LIKE ?", + "recon-del", "%assistant%") + + require.NoError(t, env.engine.ReconcileWatchRoots( + context.Background(), []string{env.opencodeDir}, false, + )) + env.engine.SyncAll(context.Background(), nil) + + // Assert the observable outcome rather than which pass did the write: + // the removed assistant turn must no longer be archived. + for _, m := range fetchMessages(t, env.db, "opencode:recon-del") { + assert.NotContains(t, m.Content, "drop answer", + "deleted child content must not remain archived") + } +} + +// TestOpenCodeSameCountChildReplacementIsDetected covers a replacement that +// preserves both child counts and leaves every new timestamp below the session +// row's already-higher watermark, so neither the watermark nor the counts move. +func TestOpenCodeSameCountChildReplacementIsDetected(t *testing.T) { + env := setupSingleAgentTestEnv(t, parser.AgentOpenCode) + oc := createOpenCodeDB(t, env.opencodeDir) + oc.addProject(t, "proj", "/home/user/code/app") + seedOpenCodeSQLiteTextSession( + t, oc, "proj", "swap-session", + 1779012000000, 1779099999000, + "original prompt", "original answer", + ) + require.Equal(t, 1, env.engine.SyncAll(context.Background(), nil).Synced) + + // Same number of messages and parts, timestamps still below the session + // row's watermark, but different rows and different content. + oc.replaceTextContent( + t, "swap-session", "swapped prompt", "swapped answer", 1779012500000, + ) + + stats := env.engine.SyncAll(context.Background(), nil) + assert.Equal(t, 1, stats.Synced, + "a same-count child replacement below the session watermark must "+ + "still change the fingerprint") +} + +// TestOpenCodeMetadataUpdateBelowWatermarkIsDetected covers a project worktree +// rename whose timestamp lands below an already-higher child watermark. The +// composite MAX cannot move in that case, so the digest has to carry the +// session and project timestamps in their own right. +func TestOpenCodeMetadataUpdateBelowWatermarkIsDetected(t *testing.T) { + env := setupSingleAgentTestEnv(t, parser.AgentOpenCode) + oc := createOpenCodeDB(t, env.opencodeDir) + oc.addProject(t, "proj", "/home/user/code/original-app") + // Children hold the highest timestamp, so a later project rename below + // that value leaves MAX(...) unchanged. + seedOpenCodeSQLiteTextSession( + t, oc, "proj", "below-watermark", + 1779012000000, 1779012030000, + "stable prompt", "stable answer", + ) + oc.mustExec(t, "raise child watermark", + "UPDATE part SET time_updated = ? WHERE session_id = ?", + 1779099999000, "below-watermark") + require.Equal(t, 1, env.engine.SyncAll(context.Background(), nil).Synced) + + // Rename below the child watermark. + oc.updateProjectWorktree( + t, "proj", "/home/user/code/renamed-app", 1779013000000, + ) + + stats := env.engine.SyncAll(context.Background(), nil) + assert.Equal(t, 1, stats.Synced, + "a metadata update below the child watermark must still be detected") +} + +// TestOpenCodeMiddleRowReplacementIsDetected covers a replacement that keeps +// every aggregate the digest currently reduces to: same counts, same timestamp +// sums, and the same min/max ids because the swapped row sorts strictly between +// the extrema. Only a complete child identity can tell these apart. +func TestOpenCodeMiddleRowReplacementIsDetected(t *testing.T) { + env := setupSingleAgentTestEnv(t, parser.AgentOpenCode) + oc := createOpenCodeDB(t, env.opencodeDir) + oc.addProject(t, "proj", "/home/user/code/app") + oc.addSession(t, "mid", "proj", 1779012000000, 1779099999000) + oc.addMessage(t, "mid-msg-a", "mid", "user", 1779012000000) + // Three parts: a, m, z. The middle one gets swapped for a different id + // carrying an identical timestamp, so count, sum and extrema all hold. + oc.addTextPart(t, "mid-part-a", "mid", "mid-msg-a", "alpha", 1779012000000) + oc.addTextPart(t, "mid-part-m", "mid", "mid-msg-a", "middle", 1779012000001) + oc.addTextPart(t, "mid-part-z", "mid", "mid-msg-a", "zulu", 1779012000002) + require.Equal(t, 1, env.engine.SyncAll(context.Background(), nil).Synced) + + oc.mustExec(t, "delete middle part", + "DELETE FROM part WHERE id = ?", "mid-part-m") + oc.addTextPart( + t, "mid-part-n", "mid", "mid-msg-a", "replaced", 1779012000001, + ) + + stats := env.engine.SyncAll(context.Background(), nil) + assert.Equal(t, 1, stats.Synced, + "a middle-row replacement preserving counts, sums and extrema must "+ + "still change the fingerprint") +} diff --git a/internal/sync/perf_invariant_test.go b/internal/sync/perf_invariant_test.go index 1e8760a44..a4de5d11a 100644 --- a/internal/sync/perf_invariant_test.go +++ b/internal/sync/perf_invariant_test.go @@ -385,3 +385,83 @@ func TestRebuildLocalAndRemoteContributorsBulkWriteDiscoveredCount(t *testing.T) assert.Equal(t, large.localWrites, large.remoteWrites, "large equivalent contributors must retain equal work counts") } + +// TestWarmFullSyncDoesNotRehashClaudeArchive pins the "background sync work is +// bounded by the changed batch, not total archive size" rule against the Claude +// DB-freshness gate. +// +// providerSingleSessionFresh runs at the top of every provider process pass. Its +// last guard, providerIncrementalContentChanged, defends against a same-size, +// same-mtime, same-inode in-place rewrite by content-hashing the source, and +// that guard reads the whole stored prefix. Today the verified-source gate +// (verifiedProviderSourceState) absorbs it: a signature is content-verified once +// to earn trust, and every later pass over an unchanged source rides the trusted +// skip without reading bytes. +// +// Nothing else pins that. Losing the trusted skip — by dropping Claude's +// VerifiedLocalStat capability, widening the signature so it never repeats, or +// pruning trust between passes — would silently turn every watcher-triggered +// pass into a full re-read of the entire Claude archive, which is invisible in +// correctness tests and only shows up as daemon CPU on a large archive. Assert +// the steady-state pass reads no content and does not scale with archive size. +func TestWarmFullSyncDoesNotRehashClaudeArchive(t *testing.T) { + if testing.Short() { + t.Skip("skipping integration test") + } + + observed := make(map[int]int) + for _, claudeCount := range []int{5, 50} { + t.Run(fmt.Sprintf("claude_%d", claudeCount), func(t *testing.T) { + claudeDir := filepath.Join(t.TempDir(), "claude") + require.NoError(t, os.MkdirAll(claudeDir, 0o755)) + writeClaudeCorpus(t, claudeDir, claudeCount) + + database := openTestDB(t) + engine := NewEngine(database, EngineConfig{ + AgentDirs: map[parser.AgentType][]string{ + parser.AgentClaude: {claudeDir}, + }, + Machine: "local", + }) + t.Cleanup(engine.Close) + + // Cold pass: populates the archive and earns verified-source trust. + require.Equal(t, claudeCount, + engine.SyncAll(context.Background(), nil).Synced) + + var mu gosync.Mutex + reads := 0 + restore := computeFileHashPrefix + computeFileHashPrefix = func(path string, size int64) (string, error) { + mu.Lock() + reads++ + mu.Unlock() + return restore(path, size) + } + t.Cleanup(func() { computeFileHashPrefix = restore }) + + // Each pass leaves the archive untouched on disk. The first warm + // pass may content-verify once to earn verified-source trust; every + // pass after that must ride the trusted-signature skip. + pass := func() int { + mu.Lock() + reads = 0 + mu.Unlock() + engine.SyncAll(context.Background(), nil) + mu.Lock() + defer mu.Unlock() + return reads + } + trustEarning := pass() + steady := pass() + t.Logf("source content reads: trust-earning pass=%d steady pass=%d", + trustEarning, steady) + observed[claudeCount] = steady + }) + } + + assert.Equal(t, observed[5], observed[50], + "warm-pass source reads must not grow with archive size") + assert.Zero(t, observed[50], + "a warm pass over an unchanged archive must not re-read source content") +} diff --git a/internal/sync/provider_sync_semantics_test.go b/internal/sync/provider_sync_semantics_test.go index 0f7240f89..2a48b6480 100644 --- a/internal/sync/provider_sync_semantics_test.go +++ b/internal/sync/provider_sync_semantics_test.go @@ -30,8 +30,22 @@ func (f semanticTestFactory) Capabilities() parser.Capabilities { return f.provider.Capabilities() } -func (f semanticTestFactory) NewProvider(parser.ProviderConfig) parser.Provider { - return f.provider +type semanticTestScopedProvider struct { + *semanticTestProvider + scopes parser.ProviderBase +} + +func (p semanticTestScopedProvider) ResolveReconciliationScopes( + ctx context.Context, req parser.ReconciliationScopeRequest, +) (parser.ReconciliationScopePlan, error) { + return p.scopes.ResolveReconciliationScopes(ctx, req) +} + +func (f semanticTestFactory) NewProvider(cfg parser.ProviderConfig) parser.Provider { + return semanticTestScopedProvider{ + semanticTestProvider: f.provider, + scopes: perCallScopeProviderBase(f.provider.ProviderBase, cfg), + } } type semanticTestProvider struct { diff --git a/internal/sync/reconcile_provider_roots_test.go b/internal/sync/reconcile_provider_roots_test.go index 630da7448..8599a86c0 100644 --- a/internal/sync/reconcile_provider_roots_test.go +++ b/internal/sync/reconcile_provider_roots_test.go @@ -79,8 +79,8 @@ func TestReconcileProviderRootsDoesNotExpandAcrossProviders(t *testing.T) { } base := t.TempDir() aiderRoot := filepath.Join(base, "aider") - // claudeDir is a descendant of aiderRoot: the overlap that - // logicalRootsForWatchRoots would otherwise expand across providers. + // claudeDir is a descendant of aiderRoot: the overlap an unscoped root + // expansion would otherwise widen across providers. claudeDir := filepath.Join(aiderRoot, "claude") require.NoError(t, os.MkdirAll(claudeDir, 0o755)) diff --git a/internal/sync/reconciliation_scoping_opencode_test.go b/internal/sync/reconciliation_scoping_opencode_test.go new file mode 100644 index 000000000..b120748a0 --- /dev/null +++ b/internal/sync/reconciliation_scoping_opencode_test.go @@ -0,0 +1,110 @@ +package sync_test + +import ( + "context" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "go.kenn.io/agentsview/internal/parser" + "go.kenn.io/agentsview/internal/sync" +) + +func seedOpenCodeContainerSessions( + t *testing.T, oc *openCodeTestDB, base int64, ids ...string, +) { + t.Helper() + oc.addProject(t, "proj-1", "/home/user/code/myapp") + for _, id := range ids { + oc.addSession(t, id, "proj-1", base, base+5000) + oc.addMessage(t, id+"-msg-u", id, "user", base) + oc.addMessage(t, id+"-msg-a", id, "assistant", base+1) + oc.addTextPart(t, id+"-part-u", id, id+"-msg-u", + "original question "+id, base) + oc.addTextPart(t, id+"-part-a", id, id+"-msg-a", + "original answer "+id, base+1) + } +} + +// TestReconcileProviderRootsOpenCodeContainerSyncsAndTombstonesMembers pins +// the exact-container request: a pass asked about opencode.db itself must +// prove the container's whole virtual membership, syncing a changed member +// and reclaiming a removed one, instead of resolving a proof no member row +// can match. +func TestReconcileProviderRootsOpenCodeContainerSyncsAndTombstonesMembers( + t *testing.T, +) { + env := setupSingleAgentTestEnv(t, parser.AgentOpenCode) + oc := createOpenCodeDB(t, env.opencodeDir) + base := int64(1704067200000) + seedOpenCodeContainerSessions( + t, oc, base, "oc-container-kept", "oc-container-removed", + ) + runSyncAndAssert(t, env.engine, sync.SyncStats{ + TotalSessions: 2, Synced: 2, + }) + + // One member changes and one disappears; the pass is asked about the + // container path, not the configured root. + oc.replaceTextContent(t, "oc-container-kept", + "updated question", "updated answer", base) + oc.updateSessionTime(t, "oc-container-kept", base+9000) + oc.deleteParts(t, "oc-container-removed") + oc.deleteMessages(t, "oc-container-removed") + oc.mustExec(t, "delete session", + "DELETE FROM session WHERE id = ?", "oc-container-removed") + + require.NoError(t, env.engine.ReconcileProviderRoots( + t.Context(), parser.AgentOpenCode, []string{oc.path}, + )) + + assertMessageContent(t, env.db, "opencode:oc-container-kept", + "updated question", "updated answer") + removed, err := env.db.GetSession( + context.Background(), "opencode:oc-container-removed", + ) + require.NoError(t, err) + assert.Nil(t, removed, + "a container-scoped pass reclaims a removed member") +} + +// TestReconcileProviderRootsOpenCodeMemberPassCannotTrustPartialMembership +// pins the trust-promotion invariant end to end: a pass asked about one +// virtual member widens to the whole container, so completing it never +// promotes container-state trust over a sibling it did not verify. Without +// the widening, the member pass records one discovered and one completed +// member, trusts the already-changed container state, and the covering pass +// gate-skips the stale sibling for as long as the database stays unchanged. +func TestReconcileProviderRootsOpenCodeMemberPassCannotTrustPartialMembership( + t *testing.T, +) { + env := setupSingleAgentTestEnv(t, parser.AgentOpenCode) + oc := createOpenCodeDB(t, env.opencodeDir) + base := int64(1704067200000) + seedOpenCodeContainerSessions(t, oc, base, "oc-member-a", "oc-member-b") + runSyncAndAssert(t, env.engine, sync.SyncStats{ + TotalSessions: 2, Synced: 2, + }) + + // Both members change; the pass is asked about member A only. + oc.replaceTextContent(t, "oc-member-a", + "updated question a", "updated answer a", base) + oc.updateSessionTime(t, "oc-member-a", base+9000) + oc.replaceTextContent(t, "oc-member-b", + "updated question b", "updated answer b", base) + oc.updateSessionTime(t, "oc-member-b", base+9000) + + require.NoError(t, env.engine.ReconcileProviderRoots( + t.Context(), parser.AgentOpenCode, + []string{parser.OpenCodeSQLiteVirtualPath(oc.path, "oc-member-a")}, + )) + + // The database does not change again between the passes, so any trust + // the member pass promoted is exactly what the covering pass consults. + require.NoError(t, env.engine.ReconcileProviderRoots( + t.Context(), parser.AgentOpenCode, []string{env.opencodeDir}, + )) + + assertMessageContent(t, env.db, "opencode:oc-member-b", + "updated question b", "updated answer b") +} diff --git a/internal/sync/reconciliation_scoping_test.go b/internal/sync/reconciliation_scoping_test.go new file mode 100644 index 000000000..185b29e39 --- /dev/null +++ b/internal/sync/reconciliation_scoping_test.go @@ -0,0 +1,739 @@ +package sync + +import ( + "context" + "errors" + "os" + "path/filepath" + "runtime" + "strings" + "sync/atomic" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "go.kenn.io/agentsview/internal/db" + "go.kenn.io/agentsview/internal/parser" + "go.kenn.io/agentsview/internal/testjsonl" +) + +// writeClaudeProjectSession writes one minimal Claude session named +// .jsonl under /; the session ID is the file base name. +func writeClaudeProjectSession(t *testing.T, root, project, name string) string { + t.Helper() + path := filepath.Join(root, project, name+".jsonl") + require.NoError(t, os.MkdirAll(filepath.Dir(path), 0o755)) + require.NoError(t, os.WriteFile(path, []byte( + testjsonl.NewSessionBuilder(). + AddClaudeUser("2024-01-01T00:00:00Z", "hi "+name). + String(), + ), 0o644)) + return path +} + +// TestReconcileProviderRootsDescendantDoesNotClaimSiblingScope is the +// reproduction row: a pass asked about one project directory must not open or +// tombstone sessions under a sibling directory it never requested. +func TestReconcileProviderRootsDescendantDoesNotClaimSiblingScope(t *testing.T) { + if testing.Short() { + t.Skip("skipping integration test") + } + database := openTestDB(t) + claudeRoot := filepath.Join(t.TempDir(), "claude") + projectA := filepath.Join(claudeRoot, "projA") + projectB := filepath.Join(claudeRoot, "projB") + writeClaudeProjectSession(t, claudeRoot, "projA", "a1") + writeClaudeProjectSession(t, claudeRoot, "projA", "a2") + siblingDeleted := writeClaudeProjectSession(t, claudeRoot, "projB", "b1") + writeClaudeProjectSession(t, claudeRoot, "projB", "b2") + + engine := NewEngine(database, EngineConfig{ + AgentDirs: map[parser.AgentType][]string{ + parser.AgentClaude: {claudeRoot}, + }, + Machine: "local", + }) + t.Cleanup(engine.Close) + require.Equal(t, 4, engine.SyncAll(t.Context(), nil).Synced) + + // The sibling directory loses a source; only a pass with authority over + // projB may treat that as deletion proof. + require.NoError(t, os.Remove(siblingDeleted)) + rec := &lstatRecorder{} + engine.lstat = rec.stat + + require.NoError(t, engine.ReconcileProviderRoots( + t.Context(), parser.AgentClaude, []string{projectA}, + )) + + for _, id := range []string{"b1", "b2"} { + active, err := database.GetSession(t.Context(), id) + require.NoError(t, err) + assert.NotNil(t, active, + "a pass scoped to projA holds no tombstone authority over projB") + } + assert.Zero(t, rec.countUnder(projectB), + "a projA-scoped pass must not stat sibling sources") + assert.LessOrEqual(t, + engine.LastReconciliationResult().Metrics.MaxRehydratedSources, 2, + "rehydration must stay bounded by the requested scope") +} + +// TestReconcileProviderRootsClaudeDescendantUsesConfiguredTraversal verifies +// the provider-owned gateway: a requested project directory traverses from +// the configured projects root while admission stays inside the descendant. +func TestReconcileProviderRootsClaudeDescendantUsesConfiguredTraversal(t *testing.T) { + if testing.Short() { + t.Skip("skipping integration test") + } + database := openTestDB(t) + claudeRoot := filepath.Join(t.TempDir(), "claude") + projectA := filepath.Join(claudeRoot, "projA") + writeClaudeProjectSession(t, claudeRoot, "projA", "a1") + + engine := NewEngine(database, EngineConfig{ + AgentDirs: map[parser.AgentType][]string{ + parser.AgentClaude: {claudeRoot}, + }, + Machine: "local", + }) + t.Cleanup(engine.Close) + require.Equal(t, 1, engine.SyncAll(t.Context(), nil).Synced) + + writeClaudeProjectSession(t, claudeRoot, "projA", "new-a") + writeClaudeProjectSession(t, claudeRoot, "projB", "new-b") + + require.NoError(t, engine.ReconcileProviderRoots( + t.Context(), parser.AgentClaude, []string{projectA}, + )) + + admitted, err := database.GetSession(t.Context(), "new-a") + require.NoError(t, err) + assert.NotNil(t, admitted, + "traversal from the configured root must still discover the descendant") + sibling, err := database.GetSession(t.Context(), "new-b") + require.NoError(t, err) + assert.Nil(t, sibling, + "admission must stay bounded to the requested descendant") +} + +// TestReconcileProviderRootsProofBoundedTombstoneWithinDescendant verifies +// both halves of the tombstone guard in one pass: a missing source inside the +// descendant proof is tombstoned while an equally missing source outside the +// proof is neither paged nor touched. +func TestReconcileProviderRootsProofBoundedTombstoneWithinDescendant(t *testing.T) { + database := openTestDB(t) + const agent = parser.AgentType("scoped-descendant") + root := t.TempDir() + scoped := filepath.Join(root, "scoped") + other := filepath.Join(root, "other") + inProof := filepath.Join(scoped, "s-in.jsonl") + outOfProof := filepath.Join(other, "s-out.jsonl") + require.NoError(t, os.MkdirAll(scoped, 0o755)) + require.NoError(t, os.MkdirAll(other, 0o755)) + require.NoError(t, os.WriteFile(inProof, []byte("{}\n"), 0o600)) + require.NoError(t, os.WriteFile(outOfProof, []byte("{}\n"), 0o600)) + provider := newScopedStreamingProvider(agent) + provider.sourcesByRoot[root] = []parser.SourceRef{ + scopedTestSource(agent, inProof), + scopedTestSource(agent, outOfProof), + } + engine := NewEngine(database, EngineConfig{ + AgentDirs: map[parser.AgentType][]string{agent: {root}}, + Machine: "local", + ProviderFactories: []parser.ProviderFactory{ + scopedStreamingFactory{provider: provider}, + }, + ProviderMigrationModes: map[parser.AgentType]parser.ProviderMigrationMode{ + agent: parser.ProviderMigrationProviderAuthoritative, + }, + }) + t.Cleanup(engine.Close) + require.NoError(t, engine.ReconcileProviderRoots( + t.Context(), agent, []string{root}, + )) + + // Both sources vanish, but only the requested descendant grants proof. + require.NoError(t, os.Remove(inProof)) + require.NoError(t, os.Remove(outOfProof)) + provider.sourcesByRoot[root] = nil + rec := &lstatRecorder{} + engine.lstat = rec.stat + + require.NoError(t, engine.ReconcileProviderRoots( + t.Context(), agent, []string{scoped}, + )) + + deleted, err := database.GetSessionFull(t.Context(), "s-in") + require.NoError(t, err) + require.NotNil(t, deleted) + require.NotNil(t, deleted.DeletionCause) + assert.Equal(t, "source_missing", *deleted.DeletionCause, + "a missing source inside the proof is tombstoned in the same pass") + + survivor, err := database.GetSession(t.Context(), "s-out") + require.NoError(t, err) + assert.NotNil(t, survivor, + "a paged row outside the proof is retained even when its source is gone") + assert.Zero(t, rec.countUnder(other), + "ownership paging must stay inside the requested scope") +} + +// TestReconcileProviderRootsSharedGatewayTraversesOnce pins the traversal +// grouping: a request naming several descendants under one configured +// gateway walks the gateway once, admitting each descendant against its own +// proof from that single stream. +func TestReconcileProviderRootsSharedGatewayTraversesOnce(t *testing.T) { + database := openTestDB(t) + const agent = parser.AgentType("scoped-shared-gateway") + root := t.TempDir() + first := filepath.Join(root, "first") + second := filepath.Join(root, "second") + firstSource := filepath.Join(first, "s-first.jsonl") + secondSource := filepath.Join(second, "s-second.jsonl") + require.NoError(t, os.MkdirAll(first, 0o755)) + require.NoError(t, os.MkdirAll(second, 0o755)) + require.NoError(t, os.WriteFile(firstSource, []byte("{}\n"), 0o600)) + require.NoError(t, os.WriteFile(secondSource, []byte("{}\n"), 0o600)) + provider := newScopedStreamingProvider(agent) + provider.sourcesByRoot[root] = []parser.SourceRef{ + scopedTestSource(agent, firstSource), + scopedTestSource(agent, secondSource), + } + engine := NewEngine(database, EngineConfig{ + AgentDirs: map[parser.AgentType][]string{agent: {root}}, + Machine: "local", + ProviderFactories: []parser.ProviderFactory{ + scopedStreamingFactory{provider: provider}, + }, + ProviderMigrationModes: map[parser.AgentType]parser.ProviderMigrationMode{ + agent: parser.ProviderMigrationProviderAuthoritative, + }, + }) + t.Cleanup(engine.Close) + + require.NoError(t, engine.ReconcileProviderRoots( + t.Context(), agent, []string{first, second}, + )) + + assert.Equal(t, int32(1), provider.streamCalls.Load(), + "descendants sharing a gateway must share one discovery walk") + for _, id := range []string{"s-first", "s-second"} { + admitted, err := database.GetSession(t.Context(), id) + require.NoError(t, err) + assert.NotNil(t, admitted, + "each descendant is admitted against its own proof from the shared walk") + } +} + +// TestReconcileProviderRootsUnresolvedRootsAreBoundedNoOp covers the +// negative-space rows: blank, unrelated, remote, and empty requests complete +// before any spool allocation and never widen into other scopes. +func TestReconcileProviderRootsUnresolvedRootsAreBoundedNoOp(t *testing.T) { + if testing.Short() { + t.Skip("skipping integration test") + } + database := openTestDB(t) + claudeRoot := filepath.Join(t.TempDir(), "claude") + writeClaudeProjectSession(t, claudeRoot, "proj", "keep") + engine := NewEngine(database, EngineConfig{ + AgentDirs: map[parser.AgentType][]string{ + parser.AgentClaude: {claudeRoot}, + }, + Machine: "local", + }) + t.Cleanup(engine.Close) + require.Equal(t, 1, engine.SyncAll(t.Context(), nil).Synced) + + var spools atomic.Int32 + factory := engine.reconciliationSpoolFactory + engine.reconciliationSpoolFactory = func( + path string, + ) (reconciliationSpoolStore, error) { + spools.Add(1) + return factory(path) + } + + for _, tc := range []struct { + name string + roots []string + wantRemote int + }{ + {name: "blank", roots: []string{""}}, + {name: "unrelated", roots: []string{t.TempDir()}}, + {name: "remote", roots: []string{"s3://bucket/prefix"}, wantRemote: 1}, + {name: "empty", roots: nil}, + } { + t.Run(tc.name, func(t *testing.T) { + require.NoError(t, engine.ReconcileProviderRoots( + t.Context(), parser.AgentClaude, tc.roots, + )) + assert.Zero(t, spools.Load(), + "an unresolved request must complete before spool allocation") + result := engine.LastReconciliationResult() + assert.True(t, result.Complete) + assert.Equal(t, tc.wantRemote, result.Metrics.ExcludedRemoteRoots) + active, err := database.GetSession(t.Context(), "keep") + require.NoError(t, err) + assert.NotNil(t, active, + "an unresolved request holds no authority over stored sessions") + }) + } +} + +// TestReconcileProviderRootsCaseVariantRootAdmitsAsExact verifies platform +// path equality: on Windows a case-variant spelling of a configured root +// resolves to the exact configured scope instead of an empty pass. +func TestReconcileProviderRootsCaseVariantRootAdmitsAsExact(t *testing.T) { + if runtime.GOOS != "windows" { + t.Skip("case-variant admission is a Windows filesystem property") + } + if testing.Short() { + t.Skip("skipping integration test") + } + database := openTestDB(t) + claudeRoot := filepath.Join(t.TempDir(), "claude") + removed := writeClaudeProjectSession(t, claudeRoot, "proj", "old-session") + variant := strings.ToUpper(claudeRoot) + require.NotEqual(t, claudeRoot, variant) + + engine := NewEngine(database, EngineConfig{ + AgentDirs: map[parser.AgentType][]string{ + parser.AgentClaude: {claudeRoot}, + }, + Machine: "local", + }) + t.Cleanup(engine.Close) + require.Equal(t, 1, engine.SyncAll(t.Context(), nil).Synced) + + require.NoError(t, os.Remove(removed)) + writeClaudeProjectSession(t, claudeRoot, "proj", "new-session") + + require.NoError(t, engine.ReconcileProviderRoots( + t.Context(), parser.AgentClaude, []string{variant}, + )) + + admitted, err := database.GetSession(t.Context(), "new-session") + require.NoError(t, err) + assert.NotNil(t, admitted, + "a case-variant of the configured root must not produce empty discovery") + gone, err := database.GetSessionFull(t.Context(), "old-session") + require.NoError(t, err) + require.NotNil(t, gone) + require.NotNil(t, gone.DeletionCause) + assert.Equal(t, "source_missing", *gone.DeletionCause, + "a case-variant request carries the exact root's full authority") +} + +// scopedStreamingProvider is a multi-root fake whose discovery reflects the +// roots each construction binds, so a failure injected for one root fails +// only the scope the engine built for that root. +type scopedStreamingProvider struct { + parser.ProviderBase + sourcesByRoot map[string][]parser.SourceRef + failRoots map[string]bool + // findable maps full session IDs to the source FindSource resolves for + // them, standing in for a member that lives under another configured + // root than the one a scoped pass streamed. + findable map[string]parser.SourceRef + // streamCalls is pointer-shared across the per-call copies NewProvider + // returns, so tests observe every discovery walk on one counter. + streamCalls *atomic.Int32 +} + +func (p *scopedStreamingProvider) FindSource( + _ context.Context, req parser.FindSourceRequest, +) (parser.SourceRef, bool, error) { + if source, ok := p.findable[req.FullSessionID]; ok { + return source, true, nil + } + return parser.SourceRef{}, false, nil +} + +func (p *scopedStreamingProvider) DiscoverEach( + ctx context.Context, yield func(parser.SourceRef) error, +) error { + if err := ctx.Err(); err != nil { + return err + } + p.streamCalls.Add(1) + for _, root := range p.Config.Roots { + if p.failRoots[root] { + return errors.New("scoped discovery failure") + } + for _, source := range p.sourcesByRoot[root] { + if err := yield(source); err != nil { + return err + } + } + } + return nil +} + +func (p *scopedStreamingProvider) WatchPlan( + context.Context, +) (parser.WatchPlan, error) { + return parser.WatchPlan{}, nil +} + +func (p *scopedStreamingProvider) SourcesForChangedPath( + _ context.Context, req parser.ChangedPathRequest, +) ([]parser.SourceRef, error) { + for _, sources := range p.sourcesByRoot { + for _, source := range sources { + if source.DisplayPath == req.Path { + return []parser.SourceRef{source}, nil + } + } + } + return nil, nil +} + +func (p *scopedStreamingProvider) Fingerprint( + _ context.Context, source parser.SourceRef, +) (parser.SourceFingerprint, error) { + return parser.SourceFingerprint{Key: source.FingerprintKey}, nil +} + +func (p *scopedStreamingProvider) Parse( + _ context.Context, req parser.ParseRequest, +) (parser.ParseOutcome, error) { + path := req.Source.DisplayPath + started := time.Unix(1704067200, 0) + return parser.ParseOutcome{ + Results: []parser.ParseResultOutcome{{ + Result: parser.ParseResult{Session: parser.ParsedSession{ + ID: strings.TrimSuffix(filepath.Base(path), ".jsonl"), + Agent: p.Def.Type, + Project: "project", Machine: "local", + StartedAt: started, EndedAt: started, + File: parser.FileInfo{Path: path}, + }}, + DataVersion: parser.DataVersionCurrent, + }}, + ResultSetComplete: true, + }, nil +} + +type scopedStreamingFactory struct{ provider *scopedStreamingProvider } + +func (f scopedStreamingFactory) Definition() parser.AgentDef { + return f.provider.Definition() +} + +func (f scopedStreamingFactory) Capabilities() parser.Capabilities { + return f.provider.Capabilities() +} + +func (f scopedStreamingFactory) NewProvider( + cfg parser.ProviderConfig, +) parser.Provider { + // Per-call copy: DiscoverEach walks this construction's roots, and the + // shared instance must not be written — sync workers construct providers + // concurrently. The maps and the stream counter stay shared so tests + // mutate and observe one place. + provider := *f.provider + provider.Config = cfg.Clone() + return &provider +} + +func newScopedStreamingProvider( + agent parser.AgentType, +) *scopedStreamingProvider { + return &scopedStreamingProvider{ + ProviderBase: parser.ProviderBase{ + Def: parser.AgentDef{Type: agent, FileBased: true}, + Caps: parser.Capabilities{Source: parser.SourceCapabilities{ + DiscoverSources: parser.CapabilitySupported, + StreamingDiscovery: parser.CapabilitySupported, + WatchSources: parser.CapabilitySupported, + FindSource: parser.CapabilitySupported, + }}, + }, + sourcesByRoot: make(map[string][]parser.SourceRef), + failRoots: make(map[string]bool), + findable: make(map[string]parser.SourceRef), + streamCalls: &atomic.Int32{}, + } +} + +// TestReconcileProviderRootsScopedVirtualMemberChecksRelocationWhenContainerGone +// pins the relocation guard for a virtual member whose home container is +// itself deleted: the stale spelling resolves nowhere, so no branch-level +// guard fires, and only the shared pre-tombstone check can distinguish a +// member that moved to another configured root from one that is gone. The +// moved member is preserved; a sibling the provider resolves nowhere is +// reclaimed in the same pass. +func TestReconcileProviderRootsScopedVirtualMemberChecksRelocationWhenContainerGone( + t *testing.T, +) { + database := openTestDB(t) + const agent = parser.AgentType("scoped-virtual-relocation") + rootOne := t.TempDir() + rootTwo := t.TempDir() + // The container under rootOne is already gone; its two stored members + // differ only in whether the provider still resolves them elsewhere. + container := filepath.Join(rootOne, "traces", "chat.db") + movedPath := container + "#moved" + gonePath := container + "#gone" + for id, path := range map[string]string{ + "moved": movedPath, "gone": gonePath, + } { + p := path + require.NoError(t, database.UpsertSession(db.Session{ + ID: id, Agent: string(agent), Project: "proj", + Machine: "local", FilePath: &p, + })) + } + require.NoError(t, database.BaselineActiveSessionSourcePaths( + t.Context(), "local", []db.SessionSourcePath{ + {Agent: string(agent), FilePath: movedPath}, + {Agent: string(agent), FilePath: gonePath}, + }, + )) + + provider := newScopedStreamingProvider(agent) + provider.findable["moved"] = scopedTestSource( + agent, filepath.Join(rootTwo, "traces", "chat.db")+"#moved", + ) + engine := NewEngine(database, EngineConfig{ + AgentDirs: map[parser.AgentType][]string{agent: {rootOne, rootTwo}}, + Machine: "local", + ProviderFactories: []parser.ProviderFactory{ + scopedStreamingFactory{provider: provider}, + }, + ProviderMigrationModes: map[parser.AgentType]parser.ProviderMigrationMode{ + agent: parser.ProviderMigrationProviderAuthoritative, + }, + }) + t.Cleanup(engine.Close) + + // Only rootOne is requested, so the pass holds no full-root coverage + // and never streams rootTwo. + require.NoError(t, engine.ReconcileProviderRoots( + t.Context(), agent, []string{rootOne}, + )) + + survivor, err := database.GetSession(t.Context(), "moved") + require.NoError(t, err) + assert.NotNil(t, survivor, + "a member the provider resolves under another root is a move") + gone, err := database.GetSessionFull(t.Context(), "gone") + require.NoError(t, err) + require.NotNil(t, gone) + require.NotNil(t, gone.DeletionCause) + assert.Equal(t, "source_missing", *gone.DeletionCause, + "a member the provider resolves nowhere is reclaimed") +} + +func scopedTestSource(agent parser.AgentType, path string) parser.SourceRef { + return parser.SourceRef{ + Provider: agent, Key: path, DisplayPath: path, FingerprintKey: path, + } +} + +// TestReconcileProviderRootsScopedFailureCommitsHealthySiblingScope pins +// scope-level partial success within one provider: the healthy scope commits +// its proof-bounded tombstone while the failed scope returns only its own +// retry roots. +func TestReconcileProviderRootsScopedFailureCommitsHealthySiblingScope(t *testing.T) { + database := openTestDB(t) + const agent = parser.AgentType("scoped-stream") + rootOne := t.TempDir() + rootTwo := t.TempDir() + pathOne := filepath.Join(rootOne, "s1.jsonl") + pathTwo := filepath.Join(rootTwo, "s2.jsonl") + require.NoError(t, os.WriteFile(pathOne, []byte("{}\n"), 0o600)) + require.NoError(t, os.WriteFile(pathTwo, []byte("{}\n"), 0o600)) + provider := newScopedStreamingProvider(agent) + provider.sourcesByRoot[rootOne] = []parser.SourceRef{ + scopedTestSource(agent, pathOne), + } + provider.sourcesByRoot[rootTwo] = []parser.SourceRef{ + scopedTestSource(agent, pathTwo), + } + engine := NewEngine(database, EngineConfig{ + AgentDirs: map[parser.AgentType][]string{agent: {rootOne, rootTwo}}, + Machine: "local", + ProviderFactories: []parser.ProviderFactory{ + scopedStreamingFactory{provider: provider}, + }, + ProviderMigrationModes: map[parser.AgentType]parser.ProviderMigrationMode{ + agent: parser.ProviderMigrationProviderAuthoritative, + }, + }) + t.Cleanup(engine.Close) + require.NoError(t, engine.ReconcileProviderRoots( + t.Context(), agent, []string{rootOne, rootTwo}, + )) + + require.NoError(t, os.Remove(pathOne)) + provider.sourcesByRoot[rootOne] = nil + provider.failRoots[rootTwo] = true + + err := engine.ReconcileProviderRoots( + t.Context(), agent, []string{rootOne, rootTwo}, + ) + + require.Error(t, err) + var retryErr reconciliationRetryRootError + require.ErrorAs(t, err, &retryErr) + assert.ElementsMatch(t, []string{rootTwo}, + retryErr.ReconciliationRetryRoots(), + "the failed scope retries at the caller's own width") + gone, getErr := database.GetSessionFull(t.Context(), "s1") + require.NoError(t, getErr) + require.NotNil(t, gone) + require.NotNil(t, gone.DeletionCause) + assert.Equal(t, "source_missing", *gone.DeletionCause, + "the healthy scope still commits its proof-bounded tombstone") + survivor, getErr := database.GetSession(t.Context(), "s2") + require.NoError(t, getErr) + assert.NotNil(t, survivor, "the failed scope preserves its sessions") +} + +// TestReconcileProviderRootsContractViolationFailsScopeClosed verifies the +// fail-closed path: a provider emitting a source outside both traversal and +// proof fails that scope, preserves its sessions, and returns retry roots. +func TestReconcileProviderRootsContractViolationFailsScopeClosed(t *testing.T) { + database := openTestDB(t) + const agent = parser.AgentType("scoped-violation") + root := t.TempDir() + outside := filepath.Join(t.TempDir(), "outside.jsonl") + live := filepath.Join(root, "live.jsonl") + require.NoError(t, os.WriteFile(live, []byte("{}\n"), 0o600)) + require.NoError(t, os.WriteFile(outside, []byte("{}\n"), 0o600)) + provider := newScopedStreamingProvider(agent) + provider.sourcesByRoot[root] = []parser.SourceRef{ + scopedTestSource(agent, live), + } + engine := NewEngine(database, EngineConfig{ + AgentDirs: map[parser.AgentType][]string{agent: {root}}, + Machine: "local", + ProviderFactories: []parser.ProviderFactory{ + scopedStreamingFactory{provider: provider}, + }, + ProviderMigrationModes: map[parser.AgentType]parser.ProviderMigrationMode{ + agent: parser.ProviderMigrationProviderAuthoritative, + }, + }) + t.Cleanup(engine.Close) + require.NoError(t, engine.ReconcileProviderRoots( + t.Context(), agent, []string{root}, + )) + + require.NoError(t, os.Remove(live)) + provider.sourcesByRoot[root] = []parser.SourceRef{ + scopedTestSource(agent, outside), + } + + err := engine.ReconcileProviderRoots(t.Context(), agent, []string{root}) + + require.Error(t, err) + var retryErr reconciliationRetryRootError + require.ErrorAs(t, err, &retryErr) + assert.ElementsMatch(t, []string{root}, + retryErr.ReconciliationRetryRoots()) + assert.Equal(t, 1, engine.LastReconciliationResult().ProviderFailures) + preserved, getErr := database.GetSession(t.Context(), "live") + require.NoError(t, getErr) + assert.NotNil(t, preserved, + "a contract violation withholds deletion authority for the scope") + spooled, getErr := database.GetSession(t.Context(), "outside") + require.NoError(t, getErr) + assert.Nil(t, spooled, + "a source outside traversal and proof must never be spooled") +} + +// TestReconcilePartialRequestCoveringAllRootsKeepsFullAuthority is the +// full-coverage preservation row: a partial request naming every configured +// root keeps exactly the authority a full pass has. +func TestReconcilePartialRequestCoveringAllRootsKeepsFullAuthority(t *testing.T) { + if testing.Short() { + t.Skip("skipping integration test") + } + database := openTestDB(t) + rootOne := filepath.Join(t.TempDir(), "claude-one") + rootTwo := filepath.Join(t.TempDir(), "claude-two") + writeClaudeProjectSession(t, rootOne, "proj", "keep-one") + removedOne := writeClaudeProjectSession(t, rootOne, "proj", "gone-one") + writeClaudeProjectSession(t, rootTwo, "proj", "keep-two") + removedTwo := writeClaudeProjectSession(t, rootTwo, "proj", "gone-two") + + engine := NewEngine(database, EngineConfig{ + AgentDirs: map[parser.AgentType][]string{ + parser.AgentClaude: {rootOne, rootTwo}, + }, + Machine: "local", + }) + t.Cleanup(engine.Close) + require.Equal(t, 4, engine.SyncAll(t.Context(), nil).Synced) + + require.NoError(t, os.Remove(removedOne)) + require.NoError(t, os.Remove(removedTwo)) + + require.NoError(t, engine.ReconcileWatchRoots( + t.Context(), []string{rootOne, rootTwo}, false, + )) + + for _, id := range []string{"gone-one", "gone-two"} { + gone, err := database.GetSessionFull(t.Context(), id) + require.NoError(t, err) + require.NotNil(t, gone) + require.NotNil(t, gone.DeletionCause) + assert.Equal(t, "source_missing", *gone.DeletionCause) + } + for _, id := range []string{"keep-one", "keep-two"} { + active, err := database.GetSession(t.Context(), id) + require.NoError(t, err) + assert.NotNil(t, active) + } +} + +// TestSyncPathsMissingSourceResolvesReplacementAcrossRoots preserves the +// deliberate replacement-index bypass: proving a replacement must span the +// provider's full configured scope even when the pass itself is narrow. +func TestSyncPathsMissingSourceResolvesReplacementAcrossRoots(t *testing.T) { + if testing.Short() { + t.Skip("skipping integration test") + } + database := openTestDB(t) + rootOne := filepath.Join(t.TempDir(), "claude-one") + rootTwo := filepath.Join(t.TempDir(), "claude-two") + moved := writeClaudeProjectSession(t, rootOne, "proj", "moved-session") + require.NoError(t, os.MkdirAll(filepath.Join(rootTwo, "proj"), 0o755)) + + engine := NewEngine(database, EngineConfig{ + AgentDirs: map[parser.AgentType][]string{ + parser.AgentClaude: {rootOne, rootTwo}, + }, + Machine: "local", + }) + t.Cleanup(engine.Close) + require.Equal(t, 1, engine.SyncAll(t.Context(), nil).Synced) + + require.NoError(t, os.Rename( + moved, filepath.Join(rootTwo, "proj", "moved-session.jsonl"), + )) + + require.NoError(t, engine.SyncPathsContext(t.Context(), []string{moved})) + + active, err := database.GetSession(t.Context(), "moved-session") + require.NoError(t, err) + assert.NotNil(t, active, + "a surviving same-identity copy under another configured root is a replacement") +} + +// TestStoredSourceDBHintScopesPreservesFields is the adapter-fidelity row: +// parser scopes convert to database scopes without losing either field. +func TestStoredSourceDBHintScopesPreservesFields(t *testing.T) { + converted := storedSourceDBHintScopes([]parser.StoredSourceHintScope{ + {Path: "/archive/state.db", IncludeVirtualMembers: true}, + {Path: "/archive/sessions"}, + }) + assert.Equal(t, []db.StoredSourcePathHintScope{ + {Path: "/archive/state.db", IncludeVirtualMembers: true}, + {Path: "/archive/sessions"}, + }, converted) +} diff --git a/internal/sync/reconciliation_scoping_zed_test.go b/internal/sync/reconciliation_scoping_zed_test.go new file mode 100644 index 000000000..1ad880c0f --- /dev/null +++ b/internal/sync/reconciliation_scoping_zed_test.go @@ -0,0 +1,121 @@ +package sync_test + +import ( + "database/sql" + "path/filepath" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "go.kenn.io/agentsview/internal/dbtest" + "go.kenn.io/agentsview/internal/parser" + "go.kenn.io/agentsview/internal/sync" +) + +// TestReconcileProviderRootsZedContainerPassReclaimsRemovedMember pins the +// container topology for the shared multi-session base at engine level, on a +// provider that is not OpenCode-family: a pass asked about the Zed threads.db +// itself proves the container's whole virtual membership, so a removed thread +// is reclaimed and a surviving thread keeps its session, instead of the +// bare-path proof admitting no member and completing as a no-op. +func TestReconcileProviderRootsZedContainerPassReclaimsRemovedMember( + t *testing.T, +) { + zedDir := t.TempDir() + dbPath := filepath.Join(zedDir, "threads", "threads.db") + createZedThreadsDB(t, dbPath, []zedThreadFixture{ + { + id: "kept", summary: "Kept thread", + updatedAt: "2026-06-09T02:30:00Z", dataType: "json", + data: []byte(`{"messages":[]}`), + }, + { + id: "removed", summary: "Removed thread", + updatedAt: "2026-06-09T02:31:00Z", dataType: "json", + data: []byte(`{"messages":[]}`), + }, + }) + database := dbtest.OpenTestDB(t) + engine := sync.NewEngine(database, sync.EngineConfig{ + AgentDirs: map[parser.AgentType][]string{parser.AgentZed: {zedDir}}, + Machine: "local", + }) + t.Cleanup(engine.Close) + require.Equal(t, 2, engine.SyncAll(t.Context(), nil).Synced) + + conn, err := sql.Open("sqlite3", dbPath) + require.NoError(t, err) + _, err = conn.Exec("DELETE FROM threads WHERE id = 'removed'") + require.NoError(t, conn.Close()) + require.NoError(t, err) + + require.NoError(t, engine.ReconcileProviderRoots( + t.Context(), parser.AgentZed, []string{dbPath}, + )) + + removed, err := database.GetSession(t.Context(), "zed:removed") + require.NoError(t, err) + assert.Nil(t, removed, + "a container-scoped pass reclaims a removed member") + kept, err := database.GetSession(t.Context(), "zed:kept") + require.NoError(t, err) + assert.NotNil(t, kept, "a surviving member keeps its session") +} + +// TestReconcileProviderRootsZedContainerPassPreservesMovedMember pins the +// relocation guard for persistent-archive members under scoped container +// authority: a thread that moved from the requested container to another +// configured root's container is a move, not a deletion, so a pass scoped to +// the source container must preserve its session even though its own stream +// no longer yields the member. +func TestReconcileProviderRootsZedContainerPassPreservesMovedMember( + t *testing.T, +) { + firstDir := t.TempDir() + secondDir := t.TempDir() + firstDB := filepath.Join(firstDir, "threads", "threads.db") + secondDB := filepath.Join(secondDir, "threads", "threads.db") + moved := zedThreadFixture{ + id: "moved", summary: "Moved thread", + updatedAt: "2026-06-09T02:30:00Z", dataType: "json", + data: []byte(`{"messages":[]}`), + } + createZedThreadsDB(t, firstDB, []zedThreadFixture{moved}) + createZedThreadsDB(t, secondDB, nil) + + database := dbtest.OpenTestDB(t) + engine := sync.NewEngine(database, sync.EngineConfig{ + AgentDirs: map[parser.AgentType][]string{ + parser.AgentZed: {firstDir, secondDir}, + }, + Machine: "local", + }) + t.Cleanup(engine.Close) + require.Equal(t, 1, engine.SyncAll(t.Context(), nil).Synced) + + // The thread moves between containers; only the source container is + // reconciled, so the pass cannot see the destination's membership. + firstConn, err := sql.Open("sqlite3", firstDB) + require.NoError(t, err) + _, err = firstConn.Exec("DELETE FROM threads WHERE id = 'moved'") + require.NoError(t, firstConn.Close()) + require.NoError(t, err) + secondConn, err := sql.Open("sqlite3", secondDB) + require.NoError(t, err) + _, err = secondConn.Exec(`INSERT INTO threads ( + id, summary, updated_at, data_type, data, + parent_id, folder_paths, created_at + ) VALUES ('moved', 'Moved thread', '2026-06-09T02:32:00Z', 'json', + '{"messages":[]}', NULL, '', '')`) + require.NoError(t, secondConn.Close()) + require.NoError(t, err) + + require.NoError(t, engine.ReconcileProviderRoots( + t.Context(), parser.AgentZed, []string{firstDB}, + )) + + survivor, err := database.GetSession(t.Context(), "zed:moved") + require.NoError(t, err) + assert.NotNil(t, survivor, + "a member found under another configured root is a move, not a deletion") +} diff --git a/internal/sync/test_helpers_test.go b/internal/sync/test_helpers_test.go index dd687a2f4..fa9067b0f 100644 --- a/internal/sync/test_helpers_test.go +++ b/internal/sync/test_helpers_test.go @@ -202,10 +202,16 @@ var ( kiroSQLiteFixtureCache stdsync.Map ) +// openCodeLikeSchema mirrors the production OpenCode container schema, +// including the project/message/part time_updated columns. Those columns are +// the per-session change signal (openCodeCompositeMtimeExpr); a fixture that +// omitted them could not model shared-container freshness at all, which is how +// the whole-container re-parse regression went unnoticed. const openCodeLikeSchema = ` CREATE TABLE project ( id TEXT PRIMARY KEY, - worktree TEXT NOT NULL + worktree TEXT NOT NULL, + time_updated INTEGER NOT NULL DEFAULT 0 ); CREATE TABLE session ( id TEXT PRIMARY KEY, @@ -219,15 +225,21 @@ const openCodeLikeSchema = ` id TEXT PRIMARY KEY, session_id TEXT NOT NULL, data TEXT NOT NULL, - time_created INTEGER NOT NULL + time_created INTEGER NOT NULL, + time_updated INTEGER NOT NULL DEFAULT 0 ); CREATE TABLE part ( id TEXT PRIMARY KEY, session_id TEXT NOT NULL, message_id TEXT NOT NULL, data TEXT NOT NULL, - time_created INTEGER NOT NULL + time_created INTEGER NOT NULL, + time_updated INTEGER NOT NULL DEFAULT 0 ); + CREATE INDEX message_session_time_created_id_idx + ON message (session_id, time_created, id); + CREATE INDEX part_session_idx ON part (session_id); + CREATE INDEX part_message_id_id_idx ON part (message_id, id); ` const kiroSQLiteSchema = ` @@ -430,8 +442,22 @@ func (oc *openCodeTestDB) addProject( ) { t.Helper() oc.mustExec(t, "insert project", - "INSERT INTO project (id, worktree) VALUES (?, ?)", - id, worktree, + "INSERT INTO project (id, worktree, time_updated) VALUES (?, ?, ?)", + id, worktree, 0, + ) +} + +// updateProjectWorktree renames a project's worktree and bumps its +// time_updated, matching production OpenCode. The bump is what lets every +// session in that project re-resolve its cwd/project without the container +// stat acting as a blunt whole-archive invalidator. +func (oc *openCodeTestDB) updateProjectWorktree( + t *testing.T, id, worktree string, timeUpdated int64, +) { + t.Helper() + oc.mustExec(t, "update project worktree", + "UPDATE project SET worktree = ?, time_updated = ? WHERE id = ?", + worktree, timeUpdated, id, ) } @@ -471,9 +497,9 @@ func (oc *openCodeTestDB) addMessage( require.NoError(t, err, "marshal message") oc.mustExec(t, "insert message", `INSERT INTO message - (id, session_id, data, time_created) - VALUES (?, ?, ?, ?)`, - id, sessionID, string(data), timeCreated, + (id, session_id, data, time_created, time_updated) + VALUES (?, ?, ?, ?, ?)`, + id, sessionID, string(data), timeCreated, timeCreated, ) } @@ -483,8 +509,11 @@ func (oc *openCodeTestDB) updateMessageData( t.Helper() raw, err := json.Marshal(data) require.NoError(t, err, "marshal message update") + // Production OpenCode bumps time_updated on an in-place row edit; the + // per-session composite freshness signal depends on it. oc.mustExec(t, "update message data", - "UPDATE message SET data = ? WHERE id = ?", + `UPDATE message SET data = ?, time_updated = time_updated + 1 + WHERE id = ?`, string(raw), id, ) } @@ -502,9 +531,9 @@ func (oc *openCodeTestDB) addTextPart( require.NoError(t, err, "marshal text part") oc.mustExec(t, "insert part", `INSERT INTO part - (id, session_id, message_id, data, time_created) - VALUES (?, ?, ?, ?, ?)`, - id, sessionID, messageID, string(data), timeCreated, + (id, session_id, message_id, data, time_created, time_updated) + VALUES (?, ?, ?, ?, ?, ?)`, + id, sessionID, messageID, string(data), timeCreated, timeCreated, ) } @@ -523,9 +552,9 @@ func (oc *openCodeTestDB) addToolPart( require.NoError(t, err, "marshal tool part") oc.mustExec(t, "insert tool part", `INSERT INTO part - (id, session_id, message_id, data, time_created) - VALUES (?, ?, ?, ?, ?)`, - id, sessionID, messageID, string(data), timeCreated, + (id, session_id, message_id, data, time_created, time_updated) + VALUES (?, ?, ?, ?, ?, ?)`, + id, sessionID, messageID, string(data), timeCreated, timeCreated, ) } diff --git a/internal/sync/watch_failure_test.go b/internal/sync/watch_failure_test.go index 419fd41a3..f5ec46059 100644 --- a/internal/sync/watch_failure_test.go +++ b/internal/sync/watch_failure_test.go @@ -51,11 +51,24 @@ func (factory partialFailureStreamingFactory) Capabilities() parser.Capabilities return factory.provider.Capabilities() } +type partialFailureStreamingScopedProvider struct { + *partialFailureStreamingProvider + scopes parser.ProviderBase +} + +func (p partialFailureStreamingScopedProvider) ResolveReconciliationScopes( + ctx context.Context, req parser.ReconciliationScopeRequest, +) (parser.ReconciliationScopePlan, error) { + return p.scopes.ResolveReconciliationScopes(ctx, req) +} + func (factory partialFailureStreamingFactory) NewProvider( cfg parser.ProviderConfig, ) parser.Provider { - factory.provider.Config = cfg.Clone() - return factory.provider + return partialFailureStreamingScopedProvider{ + partialFailureStreamingProvider: factory.provider, + scopes: perCallScopeProviderBase(factory.provider.ProviderBase, cfg), + } } type fingerprintCountingProvider struct { @@ -117,11 +130,24 @@ func (f changedPathFailureFactory) Capabilities() parser.Capabilities { return f.provider.Capabilities() } +type changedPathFailureScopedProvider struct { + *changedPathFailureProvider + scopes parser.ProviderBase +} + +func (p changedPathFailureScopedProvider) ResolveReconciliationScopes( + ctx context.Context, req parser.ReconciliationScopeRequest, +) (parser.ReconciliationScopePlan, error) { + return p.scopes.ResolveReconciliationScopes(ctx, req) +} + func (f changedPathFailureFactory) NewProvider( cfg parser.ProviderConfig, ) parser.Provider { - f.provider.Config = cfg.Clone() - return f.provider + return changedPathFailureScopedProvider{ + changedPathFailureProvider: f.provider, + scopes: perCallScopeProviderBase(f.provider.ProviderBase, cfg), + } } type changedPathContextKey struct{}