Skip to content

Commit 9e28f7e

Browse files
committed
docs: ARCHITECTURE + STRUCTURE for plan v6 deferred compaction marker
Updates ARCHITECTURE.md to reflect the new deferred compaction-marker drain pattern (plan v6) and the cache-busting-signals abstraction threaded through postprocess. Specifically: - Compaction markers: rewritten to describe the in-transaction pending blob written at historian publish, the next-pass applyDeferredCompactionMarker drain via MarkerUpdateOutcome, CAS-clear semantics, eager-path direct calls (/ctx-flush, /ctx-recomp), restart-safe rehydration via getSessionsWithPendingMarker, and session.compacted / session.deleted CAS-clear. - New Cache-busting signals section: documents the per-pass facts in cache-busting-signals.ts (historyRebuiltThisPass, historyRefreshExplicitBeforePrepare, compartmentInjectionRebuiltFromDb, canConsumeDeferredLate, phaseJustAwaitedPublication), explains the historyWasConsumedThisPass drain decision, and references the degradedCacheCountBySession warning threshold. - Schema migrations line bumped to v1-v13 (v12 orphan embedding cleanup + cascade verification, v13 pending_compaction_marker_state). Updates STRUCTURE.md to match the v1-v13 migration range. No code changes.
1 parent 72d1f1b commit 9e28f7e

2 files changed

Lines changed: 11 additions & 6 deletions

File tree

ARCHITECTURE.md

Lines changed: 10 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -199,10 +199,10 @@
199199
- Location: `src/features/magic-context/plugin-messages.ts`
200200
- Pattern: Vestigial — superseded by RPC. Module remains for forward-compat with older TUI plugin versions that may still poll it; no active runtime callers in current code.
201201

202-
**Compaction markers:**
203-
- Purpose: Inject OpenCode-compatible compaction boundaries into the message table so `filterCompacted` stops at historian's last compartment boundary, shrinking the transform-input array.
204-
- Location: `src/features/magic-context/compaction-marker.ts`, `src/hooks/magic-context/compaction-marker-manager.ts`
205-
- Pattern: Write summary/compaction rows into OpenCode's DB after historian publishes; filter them out from raw reads. Stable feature (default `compaction_markers: true` since v0.16.x); raw-history readers strip `summary=true` / `finish="stop"` rows to preserve original ordinals.
202+
**Compaction markers (deferred drain, plan v6):**
203+
- Purpose: Inject OpenCode-compatible compaction boundaries into the message table so `filterCompacted` stops at historian's last compartment boundary, shrinking the transform-input array. Marker movement is deferred from historian publish into the next materializing transform pass so a single cache-bust cycle covers both the `<session-history>` rebuild AND the marker boundary advance.
204+
- Location: `src/features/magic-context/compaction-marker.ts`, `src/hooks/magic-context/compaction-marker-manager.ts`, `src/features/magic-context/storage-meta-persisted.ts` (pending blob helpers).
205+
- Pattern: Historian / compressor incremental runners write the prospective new boundary (`{ordinal, endMessageId, publishedAt}`) into `session_meta.pending_compaction_marker_state` in the same transaction that publishes new compartments. The next consuming transform pass that drains `deferredHistoryRefreshSessions` calls `applyDeferredCompactionMarker(...)`, which validates the pending target against the latest stored compartment via `getCompartmentsByEndMessageId(...)` plus an OpenCode-message existence check via `getOpenCodeMessageById(...)`, then sequences `removeCompactionMarker` → `injectCompactionMarker`. Returns a tagged `MarkerUpdateOutcome` (`applied` | `already-current` | `stale-skip` | `retryable-failure`); only `retryable-failure` preserves the deferred-history signal so the next pass retries. CAS-clear (`clearPendingCompactionMarkerStateIf`) on success guards against publish/drain races within and across processes. Eager paths (`/ctx-flush`, `/ctx-recomp`) call the marker manager directly and CAS-clear any stale pending blob. Restart-safe: hook init calls `getSessionsWithPendingMarker(...)` to rehydrate deferred sets so the next pass after restart still drains. `event-handler` CAS-clears pending state on `session.compacted` (provider already advanced the boundary) and on `session.deleted` via cascade. Raw-history readers strip `summary=true` / `finish="stop"` rows to preserve original ordinals. Stable feature, default `compaction_markers: true` since v0.16.x; deferred drain since v0.19 (plan v6).
206206

207207
**Auto-update checker:**
208208
- Purpose: Self-update the cached `@latest` plugin install once per plugin process — OpenCode's plugin cache no longer auto-updates.
@@ -239,6 +239,11 @@
239239
- Location: `src/features/magic-context/storage-meta-shared.ts`, `src/features/magic-context/storage-meta-persisted.ts`, `src/features/magic-context/storage-meta-session.ts`, `src/features/magic-context/storage-meta.ts`
240240
- Pattern: `session_meta` SQLite table with `ensureColumn()` and versioned migrations; typed row interfaces with runtime guards; NULL coercion in `isSessionMetaRow()` so legacy rows don't trigger fallback-to-defaults on every read.
241241

242+
**Cache-busting signals (plan v6):**
243+
- Purpose: Surface durable per-pass facts the postprocess phase uses to decide whether the v12 deferred-history drain, the deferred-marker drain, and the deferred-materialization drain are eligible to fire — without re-reading transform state.
244+
- Location: `src/hooks/magic-context/cache-busting-signals.ts`, threaded into `RunPostTransformPhaseArgs` (`historyRebuiltThisPass`, `historyRefreshExplicitBeforePrepare`, `compartmentInjectionRebuiltFromDb`, `canConsumeDeferredLate`, `phaseJustAwaitedPublication`, etc.).
245+
- Pattern: Captured at well-defined points in `transform.ts` (e.g. `historyRefreshExplicitBeforePrepare` is read immediately before `prepareCompartmentInjection`, not later) so concurrent transform passes don't clobber each other's signals. The drain decision (`historyWasConsumedThisPass`) combines `historyRebuiltThisPass && (canConsumeDeferredLate || phaseJustAwaitedPublication || explicitRebuildHappened) && materializationSatisfied`. Degraded-cache state (null-boundary rebuild) is tracked by `degradedCacheCountBySession` in postprocess; entry logs in `inject-compartments.ts` and a warning at `DEGRADE_CACHE_WARNING_THRESHOLD=10` consecutive degraded rebuilds.
246+
242247
## Entry Points
243248

244249
**CLI entry:**
@@ -327,7 +332,7 @@ Magic Context runs in three effective modes depending on `ctx_reduce_enabled` an
327332

328333
**Storage:** Use the SQLite database created by `src/features/magic-context/storage-db.ts` under the cortexkit data directory resolved by `src/shared/data-path.ts` (`~/.local/share/cortexkit/magic-context/context.db` on Linux/macOS, XDG-equivalent on Windows). Legacy OpenCode-plugin-folder DBs are migrated forward on first boot. The same DB is shared cross-harness between OpenCode and Pi; session-scoped tables include a `harness` discriminator (`'opencode'` / `'pi'`) while project-scoped tables (memories, git commits) are shared.
329334

330-
**Schema migrations:** `src/features/magic-context/migrations.ts` declares versioned migrations v1–v11 (v10 added `tool_owner_message_id` for composite tool-tag identity; v11 added `todo_synthetic_*` columns for synthetic-todowrite). Migration runner uses `schema_migrations` table with version-ordered execution and sibling-startup race protection (duplicate-insert is tolerated).
335+
**Schema migrations:** `src/features/magic-context/migrations.ts` declares versioned migrations v1–v13 (v10 added `tool_owner_message_id` for composite tool-tag identity; v11 added `todo_synthetic_*` columns for synthetic-todowrite; v12 cleaned orphan `memory_embeddings` rows and verified embedding cascade behavior; v13 added `pending_compaction_marker_state` for the plan-v6 deferred-marker drain). Migration runner uses `schema_migrations` table with version-ordered execution and sibling-startup race protection (duplicate-insert is tolerated).
331336

332337
**Harness-aware behavior:** `src/shared/harness.ts` exposes `setHarness()`/`getHarness()` for the runtime to identify itself; production INSERTs into session-scoped tables tag rows with the current harness. Pi-specific session-resolution paths are skipped on OpenCode and vice versa.
333338

STRUCTURE.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -103,7 +103,7 @@
103103
- `src/hooks/magic-context/todo-view.ts`: Build the deterministic synthetic todowrite tool part and compute its hash-based `call_id`.
104104
- `src/features/magic-context/storage-db.ts`: Create durable storage; run versioned migrations; resolve runtime SQLite backend.
105105
- `src/features/magic-context/storage-meta-persisted.ts`: Read and write per-session persisted scalars and JSON blobs.
106-
- `src/features/magic-context/migrations.ts`: Versioned schema migrations v1–v11.
106+
- `src/features/magic-context/migrations.ts`: Versioned schema migrations v1–v13.
107107
- `src/features/magic-context/message-index.ts`: FTS-backed raw-message index for `ctx_search`.
108108
- `src/features/magic-context/search.ts`: Unified retrieval over memories, raw messages, and git commits.
109109

0 commit comments

Comments
 (0)