mv-backing-host.md

Backing-host capability

Stability: Beta — see Stability Tiers.

Invariant: MV-014, MV-015

The module-author contract for hosting a materialized view's stored rows: the privileged surface a virtual-table module must implement so the engine can maintain, refill, and enforce constraints over a maintained table whose storage it does not own.

The privileged surface the engine needs from the maintained table's hosting module is factored into a module-neutral capability, BackingHost (vtab/backing-host.ts), resolved per table via the optional VirtualTableModule.getBackingHost(db, schemaName, tableName) — presence of the method is the capability, mirroring getMappingAdvertisements. The "backing table" the capability hosts is the maintained table itself. One BackingHost instance corresponds to one live table incarnation: a drop+recreate (an explicit drop + re-create, or the declarative differ's body-change rebuild) yields a new host whose ownsConnection rejects the previous incarnation's connections, so a stale same-name connection is never adopted. Refresh's shape reshape is deliberately not such an event — it reconciles the live table in place (the host module's alterTable), preserving the incarnation and its connections.

The surface (see the doc comments in vtab/backing-host.ts for the full contract):

Member	Role
ownsConnection(conn)	True when conn is a live connection to this backing incarnation — how the engine re-finds the coordinated backing connection among the Database's registered connections.
connect()	Fresh VirtualTableConnection; the engine registers it so coordinated commit/rollback (savepoint replay included) covers its pending state in lockstep with the source write.
applyMaintenance(conn, ops)	Privileged, ordered MaintenanceOp application into conn's pending transaction state. Bypasses user-DML read-only enforcement, keeps secondary-index / change-tracking bookkeeping, and returns the effective BackingRowChanges realized.
replaceContents(rows, onDuplicateKey?)	Atomic replacement of the committed contents (create-fill / refresh). Throws onDuplicateKey() on a duplicate PK; concurrent readers see pre- or post-swap state, never partial.
scanEffective(conn, { equalityPrefix?, descending? })	Reads-own-writes scan over conn's effective state (pending over committed) in PK order, honoring equalityPrefix as a seek + early-terminate prefix range — the covering-UNIQUE enforcement scan.

Contract highlights:

• Cost. PK-ordered storage with O(log n) keyed upsert/delete/point-lookup and an ordered prefix-range scan are required. This keeps every bounded-delta arm (delete-by-prefix included) and the covering-UNIQUE prefix lookup module-agnostic. A module that cannot provide the ordered prefix scan must not advertise the capability — there is deliberately no per-arm gating (both real host candidates are ordered-KV, and per-module arm gating would fragment the maintenance planner).
• Effective-change reporting is part of the contract, not an optimization: the MV-over-MV cascade routes each returned BackingRowChange back through maintainRowTime, so over- or under-reporting corrupts consumer MVs. No-op ops yield nothing; replace-all yields the minimal keyed diff.
• Transactionality. applyMaintenance writes the connection's pending state; commit/rollback ride the registered VirtualTableConnection's generic begin/commit/rollback/savepoint surface.
• Read-only to user DML — engine-owned. A maintained table's rows are derived; only the privileged surface may write them, and that boundary is enforced by the engine, not owed by the host module. The planner rewrites user DML naming a maintained table to write-through against the body's base source, and the runtime DML executor carries a READONLY backstop (runtime/emit/dml-executor.ts assertNotMaintainedTableTarget) that rejects any mutation plan whose target still carries a derivation — the second net catching a plan-time mis-dispatch before a direct write can silently diverge the derived contents. applyMaintenance/replaceContents (and the reconcile / rehydrate-refill paths) bypass both by construction. A host module implements no user-DML permission check of its own; a direct programmatic update() on the backing is the embedder's responsibility (same trust level as holding the privileged surface).
• Concurrency. The engine adds no latching around the privileged surface; each host owns its own discipline under the VtabConcurrencyMode its module declares (the memory host's pending layer is private to the connection and mutated synchronously, so it needs none).

The engine resolves the host through resolveBackingHost (runtime/emit/materialized-view-helpers.ts); the memory implementation is a thin adapter over MemoryTableManager (MemoryTableModule.getBackingHost), delegating to applyMaintenanceToLayer, replaceBaseLayer, and the layer scan. USING <module>(...) selects any capability-bearing module as the host: the create builder gates on getBackingHost presence (a capability-less or unknown module is a sited error), and buildBackingTableSchema re-checks as defense-in-depth for the catalog-import path. One soft edge rides on alterTable rather than the capability: source column-rename propagation renames the backing's shifted columns through the host module's alterTable; a host without it leaves the MV stale (recoverable by refresh) instead of renaming in place — see docs/module-authoring.md § Backing Host.

The store host (using store)

The store module (@quereus/store) is the second realized host — StoreBackingHost (src/common/backing-host.ts) makes create materialized view … using store a persistent backing end-to-end. Its pending state is the table's shared TransactionCoordinator: privileged ops queue into the same pending view the store's read paths merge over the committed store, so a mid-transaction select from the MV reaches the pending maintenance through the substrate's reads-own-writes merge (isolated merged read → empty overlay → StoreTable.query → pending merge). Visibility is per-table-coordinator — every connection to the backing shares one pending view — rather than the memory host's per-connection layering; contract-conformant, since the contract requires reads-own-writes on the writing connection, not cross-connection invisibility.

The registered 'store' module is the isolation wrapper (createIsolatedStoreModule → IsolationModule(StoreModule)); the wrapper forwards getBackingHost conditionally (constructor-assigned only when the underlying implements it, so method presence still is the capability). Backing writes are all privileged, so the per-connection overlay stays empty for backing tables: at commit/rollback the backing's IsolatedConnection flushes/discards a no-op empty overlay while the host's own StoreConnection commits/rolls back the coordinator — disjoint state, so ordering between the two registered connections is immaterial.

Store-specific semantics to know:

• replaceContents commits an open coordinator transaction first. A committed bulk replace is effectively DDL-committing on a store-backed table (the posture renameTable already takes) and observably matches memory's in-flight-layer drain in replaceBaseLayer — pinned by a refresh-in-transaction parity test.
• Backing text PK columns key under the store's table-level key collation K (using store(collation = 'BINARY' | 'NOCASE'), default NOCASE) exactly as create table … using store reconciles implicit-collation text PK columns. Case-variant backing keys therefore collapse under the default — a body keyed solely on a case-varying text column trips the "must be a set" gate where a memory backing (BINARY column default) would accept it; pass collation = 'BINARY' for byte-exact keys.
• No store data-change events from privileged writes. The MV-over-MV cascade consumes the returned BackingRowChanges directly, and the sync layer must not replicate derived rows (sources replicate; each replica derives).
• Catalog. A store-hosted maintained table persists two catalog entries: its ordinary table bundle (landed via the lazy first-access DDL save during the create fill — replaceContents opens the data store) plus the create materialized view entry under the reserved MV key. On reopen, rehydrate phase 1 connects the table bundle as a plain table; phase 3's MV entry then adopts or refills it — the precondition the rehydrate/adopt phasing builds on. (A memory-hosted MV in a store database persists exactly one catalog entry — the MV form — and always refills on reopen.)

Cross-module atomicity

Invariant: MV-024

With the backing in module B and the body's sources in module A, one source-write transaction spans both modules' connections. The Database's coordinated commit covers them — the backing delta and the source write commit or roll back together in normal operation — but coordinated commit is not two-phase commit: with two durable modules, a crash between their commit acknowledgements can leave source and backing divergent on disk. The accepted position is to document this window rather than restrict module combinations: catalog rehydrate refills the maintained table from the body by default (import re-materializes; a durable module's own pre-rehydrated table at the MV's name is dropped and refilled), so any divergence self-heals at the next open.

Adopt-without-refill fast path. At rehydrate, a pre-existing durable backing is adopted — registered as-is, body not re-run — iff ALL of:

1. a derivation-less table exists at <name> — the MV's own name — whose module equals the entry's declared backing module (a different-module table fails the entry with CONSTRAINT — not ours to drop; a maintained table already at the name likewise fails the entry; absence just creates+fills);
2. the persisted backing's shape matches the re-planned body (backingShapeMatches over deriveBackingShape — names, logical types, not-null, collation, physical PK);
3. the body hash agrees — automatic by construction: the catalog persists DDL and import re-parses it, recomputing the hash from the same canonical definition, so there is no independent persisted hash to diverge;
4. every source the body reads lives in the same module as the backing, and every source that is itself a maintained table (MV-over-MV) was itself adopted this rehydration — the shared adopt ledger is keyed by lowercased qualified schema.<tablename> (a refilled upstream may hold new content — its dependents refill too; an adopted upstream is unchanged, so trust composes through the fixpoint rounds);
5. trust basis, capability-dependent (importCatalog's trustBackings option, decided per-entry — see below):
   • Non-atomic provider (no beginAtomicBatch): the host attested a clean shutdown and this MV was not stale-at-close.
   • Atomic provider (beginAtomicBatch present — LevelDB shared-root, IndexedDB single-db): gate 5 is dropped for same-module backings — gate 4 alone governs — so a non-stale backing adopts after a crash too. Logical staleness is excluded instead by the crash-durable stale-MV set (!durableStale.has(name)).

Any failed gate falls back to the drop+refill above. Gate 5 guards two distinct windows:

• the crash-divergence window — a source write and its same-module backing write torn apart by a crash, leaving them divergent on disk (the DDL-level gates 2–3 are blind to content divergence, so an unsound adopt would resurrect it forever); and
• the logical-staleness window — an MV whose row-time maintenance was detached mid-session (a body-relevant table_modified on a source — an ALTER changing columns or physical PK, or a value-semantics type/collation change on a column the body reads; a constraint/index/stats-only change instead recompiles the dependent in place) so later source writes never reached the backing, even with no crash at all. Adopting such a backing serves permanently-behind content.

The store module's coordinator is module-wide (one TransactionCoordinator shared by every table of the module), so a provider that exposes KVStoreProvider.beginAtomicBatch() (a shared durable commit domain — LevelDB's shared-root sublevels, IndexedDB's single-database object stores; see @quereus/store README § Atomic multi-store commit) commits a source write and its same-module backing write in one all-or-nothing batch — closing the crash-divergence window. It does not close the logical-staleness window (that is not a torn commit), so dropping gate 5 in the atomic domain needs a crash-durable staleness signal.

The two trust bases. rehydrateCatalog checks the capability by method presence (typeof provider.beginAtomicBatch === 'function', matching the coordinator's own gate) and decides per-entry trust:

• Atomic provider — gate 5 drops; gate 4 alone governs same-module backings. Logical staleness is excluded by a durable stale-MV set: a reserved \x00meta\x00stale_mvs catalog entry whose value is the JSON array of qualified schema.mv names currently stale. Unlike the marker it is persistent current-truth, not single-use — StoreModule overwrites it (a sync: true point-write) whenever the stale set changes during a session (recomputed in the engine schema-change listener, which runs after the engine's own MV manager has flipped derivation.stale, since the engine subscribes first) and rewrites it at clean close; rehydrateCatalog only reads it (never deletes it). A crash leaves the last synced value intact, so a non-stale backing adopts after a crash (trustBackings: !durableStale.has(name)) while a stale-at-close MV still refills. A present-but-unparseable entry degrades to refill-everything; an absent entry (a store written before this entry existed — the upgrade path) falls back to the marker path below.
  • The write is gated on the same capability as the read (StoreModule.atomicProvider): a non-atomic session writes the set nowhere — not incrementally, not at close. This is load-bearing, not merely an fsync saving: the set's whole soundness rests on every entry having been maintained under the atomic no-tear guarantee, so a session that lacks it must never author one (see Rejected alternatives).
• Non-atomic provider — the clean-shutdown marker is the trust basis, and the durable stale-MV set is never written (see the gating note above). StoreModule.closeAll writes a reserved \x00meta\x00clean_shutdown entry after every batch has flushed, and rehydrateCatalog consumes it (read + immediate delete — single-use). A crash leaves no marker; a second rehydrate without an intervening clean close finds none either — both refill, and divergence self-heals. The marker value is the JSON stale-at-close set, threaded into per-entry trust (trustBackings: trusted && !staleAtClose.has(name)) so a stale-at-close MV refills while every live-at-close MV keeps the fast path; a missing or unparseable payload degrades to refill-everything. (The marker is still consumed in the atomic branch too, for single-use hygiene — its trust bit is simply ignored there.)

Durability ordering (soundness). The unsound case to prevent is a source's content-significant schema change being durable while the stale-set entry recording the resulting staleness is not — a reopen would then adopt a behind MV. The store persists the source DDL eagerly (a non-sync put inside module.alterTable) before the table_modified event fires; the stale-set write rides the same persistQueue in the event handler, after, with sync: true. On a WAL-ordered backend the sync flushes everything queued before it (the source DDL included), so a crash before the sync loses both (→ reopen sees old source + old stale-set → sound adopt) and a crash after has both (→ refill → sound). This is the same sync-point-write discipline the marker-consume delete uses, and carries the same documented backend caveat (below).

A body that cannot plan during a trusted import does not drop the backing (it errors per-entry with the backing preserved as a plain table). MV-over-MV ordering inside the fixpoint has its own gate: a dependent's body plans against its upstream's phase-1 plain table, so plan failure cannot order the rounds — instead ImportCatalogOptions.pendingDerivations (the qualified names of the session's still-pending MV entries) defers any entry whose body reads a pending maintained table to a later fixpoint round (a per-entry error, backing preserved; the preserved rows are what the retry round adopts). A declared-column arity mismatch (a select * body widened under an explicit mv(a, b) list) errors the same way: the entry can never materialize, so it errors per-entry with the backing preserved instead of dropping first. Likewise a post-adopt registration failure (the row-time gate) keeps the backing registered as a plain table rather than destroying durable rows. An adopted MV record is byte-identical to a refilled one (same canonical DDL, same hash formula), so adopt-vs-refill leaves identical catalog bytes, and the adopted backing keeps its persisted __stats__ row count.

Two trust caveats, both closed for any backend with a durability knob and sharing the same sync-point-write discipline:

• Marker durability under power loss. The consume-side delete and the session's data writes land in separate KV stores with independent flushing, so an unsynced marker delete lost to power loss could resurrect the marker and let the next open adopt across a genuine crash window. The marker-consume delete is therefore issued with sync: true — a backend-honored WriteOptions durability hint on the KVStore point-write surface (LevelDB fsyncs the delete; IndexedDB requests durability: 'strict') — forcing it durable before any of the session's data writes can become durable. A backend without a durability knob silently no-ops the hint (in-memory has no crash; any future sync-less backend is documented best-effort — losing the marker is conservative, since the next open simply refills).
• Stale-set durability under power loss. In the atomic domain the durable stale-MV set is the adopt fast path's logical-staleness basis, so it must be durable no-later-than the source DDL that caused the staleness. It is written with the same sync: true point-write riding the per-event persistQueue (after the source DDL is queued), giving the WAL-ordered durability argument above. The same backend caveat applies — a sync-less backend no-ops the hint, and losing the stale-set entry there is conservative (an absent entry falls back to the marker path; an over-naming entry only forces a sound refill).

Both caveats rest on WAL ordering rather than on a single atomic write. Folding the stale-set write into one batch() with the source DDL would remove that dependency; it is the remaining portability step, tracked in docs/todo.md.

Rejected alternatives

• Restrict which module combinations may host an MV (forbid a durable backing over a durable source in a different module), rather than documenting the cross-module crash window. Rejected: the window self-heals — rehydrate refills the maintained table from its body by default — so the restriction would cost real capability to close a hazard that already resolves at the next open.
• Per-arm capability gating — let a module advertise BackingHost without the ordered prefix-range scan, and disable the arms that need it. Rejected: it would fragment the maintenance planner across module capabilities, and both real host candidates (memory, store) are ordered key-value stores anyway. A module that cannot provide the ordered prefix scan simply does not advertise the capability.
• Write the durable stale-MV set unconditionally, in non-atomic sessions too. Rejected as unsound: a persistent non-atomic provider could leave a torn-but-"not-stale" set on disk that a later atomic reopen would trust, adopting a divergent backing forever. Gating the write on the same capability as the read makes that hole structurally impossible — a torn writer authors no set — so the read's trust never depends on an assumption that a store's capability never changes.