desc_log.md

descriptor log

Durable append-only relation-shape history; the decode-side catalog oracle. Owned by src/catalog/desc_log.rs, populated by src/source/catalog_capture.rs at catalog-commit boundaries, read wait-free by every decode site.

Why a log, not a live cache

Decoders ask a point-in-time question — descriptor(rfn, L) for a record at LSN L — while a live shadow-PG query answers "descriptor now". Bridging that gap with caches, invalidation epochs, baseline ledgers, and drop sweeps re-implements catalog time travel piecemeal, each piece with its own race window. The log stores the history itself: per-key version chains with valid_from bounds, so both bounds of every interval hold by construction and lookups are a binary search, no locks, no SQL, no replay gate.

Capture at boundaries

The filter classifies any write below FirstNormalObjectId (or a tracked relocated catalog filenode) as catalog and marks the writing xid dirty (filter.md); mid-xact XLOG_XACT_INVALIDATIONS records re-dirty an xid whose catalog writes precede the restart resume floor. At that xact's commit it builds a BoundaryInfo: the drain xid (prepared xid for COMMIT PREPARED), affected user oids (pump-side pg_class decodes ∪ the commit record's relcache invalidations), the xact tree's first catalog-touch LSN, and a capture-all flag (whole-relcache inval, pg_namespace catcache / whole-catalog inval, or a write to a catalog whose effects invals don't enumerate — see future/catalog_capture_completeness.md). Classification off the commit record alone is restart-safe: it carries the xact tree's full inval set, so a boundary is recognized even when every catalog record replayed before a crash.

BoundaryHoldSink sequences the boundary inside the publication hold (source.md):

flush predecessors → hold (shadow applies through next_lsn) →
capture (SQL fan-out → entries + events → append + fdatasync →
index publish → events into XactBuffer) → forward commit record

Nothing past the commit exists on wire, archive, or worker queue during capture, so the SQL snapshot is the commit's catalog state (pg_last_wal_replay_lsn() == next_lsn, enforced fatal), and any record reaching a decoder already has coverage.

Entries, events, intervals

Per captured oid, capture diffs the fresh descriptor against the oid's log predecessor:

• none/Dropped predecessor → Present entry + Added event
• shape change → Present entry (+ Changed when compute_schema_diff is non-empty; the diff is attribute-based, renames recapture silently)
• filenode rotation (rewrite/TRUNCATE/SET TABLESPACE) → Retired entry closing the old rfn chain, no event — AccessExclusiveLock means no decode query lands past the rotation; the entry exists so GC drops the chain and buggy callers fail closed
• absent from capture with a Present predecessor → Dropped tombstone + event at next_lsn

valid_from biases early — a descriptor is a backward-compatible reader of older tuples (missing attrs → default/NULL; dropped columns keep their physical slots), never the reverse. Sources in preference order: the rfn's XLOG_SMGR_CREATE marker (pump-side map, before any page write), the oid's first pg_class touch in the xact, the tree's first catalog touch. Events enter the drain keyed at valid_from, sorted with config events at drain open.

Bias-early is only sound when the final descriptor provably reads every tuple in the interval. compatible_reader (src/catalog/compat.rs) is the predicate, and it classifies rejects by physical consequence:

• proven: metadata-only changes (renames, defaults) and appended nullable / missing-value columns
• Benign — declared shape drifts, every byte reads the same: typmod, and attstorage. The walk reads attlen/attalign/attbyval only and varlena / numeric datums are self-describing, so bias-early still holds and nothing is published
• Physical — no descriptor reads both formats: type oid, typlen, alignment, attnum reuse, missing-value edit, not-null append without a missing value, relkind / persistence / toast-relation change

A physical in-place change (same rfn, no rotation) publishes an Ambiguity interval [first_touch, next_lsn) alongside the final Present, which lands at next_lsn: within the interval no single descriptor provably decodes the rfn's rows. One interval per identity key the verdict can name — Rfn then Oid, fixed order since batch equality and digest gate append idempotency — so the rfn-keyed decode path and the oid-keyed truncate path see the same fence without either consulting the other's map. Database scope stays the conservative fallback for unenumerable relations. Reason is UnknownMutationPosition: only the first catalog touch is tracked, so the mutation's exact position inside the interval is unknown. Ambiguities are batch records like entries — replay-from-log reproduces the same intervals every boot.

PG rewrites for every ALTER that moves tuple layout, and a rotation skips the predicate, so a physical in-place verdict describes shapes PG does not currently produce. The fence is a guard against unmodelled ones, not a routine path.

Toast rels ('t') capture entries and Dropped events only (the retire ledger consumes those); indexes are excluded entirely.

Pending capture

Pending capture samples the same relations mid-transaction. At wal_level=logical PG logs XLOG_XACT_INVALIDATIONS from every CommandCounterIncrement, and a relation's layout cannot move except at one, so those records are exactly the sample points. Inside a dirty xact each becomes a BoundaryKind::Command boundary, holding publication the same way a commit does; capture reads the relations that command's invals name off the bridge worker's SCAN at the parked position, where the transaction's own catalog rows sit on-page uncommitted (shadow.md Bridge worker). Shapes land in PendingCatalog (src/catalog/pending.rs), keyed by the tree root the pump knew at capture, valid_from at the boundary or at the generation's smgr marker when the relation was born in this xact.

A pending descriptor is visible only to records of the transaction that wrote it, and becomes durable only at that transaction's commit. Nothing speculative reaches the log, so abort is a map removal: the abort record names every member the filter drained and their slots die on the pump, ahead of any later boundary that could promote them. A subxact abort drops the slots its own xid wrote and leaves the parent's.

At the commit, the member keys fold under the top and the slots enter the batch as Present entries at their own positions, ahead of the commit shape — one entry per (relation, command boundary) instead of one per relation. That is what shrinks the fence: an unproven in-place change whose relation the timeline covers publishes its Ambiguity only over [first_touch, first boundary), because rows past that boundary have an exact shape recorded and rows before it predate the transaction's first CommandCounterIncrement — a command sees the catalog as of its own start, so the predecessor reads them. The stash resolution folds the same chain per record, so a record inside a covered run decodes under the shape that run saw rather than the commit-time descriptor.

Every failure degrades the transaction to commit-time capture, which is sound: CaptureAll (whole-relcache flush or namespace catcache — a full catalog scan per command is the shape that makes holds expensive), CapExceeded (pending_max_boundaries_per_xact), HoldBudget (pending_max_hold_ms cumulative), ReplayMismatch (shadow was not parked where the boundary said — unrecoverable, replay cannot rewind), QueryError. A degraded transaction's slots still promote, since each is an exact shape at an exact position; what degradation costs is the coverage claim, so its fence stands. A boundary whose inval set names no user relation skips the hold entirely.

Replay-from-log

Every boundary appends a batch keyed captured_at = next_lsn — a zero-entry stub when nothing changed. Boot loads ckpt + tail, then the WAL re-read finds each boundary's batch already stored and derives events from the stored entries against predecessor_before(oid, captured_at) (the historical predecessor, never the loaded head) — no SQL, identical events every replay. A miss with shadow replayed past the boundary means the log lost coverage: fatal, remedy --ignore-cursor (which deletes the log) or re-bootstrap. The manifest version gates pre-log spill dirs the same way (ops.md).

Seed + coverage horizon

An empty log seeds one batch from fetch_all_descriptors (every rel relkind IN ('r','p','m','t'), oid ≥ 16384) at the raw resume position, entries valid from the aligned start, persisting covered_through in the ckpt. The aligned-prefix re-read decodes against the seed; boundaries at or below covered_through skip capture and event replay (baked into the snapshot); NotCovered at or below it is a counted row skip (rel died pre-snapshot). Every boot also runs a boot-Added pass over the log's active Present set — auto-create namespaces and opted-in mapped rels get their idempotent CREATE TABLE IF NOT EXISTS at attach, and newly enabled config picks up existing rels without log mutation.

Decode reads

descriptor_at(rfn, lsn) / descriptor_by_oid_at(oid, lsn) return Present | Ambiguous | Dropped | Retired | NotCovered | ForeignDb. The rfn key carries its own database, so the rfn lookups scope themselves (db_node == 0 shared locators pass — shared catalogs hold no descriptors). A bare oid does not: oids repeat across databases, so production oid callers take descriptor_by_oid_in_db_at_spanned(db_oid, oid, lsn), which compares the supplied database with the log's identity and counts lookups_foreign_db before any chain read. XLOG_HEAP_TRUNCATE apply is the caller — it names oids and its own dbId (xact.md). Unscoped descriptor_by_oid_at remains for callers that already hold the log's own database, i.e. tests. Ambiguity precedes the chain — a chain entry inside an ambiguous interval is not proven safe for rows there, so the interval check runs first (rfn/oid scope, then database scope). The _spanned twins return the Present descriptor plus its entry's valid_from under one index read (two reads could interleave with a bias-early append and pair a stale descriptor with a fresh span); present_before serves historical predecessors:

• worker buffering: Present decodes; ForeignDb and horizon/xid-0 NotCovered are counted skips; NotCovered/Dropped with a live xid stash for commit-time resolution (xact.md Commit-time stash); Ambiguous stashes when the filenode is marker-proven and fails closed when it is not (a markerless stash keeps no payload, so discarding would be silent row loss); Retired skips (rows can't outlive the rotation)
• stash resolution at commit next_lsn: Present toast → chunk decode behind its marker barrier, Present ordinary → raw decode under the commit-resolution descriptor. Resolution asks at next_lsn, which sits past every interval the commit published, so the fence is a separate query — ambiguities_intersecting(rfn, oid, first stashed lsn, next_lsn) rides the verdict and the drain fails closed per record whose own source_lsn lands inside one. Tombstones discard: under AccessExclusiveLock no row on a dropped or rotated filenode outlives the commit
• planning: descriptors ride each heap's envelope from the buffering / stash step; the planner never re-resolves (emitter.md)
• TRUNCATE fan-out resolves by oid; the barrier apply falls back to the rfn chain's last Present when the truncating commit itself retired the rfn (rotation's Retired lands before the truncate record)

Storage

desc_log.ckpt + desc_log.tail under the spill dir. Shared binary header binds pg major, system id, timeline, db oid, and segment size — mismatch fatal, mirroring the manifest's foreign-source gate. Frames are [len u32][crc32c][body]; the ckpt (written via fs::write_atomic) carries a meta frame (covered_through, floor_at_write) plus compacted batches; the tail is fdatasynced per boundary. A torn final frame truncates durably at load; interior CRC failure is fatal. One writer mutex serialises append and GC; readers take an RwLock'd index snapshot published only after fsync.

GC runs off the pump task, fed each persisted floor over a watch channel (ops.md) so a ckpt rewrite never stalls WAL consumption into the source's wal_sender_timeout; failures set a fatal the pump surfaces on its next turn. Boundary capture still shares the writer mutex, so a boundary landing mid-compaction blocks its hold. Retention: per key the entry active at the floor survives when Present; a Dropped/Retired there drops the whole at-or-below chain (nothing above can reference it — records predate the drop and the floor never exceeds the re-read start); batches above the floor survive whole, stubs included, and an interval survives while through_lsn is above the floor. Thresholds: ≥512 droppable entries or an 8 MiB tail.

Identity keys the full physical RelFileNode: relfilenumbers are unique only per database of one tablespace (future/TABLESPACES.md §0), so (db_node, rel_node) alone can alias two live relations. Capture resolves the pg_class.reltablespace 0 sentinel to the database's dattablespace, making stored rfns directly comparable to WAL locators' physical spcOid.

Metrics

walshadow_desc_capture_* (sql / log_replay / skipped_covered / capture_all / rels / seconds), walshadow_desc_events_*, walshadow_desc_log_* gauges + GC counters, walshadow_desc_lookups_* by result. Capture time counts inside the boundary-hold duration.

Pending capture adds walshadow_pending_captures_total, walshadow_pending_rels_total, walshadow_pending_holds_total, walshadow_pending_hold_seconds_total, walshadow_pending_entries_promoted_total, walshadow_pending_entries_dropped_abort_total, walshadow_pending_ambiguities_suppressed_total and walshadow_pending_degraded_total{reason}. Overlay-scan cost and its unresolvable-parentage count sit on the bridge families (walshadow_bridge_scan_*), which only pending scans populate — a committed read passes no transaction, so nothing is left to misattribute.