compute: delay retracting a dropped export's lifecycle rows [deferred, unsound] - #38417
Closed
antiguru wants to merge 17 commits into
Closed
Conversation
Record each compute export's lifecycle as an append-only log,
`mz_introspection.mz_compute_lifecycle_events_per_worker`, rather than as more
timestamp columns on the hydration time relation.
Two things stop timestamp columns from carrying the lifecycle. The stages do not
share a grain: `installed`, `started` and `hydrated` are per-worker facts, since
each worker hydrates its own fragment of the dataflow, while whether the output
is durable is a property of the sink as a whole, maintained on one elected
worker. And a timestamp cannot say why the next stage has not happened, so a
NULL cannot tell a replacement materialized view waiting for a cutover apart
from an index that will never write.
export_id text not null
worker_id uint8 not null
event text not null
occurred_at timestamptz not null
reason text nullable
details jsonb nullable
`installed`, `started` and `hydrated` are logged by every worker. The write
stages are logged only by the worker that maintains the sink's shared write
frontier, so they appear once per object and the row records which worker was
elected. An index emits the first three and stops, which is the index degeneracy
of the lifecycle falling out of the model rather than being special-cased.
`hydrated` reads the dataflow's own progress frontier, the compute probe, not
the reported output frontier. The output frontier folds in the write frontier,
which makes it a measure of durability, and for a sink-backed collection it is
not even uniform across workers: `mint` clears the shared frontier on every
non-elected worker, where it is the empty antichain and contributes nothing to
the meet. `mz_compute_hydration_times_per_worker` is unchanged, so `hydrated_at`
and `time_ns` keep reporting exactly what they reported before, and the new
relation carries the dataflow reading alongside.
The write stages are gated on hydration. Before it the sink has produced nothing
and read-only mode is holding nothing back, and every collection starts
read-only, so reporting a block from installation would put a `write_blocked`
and a `write_unblocked` on essentially every materialized view, both ahead of
`hydrated`. Gating also keeps `written` ordered after `hydrated`, which it is not
otherwise: `apply_refresh` rounds a `REFRESH` materialized view's frontier up to
the next refresh time off its input frontier, before the dataflow computes
anything, so the sink writes an empty batch for the pre-refresh window and the
shard's upper passes the as-of while the dataflow is still hydrating. That also
means a refresh schedule advances writing rather than blocking it, so there is no
`refresh` cause for `write_blocked` to report.
`details` carries the dataflow's as-of, which every stage is defined relative to:
without it the interval between two stages says nothing about how much work was
done, since a replacement materialized view with a far behind as-of is a
completely different amount of work at the same duration.
Part of CPU-226
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018ZVCMBSLdxzGus78ZKWhZz
`observe_hydration` took a bool, leaving the choice of frontier and the handling of an empty one to its two call sites. Those call sites had already diverged. `report_frontiers` compares the as-of against a probe or write frontier with no emptiness check; `process_subscribes` compares it against a batch upper and guards with `matches!(response, SubscribeResponse::Batch(_))`, whose comment claims that filtering out `DroppedAt` is enough to stop a cancelled subscribe from reading as hydrated. It is not enough. A subscribe signals completion by sending a batch at the empty frontier, so the completion batch passes the `matches!` and the empty antichain is the maximum of the order. `SUBSCRIBE ... UP TO x AS OF x` is legal, only attaching an `EqualSubscribeBounds` notice, and it emits no rows at all: its `up_to` filter admits no times. It then logs `hydrated` for a dataflow that computed nothing. Take the frontier itself and decide inside, so there is one definition of what counts as progress. An empty frontier reports completion, not hydration, whether it arrives as `DroppedAt`, as a completion batch, or from anywhere else. A subscribe that genuinely hydrated has already logged it from the preceding batch, so nothing is lost. Part of CPU-226 Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ZVCMBSLdxzGus78ZKWhZz
`written` compared the as-of against the sink's write frontier. That frontier is
the output shard's upper, a property of the shard rather than of this replica,
and `as_of_selection::apply_downstream_storage_constraints` bounds the as-of to
one step below the upper for a non-empty storage export. So for any shard that
already holds data the comparison is true from the moment the dataflow is
installed, and the stage says only that somebody once wrote the output.
The read-only gate did not fix this, it deferred it by one poll. A replica that
is permitted to write is not thereby the replica that wrote. Three cases got a
`written` for appending nothing:
* a replica added by raising the replication factor,
* a replica whose process restarted,
* a read-only replica at cutover, where `write_unblocked` and `written` land in
the same call and the interval between them is always zero.
The third is the case the relation exists to measure, and the previous NOTE at
this site claimed the guard prevented exactly what it permitted.
Latch the frontier the first time writes are permitted and require it to advance
past that baseline. The latch is deliberately before the hydration gate rather
than after it: latching later would fold this replica's own early writes into the
baseline and never report them, while latching before the block is lifted would
measure against an upper the previous writer goes on to advance.
`written` now means that this replica's sink advanced the shard beyond where it
stood when the replica was allowed to write. For a fresh materialized view, whose
as-of is not stepped back, that is the first real append, unchanged from before.
Part of CPU-226
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018ZVCMBSLdxzGus78ZKWhZz
The relation promises at most one row per export, worker and stage, and a reader relies on it: an event's `occurred_at` is taken directly, without aggregating first. That promise was kept by three unrelated mechanisms, none of them in the demux that owns the log. `CollectionState::logged_stages` covered four stages, `hydration_timestamps.started_at` covered `started`, and `installed` relied on `handle_export` running once per export. `log_lifecycle_at` pushed unconditionally, and `CollectionLogging::log_lifecycle` is public and accepts any stage, including the two the demux already emits from its own events. A caller passing either produced a duplicate row that nothing rejected. The doc comment stated the contract, the type did not. Key the retained rows by stage and ignore a stage already logged. The bound on the map is now structural rather than incidental, the back-fill of `started` in `handle_hydration` no longer needs to know whether `handle_hydration_start` already ran, and a caller that cannot cheaply tell whether it has reported a stage does not have to. Part of CPU-226 Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ZVCMBSLdxzGus78ZKWhZz
Three cleanups to the boundary the write lifecycle stages depend on. `frontier_owner` lived in the v1 materialized view sink and was named for a property v1 does not own. The election belongs to the persist sink protocol that both implementations share, so it moves to `crate::sink`, and all four call sites now spell it the same way. `materialized_view_v2` had been reaching for it twice in one file under two different paths. `metric_sink` carries a byte-identical expression and must keep it. Its shared frontier is written by every worker, before the early return for the inactive ones, so it has no elected owner at all, and the worker it does elect is the one that folds metrics into the registry. The obvious next cleanup would be to call `frontier_owner` there and silently couple two independent elections, so both sites now say why they are separate. `sink_write_frontier` and `owns_sink_frontier` were two public fields whose coupling lived only in a doc comment. Three sinks set the frontier and two set the flag. A future sink that sets the frontier and forgets the flag loses every write stage with no error, and one that sets it wrongly reports having written everything immediately, because a non-owning worker's copy is cleared to the empty antichain. One setter takes both, and the fields are now private. Also correct the `ExportState::as_of` comment: a dataflow retained across reconciliation logs no new `Export` event, so the field holds the as-of it was installed with rather than the collection's current one. Part of CPU-226 Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ZVCMBSLdxzGus78ZKWhZz
A dataflow's exports share one suspension token, so its computation begins only once every export has been scheduled. `handle_schedule` says so in a comment and then reports hydration start for the single export the command named. For a dataflow exporting A and B, scheduled at t1 and t2, the dataflow starts at t2 but A's `started` reads t1. A claims to have started while its dataflow was still suspended, and `hydrated - started` overstates the compute time by t2 - t1. Report the start only on the token release that actually unsuspends the dataflow, and report it for every export of that dataflow. The start is a property of the dataflow, so all its exports share the instant, and now they share it because that is when computation began rather than by approximation. This is latent today: nothing in production ships more than one export per dataflow, since `export_index` and `export_sink` each insert one. It affects `mz_compute_hydration_times_per_worker.started_at` equally. Part of CPU-226 Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ZVCMBSLdxzGus78ZKWhZz
Rejecting the empty frontier in `observe_hydration` was wrong. An inputless collection, such as an index or materialized view on `SELECT 1`, emits its rows and drops its capability, so its frontier goes from the initial time straight to the empty antichain without ever holding a non-empty value beyond its as-of. Rejecting empty leaves such an export permanently unhydrated in the lifecycle relation, and because the write stages are gated on `hydrated`, a materialized view of that shape reports no write stage ever. It also contradicted the durability reading. `CollectionState::hydrated` compares the as-of against the reported output frontier with no emptiness check, so the existing relation calls these exports hydrated. The two relations would have disagreed for the same object. Emptiness therefore cannot distinguish completion from cancellation, and a caller that can observe a dataflow ending without computing its as-of has to exclude that itself. `process_subscribes` is the one such caller: `DroppedAt` carries the empty antichain for a subscribe cancelled mid-hydration, so it keeps the batch check that excludes it. A subscribe that runs to its `up_to` also finishes with an empty upper, but it carries that in a batch and did compute through its as-of. The frontier-taking signature stays. Passing the frontier rather than a bool is what gives the emptiness question a single answer instead of one per call site. Part of CPU-226 Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ZVCMBSLdxzGus78ZKWhZz
The latched baseline did not work, for two independent reasons. It latched a placeholder. `mint` initializes the shared sink frontier to `Antichain::from_elem(Timestamp::MIN)` and only fills in the real upper once its `persist_frontiers` stream has read it back. The latch runs from the first `report_frontiers` poll that sees writes permitted, which is normally before that, so the baseline was `[MIN]` and any real upper compared greater. `written` fired at once, which is what the baseline was meant to prevent. And no reading of that frontier can attribute a write. Every replica's `mint` reads the shard's upper back from persist into its own shared frontier, so the frontier advances on every replica when any one of them wins the append. A baseline latched from it, placeholder or not, says only that the shard moved while this replica was eligible to write. Go back to comparing against the as-of and state the semantics instead of implying stronger ones. `written` means the output is durable through the as-of and this replica was permitted to write. Replicas of a cluster produce identical batches and race to append, so which replica won is not an operationally meaningful question, but it does mean `written` lands with `hydrated` on a restarted or scaled-out replica and with `write_unblocked` at a cutover, and a reader has to know that. Real attribution needs a signal from this replica's own append path. That has to cross workers, because `next_append_worker` rotates independently of `sink::frontier_owner`, so it is left as a TODO rather than approximated. Part of CPU-226 Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ZVCMBSLdxzGus78ZKWhZz
…nator A consumer rolling the per-worker relation up per object has to know whether every worker has reached a stage. `mz_compute_hydration_times_per_worker` lets it ask without knowing the worker count, because it holds one row per export and worker from installation with a nullable `time_ns`, so `count(*) = count(time_ns)` is the test. The introspection subscribe for `ComputeHydrationTimes` uses exactly that. An append-only log has no NULLs to count: a worker that has not hydrated has no `hydrated` row. The equivalent is `installed`, which every worker logs unconditionally when the export is created, so the count of `installed` events is the number of workers reporting on that export. That makes it a contract the relation owes its readers rather than an incidental property of the implementation, and it was written down nowhere. Say it in the ontology description, where a reader of the relation will find it, and in the design doc alongside the grain rules, together with the point that which stages have which grain is fixed by the vocabulary rather than varying per object or cluster, so a consumer never needs a catalog join to interpret a count. Part of CPU-226 Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ZVCMBSLdxzGus78ZKWhZz
The design doc argued that withholding `written` while writes are blocked keeps another writer's progress from being attributed to this replica. That is true of a blocked replica and says nothing about a permitted one, which leaves the reader with a stronger impression of the stage than it earns. Every replica's `mint` reads the output shard's upper back from persist into its own shared frontier, so the frontier advances on every replica when any one of them wins the append. State that, state the two cases where the implied interval is therefore zero, `written` landing with `hydrated` on a restarted or scaled-out replica and with `write_unblocked` at a cutover, and record why the obvious tightening does not work: a frontier latched when writes are first permitted sees the same concurrent advance, and the value available at the first observation is usually the placeholder `mint` starts from rather than a real upper. Attribution needs a signal from the replica's own append path, which has to cross workers because `next_append_worker` rotates independently of the frontier owner. That goes under "Follow-up work" with the open question of whether the distinction earns the machinery, given that the racing replicas append identical batches. Part of CPU-226 Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ZVCMBSLdxzGus78ZKWhZz
The stages do not all describe the same object. `hydrated` and the write stages belong to one export: hydration reads that export's own progress, and the write stages read its sink. `installed` and `started` describe the dataflow, which can maintain more than one export, and `started` is the instant that dataflow was unsuspended, shared by every export it maintains. Keying the relation by export is still right. The identifier a consumer has is the export id, which is what `mz_objects.id` holds, while dataflow ids are per worker and internal to a replica. A dataflow-grain relation could not hold hydration or the write stages at all, so splitting along the seam would produce two relations instead of one and make the most basic question a join through the export-to-dataflow map. Carry the dataflow id as a column instead. The events whose cause is dataflow-wide become recognizable as such, `SELECT DISTINCT dataflow_id, event, occurred_at` recovers the dataflow-level facts, and the redundancy is visible rather than implied. The relation also ends up carrying the export-to-dataflow mapping that `mz_compute_exports_per_worker` holds, for eight bytes a row. No new migration step. `make_mz_indexes` inlines each log's `index_by` column names, so `mz_indexes` moves again, but `MigrationStep::replacement` records only a version and an object, and the step this change already added at the current dev version covers any further change to that SQL within the same version. `make_mz_sources` inlines no columns, so it does not move. Part of CPU-226 Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ZVCMBSLdxzGus78ZKWhZz
antiguru
force-pushed
the
claude/lifecycle-retraction-delay-js1ycm
branch
from
August 24, 2026 11:28
8ea2b6a to
069d6fe
Compare
`lifecycle_event_in_dataflow` declared `dataflow_id` a many-to-one foreign key into `dataflow_global_id_per_worker.id`, which it is not. `ComputeLog::DataflowGlobal` keys that relation on `(id, worker_id, global_id)`, one row per object rendered in the dataflow, so the advertised join both crosses workers and multiplies every lifecycle event by the number of rendered objects. Adding `worker_id` through `extra_key_columns` would not fix it, because the fan-out over `global_id` remains: the relation is a dataflow-to-object mapping, not a dataflow entity, and no ontology entity is keyed by `(dataflow_id, worker_id)` for the link to point at. This metadata is acted on rather than merely read. `mz_ontology_link_types` is what agents are pointed at to discover join paths, so a wrong cardinality produces a query that silently fans out instead of a description that merely reads oddly. Remove the link and put the join guidance where it cannot mislead a planner: the column description now says dataflow ids are worker-scoped, names `(dataflow_id, worker_id)` against `mz_compute_exports_per_worker` as the way to reach a dataflow's exports, and warns that the global-ids relation fans out. The surviving `lifecycle_event_of` link is `MapsTo`, which carries no cardinality and so claims nothing false, but it joins two per-worker relations on `export_id` alone. Its `note` now records that the exact join is `(export_id, worker_id)`. Part of CPU-226 Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ZVCMBSLdxzGus78ZKWhZz
antiguru
force-pushed
the
claude/lifecycle-retraction-delay-js1ycm
branch
from
August 24, 2026 11:38
069d6fe to
c56501a
Compare
The description claimed dataflow ids are worker-scoped and only meaningful paired with `worker_id`. They are not. Each worker assigns indices from a counter advanced by the same command sequence, so a dataflow carries the same index on every worker of a replica, and `dataflow_id` identifies the dataflow rather than a per-worker artifact. It is still a replica-local index and not a catalog id, which is the part that matters for a consumer. Dropping the `lifecycle_event_in_dataflow` link remains right, for the other reason given: `mz_compute_dataflow_global_ids_per_worker` holds one row per object rendered in a dataflow, so `dataflow_id` is not unique there even within a single worker, and a foreign key claiming otherwise makes a generated join fan out. Part of CPU-226 Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ZVCMBSLdxzGus78ZKWhZz
`cluster.slt` lists each per-replica introspection index's key columns with their positions, so a column added to an unkeyed log shifts every position after it. `cockroach/srfs.slt` generates a series over each relation's column count, so it gains a row per instance. Both were missed when the column was added. The position listing in `cluster.slt` is a third section of that file, distinct from the two counts already updated there, and searching the file for the relation's name finds it only if the search is not anchored on those counts. Part of CPU-226 Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ZVCMBSLdxzGus78ZKWhZz
antiguru
force-pushed
the
claude/lifecycle-retraction-delay-js1ycm
branch
from
August 24, 2026 12:14
c56501a to
5981d21
Compare
Four corrections, none changing what the relation reports. `sink_write_frontier` was left public. Its own doc comment says it is set through `set_sink_write_frontier` together with `owns_sink_frontier`, and the point of that setter is that the pairing cannot be forgotten, which a public field defeats. `report_frontiers` is the only other reader and is in the same module. `handle_schedule` scanned every collection to find the dataflow's exports. A `Schedule` arrives once per dataflow, so a replica starting N dataflows made N full traversals of the collection map on the timely worker thread, quadratic in exactly the phase where start-up latency is the thing being measured. Two strong references to the dataflow index mean the named collection is the only export, so take that case directly. `drop_collection` releases a suspension token without the check `handle_schedule` now makes, so for a multi-export dataflow it can be the release that unsuspends the computation while reporting no start. Unreachable while every dataflow reaching a replica has exactly one export, which `SequentialHydration::absorb_command` requires, but silent divergence between two paths that release the same token deserves a note rather than nothing. `CollectionLogging::log_lifecycle` still told callers the demux does not deduplicate. It does, since the retained rows became keyed by stage, and two handlers rely on it. A caller trusting the stale comment would add a redundant guard, or trust it in the other direction. Part of CPU-226 Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ZVCMBSLdxzGus78ZKWhZz
The lifecycle log retracts an export's rows the moment it is dropped, so an
object's history vanishes with the object. That loses exactly the episodes worth
looking at: a dataflow that was dropped before it hydrated, or one whose
hydration is the reason someone is reading the log at all. It also means a short
lived dataflow can come and go inside one introspection interval and leave no
trace.
Delay the retraction instead, and record the drop as a stage of its own.
Two delays, because the two populations churn at completely different rates. A
transient export is created per peek and per subscribe, so retaining those for
minutes costs hundreds of megabytes on a busy replica: at eight workers and two
hundred peeks per second, a five minute window holds around 1.4 million rows.
A few seconds is enough for a reader to observe them and costs single digit
megabytes. A user object is dropped by DDL, so even a thousand drops inside the
window is under ten megabytes, and there the history is worth keeping.
compute_lifecycle_retraction_delay default 5 min
compute_lifecycle_retraction_delay_transient default 5 s
Both are floored at the logging interval in code rather than by convention. The
demux rounds update timestamps up to that interval, so a shorter delay can round
to the same timestamp as the insertion and leave the rows never separately
visible, which would defeat the point.
The delays are read per batch rather than at construction, so an
`UpdateConfiguration` command takes effect without recreating the logging
dataflow, and a per-replica override applies to a replica under investigation
without touching the rest.
The `dropped` stage is what makes the delay readable. Without it a lingering row
says only that an export reached some stage, not whether it still exists, so
"hydrated but never written" could not be told apart from "dropped before it
wrote". With it the object's last event names its fate and the delay is pure
retention, and a row whose export id no longer appears in `mz_objects` is
explained by its own `dropped` event rather than reading as a leak.
Part of CPU-226
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018ZVCMBSLdxzGus78ZKWhZz
The design doc says `catalog_server_explain.slt` needs no change when a builtin log is added, on the grounds that its query filters `o.id NOT LIKE 'si%'` and so never enumerates a per-replica introspection index. The filter is real, but the conclusion does not follow. The plans already in the file embed the inlined builtin `VALUES` sets as `Constant (N rows)` nodes, so every count over a catalog relation that gained a row moves, and adding an ontology entity and link moves two more. Record the question that catches this: not whether a new plan appears, but whether the existing plans change. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ZVCMBSLdxzGus78ZKWhZz
antiguru
force-pushed
the
claude/lifecycle-retraction-delay-js1ycm
branch
from
August 24, 2026 13:02
5981d21 to
4541a11
Compare
antiguru
changed the title
antiguru
force-pushed
the
claude/hydration-visibility-compute-js1ycm
branch
from
August 27, 2026 14:54
fb01c00 to
fe9c7b4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Blocker: delayed retraction is unsound while ids can be reused
The delay retracts an export's rows at a future timestamp while removing its
ExportStateimmediately. Nothing records that a retraction is pending, so a new export with the sameGlobalIdinside the window inserts a fresh row and the relation ends up holding two rows with the same(export_id, worker_id, event)from different incarnations. That breaks the at-most-once property the relation documents and thattest/testdrive/compute-lifecycle-events.tdasserts undermax-tries=1, and it corrupts any duration a reader computes withmax(occurred_at) FILTER (...).Transient ids are reused because
TransientIdGen::new()starts at 1 per process (src/repr/src/global_id.rs:123-129), so a plain environmentd restart — replicas survive and reconcile, transient dataflows are dropped, new generation reissuest1…— collides inside the window.Non-transient ids can also be reused during reconciliation, so the window is not a transient-only problem. That kills the obvious fix: excluding transient exports from the delay does not make this sound, and the split between
compute_lifecycle_retraction_delayandcompute_lifecycle_retraction_delay_transientis not the axis that matters. Any future attempt needs to make retraction and re-insertion of the same id ordered — for example apending_retractionsmap flushed at the current timestamp whenhandle_exportsees an id whose retraction is still outstanding — before the delay is worth having at all.Note the base PR is unaffected: it retracts at the drop timestamp, so no window exists in which two incarnations of an id coexist.
Two further findings, unaddressed
compute_lifecycle_retraction_delaypanics the replica ints_at'sexpect("must fit"). There is a lower clamp to the logging interval but no upper one, andALTER SYSTEM SETreaches it.export_idis absent frommz_compute_exports_per_workerfor up to the delay. The ontology description was written beforedroppedexisted and still implies a lifecycle row means a live export, and the design doc and the testdrive orphan check disagree about whether an orphaned row is expected.Original motivation
The lifecycle log retracts an export's rows the moment it is dropped, so an object's history vanishes with the object. That loses exactly the episodes worth looking at: a dataflow dropped before it hydrated, or one whose hydration is the reason someone opened the log. A short-lived dataflow can also come and go inside one introspection interval and leave no trace.
Part of CPU-226.
What is in the diff
Two commits on top of #38403: the retraction delay with a
droppedstage and two dyncfgs, and an unrelated design-doc correction aboutcatalog_server_explain.slt.The
droppedstage is the part worth keeping from this attempt. Without it a lingering row says only that an export reached some stage, not whether it still exists, so "hydrated but never written" cannot be told apart from "dropped before it wrote".The storage reasoning also stands independently of the soundness problem: a transient export exists per peek and per subscribe, so at eight workers and two hundred peeks per second a five-minute window holds around 1.4 million rows, hundreds of megabytes, while a user object is dropped by DDL and a thousand drops inside the window is under ten megabytes. Whatever replaces this will still need to treat those two populations differently, just not by delay alone.