Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
source: src/adapter/src/catalog/open/builtin_schema_migration.rs
revision: c317ceee3c
revision: 46f729653a
---

# adapter::catalog::open::builtin_schema_migration
Expand All @@ -20,7 +20,7 @@ Three `Replacement` steps at version `26.38.0-dev.0` cover: `mz_catalog.mz_sinks
A `Replacement` step at version `26.38.0-rc.2` covers `mz_internal.mz_cluster_reconfigurations`, whose MV definition changed to include `arrangement_compression` in the `changes` diff column.
Three `Replacement` steps at version `26.39.0-dev.0` cover: `mz_catalog.mz_audit_events` (because the MV gained a `metric-sink` arm in its `object_type` CASE expression, changing its SQL fingerprint); `mz_catalog.mz_tables` and `mz_catalog.mz_views` (reflecting their conversion from `BuiltinTable` to `BuiltinMaterializedView` over `mz_catalog_raw`). The `mz_indexes` step originally at `26.39.0-dev.0` (covering `mz_object_graph_edges_ind`) was superseded and is no longer present.
Three `Replacement` steps at version `26.40.0-dev.0` cover: `mz_catalog.mz_indexes` (required because adding the `mz_cluster_replica_resource_usage` builtin log changes the VALUES set inlined by `make_mz_indexes`, altering the MV's SQL fingerprint, superseding the earlier `mz_metric_sinks_ind` step); `mz_catalog.mz_sources` (required because `make_mz_sources` also inlines the builtin-log set as VALUES, so adding a builtin log changes its SQL fingerprint); and `mz_internal.mz_type_pg_metadata` (because the table gained a trailing `typsend` column).
When applying replacement migrations, `mz_storage_usage_by_shard` and `mz_object_hydration_history` are excluded from data-destroying replacement plans. `mz_storage_usage_by_shard` retains billing data; `mz_object_hydration_history` cannot be rebuilt from any other source. Both tables participate in forced evolution (which keeps the shard and its rows) but not in forced replacement. `validate_migration_steps` asserts that any declared `Replacement` step for `mz_object_hydration_history` is a deliberate, explicit choice rather than an accidental schema change. `validate_migration_steps` also asserts that `mz_cluster_replica_frontiers` is never declared as a migration target: the 0dt caught-up gate reads the leader's `mz_cluster_replica_frontiers` shard for the live frontiers it checks every collection against, so a `Replacement` step would hand the read-only environment a self-owned shard, causing the gate to compare itself against itself rather than against the leader.
The `participates_in_forced_migration(builtin, mechanism)` function encapsulates which builtins take part in forced migrations: tables participate unless they are `mz_storage_usage_by_shard` (for any mechanism) or `mz_object_hydration_history` (for `Replacement`); materialized views always participate; sources participate unless they are `mz_catalog_raw`; logs, views, types, functions, indexes, and connections never participate.
When applying replacement migrations, `mz_storage_usage_by_shard`, `mz_object_hydration_history`, and `mz_replica_hydration_history` are excluded from data-destroying replacement plans. `mz_storage_usage_by_shard` retains billing data; the two hydration history tables cannot be rebuilt from any other source. All three tables participate in forced evolution (which keeps the shard and its rows) but not in forced replacement. `validate_migration_steps` asserts that any declared `Replacement` step for `mz_object_hydration_history` or `mz_replica_hydration_history` is a deliberate, explicit choice rather than an accidental schema change. `validate_migration_steps` also asserts that `mz_cluster_replica_frontiers` is never declared as a migration target: the 0dt caught-up gate reads the leader's `mz_cluster_replica_frontiers` shard for the live frontiers it checks every collection against, so a `Replacement` step would hand the read-only environment a self-owned shard, causing the gate to compare itself against itself rather than against the leader.
The `participates_in_forced_migration(builtin, mechanism)` function encapsulates which builtins take part in forced migrations: tables participate unless they are `mz_storage_usage_by_shard` (for any mechanism) or `mz_object_hydration_history` / `mz_replica_hydration_history` (for `Replacement`); materialized views always participate; sources participate unless they are `mz_catalog_raw`; logs, views, types, functions, indexes, and connections never participate.
When the source and target versions differ and the source version is a dev build, `Migration::run` forces evolution-mode migration even without an explicit `force_migration` config, avoiding version-based filter failures in dev environments.
In forced dev-to-dev migrations, `migrate_evolve_one` skips builtins that do not yet have a shard registered. Brand-new builtins in a given version may not have their shards allocated until the leader completes bootstrap; excluding them avoids "missing shard ID" errors on read-only replicas.
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
---
source: src/adapter/src/catalog/open/builtin_schema_migration_tests.rs
revision: 3a822d8457
revision: 46f729653a
---

# adapter::catalog::open::builtin_schema_migration_tests

Contains turmoil-based integration tests for `builtin_schema_migration`, verifying that schema migrations applied across multiple simulated versions and concurrent processes produce consistent results.
The main test generates random builtin tables and sources, evolves their schemas across versions with random migration steps (evolution or replacement), spawns multiple concurrent processes per version, and asserts that all processes converge on the same migration outcome despite random crashes.
`test_migration_steps_resolve_to_builtins` verifies that every step in the `MIGRATIONS` list names a builtin that currently exists with the declared `CatalogItemType`. A step naming a builtin that no longer exists or that changed type would panic `validate_migration_steps` at catalog open, which only the upgrade nightly catches; this unit test catches the same failure cheaply at CI time.
`hydration_history_forced_migration_policy` verifies that `mz_object_hydration_history` participates in a forced evolution migration but not in a forced replacement migration, confirming the exemption in `participates_in_forced_migration` that prevents accidental data loss when schema changes are force-applied during dev upgrades.
`hydration_history_forced_migration_policy` verifies that both `mz_object_hydration_history` and `mz_replica_hydration_history` participate in a forced evolution migration but not in a forced replacement migration, confirming the exemptions in `participates_in_forced_migration` that prevent accidental data loss when schema changes are force-applied during dev upgrades.
6 changes: 3 additions & 3 deletions doc/developer/generated/adapter/coord/_module.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
source: src/adapter/src/coord.rs
revision: a1bcaebfe6
revision: 46f729653a
---

# adapter::coord
Expand All @@ -9,14 +9,14 @@ The core coordinator: `coord.rs` defines the `Coordinator` struct (the central s
The coordinator owns the `Catalog`, active compute sinks, pending peeks, read-policy manager, timeline oracles, and all inter-subsystem handles (controller, storage collections, secrets, orchestrator). The `Coordinator` struct holds a `reconcile_now: Arc<Notify>` field that wakes the cluster controller task to reconcile immediately after catalog transactions that change durable cluster state, rather than waiting out the tick interval.
The file also defines `IdPool`, a pre-allocated pool of user `GlobalId` integers that amortizes per-DDL persist writes by reserving batches of IDs at once; the pool is owned by the coordinator and access is serialized through its single-threaded event loop.
`ArrangementSizeRecord` is a struct for rows destined for `mz_object_arrangement_size_history`, carrying `replica_id`, `object_id`, `size`, and `hydration_complete`; records are prepared off-thread by the arrangement sizes snapshot task and stamped with a collection timestamp on the coordinator loop at write time.
Child modules partition the coordinator's responsibilities: `command_handler` handles external `Command` messages; `message_handler` handles internal async `Message` responses; `sequencer` executes SQL plans; `appends` manages table and builtin-table writes; `catalog_implications` derives and applies downstream effects from catalog state changes, including pushing replica-scoped dyncfg override layers to the compute and storage controllers via `push_replica_dyncfg_overrides`; `ddl` wraps catalog transactions; `peek` and `read_policy` manage query execution and compaction; `read_then_write` implements the read-then-write protocol for DML (INSERT, UPDATE, DELETE) including dependency validation; `group_sync` computes the diff between sync-managed and manually-granted role memberships for SSO group synchronization; `timestamp_selection` and `timeline` handle temporal reasoning; `catalog_serving` serves catalog snapshots; `info_metrics` owns the catalog `*_info` Prometheus series background task; `cluster_controller` scaffolds the cluster controller task that reconciles durable cluster state with the live controller; `hydration_history` implements the periodic sweep that populates `mz_internal.mz_object_hydration_history`; and supporting modules cover unified compute introspection subscribes, consistency checking, index management, and statement logging.
Child modules partition the coordinator's responsibilities: `command_handler` handles external `Command` messages; `message_handler` handles internal async `Message` responses; `sequencer` executes SQL plans; `appends` manages table and builtin-table writes; `catalog_implications` derives and applies downstream effects from catalog state changes, including pushing replica-scoped dyncfg override layers to the compute and storage controllers via `push_replica_dyncfg_overrides`; `ddl` wraps catalog transactions; `peek` and `read_policy` manage query execution and compaction; `read_then_write` implements the read-then-write protocol for DML (INSERT, UPDATE, DELETE) including dependency validation; `group_sync` computes the diff between sync-managed and manually-granted role memberships for SSO group synchronization; `timestamp_selection` and `timeline` handle temporal reasoning; `catalog_serving` serves catalog snapshots; `info_metrics` owns the catalog `*_info` Prometheus series background task; `cluster_controller` scaffolds the cluster controller task that reconciles durable cluster state with the live controller; `hydration_history` implements the periodic sweep that populates `mz_internal.mz_object_hydration_history`; `metric_sink` manages coordinator-installed curated metric sinks (non-catalog, transient-id sinks installed on every replica at bootstrap and on each new replica, driven by the `MetricSinkStageReady` pipeline); and supporting modules cover unified compute introspection subscribes, consistency checking, index management, and statement logging.
The `Coordinator` struct holds a `scoped_frontend: Option<Arc<SystemParameterFrontend>>` field populated by `Command::InstallScopedSystemParameterFrontend`. `reconcile_scoped_system_parameters` persists the diff between the current durable working copy and the desired `ScopedParameters` via `Op::UpdateScopedSystemParameters`, which also updates the in-memory working copy and introspection relations. `scoped_overrides_create_op` evaluates scoped overrides for freshly-created clusters and replicas from explicit `ClusterEvalContext` / `ReplicaEvalContext` values and returns an `Op::UpdateScopedSystemParameters` to fold into the same transaction that creates them, so the committed diff drives the replica-scoped controller push before `create_replica`. `replica_dyncfg_overrides` renders the replica-local scoped overrides from the catalog working copy as `ConfigUpdates` grouped by cluster (sparse: only replicas with an override are present). `push_replica_dyncfg_overrides` propagates those overrides to both controllers' per-replica dyncfg layers via the outer `controller.update_replica_dyncfg_overrides`, then re-pushes the environment-wide configs so existing replicas observe any changed or removed overrides. `cluster_scoped_optimizer_overrides` retrieves the cluster-coherent `OptimizerFeatureOverrides` from the catalog working copy for use at plan time.
The `Coordinator` struct holds a `catalog_info_metrics_registry: MetricsRegistry` field used to hand the metrics registry to the catalog info-metrics background task spawned at bootstrap.
Bootstrap handles derived builtin storage collections (builtin MVs) separately: after registering input-less collections in dependency order, it bumps their sinces based on transitive dependency frontiers to satisfy as-of selection invariants. Bootstrap restores compaction policies for all MV version IDs (via `global_ids()`, not just the write ID) so that no version blocks capability propagation through the `primary` ownership chain and pins compaction. When applying replacement MVs, each `CollectionDescription` receives a `primary` field pointing to its predecessor's latest collection ID, chaining shard ownership from the oldest version through each replacement in order. Bootstrap installs the replica-local scoped overrides via `replica_dyncfg_overrides` and pushes them to the outer controller before any replica is created, so provisioning-time config values (e.g. `TimelyConfig`, expiration offsets) that are frozen into the replica process at provisioning time resolve against the correct overrides. Bootstrap then calls `push_replica_dyncfg_overrides` once after all replicas exist so that existing replicas observe startup-time override values. Bootstrap accepts a `hydrate_migrated_mvs: bool` flag; when true, replacement-migrated builtin MVs call `allow_writes_in_read_only` so they hydrate before cut-over. Bootstrap calls `spawn_cluster_controller_task` to start the cluster controller background task.
The `clusters_caught_up_check` built during `serve` excludes all new builtin collections (tables, sources, and MVs) and, when the leader version predates `MIN_LEADER_VERSION_FOR_MIGRATED_MV_WRITES` (v26.17) or the `ENABLE_0DT_HYDRATE_MIGRATED_BUILTIN_MVS` flag is off, also excludes replacement-migrated MVs and their transitive dependents from the caught-up frontier check. New builtin collections have no writer until the deployment promotes. `MIN_LEADER_VERSION_FOR_MIGRATED_MV_WRITES` is the minimum leader version required for replacement-migrated MVs to write their new shards while read-only; older leaders do not keep the catalog shard frontier current, which would stall those MVs at a stale frontier.
The `Message` enum includes a `ClusterControllerRequest(cluster_controller::ClusterControllerRequest)` variant, carrying one pull/apply call from the cluster controller task to be answered on the main coordinator message loop.
`Message::ArrangementSizesWrite(Vec<ArrangementSizeRecord>)` carries the records prepared by the off-thread arrangement sizes snapshot task back to the coordinator loop for stamping and appending to `mz_object_arrangement_size_history`.
`Message::HydrationHistorySchedule` and `Message::HydrationHistoryRun` drive the two-phase hydration-history sweep: `HydrationHistorySchedule` fires the timer that picks when the next sweep starts, and `HydrationHistoryRun` triggers the actual collection attempt. The coordinator tracks the replica cursor (`hydration_history_replica_cursor: Option<ReplicaId>`) for round-robin replica selection and the in-flight sweep task (`hydration_history_sweep: Option<AbortOnDropHandle<()>>`); the task is single-flight and coordinator-owned, so dropping the handle aborts the sweep.
`Message::HydrationHistorySchedule` and `Message::HydrationHistoryRun` drive the two-phase hydration-history sweep: `HydrationHistorySchedule` fires the timer that picks when the next sweep starts, and `HydrationHistoryRun` triggers the actual collection attempt. The coordinator tracks the replica cursor (`hydration_history_replica_cursor: Option<ReplicaId>`) for round-robin replica selection and the in-flight sweep task (`hydration_history_sweep: Option<AbortOnDropHandle<()>>`); the task is single-flight and coordinator-owned, so dropping the handle aborts the sweep. On coordinator shutdown, the sweep is explicitly aborted and awaited (`abort_and_wait`) before the coordinator runtime releases timestamp-oracle workers, preserving dependency ordering for timestamp-oracle clients held by the sweep's background client.
`ship_dataflow`, `try_ship_dataflow`, and `ship_dataflow_and_notice_builtin_table_updates` accept `DataflowDescription<LirRelationExpr>`.
`ExecuteContextInner` carries a `response_barriers: Vec<BuiltinTableAppendNotify>` field (Debug-ignored). `ExecuteContext::from_parts` is a convenience wrapper around `from_parts_with_response_barriers`, which accepts an explicit barrier list. `into_parts` returns the barrier list as a fifth element alongside `tx`, `internal_cmd_tx`, `session`, and `extra`; callers that repack an `ExecuteContext` must preserve and forward these barriers. `delay_response_until` appends a `BuiltinTableAppendCompletion` barrier; on retirement, the response is held until all barriers resolve. The parts returned by `into_parts` lose the `Drop` backstop that answers the client on shutdown, so they must not be held across an await point (a bare `ClientTransmitter` panics when dropped unsent). When `retire` must wait on response barriers, it keeps `self` intact across the wait so the `Drop` backstop remains active; the barriers are cleared before the recursive call so the recursion terminates.
`ExplainTimestampStage` has four variants in order: `Optimize`, `RealTimeRecency`, `LinearizeTimestamp` (carries `ExplainTimestampLinearizeTimestamp`: validity, format, optimized_plan, cluster_id, source_ids, when, real_time_recency_ts), and `Finish` (carries `ExplainTimestampFinish`: validity, format, cluster_id, source_ids, when, real_time_recency_ts, timeline_context, oracle_read_ts). The `LinearizeTimestamp` stage reads the oracle timestamp off the coordinator loop and passes it forward as `oracle_read_ts` in `ExplainTimestampFinish`.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
source: src/adapter/src/coord/catalog_implications.rs
revision: c69fde3d50
revision: 41e1741ca3
---

# adapter::coord::catalog_implications
Expand Down
2 changes: 1 addition & 1 deletion doc/developer/generated/adapter/coord/ddl.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
source: src/adapter/src/coord/ddl.rs
revision: 39dcae2fba
revision: 41e1741ca3
---

# adapter::coord::ddl
Expand Down
Loading