diff --git a/doc/developer/generated/adapter/catalog/open/builtin_schema_migration.md b/doc/developer/generated/adapter/catalog/open/builtin_schema_migration.md index b8f5dd4585035..187e79c79c626 100644 --- a/doc/developer/generated/adapter/catalog/open/builtin_schema_migration.md +++ b/doc/developer/generated/adapter/catalog/open/builtin_schema_migration.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/catalog/open/builtin_schema_migration.rs -revision: c317ceee3c +revision: 46f729653a --- # adapter::catalog::open::builtin_schema_migration @@ -20,7 +20,7 @@ Three `Replacement` steps at version `26.38.0-dev.0` cover: `mz_catalog.mz_sinks A `Replacement` step at version `26.38.0-rc.2` covers `mz_internal.mz_cluster_reconfigurations`, whose MV definition changed to include `arrangement_compression` in the `changes` diff column. Three `Replacement` steps at version `26.39.0-dev.0` cover: `mz_catalog.mz_audit_events` (because the MV gained a `metric-sink` arm in its `object_type` CASE expression, changing its SQL fingerprint); `mz_catalog.mz_tables` and `mz_catalog.mz_views` (reflecting their conversion from `BuiltinTable` to `BuiltinMaterializedView` over `mz_catalog_raw`). The `mz_indexes` step originally at `26.39.0-dev.0` (covering `mz_object_graph_edges_ind`) was superseded and is no longer present. Three `Replacement` steps at version `26.40.0-dev.0` cover: `mz_catalog.mz_indexes` (required because adding the `mz_cluster_replica_resource_usage` builtin log changes the VALUES set inlined by `make_mz_indexes`, altering the MV's SQL fingerprint, superseding the earlier `mz_metric_sinks_ind` step); `mz_catalog.mz_sources` (required because `make_mz_sources` also inlines the builtin-log set as VALUES, so adding a builtin log changes its SQL fingerprint); and `mz_internal.mz_type_pg_metadata` (because the table gained a trailing `typsend` column). -When applying replacement migrations, `mz_storage_usage_by_shard` and `mz_object_hydration_history` are excluded from data-destroying replacement plans. `mz_storage_usage_by_shard` retains billing data; `mz_object_hydration_history` cannot be rebuilt from any other source. Both tables participate in forced evolution (which keeps the shard and its rows) but not in forced replacement. `validate_migration_steps` asserts that any declared `Replacement` step for `mz_object_hydration_history` is a deliberate, explicit choice rather than an accidental schema change. `validate_migration_steps` also asserts that `mz_cluster_replica_frontiers` is never declared as a migration target: the 0dt caught-up gate reads the leader's `mz_cluster_replica_frontiers` shard for the live frontiers it checks every collection against, so a `Replacement` step would hand the read-only environment a self-owned shard, causing the gate to compare itself against itself rather than against the leader. -The `participates_in_forced_migration(builtin, mechanism)` function encapsulates which builtins take part in forced migrations: tables participate unless they are `mz_storage_usage_by_shard` (for any mechanism) or `mz_object_hydration_history` (for `Replacement`); materialized views always participate; sources participate unless they are `mz_catalog_raw`; logs, views, types, functions, indexes, and connections never participate. +When applying replacement migrations, `mz_storage_usage_by_shard`, `mz_object_hydration_history`, and `mz_replica_hydration_history` are excluded from data-destroying replacement plans. `mz_storage_usage_by_shard` retains billing data; the two hydration history tables cannot be rebuilt from any other source. All three tables participate in forced evolution (which keeps the shard and its rows) but not in forced replacement. `validate_migration_steps` asserts that any declared `Replacement` step for `mz_object_hydration_history` or `mz_replica_hydration_history` is a deliberate, explicit choice rather than an accidental schema change. `validate_migration_steps` also asserts that `mz_cluster_replica_frontiers` is never declared as a migration target: the 0dt caught-up gate reads the leader's `mz_cluster_replica_frontiers` shard for the live frontiers it checks every collection against, so a `Replacement` step would hand the read-only environment a self-owned shard, causing the gate to compare itself against itself rather than against the leader. +The `participates_in_forced_migration(builtin, mechanism)` function encapsulates which builtins take part in forced migrations: tables participate unless they are `mz_storage_usage_by_shard` (for any mechanism) or `mz_object_hydration_history` / `mz_replica_hydration_history` (for `Replacement`); materialized views always participate; sources participate unless they are `mz_catalog_raw`; logs, views, types, functions, indexes, and connections never participate. When the source and target versions differ and the source version is a dev build, `Migration::run` forces evolution-mode migration even without an explicit `force_migration` config, avoiding version-based filter failures in dev environments. In forced dev-to-dev migrations, `migrate_evolve_one` skips builtins that do not yet have a shard registered. Brand-new builtins in a given version may not have their shards allocated until the leader completes bootstrap; excluding them avoids "missing shard ID" errors on read-only replicas. diff --git a/doc/developer/generated/adapter/catalog/open/builtin_schema_migration_tests.md b/doc/developer/generated/adapter/catalog/open/builtin_schema_migration_tests.md index 95fd0d9e4e19d..164c99e521098 100644 --- a/doc/developer/generated/adapter/catalog/open/builtin_schema_migration_tests.md +++ b/doc/developer/generated/adapter/catalog/open/builtin_schema_migration_tests.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/catalog/open/builtin_schema_migration_tests.rs -revision: 3a822d8457 +revision: 46f729653a --- # adapter::catalog::open::builtin_schema_migration_tests @@ -8,4 +8,4 @@ revision: 3a822d8457 Contains turmoil-based integration tests for `builtin_schema_migration`, verifying that schema migrations applied across multiple simulated versions and concurrent processes produce consistent results. The main test generates random builtin tables and sources, evolves their schemas across versions with random migration steps (evolution or replacement), spawns multiple concurrent processes per version, and asserts that all processes converge on the same migration outcome despite random crashes. `test_migration_steps_resolve_to_builtins` verifies that every step in the `MIGRATIONS` list names a builtin that currently exists with the declared `CatalogItemType`. A step naming a builtin that no longer exists or that changed type would panic `validate_migration_steps` at catalog open, which only the upgrade nightly catches; this unit test catches the same failure cheaply at CI time. -`hydration_history_forced_migration_policy` verifies that `mz_object_hydration_history` participates in a forced evolution migration but not in a forced replacement migration, confirming the exemption in `participates_in_forced_migration` that prevents accidental data loss when schema changes are force-applied during dev upgrades. +`hydration_history_forced_migration_policy` verifies that both `mz_object_hydration_history` and `mz_replica_hydration_history` participate in a forced evolution migration but not in a forced replacement migration, confirming the exemptions in `participates_in_forced_migration` that prevent accidental data loss when schema changes are force-applied during dev upgrades. diff --git a/doc/developer/generated/adapter/coord/_module.md b/doc/developer/generated/adapter/coord/_module.md index 375a43b53aaee..1de82f7554a7e 100644 --- a/doc/developer/generated/adapter/coord/_module.md +++ b/doc/developer/generated/adapter/coord/_module.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord.rs -revision: a1bcaebfe6 +revision: 46f729653a --- # adapter::coord @@ -9,14 +9,14 @@ The core coordinator: `coord.rs` defines the `Coordinator` struct (the central s The coordinator owns the `Catalog`, active compute sinks, pending peeks, read-policy manager, timeline oracles, and all inter-subsystem handles (controller, storage collections, secrets, orchestrator). The `Coordinator` struct holds a `reconcile_now: Arc` field that wakes the cluster controller task to reconcile immediately after catalog transactions that change durable cluster state, rather than waiting out the tick interval. The file also defines `IdPool`, a pre-allocated pool of user `GlobalId` integers that amortizes per-DDL persist writes by reserving batches of IDs at once; the pool is owned by the coordinator and access is serialized through its single-threaded event loop. `ArrangementSizeRecord` is a struct for rows destined for `mz_object_arrangement_size_history`, carrying `replica_id`, `object_id`, `size`, and `hydration_complete`; records are prepared off-thread by the arrangement sizes snapshot task and stamped with a collection timestamp on the coordinator loop at write time. -Child modules partition the coordinator's responsibilities: `command_handler` handles external `Command` messages; `message_handler` handles internal async `Message` responses; `sequencer` executes SQL plans; `appends` manages table and builtin-table writes; `catalog_implications` derives and applies downstream effects from catalog state changes, including pushing replica-scoped dyncfg override layers to the compute and storage controllers via `push_replica_dyncfg_overrides`; `ddl` wraps catalog transactions; `peek` and `read_policy` manage query execution and compaction; `read_then_write` implements the read-then-write protocol for DML (INSERT, UPDATE, DELETE) including dependency validation; `group_sync` computes the diff between sync-managed and manually-granted role memberships for SSO group synchronization; `timestamp_selection` and `timeline` handle temporal reasoning; `catalog_serving` serves catalog snapshots; `info_metrics` owns the catalog `*_info` Prometheus series background task; `cluster_controller` scaffolds the cluster controller task that reconciles durable cluster state with the live controller; `hydration_history` implements the periodic sweep that populates `mz_internal.mz_object_hydration_history`; and supporting modules cover unified compute introspection subscribes, consistency checking, index management, and statement logging. +Child modules partition the coordinator's responsibilities: `command_handler` handles external `Command` messages; `message_handler` handles internal async `Message` responses; `sequencer` executes SQL plans; `appends` manages table and builtin-table writes; `catalog_implications` derives and applies downstream effects from catalog state changes, including pushing replica-scoped dyncfg override layers to the compute and storage controllers via `push_replica_dyncfg_overrides`; `ddl` wraps catalog transactions; `peek` and `read_policy` manage query execution and compaction; `read_then_write` implements the read-then-write protocol for DML (INSERT, UPDATE, DELETE) including dependency validation; `group_sync` computes the diff between sync-managed and manually-granted role memberships for SSO group synchronization; `timestamp_selection` and `timeline` handle temporal reasoning; `catalog_serving` serves catalog snapshots; `info_metrics` owns the catalog `*_info` Prometheus series background task; `cluster_controller` scaffolds the cluster controller task that reconciles durable cluster state with the live controller; `hydration_history` implements the periodic sweep that populates `mz_internal.mz_object_hydration_history`; `metric_sink` manages coordinator-installed curated metric sinks (non-catalog, transient-id sinks installed on every replica at bootstrap and on each new replica, driven by the `MetricSinkStageReady` pipeline); and supporting modules cover unified compute introspection subscribes, consistency checking, index management, and statement logging. The `Coordinator` struct holds a `scoped_frontend: Option>` field populated by `Command::InstallScopedSystemParameterFrontend`. `reconcile_scoped_system_parameters` persists the diff between the current durable working copy and the desired `ScopedParameters` via `Op::UpdateScopedSystemParameters`, which also updates the in-memory working copy and introspection relations. `scoped_overrides_create_op` evaluates scoped overrides for freshly-created clusters and replicas from explicit `ClusterEvalContext` / `ReplicaEvalContext` values and returns an `Op::UpdateScopedSystemParameters` to fold into the same transaction that creates them, so the committed diff drives the replica-scoped controller push before `create_replica`. `replica_dyncfg_overrides` renders the replica-local scoped overrides from the catalog working copy as `ConfigUpdates` grouped by cluster (sparse: only replicas with an override are present). `push_replica_dyncfg_overrides` propagates those overrides to both controllers' per-replica dyncfg layers via the outer `controller.update_replica_dyncfg_overrides`, then re-pushes the environment-wide configs so existing replicas observe any changed or removed overrides. `cluster_scoped_optimizer_overrides` retrieves the cluster-coherent `OptimizerFeatureOverrides` from the catalog working copy for use at plan time. The `Coordinator` struct holds a `catalog_info_metrics_registry: MetricsRegistry` field used to hand the metrics registry to the catalog info-metrics background task spawned at bootstrap. Bootstrap handles derived builtin storage collections (builtin MVs) separately: after registering input-less collections in dependency order, it bumps their sinces based on transitive dependency frontiers to satisfy as-of selection invariants. Bootstrap restores compaction policies for all MV version IDs (via `global_ids()`, not just the write ID) so that no version blocks capability propagation through the `primary` ownership chain and pins compaction. When applying replacement MVs, each `CollectionDescription` receives a `primary` field pointing to its predecessor's latest collection ID, chaining shard ownership from the oldest version through each replacement in order. Bootstrap installs the replica-local scoped overrides via `replica_dyncfg_overrides` and pushes them to the outer controller before any replica is created, so provisioning-time config values (e.g. `TimelyConfig`, expiration offsets) that are frozen into the replica process at provisioning time resolve against the correct overrides. Bootstrap then calls `push_replica_dyncfg_overrides` once after all replicas exist so that existing replicas observe startup-time override values. Bootstrap accepts a `hydrate_migrated_mvs: bool` flag; when true, replacement-migrated builtin MVs call `allow_writes_in_read_only` so they hydrate before cut-over. Bootstrap calls `spawn_cluster_controller_task` to start the cluster controller background task. The `clusters_caught_up_check` built during `serve` excludes all new builtin collections (tables, sources, and MVs) and, when the leader version predates `MIN_LEADER_VERSION_FOR_MIGRATED_MV_WRITES` (v26.17) or the `ENABLE_0DT_HYDRATE_MIGRATED_BUILTIN_MVS` flag is off, also excludes replacement-migrated MVs and their transitive dependents from the caught-up frontier check. New builtin collections have no writer until the deployment promotes. `MIN_LEADER_VERSION_FOR_MIGRATED_MV_WRITES` is the minimum leader version required for replacement-migrated MVs to write their new shards while read-only; older leaders do not keep the catalog shard frontier current, which would stall those MVs at a stale frontier. The `Message` enum includes a `ClusterControllerRequest(cluster_controller::ClusterControllerRequest)` variant, carrying one pull/apply call from the cluster controller task to be answered on the main coordinator message loop. `Message::ArrangementSizesWrite(Vec)` carries the records prepared by the off-thread arrangement sizes snapshot task back to the coordinator loop for stamping and appending to `mz_object_arrangement_size_history`. -`Message::HydrationHistorySchedule` and `Message::HydrationHistoryRun` drive the two-phase hydration-history sweep: `HydrationHistorySchedule` fires the timer that picks when the next sweep starts, and `HydrationHistoryRun` triggers the actual collection attempt. The coordinator tracks the replica cursor (`hydration_history_replica_cursor: Option`) for round-robin replica selection and the in-flight sweep task (`hydration_history_sweep: Option>`); the task is single-flight and coordinator-owned, so dropping the handle aborts the sweep. +`Message::HydrationHistorySchedule` and `Message::HydrationHistoryRun` drive the two-phase hydration-history sweep: `HydrationHistorySchedule` fires the timer that picks when the next sweep starts, and `HydrationHistoryRun` triggers the actual collection attempt. The coordinator tracks the replica cursor (`hydration_history_replica_cursor: Option`) for round-robin replica selection and the in-flight sweep task (`hydration_history_sweep: Option>`); the task is single-flight and coordinator-owned, so dropping the handle aborts the sweep. On coordinator shutdown, the sweep is explicitly aborted and awaited (`abort_and_wait`) before the coordinator runtime releases timestamp-oracle workers, preserving dependency ordering for timestamp-oracle clients held by the sweep's background client. `ship_dataflow`, `try_ship_dataflow`, and `ship_dataflow_and_notice_builtin_table_updates` accept `DataflowDescription`. `ExecuteContextInner` carries a `response_barriers: Vec` field (Debug-ignored). `ExecuteContext::from_parts` is a convenience wrapper around `from_parts_with_response_barriers`, which accepts an explicit barrier list. `into_parts` returns the barrier list as a fifth element alongside `tx`, `internal_cmd_tx`, `session`, and `extra`; callers that repack an `ExecuteContext` must preserve and forward these barriers. `delay_response_until` appends a `BuiltinTableAppendCompletion` barrier; on retirement, the response is held until all barriers resolve. The parts returned by `into_parts` lose the `Drop` backstop that answers the client on shutdown, so they must not be held across an await point (a bare `ClientTransmitter` panics when dropped unsent). When `retire` must wait on response barriers, it keeps `self` intact across the wait so the `Drop` backstop remains active; the barriers are cleared before the recursive call so the recursion terminates. `ExplainTimestampStage` has four variants in order: `Optimize`, `RealTimeRecency`, `LinearizeTimestamp` (carries `ExplainTimestampLinearizeTimestamp`: validity, format, optimized_plan, cluster_id, source_ids, when, real_time_recency_ts), and `Finish` (carries `ExplainTimestampFinish`: validity, format, cluster_id, source_ids, when, real_time_recency_ts, timeline_context, oracle_read_ts). The `LinearizeTimestamp` stage reads the oracle timestamp off the coordinator loop and passes it forward as `oracle_read_ts` in `ExplainTimestampFinish`. diff --git a/doc/developer/generated/adapter/coord/catalog_implications.md b/doc/developer/generated/adapter/coord/catalog_implications.md index 843cdcf0f7444..fde8f064becc6 100644 --- a/doc/developer/generated/adapter/coord/catalog_implications.md +++ b/doc/developer/generated/adapter/coord/catalog_implications.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/catalog_implications.rs -revision: c69fde3d50 +revision: 41e1741ca3 --- # adapter::coord::catalog_implications diff --git a/doc/developer/generated/adapter/coord/ddl.md b/doc/developer/generated/adapter/coord/ddl.md index db67c1e95b919..306f0e3c38119 100644 --- a/doc/developer/generated/adapter/coord/ddl.md +++ b/doc/developer/generated/adapter/coord/ddl.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/ddl.rs -revision: 39dcae2fba +revision: 41e1741ca3 --- # adapter::coord::ddl diff --git a/doc/developer/generated/adapter/coord/hydration_history.md b/doc/developer/generated/adapter/coord/hydration_history.md index ce12019aed519..57b540f27b3de 100644 --- a/doc/developer/generated/adapter/coord/hydration_history.md +++ b/doc/developer/generated/adapter/coord/hydration_history.md @@ -1,23 +1,25 @@ --- source: src/adapter/src/coord/hydration_history.rs -revision: a1bcaebfe6 +revision: 46f729653a --- # `adapter::coord::hydration_history` -Durable history collection for completed compute-object hydration episodes. +Durable history collection for completed object and replica hydration episodes. ## Overview -Each sweep visits one user replica, installs a replica-targeted subscribe that diffs that replica's live hydration timestamps against the durable history table, and appends missing rows through the timestamped OCC write path. Including the history table in the read expression makes the write idempotent across concurrent `environmentd` processes: two collectors that compute the same row race for one write timestamp, and the loser observes the winner's append through its own subscribe and finds nothing left to write. +Each sweep visits one user replica, installs a replica-targeted subscribe that diffs that replica's live hydration timestamps against the durable history tables, and appends missing rows through the timestamped OCC write path. Including each history table in its read expression makes the write idempotent across concurrent `environmentd` processes: two collectors that compute the same row race for one write timestamp, and the loser observes the winner's append through its own subscribe and finds nothing left to write. -One replica is sampled per interval, so an environment with N eligible replicas revisits each one approximately every `N * interval`. Collection is sampling, not an event log. An episode whose live row is retracted before its replica's turn in the sweep is not recorded because the evidence is gone. +One replica is sampled per interval, so an environment with N eligible replicas revisits each one approximately every `N * interval`. Collection is sampling, not an event log. Replica history records only the latest completed episode visible in a sweep. Intermediate episodes and intervals retracted before collection leave no evidence and are not recorded. ## Key Types -**`Sweep`** — Context for one sweep run, holding the `PeekClient`, catalog reference, history table ID, metrics handle, wall time, and a `cutoff` string (RFC 3339 timestamp). The two operations are: -- `collect` — appends one replica's completed episodes that the history table is missing -- `retain` — retracts one bounded batch of rows that have aged out of the retention window +**`ReplicaTarget`** — A user replica eligible for one collection step, carrying `cluster_id`, `replica_id`, and `process_count`. The `process_count` is used by `replica_collection_sql` to gate the write on all configured processes having reported resource usage. + +**`Sweep`** — Context for one sweep run, holding the `PeekClient`, catalog reference, `object_history_id` and `replica_history_id` table IDs, metrics handle, wall time, and a `cutoff` string (RFC 3339 timestamp). The two operations are: +- `collect` — appends one replica's completed object and replica episodes that their respective history tables are missing +- `retain` — retracts one bounded batch of rows from each history table that have aged out of the retention window ## Scheduling @@ -25,13 +27,17 @@ One replica is sampled per interval, so an environment with N eligible replicas `Coordinator::run_hydration_history_collection` dispatches the sweep as a background task. The task runs `collect` against the selected user replica and `retain` against the catalog server cluster, then reschedules. The sweep handle is stored on the `Coordinator` so it is aborted when the coordinator drops. -## Collection Query +## Collection Queries + +`object_collection_sql` builds a `SELECT` that joins `mz_compute_hydration_times_per_worker` against `mz_object_hydration_history` with an anti-join, filtering to fully-hydrated user indexes and materialized views (all workers have a `hydrated_at`) whose episodes are not yet recorded. The cutoff and the anti-join sit outside the aggregate so they do not interfere with the per-worker completeness check. -`collect_sql` builds a `SELECT` that joins `mz_compute_hydration_times_per_worker` against `mz_object_hydration_history` with an anti-join, filtering to fully-hydrated objects (all workers have a `hydrated_at`) whose episodes are not yet recorded. The cutoff and the anti-join sit outside the aggregate so they do not interfere with the per-worker completeness check. +`replica_collection_sql` implements a gaps-and-islands algorithm over compute export hydration intervals to find the latest completed hydration episode that is disconnected from any still-open interval. Transient exports (those with IDs starting with `t`) are excluded. The query also waits until every configured replica process (`process_count`) has reported resource usage before committing the episode, capturing `peak_memory_bytes` (cgroup `memory_peak`) and `peak_disk_bytes` (statvfs `fs_used_peak`, falling back to cgroup `swap_peak`). ## Retention -`retention_sql` returns a bounded batch (`RETENTION_BATCH_SIZE` = 1000) of the oldest rows aged past the cutoff, using a subquery so the `LIMIT` applies inside the relation expression rather than as a top-level `RowSetFinishing` that the OCC path cannot apply. +`object_retention_sql` returns a bounded batch (`RETENTION_BATCH_SIZE` = 1000) of the oldest `mz_object_hydration_history` rows aged past the cutoff. +`replica_retention_sql` returns the same bounded batch for `mz_replica_hydration_history`. +Both use a subquery so the `LIMIT` applies inside the relation expression rather than as a top-level `RowSetFinishing` that the OCC path cannot apply. ## Constants @@ -44,6 +50,6 @@ One replica is sampled per interval, so an environment with N eligible replicas ## Helper Functions -- `next_replica` — advances a cursor through replicas sorted by ID, wrapping at the end, so each sweep visits a different replica +- `next_replica` — advances a cursor through `ReplicaTarget` entries sorted by ID, wrapping at the end, so each sweep visits a different replica - `environment_schedule_offset` — derives a stable per-environment offset from a SHA-256 hash of the environment ID - `plan_mutation` — plans a `SELECT` statement as the read side of a `ReadThenWritePlan`, and validates that the selection's column types match the target table diff --git a/doc/developer/generated/adapter/coord/introspection.md b/doc/developer/generated/adapter/coord/introspection.md index 60953ddcfbb5d..196f77e5a5a75 100644 --- a/doc/developer/generated/adapter/coord/introspection.md +++ b/doc/developer/generated/adapter/coord/introspection.md @@ -1,11 +1,11 @@ --- source: src/adapter/src/coord/introspection.rs -revision: 7f6c52776d +revision: 41e1741ca3 --- # adapter::coord::introspection Implements unified compute introspection: the process of collecting introspection data exported by individual replicas through their logging indexes and writing that data, tagged with the respective replica ID, to unified storage collections. -`install_introspection_subscribes` installs all defined introspection subscribes on a given replica (and `bootstrap_introspection_subscribes` calls it for all existing replicas during coordinator startup); `handle_introspection_subscribe_batch` processes each batch response, writing updates to the corresponding storage-managed collection and reinstalling failed subscribes on disconnect; `drop_introspection_subscribes` removes all subscribes installed on a replica before it is dropped. +`all_cluster_replicas` returns every `(ClusterId, ReplicaId)` pair currently in the catalog and is shared by both introspection subscribes and curated metric sinks so those callers cannot drift on what "all replicas" means. `install_introspection_subscribes` installs all defined introspection subscribes on a given replica (and `bootstrap_introspection_subscribes` calls it for all existing replicas during coordinator startup); `handle_introspection_subscribe_batch` processes each batch response, writing updates to the corresponding storage-managed collection and reinstalling failed subscribes on disconnect; `drop_introspection_subscribes` removes all subscribes installed on a replica before it is dropped. Each introspection subscribe is sequenced through a multi-stage pipeline (`OptimizeMir` → `TimestampOptimizeLir` → `Finish`) using the `sequence_staged` driver. The optimizer config for introspection subscribes includes cluster-coherent scoped overrides via `Coordinator::cluster_scoped_optimizer_overrides`. `IntrospectionSubscribe` tracks a `first_data_at: Option` field recording when the subscribe first appended data to its target storage collection in the current process. Rows in that collection before that point may describe a previous environmentd process or a prior replica incarnation and must not be trusted. `invalidate_introspection_freshness(replica_id)` clears `first_data_at` for all subscribes targeting a given replica when a cluster event reports the replica offline or restarted. `fresh_introspection_replicas(introspection_type, margin)` returns the string replica IDs whose subscribe of the given type delivered data at least `margin` ago; consumers such as the arrangement sizes snapshot use this to exclude stale replicas. diff --git a/doc/developer/generated/adapter/coord/message_handler.md b/doc/developer/generated/adapter/coord/message_handler.md index 9abc3c69e1562..3f6594d86f55b 100644 --- a/doc/developer/generated/adapter/coord/message_handler.md +++ b/doc/developer/generated/adapter/coord/message_handler.md @@ -1,12 +1,12 @@ --- source: src/adapter/src/coord/message_handler.rs -revision: a1bcaebfe6 +revision: 41e1741ca3 --- # adapter::coord::message_handler Implements `Coordinator::handle_message`, the main dispatch for internal `Message` variants flowing through the coordinator's event loop. -Handles controller responses (compute peek results, subscribe batches, copy-to responses, watch-set notifications), timer ticks (group commit, timeline advancement, storage usage collection and pruning), staged-pipeline continuations (peek, create index, create view, create materialized view, subscribe, introspection subscribe, explain timestamp, secret, cluster), linearized read delivery, deferred statement execution, private-link VPC endpoint events, cluster controller requests (`Message::ClusterControllerRequest`, dispatched to `handle_cluster_controller_request`), and hydration-history scheduling (`Message::HydrationHistorySchedule` dispatched to `schedule_hydration_history_collection`; `Message::HydrationHistoryRun` dispatched to `run_hydration_history_collection`). +Handles controller responses (compute peek results, subscribe batches, copy-to responses, watch-set notifications), timer ticks (group commit, timeline advancement, storage usage collection and pruning), staged-pipeline continuations (peek, create index, create metric sink, create view, create materialized view, subscribe, introspection subscribe, metric sink (coordinator-installed curated sinks), explain timestamp, secret, cluster), linearized read delivery, deferred statement execution, private-link VPC endpoint events, cluster controller requests (`Message::ClusterControllerRequest`, dispatched to `handle_cluster_controller_request`), and hydration-history scheduling (`Message::HydrationHistorySchedule` dispatched to `schedule_hydration_history_collection`; `Message::HydrationHistoryRun` dispatched to `run_hydration_history_collection`). This is the heart of the coordinator's reactive loop; every asynchronous response from the storage/compute layers arrives here. `storage_usage_update` obtains a write timestamp from the oracle, allocates a single durable batch id via `Catalog::allocate_storage_usage_id`, builds `BuiltinTableUpdate` rows via `pack_storage_usage_update` for each shard, and submits them via `builtin_table_update().execute()` without going through `catalog_transact_inner`. In read-only mode, `storage_usage_fetch` logs an info message and reschedules via `Message::StorageUsageSchedule` without performing any shard scan or writes. diff --git a/doc/developer/generated/adapter/coord/sequencer/inner/cluster.md b/doc/developer/generated/adapter/coord/sequencer/inner/cluster.md index 7a60d5bf9df37..02c8152e2d4e0 100644 --- a/doc/developer/generated/adapter/coord/sequencer/inner/cluster.md +++ b/doc/developer/generated/adapter/coord/sequencer/inner/cluster.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/sequencer/inner/cluster.rs -revision: e3fe017dcb +revision: b2419281fb --- # adapter::coord::sequencer::inner::cluster diff --git a/doc/developer/generated/adapter/coord/sequencer/inner/create_metric_sink.md b/doc/developer/generated/adapter/coord/sequencer/inner/create_metric_sink.md index 7deccf1cbe219..7d26d12f3041e 100644 --- a/doc/developer/generated/adapter/coord/sequencer/inner/create_metric_sink.md +++ b/doc/developer/generated/adapter/coord/sequencer/inner/create_metric_sink.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/sequencer/inner/create_metric_sink.rs -revision: a702b8be70 +revision: 41e1741ca3 --- # adapter::coord::sequencer::inner::create_metric_sink diff --git a/doc/developer/generated/adapter/metrics.md b/doc/developer/generated/adapter/metrics.md index 1a81e0b53a94d..d7d134c9fe6fc 100644 --- a/doc/developer/generated/adapter/metrics.md +++ b/doc/developer/generated/adapter/metrics.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/metrics.rs -revision: a1bcaebfe6 +revision: 46f729653a --- # adapter::metrics diff --git a/doc/developer/generated/adapter/optimize/metric_sink.md b/doc/developer/generated/adapter/optimize/metric_sink.md index ad09d691f3b41..e34800a80ae4d 100644 --- a/doc/developer/generated/adapter/optimize/metric_sink.md +++ b/doc/developer/generated/adapter/optimize/metric_sink.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/optimize/metric_sink.rs -revision: a702b8be70 +revision: 41e1741ca3 --- # adapter::optimize::metric_sink @@ -25,4 +25,4 @@ No rows are filtered: the operator needs every row, including invalid ones, to c `METRIC_NAME_PATTERN` is the Prometheus metric name grammar regexp (`^[a-zA-Z_:][a-zA-Z0-9_:]*$`), stored as a constant for use in the MIR `IsRegexpMatchCaseSensitive` scalar. -The `Optimizer` struct for metric sinks implements `Optimize` (MIR stage, producing `GlobalMirPlan`) and `Optimize` (LIR stage, producing `GlobalLirPlan`). Like `CREATE INDEX`, the pipeline starts directly from the `GlobalId` of the collection to export rather than lowering a new relational expression from HIR. Unlike a materialized view sink, there is no persist shard, so there is no storage-metadata stage. +The `Optimizer` struct for metric sinks implements `Optimize` (MIR stage, producing `GlobalMirPlan`) and `Optimize` (LIR stage, producing `GlobalLirPlan`). The source is described by `MetricSinkFrom`: `MetricSinkFrom::Id(GlobalId)` starts from an existing catalog collection (like `CREATE INDEX`, no HIR to lower), while `MetricSinkFrom::Query { expr, desc }` lowers a planned HIR query (what a coordinator-installed curated sink built from SQL uses). In both cases the row-wise shaping is appended in MIR and the dataflow exports a single `MetricSink`. The `MetricSink` wrapper also carries an optional `label` field for the `sink` label on health gauges; `None` defaults to the sink's `GlobalId` (for user sinks), while curated sinks pass their stable name. Unlike a materialized view sink, there is no persist shard, so there is no storage-metadata stage. diff --git a/doc/developer/generated/avro/util.md b/doc/developer/generated/avro/util.md index 62ffebc3ae406..80a8c844295ec 100644 --- a/doc/developer/generated/avro/util.md +++ b/doc/developer/generated/avro/util.md @@ -1,6 +1,6 @@ --- source: src/avro/src/util.rs -revision: de06eb12d9 +revision: 4e012ea88d --- Contains low-level encoding and decoding primitives shared across the crate. diff --git a/doc/developer/generated/catalog/builtin/_module.md b/doc/developer/generated/catalog/builtin/_module.md index 2143ac9c7c877..871eec592fd87 100644 --- a/doc/developer/generated/catalog/builtin/_module.md +++ b/doc/developer/generated/catalog/builtin/_module.md @@ -1,13 +1,13 @@ --- source: src/catalog/src/builtin.rs -revision: c317ceee3c +revision: 46f729653a --- # catalog::builtin Defines every built-in catalog object hardcoded into Materialize: system tables (`BuiltinTable`), views (`BuiltinView`), materialized views (`BuiltinMaterializedView`), indexes (`BuiltinIndex`), types (`BuiltinType`), sources (`BuiltinSource`), logs (`BuiltinLog`), connections (`BuiltinConnection`), cluster definitions (`BuiltinCluster`), roles, and schemas. `BuiltinClusterReplica` and `BUILTIN_CLUSTER_REPLICAS` are absent; the durable managed cluster config (specifically `replication_factor`) is the single source of truth for each builtin cluster's replica set, and `reconcile_builtin_cluster_replicas` in the adapter converges it at catalog open. Key types include `Builtin` (a generic enum with variants Log, Table, View, MaterializedView, Type, Func, Source, Index, and Connection, each wrapping the corresponding builtin struct), `BUILTINS` (the exhaustive static list used at catalog open time), and constants like `BUILTIN_PREFIXES` and `RUNTIME_ALTERABLE_FINGERPRINT_SENTINEL`. -`MZ_DATABASES`, `MZ_CLUSTERS`, `MZ_CLUSTER_REPLICAS`, `MZ_CLUSTER_SCHEDULES`, `MZ_CLUSTER_RECONFIGURATIONS`, `MZ_CLUSTER_AUTO_SCALING_STRATEGIES`, `MZ_KAFKA_SOURCES`, `MZ_POSTGRES_SOURCES`, `MZ_POSTGRES_SOURCE_TABLES`, `MZ_MYSQL_SOURCE_TABLES`, `MZ_SQL_SERVER_SOURCE_TABLES`, `MZ_KAFKA_SOURCE_TABLES`, `MZ_KAFKA_CONNECTIONS`, `MZ_SSH_TUNNEL_CONNECTIONS`, `MZ_AWS_CONNECTIONS`, `MZ_AWS_PRIVATELINK_CONNECTIONS`, `MZ_KAFKA_SINKS`, `MZ_ICEBERG_SINKS`, `MZ_SINKS`, and `MZ_METRIC_SINKS` are all `BuiltinMaterializedView` objects backed by queries over the catalog. `MZ_CLUSTER_RECONFIGURATIONS` and `MZ_CLUSTER_AUTO_SCALING_STRATEGIES` each have a corresponding `BuiltinIndex` (`MZ_CLUSTER_RECONFIGURATIONS_IND` and `MZ_CLUSTER_AUTO_SCALING_STRATEGIES_IND` respectively) on `cluster_id`, registered in `BUILTINS_STATIC` alongside the other index entries. `MZ_METRIC_SINKS` has a corresponding `BuiltinIndex` (`MZ_METRIC_SINKS_IND`) on `id`, also registered in `BUILTINS_STATIC`. `MZ_CLUSTER_REPLICA_SIZE_INTERNAL` is a `BuiltinTable` that retains size-map entries including disabled sizes, allowing the `MZ_CLUSTER_REPLICAS` materialized view to resolve the `disk` column via LEFT JOIN even for replicas whose sizes have been disabled. `MZ_CLUSTER_REPLICA_FRONTIERS_DESCRIPTION` is a `SystemObjectDescription` static that identifies `MZ_CLUSTER_REPLICA_FRONTIERS` for the schema-migration guard in `builtin_schema_migration.rs`, which forbids migrating that source because the 0dt caught-up gate reads its shard to learn live frontiers. `MZ_OBJECT_HYDRATION_HISTORY_DESCRIPTION` is a `SystemObjectDescription` static that identifies `MZ_OBJECT_HYDRATION_HISTORY` for the same migration guard, which forbids migrating it via `Mechanism::Replacement` (to avoid clearing durable hydration history); `Mechanism::Evolution` is allowed. +`MZ_DATABASES`, `MZ_CLUSTERS`, `MZ_CLUSTER_REPLICAS`, `MZ_CLUSTER_SCHEDULES`, `MZ_CLUSTER_RECONFIGURATIONS`, `MZ_CLUSTER_AUTO_SCALING_STRATEGIES`, `MZ_KAFKA_SOURCES`, `MZ_POSTGRES_SOURCES`, `MZ_POSTGRES_SOURCE_TABLES`, `MZ_MYSQL_SOURCE_TABLES`, `MZ_SQL_SERVER_SOURCE_TABLES`, `MZ_KAFKA_SOURCE_TABLES`, `MZ_KAFKA_CONNECTIONS`, `MZ_SSH_TUNNEL_CONNECTIONS`, `MZ_AWS_CONNECTIONS`, `MZ_AWS_PRIVATELINK_CONNECTIONS`, `MZ_KAFKA_SINKS`, `MZ_ICEBERG_SINKS`, `MZ_SINKS`, and `MZ_METRIC_SINKS` are all `BuiltinMaterializedView` objects backed by queries over the catalog. `MZ_CLUSTER_RECONFIGURATIONS` and `MZ_CLUSTER_AUTO_SCALING_STRATEGIES` each have a corresponding `BuiltinIndex` (`MZ_CLUSTER_RECONFIGURATIONS_IND` and `MZ_CLUSTER_AUTO_SCALING_STRATEGIES_IND` respectively) on `cluster_id`, registered in `BUILTINS_STATIC` alongside the other index entries. `MZ_METRIC_SINKS` has a corresponding `BuiltinIndex` (`MZ_METRIC_SINKS_IND`) on `id`, also registered in `BUILTINS_STATIC`. `MZ_CLUSTER_REPLICA_SIZE_INTERNAL` is a `BuiltinTable` that retains size-map entries including disabled sizes, allowing the `MZ_CLUSTER_REPLICAS` materialized view to resolve the `disk` column via LEFT JOIN even for replicas whose sizes have been disabled. `MZ_CLUSTER_REPLICA_FRONTIERS_DESCRIPTION` is a `SystemObjectDescription` static that identifies `MZ_CLUSTER_REPLICA_FRONTIERS` for the schema-migration guard in `builtin_schema_migration.rs`, which forbids migrating that source because the 0dt caught-up gate reads its shard to learn live frontiers. `MZ_OBJECT_HYDRATION_HISTORY_DESCRIPTION` is a `SystemObjectDescription` static that identifies `MZ_OBJECT_HYDRATION_HISTORY` for the same migration guard, which forbids migrating it via `Mechanism::Replacement` (to avoid clearing durable hydration history); `Mechanism::Evolution` is allowed. `MZ_REPLICA_HYDRATION_HISTORY` is a `BuiltinTable` for durable replica hydration episodes, and `MZ_REPLICA_HYDRATION_HISTORY_DESCRIPTION` is its corresponding `SystemObjectDescription` static for the same migration guard, carrying the same `Mechanism::Replacement` exemption. `MZ_CONNECTIONS`, `MZ_SECRETS`, `MZ_SOURCES`, `MZ_DEFAULT_PRIVILEGES`, and `MZ_SYSTEM_PRIVILEGES` are `BuiltinMaterializedView` objects backed by queries over `mz_internal.mz_catalog_raw`. `MZ_SOURCES` is generated dynamically via `make_mz_sources()` in the `builtin` submodule rather than declared as a static, since its SQL must enumerate all builtin and log sources. `MZ_INDEXES` is also a `BuiltinMaterializedView` generated dynamically via `make_mz_indexes()` in the `mz_catalog` submodule; it inlines the full set of builtin indexes and logs as `VALUES` so that its SQL fingerprint changes whenever a builtin index or log is added or removed, triggering an automatic `MigrationStep::replacement`. Each builtin struct carries an optional `ontology` field of type `Option` that marks the object as a catalog ontology entity and provides entity-level metadata. diff --git a/doc/developer/generated/catalog/builtin/mz_internal.md b/doc/developer/generated/catalog/builtin/mz_internal.md index 0e455c1c263d4..d8152b76d7b98 100644 --- a/doc/developer/generated/catalog/builtin/mz_internal.md +++ b/doc/developer/generated/catalog/builtin/mz_internal.md @@ -1,6 +1,6 @@ --- source: src/catalog/src/builtin/mz_internal.rs -revision: c317ceee3c +revision: 46f729653a --- # catalog::builtin::mz_internal @@ -11,7 +11,7 @@ This is the largest builtin submodule, exporting 202 public items: sources, tabl **Sources** (`BuiltinSource`) — Backed by `DataSourceDesc::Catalog` or `IntrospectionType`. Key items include `MZ_CATALOG_RAW` (the raw persist-backed catalog source, system-only access), and storage statistics/status history sources such as `MZ_SOURCE_STATISTICS_RAW`, `MZ_SINK_STATISTICS_RAW`, `MZ_SOURCE_STATUS_HISTORY`, `MZ_SINK_STATUS_HISTORY`, `MZ_STATEMENT_EXECUTION_HISTORY`, `MZ_SESSION_HISTORY`, `MZ_SQL_TEXT`, `MZ_PREPARED_STATEMENT_HISTORY`, and replica metrics/status history sources. -**Tables** (`BuiltinTable`) — Connector-specific and internal metadata tables such as `MZ_MYSQL_SOURCES`, and others tracking sink/source details. `MZ_CLUSTER_SCHEDULES` is a `BuiltinMaterializedView` (not a table) backed by a query over `mz_internal.mz_catalog_raw` that derives cluster scheduling configuration from the durable catalog JSON. `MZ_POSTGRES_SOURCE_TABLES`, `MZ_MYSQL_SOURCE_TABLES`, `MZ_SQL_SERVER_SOURCE_TABLES`, and `MZ_KAFKA_SOURCE_TABLES` are `BuiltinMaterializedView` objects (not tables) backed by queries over `mz_catalog_raw`. `MZ_OBJECT_HYDRATION_HISTORY` (OID `TABLE_MZ_OBJECT_HYDRATION_HISTORY_OID`) is a `BuiltinTable` that records durable episodes of dataflow hydration, with one row per completed hydration identified by `(object_id, replica_id, installed_at)`. Columns are `object_id` (dataflow id with `SemanticType::GlobalId`; resolve to a catalog object via `mz_internal.mz_object_global_ids`), `cluster_id`, `replica_id` (`SemanticType::ReplicaId`), `installed_at` (replica-stamped, stable across environmentd restarts), `started_at` (nullable; when hydration work began, or `NULL` if none was reported), `hydrated_at` (nullable; when hydration finished), and `status` (terminal status, currently always `'hydrated'`). No relation key is declared; the anti-join in the collector enforces uniqueness, so a duplicate row must remain visible. The table has no index and is not a retained-metrics object. It is exempt from bootstrap reset and from forced shard replacement (`plan_forced_migration` skips `Mechanism::Replacement` for this table) to preserve sampled history across restarts; `validate_migration_steps` enforces this exemption via `MZ_OBJECT_HYDRATION_HISTORY_DESCRIPTION` and panics if a `Replacement` step targets the table. An `Evolution` step (which keeps the shard and its rows) is the permitted path for schema changes. It carries an `Ontology` annotation with foreign-key links to `object_global_id` (via `object_id`), `cluster` (via `cluster_id`), and `replica` (via `replica_id`). `MZ_TYPE_PG_METADATA` includes a `typsend` column (OID, not nullable) alongside `typinput` and `typreceive`; consumers that LEFT JOIN against this table must use `COALESCE` on `typsend` because the join yields NULL for types without PostgreSQL metadata. +**Tables** (`BuiltinTable`) — Connector-specific and internal metadata tables such as `MZ_MYSQL_SOURCES`, and others tracking sink/source details. `MZ_CLUSTER_SCHEDULES` is a `BuiltinMaterializedView` (not a table) backed by a query over `mz_internal.mz_catalog_raw` that derives cluster scheduling configuration from the durable catalog JSON. `MZ_POSTGRES_SOURCE_TABLES`, `MZ_MYSQL_SOURCE_TABLES`, `MZ_SQL_SERVER_SOURCE_TABLES`, and `MZ_KAFKA_SOURCE_TABLES` are `BuiltinMaterializedView` objects (not tables) backed by queries over `mz_catalog_raw`. `MZ_OBJECT_HYDRATION_HISTORY` (OID `TABLE_MZ_OBJECT_HYDRATION_HISTORY_OID`) is a `BuiltinTable` that records durable episodes of dataflow hydration, with one row per completed hydration identified by `(object_id, replica_id, installed_at)`. Columns are `object_id` (dataflow id with `SemanticType::GlobalId`; resolve to a catalog object via `mz_internal.mz_object_global_ids`), `cluster_id`, `replica_id` (`SemanticType::ReplicaId`), `installed_at` (replica-stamped, stable across environmentd restarts), `started_at` (nullable; when hydration work began, or `NULL` if none was reported), `hydrated_at` (nullable; when hydration finished), and `status` (terminal status, currently always `'hydrated'`). No relation key is declared; the anti-join in the collector enforces uniqueness, so a duplicate row must remain visible. The table has no index and is not a retained-metrics object. It is exempt from bootstrap reset and from forced shard replacement (`plan_forced_migration` skips `Mechanism::Replacement` for this table) to preserve sampled history across restarts; `validate_migration_steps` enforces this exemption via `MZ_OBJECT_HYDRATION_HISTORY_DESCRIPTION` and panics if a `Replacement` step targets the table. An `Evolution` step (which keeps the shard and its rows) is the permitted path for schema changes. It carries an `Ontology` annotation with foreign-key links to `object_global_id` (via `object_id`), `cluster` (via `cluster_id`), and `replica` (via `replica_id`). `MZ_TYPE_PG_METADATA` includes a `typsend` column (OID, not nullable) alongside `typinput` and `typreceive`; consumers that LEFT JOIN against this table must use `COALESCE` on `typsend` because the join yields NULL for types without PostgreSQL metadata. `MZ_REPLICA_HYDRATION_HISTORY` (OID `TABLE_MZ_REPLICA_HYDRATION_HISTORY_OID`) is a `BuiltinTable` that records durable episodes of cluster replica hydration. Each row represents one completed hydration episode with columns `replica_id`, `cluster_id`, `started_at` (earliest export installation in the episode), `finished_at` (nullable; latest export hydration in the episode), `object_count` (uint8, number of compute exports in the episode), `peak_memory_bytes` (nullable uint64; largest process-lifetime cgroup memory high-water mark across processes), `peak_disk_bytes` (nullable uint64; largest statvfs `fs_used_peak` or cgroup `swap_peak` across processes), and `status` (currently always `'hydrated'`). The table has `PUBLIC_SELECT` access and carries an `Ontology` annotation with foreign-key links to `cluster` (via `cluster_id`) and `replica` (via `replica_id`). It is exempt from bootstrap reset and from forced shard replacement; `validate_migration_steps` asserts that any `Replacement` step targeting it is deliberate. **Materialized views** (`BuiltinMaterializedView`) — Derived catalog views backed by queries over `mz_catalog_raw` and other sources: aggregated statistics, lag histograms, and other derived metrics. `MZ_POSTGRES_SOURCES` is a `BuiltinMaterializedView` (OID constant `MV_MZ_POSTGRES_SOURCES_OID`) that derives PostgreSQL source details (`replication_slot`, `timeline_id`) by parsing the raw catalog create SQL via `mz_internal.parse_postgres_source_details`; the MV filters to postgres sources first via `parse_catalog_create_sql` because the detail decoder must not be applied to non-postgres rows. `MZ_POSTGRES_SOURCE_TABLES` (OID `MV_MZ_POSTGRES_SOURCE_TABLES_OID`), `MZ_MYSQL_SOURCE_TABLES` (OID `MV_MZ_MYSQL_SOURCE_TABLES_OID`), `MZ_SQL_SERVER_SOURCE_TABLES` (OID `MV_MZ_SQL_SERVER_SOURCE_TABLES_OID`), and `MZ_KAFKA_SOURCE_TABLES` (OID `MV_MZ_KAFKA_SOURCE_TABLES_OID`) are `BuiltinMaterializedView` objects that derive source-export (source table) metadata by parsing catalog `create_sql` via `mz_internal.parse_source_export_details` and joining `mz_catalog.mz_sources` to filter by parent source type. Each has columns `id`, `schema_name`, and `table_name` (postgres, mysql, sql-server) or `id`, `topic`, `envelope_type`, `key_format`, and `value_format` (kafka). The postgres and sql-server external references are three-part (`[database, schema, table]`), so `schema_name` and `table_name` come from positions 1 and 2. MySQL references are two-part (`[schema, table]`), so positions 0 and 1. For kafka source tables, `envelope_type` is not nullable: Kafka defaults to `ENVELOPE NONE` when the clause is omitted, so the SQL defaults to `'none'` via `COALESCE` for rows where the parser helper returns NULL. These four MVs are marked `is_retained_metrics_object: true`. `MZ_AWS_CONNECTIONS` is a `BuiltinMaterializedView` (OID constant `MV_MZ_AWS_CONNECTIONS_OID`) that reads `Item` rows from `mz_catalog_raw`, pulls create_sql-derived fields (endpoint, region, credentials, assume-role options) via `parse_connection_details`, and keeps only aws connections. The three context-derived columns (`principal`, `external_id`, `example_trust_policy`) apply to assume-role connections only and are reconstructed from the plan-time AWS context functions (`mz_aws_account_id`, `mz_aws_external_id_prefix`, `mz_aws_connection_role_arn`). On an environment without the AWS context, assume-role connections are excluded entirely, matching the packer this view replaces. `MZ_OVERRIDDEN_SYSTEM_PARAMETERS` projects the `system_configurations` durable collection out of `mz_catalog_raw`, exposing environment-wide system parameter overrides as `(name, value)` pairs with `PUBLIC_SELECT` access and an `Ontology` annotation; only parameters with an explicit override appear (defaults are absent). `MZ_CLUSTER_SYSTEM_PARAMETERS` projects the `cluster_system_configurations` durable collection out of `mz_catalog_raw`, exposing per-cluster system parameter overrides keyed by `(cluster_id, name)`. `MZ_REPLICA_SYSTEM_PARAMETERS` does the same for the `replica_system_configurations` collection, exposing per-replica overrides keyed by `(replica_id, name)`. `MZ_COMMENTS` is a `BuiltinMaterializedView` backed by a query over `mz_catalog_raw` that reads `Comment` entries, decodes each `proto::CommentObject` variant via CASE expressions, and projects `(id, object_type, object_sub_id, comment)` rows; new `CommentObject` variants must be added to both CASE expressions in its SQL. `MZ_CLUSTER_RECONFIGURATIONS` (OID `MV_MZ_CLUSTER_RECONFIGURATIONS_OID`) is a `BuiltinMaterializedView` with a unique key on `cluster_id` that exposes the latest graceful reconfiguration record per managed cluster, with columns `cluster_id`, `status`, `deadline`, `on_timeout`, `target` (JSONB), and `changes` (JSONB). Terminal records are retained after settling so the outcome stays inspectable until a later reconfiguration overwrites it. The `status` column maps serde variant names from `ReconfigurationStatus` to kebab-case (e.g. `InProgress` -> `in-progress`); unmapped variants pass through verbatim. `changes` diffs `target` against the realized config per dimension. `MZ_CLUSTER_AUTO_SCALING_STRATEGIES` (OID `MV_MZ_CLUSTER_AUTO_SCALING_STRATEGIES_OID`) is a `BuiltinMaterializedView` with a unique key on `cluster_id` that exposes the configured autoscaling strategy and in-flight state per managed cluster; `state` is keyed by strategy type so a future strategy's state is an additional key rather than a schema change. Both views have corresponding `BuiltinIndex` constants (`MZ_CLUSTER_RECONFIGURATIONS_IND`, `MZ_CLUSTER_AUTO_SCALING_STRATEGIES_IND`) on `cluster_id`. `MZ_METRIC_SINKS` (OID `MV_MZ_METRIC_SINKS_OID`) is a `BuiltinMaterializedView` that derives metric sink metadata (`id`, `oid`, `schema_id`, `name`, `from_id`, `cluster_id`, `owner_id`) by parsing `create_sql` from `mz_catalog_raw` via `mz_internal.parse_catalog_create_sql` and filtering to rows where `type = 'metric-sink'`; it has `PUBLIC_SELECT` access, an `Ontology` annotation with foreign-key links to schema, relation, cluster, and role, and a corresponding `BuiltinIndex` (`MZ_METRIC_SINKS_IND`) on `id`. diff --git a/doc/developer/generated/cluster-controller/strategy.md b/doc/developer/generated/cluster-controller/strategy.md index 9afdaaaa92029..c3156f0760906 100644 --- a/doc/developer/generated/cluster-controller/strategy.md +++ b/doc/developer/generated/cluster-controller/strategy.md @@ -1,6 +1,6 @@ --- source: src/cluster-controller/src/strategy.rs -revision: 38a95cefe2 +revision: b2419281fb --- # cluster-controller::strategy diff --git a/doc/developer/generated/clusterd-test-driver/dataflow.md b/doc/developer/generated/clusterd-test-driver/dataflow.md index 53027eb1cef00..590e65e2683c1 100644 --- a/doc/developer/generated/clusterd-test-driver/dataflow.md +++ b/doc/developer/generated/clusterd-test-driver/dataflow.md @@ -1,6 +1,6 @@ --- source: src/clusterd-test-driver/src/dataflow.rs -revision: 5b1466a2b6 +revision: 41e1741ca3 --- # mz-clusterd-test-driver::dataflow @@ -11,6 +11,8 @@ Assembly of compute `DataflowDescription`s for the headless test driver. By default the builder lowers the caller's MIR faithfully without optimization. The `optimize` method enables the MIR dataflow optimizer (`mz_transform::optimize_dataflow`) before lowering, which is required for plans containing a `Join` whose `implementation` is `Unimplemented`. When optimizing, the builder supplies an `ImportedIndexOracle` built from the dataflow's own `index_imports` so the optimizer recognizes imported arrangements. +`DataflowBuilder` exposes the following export methods: `export_index` (arranges a collection and exports the index), `export_materialized_view` (writes a collection to a target persist shard), `export_subscribe` (streams changes as `SubscribeResponse` batches; no shard), and `export_metric_sink` (publishes a collection into the replica's in-process Prometheus registry; no shard). It also provides `import_persist` (persist-backed storage collection) and `import_index` (existing hydrated arrangement), plus `get` to retrieve a typed `MirRelationExpr::Get` for any previously imported or built id. + `DataflowBuilder::explain` renders the lowered LIR plan as `EXPLAIN PHYSICAL PLAN`-style text using a `DummyHumanizer`, without submitting a dataflow. It honors the `optimize` flag identically to `finish`, so the explained plan matches what would be shipped. The `ExplainConfig` has `redacted` pinned to `false` rather than derived from `ExplainConfig::default()`: `default()` derives `redacted` from the build's soft-assertion setting, which anonymizes literals in the release-profile binary and prints them verbatim in debug builds. Pinning `false` makes the golden output independent of the build profile. `index_dataflow` is sugar over `DataflowBuilder` for the common single-index shape. @@ -23,4 +25,4 @@ By default the builder lowers the caller's MIR faithfully without optimization. `ImportedIndexOracle` is a private `IndexOracle` implementation built from a dataflow's `index_imports`, exposing only the arrangements the dataflow itself imports. It is passed to the MIR optimizer when `optimize` is enabled. -The private `augment` function converts a lowered `DataflowDescription` into `DataflowDescription` by flattening each object's plan via `RenderPlan::try_from` and splicing `CollectionMetadata` into source and materialized-view sink entries, mirroring what `compute-client`'s `Instance::create_dataflow` does. +The private `augment` function converts a lowered `DataflowDescription` into `DataflowDescription` by flattening each object's plan via `RenderPlan::try_from` and splicing `CollectionMetadata` into source and materialized-view sink entries, mirroring what `compute-client`'s `Instance::create_dataflow` does. Subscribe and metric sink connections pass through unchanged (they write no shard and carry no storage metadata). diff --git a/doc/developer/generated/compute-client/controller/_module.md b/doc/developer/generated/compute-client/controller/_module.md index 2bea337da90a3..e7accbd8c7724 100644 --- a/doc/developer/generated/compute-client/controller/_module.md +++ b/doc/developer/generated/compute-client/controller/_module.md @@ -1,13 +1,13 @@ --- source: src/compute-client/src/controller.rs -revision: c69fde3d50 +revision: 41e1741ca3 --- # mz-compute-client::controller Provides the compute controller, which manages compute instances, their replicas, and the collections (indexes, sinks, subscribes, COPY TOs) installed on them. `ComputeController` is the top-level entry point; it creates and drops instances (each represented by an `Instance` task via `instance_client`), routes commands and responses through the compute protocol, and exposes `update_replica_dyncfg_overrides` to set per-replica dyncfg overrides (used by the scoped feature flags layer) across all instances. -`ComputeController` maintains a `replica_dyncfg_overrides: BTreeMap` field alongside the per-instance copies. This controller-level copy is consulted at replica-creation time to resolve replica-scoped configs (such as `COMPUTE_REPLICA_EXPIRATION_OFFSET` and `ENABLE_COMPUTE_REPLICA_EXPIRATION`) through the new replica's overrides before the first configuration command arrives; without it, those values would be read from the environment-wide set even when a replica-specific override exists. -When adding a replica, the controller folds the current dyncfg into the `CreateInstance` command as `initial_config` (via `specialize_command_for_replica`) so the replica seeds its worker configuration before create-time setup. A subsequent `UpdateConfiguration` still follows to carry workload class, max result size, tracing, and to sync dyncfg into persist config and metrics; the overlapping dyncfg application is idempotent. -`PeekNotification` converts a `PeekResponse::Error` to its notification form by calling `.to_string()` on the `PeekError`. +`ComputeController` maintains a `replica_dyncfg_overrides: BTreeMap` field alongside the per-instance copies. This controller-level copy is consulted at replica-creation time to resolve replica-scoped configs (such as `COMPUTE_REPLICA_EXPIRATION_OFFSET` and `ENABLE_ARRANGEMENT_DICTIONARY_COMPRESSION_ALPHA`) through the new replica's overrides before the first configuration command arrives; without it, those values would be read from the environment-wide set even when a replica-specific override exists. +When adding a replica, the controller folds the current dyncfg into the `CreateInstance` command as `initial_config` so the replica seeds its worker configuration before create-time setup. A subsequent `UpdateConfiguration` still follows to carry workload class, max result size, tracing, and to sync dyncfg into persist config and metrics; the overlapping dyncfg application is idempotent. +`PeekNotification` has a `Success` variant carrying `rows` (row count after applying `offset`/`limit`) and `result_size` (bytes), an `Error(String)` variant, and a `Canceled` variant. It is constructed from a `PeekResponse` via `PeekNotification::new`, which handles both inline (`PeekResponse::Rows`) and stashed (`PeekResponse::Stashed`) responses. Supporting modules cover per-method error types (`error`), the instance state machine (`instance`), the external instance interface (`instance_client`), replica connection management (`replica`), sequential hydration enforcement (`sequential_hydration`), and introspection routing (`introspection`). diff --git a/doc/developer/generated/compute-types/sinks.md b/doc/developer/generated/compute-types/sinks.md index e724ebd11c40d..57625c11e43d3 100644 --- a/doc/developer/generated/compute-types/sinks.md +++ b/doc/developer/generated/compute-types/sinks.md @@ -1,6 +1,6 @@ --- source: src/compute-types/src/sinks.rs -revision: 94054eb165 +revision: 41e1741ca3 --- # compute-types::sinks @@ -9,4 +9,4 @@ Defines `ComputeSinkDesc` and `ComputeSinkConnection`, the descriptor type The four connection variants are: `Subscribe` (streaming query output), `MaterializedView` (persist-backed MV), `CopyToS3Oneshot` (one-shot COPY TO S3), and `MetricSink` (writes rows into the in-process Prometheus metrics registry). `SubscribeSinkConnection` carries an `output` field (`Vec`) that specifies the ordering for rows emitted by the subscribe. `MaterializedViewSinkConnection` carries a `storage_metadata` field that is filled in by the storage/persist layer. -`MetricSinkConnection` carries no payload: the identity of the metric to update is the sink's `GlobalId`, and the sink does not write to persist. +`MetricSinkConnection` carries a `label: String` field used as the value of the `sink` const label on the sink's companion health gauges. A user-created sink passes its `GlobalId` (durable); a coordinator-installed curated sink passes a stable definition name so the label survives reboots even though the sink's `GlobalId` is transient. The sink does not write to persist. diff --git a/doc/developer/generated/compute/logging/_module.md b/doc/developer/generated/compute/logging/_module.md index a4ea60555cce6..95d0634026b4b 100644 --- a/doc/developer/generated/compute/logging/_module.md +++ b/doc/developer/generated/compute/logging/_module.md @@ -1,6 +1,6 @@ --- source: src/compute/src/logging.rs -revision: 780c9c1add +revision: 4e012ea88d --- # mz-compute::logging diff --git a/doc/developer/generated/compute/sink/metric_sink.md b/doc/developer/generated/compute/sink/metric_sink.md index 093b5c45088bc..538ad460001b5 100644 --- a/doc/developer/generated/compute/sink/metric_sink.md +++ b/doc/developer/generated/compute/sink/metric_sink.md @@ -1,6 +1,6 @@ --- source: src/compute/src/sink/metric_sink.rs -revision: d7b0d9eb90 +revision: 41e1741ca3 --- # mz-compute::sink::metric_sink @@ -15,11 +15,11 @@ The operator routes all data to one worker per process (chosen by hashing the si **`SinkState`** — the full working and published state for one metric sink. Incoming updates are buffered by timestamp in `pending_ok`/`pending_err` and folded into `working` only once the combined ok+err frontier has closed that timestamp. `working` holds a signed multiplicity per full row identity (`RowKey`). `published` is rebuilt from the live set of `working` on each healthy activation. -**`RowKey`** — `(metric_name, labels, value_bits, metric_kind, name_valid, help)`. Labels are `Vec<(String, Option)>` where `None` represents a null label value (distinct from an empty string). The name and labels lead the tuple so that a `BTreeMap` keeps all rows of one `(metric_name, labels)` series adjacent for efficient collision detection. +**`RowKey`** — `(metric_name, labels, value_bits, metric_kind, name_valid, help)`. Labels are `Vec<(String, Option)>` where `None` represents a null label value (distinct from an empty string). `metric_kind` is `Option` where `None` represents any unsupported `metric_type`; two rows differing only in their unsupported type share one identity rather than two. The name and labels lead the tuple so that a `BTreeMap` keeps all rows of one `(metric_name, labels)` series adjacent for efficient collision detection. -**`MetricKind`** — `Gauge` or `Counter`, recovered from the `metric_kind` column the planner's `shape_metric_sink_source` already computed (`0` = gauge, `1` = counter). +**`MetricKind`** — `Gauge` or `Counter`, recovered from the `metric_kind` column the planner's `shape_metric_sink_source` already computed (`0` = gauge, `1` = counter). Any other datum (including `NULL`) produces `None`, representing an unsupported metric type. -**`SinkCollector`** — a `prometheus::core::Collector` that exposes six companion gauges (`mz_metric_sink_frontier_ms`, `mz_metric_sink_errors`, `mz_metric_sink_skipped`, `mz_metric_sink_conflicts`, `mz_metric_sink_collisions`, `mz_metric_sink_null_values`) plus the user-defined series built dynamically as `MetricFamily` protos. The companion gauges carry a `sink` const label so each sink instance has distinct `Desc` ids at registration time. +**`SinkCollector`** — a `prometheus::core::Collector` that exposes six companion gauges (`mz_compute_metric_sink_frontier_ms`, `mz_compute_metric_sink_errors`, `mz_compute_metric_sink_skipped`, `mz_compute_metric_sink_conflicts`, `mz_compute_metric_sink_collisions`, `mz_compute_metric_sink_null_values`) plus the user-defined series built dynamically as `MetricFamily` protos. The companion gauges carry a `sink` const label (taken from `MetricSinkConnection::label`) so each sink instance has distinct `Desc` ids at registration time. ## Collision and conflict semantics @@ -35,4 +35,4 @@ While the sink's error count (`errors`) is nonzero, `published` is not rebuilt; ## Planner contract -The source relation is expected to carry the seven canonical columns produced by `mz_adapter::optimize::metric_sink::shape_metric_sink_source`: `metric_name`, `metric_type`, `labels` (non-null), `value`, `help` (non-null), `metric_kind`, and `name_valid`. `ColumnIndices::resolve` panics if a required column is missing; the SQL planner enforces this contract once the `CREATE METRIC SINK` planning path exists. +The source relation is expected to carry the six canonical columns resolved by `ColumnIndices::resolve`: `metric_name`, `labels` (non-null by construction), `value`, `help` (non-null by construction), `metric_kind`, and `name_valid`. These are produced by `mz_adapter::optimize::metric_sink::shape_metric_sink_source`, which also computes the `metric_kind` and `name_valid` classification columns so this module does not re-parse `metric_type` strings or re-validate `metric_name`. `ColumnIndices::resolve` panics if a required column is missing; the SQL planner enforces this contract. diff --git a/doc/developer/generated/compute/typedefs.md b/doc/developer/generated/compute/typedefs.md index 9d5f1be6e0634..c0307a004eb88 100644 --- a/doc/developer/generated/compute/typedefs.md +++ b/doc/developer/generated/compute/typedefs.md @@ -1,6 +1,6 @@ --- source: src/compute/src/typedefs.rs -revision: 98ea0cc1cc +revision: 4e012ea88d --- # mz-compute::typedefs diff --git a/doc/developer/generated/environmentd/http/_module.md b/doc/developer/generated/environmentd/http/_module.md index ab7ecd41a7b2a..b42aa048219e7 100644 --- a/doc/developer/generated/environmentd/http/_module.md +++ b/doc/developer/generated/environmentd/http/_module.md @@ -1,6 +1,6 @@ --- source: src/environmentd/src/http.rs -revision: 53e55ae9df +revision: 6e83bda250 --- # environmentd::http diff --git a/doc/developer/generated/environmentd/http/console.md b/doc/developer/generated/environmentd/http/console.md index ba9e4d6da7a63..866244c6ae6aa 100644 --- a/doc/developer/generated/environmentd/http/console.md +++ b/doc/developer/generated/environmentd/http/console.md @@ -1,6 +1,6 @@ --- source: src/environmentd/src/http/console.rs -revision: 253293ef87 +revision: 6e83bda250 --- # environmentd::http::console @@ -9,4 +9,5 @@ Provides HTTP handlers for the web console integration. `handle_console_config` returns an unauthenticated JSON response with system variable values the console needs for OIDC login (issuer URL, client ID, scopes), read from the adapter's system vars. `handle_internal_console` is a reverse-proxy handler that forwards requests from the internal HTTP server's `/internal-console` route to the upstream Materialize console URL (default `https://console.materialize.com`), rewriting the `Host` header to avoid Vercel redirect issues. This avoids CORS issues when the console is accessed through a Teleport proxy by serving static console assets from the same host. -`ConsoleProxyConfig` holds the hyper HTTPS client, upstream URL, and route prefix. +`ConsoleProxyConfig` holds the hyper HTTPS client, upstream URL, route prefix, and `preview_host_suffix` (the host of the upstream URL, under which preview builds are served as subdomains). +The handler supports selecting a console preview build via a `?preview_build=