From 00b1f0d019babee6cd47190543fdbec12ab9658f Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" Date: Wed, 2 Sep 2026 06:23:00 +0000 Subject: [PATCH 1/2] doc: update generated developer documentation Automated daily refresh of doc/developer/generated/ to reflect source changes since each doc was last written. Co-Authored-By: Claude Opus 4.6 (1M context) --- doc/developer/generated/adapter/coord/_module.md | 4 ++-- .../generated/adapter/coord/catalog_implications.md | 2 +- doc/developer/generated/adapter/coord/ddl.md | 2 +- doc/developer/generated/adapter/coord/introspection.md | 4 ++-- .../generated/adapter/coord/message_handler.md | 4 ++-- .../coord/sequencer/inner/create_metric_sink.md | 2 +- .../generated/adapter/optimize/metric_sink.md | 4 ++-- doc/developer/generated/catalog/builtin/mz_internal.md | 2 +- .../generated/clusterd-test-driver/dataflow.md | 6 ++++-- .../generated/compute-client/controller/_module.md | 8 ++++---- doc/developer/generated/compute-types/sinks.md | 4 ++-- doc/developer/generated/compute/sink/metric_sink.md | 10 +++++----- doc/developer/generated/environmentd/http/_module.md | 2 +- doc/developer/generated/environmentd/http/console.md | 5 +++-- .../generated/environmentd/http/prometheus.md | 2 +- doc/developer/generated/persist-client/_crate.md | 2 +- .../generated/persist-client/internal/_module.md | 2 +- .../generated/persist-client/internal/state.md | 2 +- doc/developer/generated/persist-client/schema.md | 2 +- doc/developer/generated/sql/plan.md | 4 ++-- doc/developer/generated/sql/plan/_module.md | 4 ++-- doc/developer/generated/sql/plan/statement/ddl.md | 2 +- .../generated/sql/plan/statement/ddl/_module.md | 2 +- doc/developer/generated/sql/session/vars.md | 2 +- doc/developer/generated/sql/session/vars/_module.md | 2 +- .../storage-client/storage_collections/_module.md | 2 +- doc/developer/generated/storage-controller/instance.md | 2 +- 27 files changed, 46 insertions(+), 43 deletions(-) diff --git a/doc/developer/generated/adapter/coord/_module.md b/doc/developer/generated/adapter/coord/_module.md index 375a43b53aaee..081a77432ca8d 100644 --- a/doc/developer/generated/adapter/coord/_module.md +++ b/doc/developer/generated/adapter/coord/_module.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord.rs -revision: a1bcaebfe6 +revision: 41e1741ca3 --- # adapter::coord @@ -9,7 +9,7 @@ The core coordinator: `coord.rs` defines the `Coordinator` struct (the central s The coordinator owns the `Catalog`, active compute sinks, pending peeks, read-policy manager, timeline oracles, and all inter-subsystem handles (controller, storage collections, secrets, orchestrator). The `Coordinator` struct holds a `reconcile_now: Arc` field that wakes the cluster controller task to reconcile immediately after catalog transactions that change durable cluster state, rather than waiting out the tick interval. The file also defines `IdPool`, a pre-allocated pool of user `GlobalId` integers that amortizes per-DDL persist writes by reserving batches of IDs at once; the pool is owned by the coordinator and access is serialized through its single-threaded event loop. `ArrangementSizeRecord` is a struct for rows destined for `mz_object_arrangement_size_history`, carrying `replica_id`, `object_id`, `size`, and `hydration_complete`; records are prepared off-thread by the arrangement sizes snapshot task and stamped with a collection timestamp on the coordinator loop at write time. -Child modules partition the coordinator's responsibilities: `command_handler` handles external `Command` messages; `message_handler` handles internal async `Message` responses; `sequencer` executes SQL plans; `appends` manages table and builtin-table writes; `catalog_implications` derives and applies downstream effects from catalog state changes, including pushing replica-scoped dyncfg override layers to the compute and storage controllers via `push_replica_dyncfg_overrides`; `ddl` wraps catalog transactions; `peek` and `read_policy` manage query execution and compaction; `read_then_write` implements the read-then-write protocol for DML (INSERT, UPDATE, DELETE) including dependency validation; `group_sync` computes the diff between sync-managed and manually-granted role memberships for SSO group synchronization; `timestamp_selection` and `timeline` handle temporal reasoning; `catalog_serving` serves catalog snapshots; `info_metrics` owns the catalog `*_info` Prometheus series background task; `cluster_controller` scaffolds the cluster controller task that reconciles durable cluster state with the live controller; `hydration_history` implements the periodic sweep that populates `mz_internal.mz_object_hydration_history`; and supporting modules cover unified compute introspection subscribes, consistency checking, index management, and statement logging. +Child modules partition the coordinator's responsibilities: `command_handler` handles external `Command` messages; `message_handler` handles internal async `Message` responses; `sequencer` executes SQL plans; `appends` manages table and builtin-table writes; `catalog_implications` derives and applies downstream effects from catalog state changes, including pushing replica-scoped dyncfg override layers to the compute and storage controllers via `push_replica_dyncfg_overrides`; `ddl` wraps catalog transactions; `peek` and `read_policy` manage query execution and compaction; `read_then_write` implements the read-then-write protocol for DML (INSERT, UPDATE, DELETE) including dependency validation; `group_sync` computes the diff between sync-managed and manually-granted role memberships for SSO group synchronization; `timestamp_selection` and `timeline` handle temporal reasoning; `catalog_serving` serves catalog snapshots; `info_metrics` owns the catalog `*_info` Prometheus series background task; `cluster_controller` scaffolds the cluster controller task that reconciles durable cluster state with the live controller; `hydration_history` implements the periodic sweep that populates `mz_internal.mz_object_hydration_history`; `metric_sink` manages coordinator-installed curated metric sinks (non-catalog, transient-id sinks installed on every replica at bootstrap and on each new replica, driven by the `MetricSinkStageReady` pipeline); and supporting modules cover unified compute introspection subscribes, consistency checking, index management, and statement logging. The `Coordinator` struct holds a `scoped_frontend: Option>` field populated by `Command::InstallScopedSystemParameterFrontend`. `reconcile_scoped_system_parameters` persists the diff between the current durable working copy and the desired `ScopedParameters` via `Op::UpdateScopedSystemParameters`, which also updates the in-memory working copy and introspection relations. `scoped_overrides_create_op` evaluates scoped overrides for freshly-created clusters and replicas from explicit `ClusterEvalContext` / `ReplicaEvalContext` values and returns an `Op::UpdateScopedSystemParameters` to fold into the same transaction that creates them, so the committed diff drives the replica-scoped controller push before `create_replica`. `replica_dyncfg_overrides` renders the replica-local scoped overrides from the catalog working copy as `ConfigUpdates` grouped by cluster (sparse: only replicas with an override are present). `push_replica_dyncfg_overrides` propagates those overrides to both controllers' per-replica dyncfg layers via the outer `controller.update_replica_dyncfg_overrides`, then re-pushes the environment-wide configs so existing replicas observe any changed or removed overrides. `cluster_scoped_optimizer_overrides` retrieves the cluster-coherent `OptimizerFeatureOverrides` from the catalog working copy for use at plan time. The `Coordinator` struct holds a `catalog_info_metrics_registry: MetricsRegistry` field used to hand the metrics registry to the catalog info-metrics background task spawned at bootstrap. Bootstrap handles derived builtin storage collections (builtin MVs) separately: after registering input-less collections in dependency order, it bumps their sinces based on transitive dependency frontiers to satisfy as-of selection invariants. Bootstrap restores compaction policies for all MV version IDs (via `global_ids()`, not just the write ID) so that no version blocks capability propagation through the `primary` ownership chain and pins compaction. When applying replacement MVs, each `CollectionDescription` receives a `primary` field pointing to its predecessor's latest collection ID, chaining shard ownership from the oldest version through each replacement in order. Bootstrap installs the replica-local scoped overrides via `replica_dyncfg_overrides` and pushes them to the outer controller before any replica is created, so provisioning-time config values (e.g. `TimelyConfig`, expiration offsets) that are frozen into the replica process at provisioning time resolve against the correct overrides. Bootstrap then calls `push_replica_dyncfg_overrides` once after all replicas exist so that existing replicas observe startup-time override values. Bootstrap accepts a `hydrate_migrated_mvs: bool` flag; when true, replacement-migrated builtin MVs call `allow_writes_in_read_only` so they hydrate before cut-over. Bootstrap calls `spawn_cluster_controller_task` to start the cluster controller background task. diff --git a/doc/developer/generated/adapter/coord/catalog_implications.md b/doc/developer/generated/adapter/coord/catalog_implications.md index 843cdcf0f7444..fde8f064becc6 100644 --- a/doc/developer/generated/adapter/coord/catalog_implications.md +++ b/doc/developer/generated/adapter/coord/catalog_implications.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/catalog_implications.rs -revision: c69fde3d50 +revision: 41e1741ca3 --- # adapter::coord::catalog_implications diff --git a/doc/developer/generated/adapter/coord/ddl.md b/doc/developer/generated/adapter/coord/ddl.md index db67c1e95b919..306f0e3c38119 100644 --- a/doc/developer/generated/adapter/coord/ddl.md +++ b/doc/developer/generated/adapter/coord/ddl.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/ddl.rs -revision: 39dcae2fba +revision: 41e1741ca3 --- # adapter::coord::ddl diff --git a/doc/developer/generated/adapter/coord/introspection.md b/doc/developer/generated/adapter/coord/introspection.md index 60953ddcfbb5d..196f77e5a5a75 100644 --- a/doc/developer/generated/adapter/coord/introspection.md +++ b/doc/developer/generated/adapter/coord/introspection.md @@ -1,11 +1,11 @@ --- source: src/adapter/src/coord/introspection.rs -revision: 7f6c52776d +revision: 41e1741ca3 --- # adapter::coord::introspection Implements unified compute introspection: the process of collecting introspection data exported by individual replicas through their logging indexes and writing that data, tagged with the respective replica ID, to unified storage collections. -`install_introspection_subscribes` installs all defined introspection subscribes on a given replica (and `bootstrap_introspection_subscribes` calls it for all existing replicas during coordinator startup); `handle_introspection_subscribe_batch` processes each batch response, writing updates to the corresponding storage-managed collection and reinstalling failed subscribes on disconnect; `drop_introspection_subscribes` removes all subscribes installed on a replica before it is dropped. +`all_cluster_replicas` returns every `(ClusterId, ReplicaId)` pair currently in the catalog and is shared by both introspection subscribes and curated metric sinks so those callers cannot drift on what "all replicas" means. `install_introspection_subscribes` installs all defined introspection subscribes on a given replica (and `bootstrap_introspection_subscribes` calls it for all existing replicas during coordinator startup); `handle_introspection_subscribe_batch` processes each batch response, writing updates to the corresponding storage-managed collection and reinstalling failed subscribes on disconnect; `drop_introspection_subscribes` removes all subscribes installed on a replica before it is dropped. Each introspection subscribe is sequenced through a multi-stage pipeline (`OptimizeMir` → `TimestampOptimizeLir` → `Finish`) using the `sequence_staged` driver. The optimizer config for introspection subscribes includes cluster-coherent scoped overrides via `Coordinator::cluster_scoped_optimizer_overrides`. `IntrospectionSubscribe` tracks a `first_data_at: Option` field recording when the subscribe first appended data to its target storage collection in the current process. Rows in that collection before that point may describe a previous environmentd process or a prior replica incarnation and must not be trusted. `invalidate_introspection_freshness(replica_id)` clears `first_data_at` for all subscribes targeting a given replica when a cluster event reports the replica offline or restarted. `fresh_introspection_replicas(introspection_type, margin)` returns the string replica IDs whose subscribe of the given type delivered data at least `margin` ago; consumers such as the arrangement sizes snapshot use this to exclude stale replicas. diff --git a/doc/developer/generated/adapter/coord/message_handler.md b/doc/developer/generated/adapter/coord/message_handler.md index 9abc3c69e1562..3f6594d86f55b 100644 --- a/doc/developer/generated/adapter/coord/message_handler.md +++ b/doc/developer/generated/adapter/coord/message_handler.md @@ -1,12 +1,12 @@ --- source: src/adapter/src/coord/message_handler.rs -revision: a1bcaebfe6 +revision: 41e1741ca3 --- # adapter::coord::message_handler Implements `Coordinator::handle_message`, the main dispatch for internal `Message` variants flowing through the coordinator's event loop. -Handles controller responses (compute peek results, subscribe batches, copy-to responses, watch-set notifications), timer ticks (group commit, timeline advancement, storage usage collection and pruning), staged-pipeline continuations (peek, create index, create view, create materialized view, subscribe, introspection subscribe, explain timestamp, secret, cluster), linearized read delivery, deferred statement execution, private-link VPC endpoint events, cluster controller requests (`Message::ClusterControllerRequest`, dispatched to `handle_cluster_controller_request`), and hydration-history scheduling (`Message::HydrationHistorySchedule` dispatched to `schedule_hydration_history_collection`; `Message::HydrationHistoryRun` dispatched to `run_hydration_history_collection`). +Handles controller responses (compute peek results, subscribe batches, copy-to responses, watch-set notifications), timer ticks (group commit, timeline advancement, storage usage collection and pruning), staged-pipeline continuations (peek, create index, create metric sink, create view, create materialized view, subscribe, introspection subscribe, metric sink (coordinator-installed curated sinks), explain timestamp, secret, cluster), linearized read delivery, deferred statement execution, private-link VPC endpoint events, cluster controller requests (`Message::ClusterControllerRequest`, dispatched to `handle_cluster_controller_request`), and hydration-history scheduling (`Message::HydrationHistorySchedule` dispatched to `schedule_hydration_history_collection`; `Message::HydrationHistoryRun` dispatched to `run_hydration_history_collection`). This is the heart of the coordinator's reactive loop; every asynchronous response from the storage/compute layers arrives here. `storage_usage_update` obtains a write timestamp from the oracle, allocates a single durable batch id via `Catalog::allocate_storage_usage_id`, builds `BuiltinTableUpdate` rows via `pack_storage_usage_update` for each shard, and submits them via `builtin_table_update().execute()` without going through `catalog_transact_inner`. In read-only mode, `storage_usage_fetch` logs an info message and reschedules via `Message::StorageUsageSchedule` without performing any shard scan or writes. diff --git a/doc/developer/generated/adapter/coord/sequencer/inner/create_metric_sink.md b/doc/developer/generated/adapter/coord/sequencer/inner/create_metric_sink.md index 7deccf1cbe219..7d26d12f3041e 100644 --- a/doc/developer/generated/adapter/coord/sequencer/inner/create_metric_sink.md +++ b/doc/developer/generated/adapter/coord/sequencer/inner/create_metric_sink.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/coord/sequencer/inner/create_metric_sink.rs -revision: a702b8be70 +revision: 41e1741ca3 --- # adapter::coord::sequencer::inner::create_metric_sink diff --git a/doc/developer/generated/adapter/optimize/metric_sink.md b/doc/developer/generated/adapter/optimize/metric_sink.md index ad09d691f3b41..e34800a80ae4d 100644 --- a/doc/developer/generated/adapter/optimize/metric_sink.md +++ b/doc/developer/generated/adapter/optimize/metric_sink.md @@ -1,6 +1,6 @@ --- source: src/adapter/src/optimize/metric_sink.rs -revision: a702b8be70 +revision: 41e1741ca3 --- # adapter::optimize::metric_sink @@ -25,4 +25,4 @@ No rows are filtered: the operator needs every row, including invalid ones, to c `METRIC_NAME_PATTERN` is the Prometheus metric name grammar regexp (`^[a-zA-Z_:][a-zA-Z0-9_:]*$`), stored as a constant for use in the MIR `IsRegexpMatchCaseSensitive` scalar. -The `Optimizer` struct for metric sinks implements `Optimize` (MIR stage, producing `GlobalMirPlan`) and `Optimize` (LIR stage, producing `GlobalLirPlan`). Like `CREATE INDEX`, the pipeline starts directly from the `GlobalId` of the collection to export rather than lowering a new relational expression from HIR. Unlike a materialized view sink, there is no persist shard, so there is no storage-metadata stage. +The `Optimizer` struct for metric sinks implements `Optimize` (MIR stage, producing `GlobalMirPlan`) and `Optimize` (LIR stage, producing `GlobalLirPlan`). The source is described by `MetricSinkFrom`: `MetricSinkFrom::Id(GlobalId)` starts from an existing catalog collection (like `CREATE INDEX`, no HIR to lower), while `MetricSinkFrom::Query { expr, desc }` lowers a planned HIR query (what a coordinator-installed curated sink built from SQL uses). In both cases the row-wise shaping is appended in MIR and the dataflow exports a single `MetricSink`. The `MetricSink` wrapper also carries an optional `label` field for the `sink` label on health gauges; `None` defaults to the sink's `GlobalId` (for user sinks), while curated sinks pass their stable name. Unlike a materialized view sink, there is no persist shard, so there is no storage-metadata stage. diff --git a/doc/developer/generated/catalog/builtin/mz_internal.md b/doc/developer/generated/catalog/builtin/mz_internal.md index 0e455c1c263d4..359a04fe99d7c 100644 --- a/doc/developer/generated/catalog/builtin/mz_internal.md +++ b/doc/developer/generated/catalog/builtin/mz_internal.md @@ -1,6 +1,6 @@ --- source: src/catalog/src/builtin/mz_internal.rs -revision: c317ceee3c +revision: 09a24b328a --- # catalog::builtin::mz_internal diff --git a/doc/developer/generated/clusterd-test-driver/dataflow.md b/doc/developer/generated/clusterd-test-driver/dataflow.md index 53027eb1cef00..590e65e2683c1 100644 --- a/doc/developer/generated/clusterd-test-driver/dataflow.md +++ b/doc/developer/generated/clusterd-test-driver/dataflow.md @@ -1,6 +1,6 @@ --- source: src/clusterd-test-driver/src/dataflow.rs -revision: 5b1466a2b6 +revision: 41e1741ca3 --- # mz-clusterd-test-driver::dataflow @@ -11,6 +11,8 @@ Assembly of compute `DataflowDescription`s for the headless test driver. By default the builder lowers the caller's MIR faithfully without optimization. The `optimize` method enables the MIR dataflow optimizer (`mz_transform::optimize_dataflow`) before lowering, which is required for plans containing a `Join` whose `implementation` is `Unimplemented`. When optimizing, the builder supplies an `ImportedIndexOracle` built from the dataflow's own `index_imports` so the optimizer recognizes imported arrangements. +`DataflowBuilder` exposes the following export methods: `export_index` (arranges a collection and exports the index), `export_materialized_view` (writes a collection to a target persist shard), `export_subscribe` (streams changes as `SubscribeResponse` batches; no shard), and `export_metric_sink` (publishes a collection into the replica's in-process Prometheus registry; no shard). It also provides `import_persist` (persist-backed storage collection) and `import_index` (existing hydrated arrangement), plus `get` to retrieve a typed `MirRelationExpr::Get` for any previously imported or built id. + `DataflowBuilder::explain` renders the lowered LIR plan as `EXPLAIN PHYSICAL PLAN`-style text using a `DummyHumanizer`, without submitting a dataflow. It honors the `optimize` flag identically to `finish`, so the explained plan matches what would be shipped. The `ExplainConfig` has `redacted` pinned to `false` rather than derived from `ExplainConfig::default()`: `default()` derives `redacted` from the build's soft-assertion setting, which anonymizes literals in the release-profile binary and prints them verbatim in debug builds. Pinning `false` makes the golden output independent of the build profile. `index_dataflow` is sugar over `DataflowBuilder` for the common single-index shape. @@ -23,4 +25,4 @@ By default the builder lowers the caller's MIR faithfully without optimization. `ImportedIndexOracle` is a private `IndexOracle` implementation built from a dataflow's `index_imports`, exposing only the arrangements the dataflow itself imports. It is passed to the MIR optimizer when `optimize` is enabled. -The private `augment` function converts a lowered `DataflowDescription` into `DataflowDescription` by flattening each object's plan via `RenderPlan::try_from` and splicing `CollectionMetadata` into source and materialized-view sink entries, mirroring what `compute-client`'s `Instance::create_dataflow` does. +The private `augment` function converts a lowered `DataflowDescription` into `DataflowDescription` by flattening each object's plan via `RenderPlan::try_from` and splicing `CollectionMetadata` into source and materialized-view sink entries, mirroring what `compute-client`'s `Instance::create_dataflow` does. Subscribe and metric sink connections pass through unchanged (they write no shard and carry no storage metadata). diff --git a/doc/developer/generated/compute-client/controller/_module.md b/doc/developer/generated/compute-client/controller/_module.md index 2bea337da90a3..e7accbd8c7724 100644 --- a/doc/developer/generated/compute-client/controller/_module.md +++ b/doc/developer/generated/compute-client/controller/_module.md @@ -1,13 +1,13 @@ --- source: src/compute-client/src/controller.rs -revision: c69fde3d50 +revision: 41e1741ca3 --- # mz-compute-client::controller Provides the compute controller, which manages compute instances, their replicas, and the collections (indexes, sinks, subscribes, COPY TOs) installed on them. `ComputeController` is the top-level entry point; it creates and drops instances (each represented by an `Instance` task via `instance_client`), routes commands and responses through the compute protocol, and exposes `update_replica_dyncfg_overrides` to set per-replica dyncfg overrides (used by the scoped feature flags layer) across all instances. -`ComputeController` maintains a `replica_dyncfg_overrides: BTreeMap` field alongside the per-instance copies. This controller-level copy is consulted at replica-creation time to resolve replica-scoped configs (such as `COMPUTE_REPLICA_EXPIRATION_OFFSET` and `ENABLE_COMPUTE_REPLICA_EXPIRATION`) through the new replica's overrides before the first configuration command arrives; without it, those values would be read from the environment-wide set even when a replica-specific override exists. -When adding a replica, the controller folds the current dyncfg into the `CreateInstance` command as `initial_config` (via `specialize_command_for_replica`) so the replica seeds its worker configuration before create-time setup. A subsequent `UpdateConfiguration` still follows to carry workload class, max result size, tracing, and to sync dyncfg into persist config and metrics; the overlapping dyncfg application is idempotent. -`PeekNotification` converts a `PeekResponse::Error` to its notification form by calling `.to_string()` on the `PeekError`. +`ComputeController` maintains a `replica_dyncfg_overrides: BTreeMap` field alongside the per-instance copies. This controller-level copy is consulted at replica-creation time to resolve replica-scoped configs (such as `COMPUTE_REPLICA_EXPIRATION_OFFSET` and `ENABLE_ARRANGEMENT_DICTIONARY_COMPRESSION_ALPHA`) through the new replica's overrides before the first configuration command arrives; without it, those values would be read from the environment-wide set even when a replica-specific override exists. +When adding a replica, the controller folds the current dyncfg into the `CreateInstance` command as `initial_config` so the replica seeds its worker configuration before create-time setup. A subsequent `UpdateConfiguration` still follows to carry workload class, max result size, tracing, and to sync dyncfg into persist config and metrics; the overlapping dyncfg application is idempotent. +`PeekNotification` has a `Success` variant carrying `rows` (row count after applying `offset`/`limit`) and `result_size` (bytes), an `Error(String)` variant, and a `Canceled` variant. It is constructed from a `PeekResponse` via `PeekNotification::new`, which handles both inline (`PeekResponse::Rows`) and stashed (`PeekResponse::Stashed`) responses. Supporting modules cover per-method error types (`error`), the instance state machine (`instance`), the external instance interface (`instance_client`), replica connection management (`replica`), sequential hydration enforcement (`sequential_hydration`), and introspection routing (`introspection`). diff --git a/doc/developer/generated/compute-types/sinks.md b/doc/developer/generated/compute-types/sinks.md index e724ebd11c40d..57625c11e43d3 100644 --- a/doc/developer/generated/compute-types/sinks.md +++ b/doc/developer/generated/compute-types/sinks.md @@ -1,6 +1,6 @@ --- source: src/compute-types/src/sinks.rs -revision: 94054eb165 +revision: 41e1741ca3 --- # compute-types::sinks @@ -9,4 +9,4 @@ Defines `ComputeSinkDesc` and `ComputeSinkConnection`, the descriptor type The four connection variants are: `Subscribe` (streaming query output), `MaterializedView` (persist-backed MV), `CopyToS3Oneshot` (one-shot COPY TO S3), and `MetricSink` (writes rows into the in-process Prometheus metrics registry). `SubscribeSinkConnection` carries an `output` field (`Vec`) that specifies the ordering for rows emitted by the subscribe. `MaterializedViewSinkConnection` carries a `storage_metadata` field that is filled in by the storage/persist layer. -`MetricSinkConnection` carries no payload: the identity of the metric to update is the sink's `GlobalId`, and the sink does not write to persist. +`MetricSinkConnection` carries a `label: String` field used as the value of the `sink` const label on the sink's companion health gauges. A user-created sink passes its `GlobalId` (durable); a coordinator-installed curated sink passes a stable definition name so the label survives reboots even though the sink's `GlobalId` is transient. The sink does not write to persist. diff --git a/doc/developer/generated/compute/sink/metric_sink.md b/doc/developer/generated/compute/sink/metric_sink.md index 093b5c45088bc..538ad460001b5 100644 --- a/doc/developer/generated/compute/sink/metric_sink.md +++ b/doc/developer/generated/compute/sink/metric_sink.md @@ -1,6 +1,6 @@ --- source: src/compute/src/sink/metric_sink.rs -revision: d7b0d9eb90 +revision: 41e1741ca3 --- # mz-compute::sink::metric_sink @@ -15,11 +15,11 @@ The operator routes all data to one worker per process (chosen by hashing the si **`SinkState`** — the full working and published state for one metric sink. Incoming updates are buffered by timestamp in `pending_ok`/`pending_err` and folded into `working` only once the combined ok+err frontier has closed that timestamp. `working` holds a signed multiplicity per full row identity (`RowKey`). `published` is rebuilt from the live set of `working` on each healthy activation. -**`RowKey`** — `(metric_name, labels, value_bits, metric_kind, name_valid, help)`. Labels are `Vec<(String, Option)>` where `None` represents a null label value (distinct from an empty string). The name and labels lead the tuple so that a `BTreeMap` keeps all rows of one `(metric_name, labels)` series adjacent for efficient collision detection. +**`RowKey`** — `(metric_name, labels, value_bits, metric_kind, name_valid, help)`. Labels are `Vec<(String, Option)>` where `None` represents a null label value (distinct from an empty string). `metric_kind` is `Option` where `None` represents any unsupported `metric_type`; two rows differing only in their unsupported type share one identity rather than two. The name and labels lead the tuple so that a `BTreeMap` keeps all rows of one `(metric_name, labels)` series adjacent for efficient collision detection. -**`MetricKind`** — `Gauge` or `Counter`, recovered from the `metric_kind` column the planner's `shape_metric_sink_source` already computed (`0` = gauge, `1` = counter). +**`MetricKind`** — `Gauge` or `Counter`, recovered from the `metric_kind` column the planner's `shape_metric_sink_source` already computed (`0` = gauge, `1` = counter). Any other datum (including `NULL`) produces `None`, representing an unsupported metric type. -**`SinkCollector`** — a `prometheus::core::Collector` that exposes six companion gauges (`mz_metric_sink_frontier_ms`, `mz_metric_sink_errors`, `mz_metric_sink_skipped`, `mz_metric_sink_conflicts`, `mz_metric_sink_collisions`, `mz_metric_sink_null_values`) plus the user-defined series built dynamically as `MetricFamily` protos. The companion gauges carry a `sink` const label so each sink instance has distinct `Desc` ids at registration time. +**`SinkCollector`** — a `prometheus::core::Collector` that exposes six companion gauges (`mz_compute_metric_sink_frontier_ms`, `mz_compute_metric_sink_errors`, `mz_compute_metric_sink_skipped`, `mz_compute_metric_sink_conflicts`, `mz_compute_metric_sink_collisions`, `mz_compute_metric_sink_null_values`) plus the user-defined series built dynamically as `MetricFamily` protos. The companion gauges carry a `sink` const label (taken from `MetricSinkConnection::label`) so each sink instance has distinct `Desc` ids at registration time. ## Collision and conflict semantics @@ -35,4 +35,4 @@ While the sink's error count (`errors`) is nonzero, `published` is not rebuilt; ## Planner contract -The source relation is expected to carry the seven canonical columns produced by `mz_adapter::optimize::metric_sink::shape_metric_sink_source`: `metric_name`, `metric_type`, `labels` (non-null), `value`, `help` (non-null), `metric_kind`, and `name_valid`. `ColumnIndices::resolve` panics if a required column is missing; the SQL planner enforces this contract once the `CREATE METRIC SINK` planning path exists. +The source relation is expected to carry the six canonical columns resolved by `ColumnIndices::resolve`: `metric_name`, `labels` (non-null by construction), `value`, `help` (non-null by construction), `metric_kind`, and `name_valid`. These are produced by `mz_adapter::optimize::metric_sink::shape_metric_sink_source`, which also computes the `metric_kind` and `name_valid` classification columns so this module does not re-parse `metric_type` strings or re-validate `metric_name`. `ColumnIndices::resolve` panics if a required column is missing; the SQL planner enforces this contract. diff --git a/doc/developer/generated/environmentd/http/_module.md b/doc/developer/generated/environmentd/http/_module.md index ab7ecd41a7b2a..b42aa048219e7 100644 --- a/doc/developer/generated/environmentd/http/_module.md +++ b/doc/developer/generated/environmentd/http/_module.md @@ -1,6 +1,6 @@ --- source: src/environmentd/src/http.rs -revision: 53e55ae9df +revision: 6e83bda250 --- # environmentd::http diff --git a/doc/developer/generated/environmentd/http/console.md b/doc/developer/generated/environmentd/http/console.md index ba9e4d6da7a63..866244c6ae6aa 100644 --- a/doc/developer/generated/environmentd/http/console.md +++ b/doc/developer/generated/environmentd/http/console.md @@ -1,6 +1,6 @@ --- source: src/environmentd/src/http/console.rs -revision: 253293ef87 +revision: 6e83bda250 --- # environmentd::http::console @@ -9,4 +9,5 @@ Provides HTTP handlers for the web console integration. `handle_console_config` returns an unauthenticated JSON response with system variable values the console needs for OIDC login (issuer URL, client ID, scopes), read from the adapter's system vars. `handle_internal_console` is a reverse-proxy handler that forwards requests from the internal HTTP server's `/internal-console` route to the upstream Materialize console URL (default `https://console.materialize.com`), rewriting the `Host` header to avoid Vercel redirect issues. This avoids CORS issues when the console is accessed through a Teleport proxy by serving static console assets from the same host. -`ConsoleProxyConfig` holds the hyper HTTPS client, upstream URL, and route prefix. +`ConsoleProxyConfig` holds the hyper HTTPS client, upstream URL, route prefix, and `preview_host_suffix` (the host of the upstream URL, under which preview builds are served as subdomains). +The handler supports selecting a console preview build via a `?preview_build=