[PoC] cluster-controller: gate graceful cut-over on lag, not hydration alone

[bosconi]

Motivation

Linear: SQL-672

A graceful ALTER CLUSTER reconfiguration cuts over to its pending replicas as soon as every collection reports hydrated on them. Hydration is as_of < output_frontier, with the as-of pinned to the collection's read frontier at the moment the replica is added. A collection whose initial snapshot takes hours therefore reports hydrated the instant the snapshot lands, with everything since the as-of still to replay. The outgoing replicas hold the cluster's frontiers current until the cut-over, so the lag is invisible; at the cut-over they are dropped and the frontiers freeze until the new replicas drain the backlog. On 2026-08-28 a production cluster cut over after a 4h08m reconfiguration with roughly 15 minutes of index lag remaining.

The cluster-autoscaling design deferred a caught-up check as a later refinement; this PR adds it. The 0dt-upgrade readiness check in caught_up.rs already applies an equivalent lag term.

Description

User-visible effect. A graceful ALTER CLUSTER (resize, availability zones, logging, EXPERIMENTAL ARRANGEMENT COMPRESSION) now retires the old replicas only once the new replicas are both hydrated and caught up to them, within a configurable allowance. Previously the old replicas could be retired while the new ones were still minutes behind, producing a period of stale results after the cut-over. Two new system parameters: cluster_reconfiguration_allowed_lag (default 60s) and enable_cluster_reconfiguration_lag_gate (default on; disabling it restores the hydration-only gate). A reconfiguration whose new replicas cannot catch up within the timeout now rolls back under the default ON TIMEOUT = ROLLBACK; ON TIMEOUT = COMMIT still cuts over at the deadline regardless.

Approach.

• The cut-over gate is readiness: hydrated, and the replica's output frontier for each collection is within the allowance of the furthest output frontier any replica of the cluster reports for it. While the outgoing replicas are present, that reference is theirs; once they are gone a replica is its own reference, so the gate cannot wedge in steady state.
• Output frontiers, not write frontiers. A materialized view's replica-reported write frontier is the persist shard upper, identical on every replica writing it, and for a REFRESH MV it jumps to the next refresh time. A gate on write frontiers would be a no-op for the former and wedge on the latter. The output frontier is meet(write_frontier, compute_probe), the replica's own progress, and is what hydrated() already measures. For indexes the two coincide.
• A second live signal. LiveSignals::hydrated_replicas also feeds the hydration burst, whose durable steady_hydrated_at field means exactly "hydration was observed". The reconfiguration strategy gets its own ready_replicas; the burst is untouched and its probe path is behaviorally identical to before.
• Instance::collections_hydrated_on_replicas becomes collections_ready_on_replicas(.., allowed_lag: Option<Timestamp>). The per-collection decision is a pure function, classify_collection_readiness, that also derives the reference, so it is testable over plain frontiers. The probe logs unhydrated and lagging separately.
• The lag predicate reference <= frontier + allowed_lag is extracted to mz_repr::frontier_within_lag and used by caught_up.rs as well; it saturates rather than panicking on overflow.
• Both cut-over paths are gated: the cluster controller's reconcile tick and the session wait-shim.
• The storage-side hydration check is unchanged.

The design doc paragraph and the ALTER CLUSTER user docs are updated to match.

Verification

New unit tests:

• mz_repr::timestamp::frontier_within_lag_tests: boundary, monotone in the allowance, empty-antichain semantics, saturation instead of panic.
• compute_client::controller::instance::tests on classify_collection_readiness: hydrated-but-lagging is not ready; the reference is the furthest hosting replica; one ready target suffices; None is a true no-op; a complete collection has nothing to lag behind; a sole replica is its own reference; no targets reads unhydrated.
• mz_cluster_controller::tests: graceful_holds_when_target_is_hydrated_but_lagging, graceful_commit_on_timeout_cuts_over_lagging_target; the full-flow seam test asserts the graceful path pulls only the readiness signal.

The new parameters are registered with the test harnesses; LaunchDarkly flags still need to be created.

The 60s default allowance matches with_0dt_caught_up_check_allowed_lag and has not been tuned for this use.

🤖 Generated with Claude Code

[def-]

QA LLM Review

1. MEDIUM -- Downsizing a saturated cluster can no longer complete; it rolls back at the deadline

src/cluster-controller/src/strategy.rs:250

The cut-over now requires the target replicas to hold the live frontier the outgoing replicas are setting, so ALTER CLUSTER SET (SIZE = <smaller>) on a cluster whose target size cannot sustain the input rate never becomes ready and, under the default ROLLBACK, fails at the deadline (24h by default) after running both replica sets for the whole window. It previously hydrated and cut over. The design doc's risk analysis covers only same-size reshapes, but a downsize is the case where the target has structurally worse throughput than the replicas it replaces.

Details

The reference is the join of output_frontier over every replica hosting the collection (src/compute-client/src/controller/instance.rs:3530), so while the outgoing replicas are present the bar is their live frontier. A target replica whose sustainable throughput is below the input rate falls further behind after hydration, so frontier_within_lag never holds and target_ready never fires.

Memory-bound downsizes were already blocked (the replica never hydrates at all). The new class is CPU-bound: the smaller replica fits the arrangement and completes its snapshot but cannot keep up afterwards. Accepting lag in exchange for a smaller size is a legitimate trade for a batch or dev cluster and is expressible today; after this change it needs ON TIMEOUT = COMMIT or enable_cluster_reconfiguration_lag_gate = false on every such ALTER.

The cost of discovering this is the full deadline. The per-collection lag is already computed on every tick, so a target whose lag is not shrinking across ticks is detectably unachievable and could be shed early (audit event plus a terminal record, as ResourceExhausted already does) instead of paying 24h of double capacity. At minimum the size-decreasing case belongs in the design doc's list of affected reconfigurations, since it is far more common than an AZ or logging reshape.

2. LOW -- A concurrent hydration-burst replica raises a cut-over bar the target is not expected to meet

src/compute-client/src/controller/instance.rs:3530

The lag reference is the join over every replica hosting the collection, which includes a hydration-burst replica running at HYDRATION SIZE. The reconfiguration target does not replace that replica and, when HYDRATION SIZE exceeds the target size, may never come within cluster_reconfiguration_allowed_lag of it, so a reconfiguration can be held open by a replica that is about to be torn down.

Details

Burst and graceful reconfiguration coexist by design, and enable_hydration_burst and enable_auto_scaling_strategy both default on, so AUTO SCALING STRATEGY ON HYDRATION (HYDRATION SIZE = '800cc') plus ALTER CLUSTER SET (SIZE = '200cc') reaches this. Burst also has no TTL: if no steady replica ever hydrates, the burst replica stays up indefinitely and the reconfiguration cannot cut over at all.

Worth noting that the two docs disagree on the intended reference. ClusterControllerRequest::ReadyReplicas says "within the configured cut-over lag of the replicas they would replace", i.e. the realized-shape set; ClusterControllerCtx::ready_replicas and the implementation say the furthest frontier of any replica. The wider reference is arguably the safer one for "the cluster's frontier will not freeze", so this may just want the narrower doc corrected, but the burst interaction is a concrete case where the two differ in outcome.