Skip to content

console: fix stale catalog data across region switches - #38631

Open
leedqin wants to merge 7 commits into
MaterializeInc:mainfrom
leedqin:console-region-switch-fixes
Open

console: fix stale catalog data across region switches#38631
leedqin wants to merge 7 commits into
MaterializeInc:mainfrom
leedqin:console-region-switch-fixes

Conversation

@leedqin

@leedqin leedqin commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Motivation

Switching regions in the console left pages fed by the app-wide SUBSCRIBEs (cluster list, object explorer) showing the previous region's catalog until a full page refresh. Three related defects, in one PR because each is half a fix without the others:

  1. The connection manager never reconnected on a region switch. WebsocketConnectionManager.handleEnvironmentChange updated currentHttpAddress but only connected when the socket was disconnected — and it was still happily connected to the previous region. Fixed by tracking the address of the most recent connection attempt and reconnecting whenever the environment's address differs, including when the switch lands mid-handshake.

  2. The global subscribe atoms served the previous region's rows during (and, if the new region is unhealthy, after) the switch. useGlobalUpsertSubscribe now resets the manager and atom to the loading state when the region changes, via a small useRegionChangeReset hook.

  3. The sync-engine collection cache was poisoned across regions (flagged by the QA review on console: fix the sync-engine cache never seeding on load #38609): hydrate on a scope change scheduled a persist of the rows still in memory, writing the old region's catalog under the new region's cache key, where the instant-load path would serve it as a complete tree. A scope change now drops the pending persist and in-memory rows, then seeds from the new scope's own cache or flushes the emptied set. The same commit clears a stale error when a reconnect's empty pre-snapshot arrives, so the UI falls back to loading instead of holding the error through the whole backoff window (the QA review's second finding).

Tips for reviewer

One commit per defect, in reverse order of the list above. Regression tests accompany each: the scope-change and error-clear tests in subscribeCollection.test.ts, the two region-switch reconnect tests in WebsocketConnectionManager.test.ts, and useSubscribe.test.tsx for the reset hook. Each was verified to fail against the code it fixes.

Checklist

  • This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
  • This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
  • If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
  • If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
  • If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.

🤖 Generated with Claude Code

A region switch re-hydrates the sync-engine collections under a new
scope while the previous region's rows are still in memory. hydrate
pointed the persist key at the new scope and scheduled a persist, so the
old region's rows were written under the new region's cache key and
served as a complete tree on the next load there. Treat a scope change
as a fresh start: drop the pending persist and the in-memory rows, then
seed from the new scope's own cache or flush the emptied set.

Also clear a stale error when an empty pre-snapshot arrives: it is
re-emitted when a reconnect attempt opens, and the UI should fall back
to loading rather than hold the error through the backoff window.
On a region switch the connection manager updated its current address
but only reconnected when the socket was disconnected. The socket was
still connected to the previous region, so app-wide subscribes (cluster
list, object explorer) kept streaming the old region's catalog until a
full page refresh. Track the address of the most recent connection
attempt and reconnect whenever the environment's address differs,
including when the switch lands while a handshake is in flight.
While the re-subscribe at the new address is in flight, the atoms fed by
useGlobalUpsertSubscribe still held the previous region's rows, so pages
like the cluster list showed another region's catalog until the new
snapshot arrived, or indefinitely if the new region is unhealthy. Reset
the manager and atom to the loading state when the region changes, via
a small useRegionChangeReset hook that the tests can exercise directly.
@def-

def- commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

QA LLM Review

1. HIGH -- a region round trip through a non-healthy region permanently kills the global subscribes

console/src/api/materialize/useSubscribe.ts:166

subscribe.reset() clears SubscribeManager.querySent, after which the manager drops every row arriving on the still-open socket until a new connection re-sends the SUBSCRIBE. When the destination region is not healthy the connection manager never attempts a connection, so switching A → (crashed or disabled region) → A leaves allObjects, allSchemas and allClusters pinned at {data: [], snapshotComplete: false, error: undefined} for the rest of the session: the cluster list, object explorer and schema pickers spin forever until the user reloads the page.

Details

handleEnvironmentChange (WebsocketConnectionManager.ts:162) resumes only when !this.target.isConnected() || this.attemptedHttpAddress !== this.currentHttpAddress. On the way out, a non-healthy destination takes the pauseConnection() branch, which neither disconnects nor attempts, so attemptedHttpAddress still holds region A's address. On the way back both disjuncts are false, so nothing reconnects. Reproduced with the file's own mock harness: after A (healthy) → B (state: "enabled", health: "crashed") → A, neither reconnect nor disconnect is ever called. A state: "disabled" destination behaves the same, and there currentHttpAddress is never even updated, since it is only assigned for enabled environments.

Meanwhile useRegionChangeReset has fired twice, and SubscribeManager.reset()resetForNewConnection() sets querySent = false. onRow returns early while that is false, and only onReadyForQuery (fires once per connection, already consumed) or setRequest (gated on readyAwaitingRequest, which reset also clears) can set it back. useAutomaticallyConnectSocket's request effect bails out because previousRequest === request for the memoized global queries. Confirmed directly against SubscribeManager with a stub socket: after reset() on a live connection, later rows produce no data and no further frame is sent.

Reachability is ordinary. EnvironmentSelect lists every region in environmentsWithHealth and sets no isOptionDisabled, and isEnvironmentReady treats health: "crashed" as ready, so a crashed region keeps the user inside /regions/:slug and the selector works normally in both directions. AppInitializer renders outside the route switch, so these managers never remount.

Fix: pair the reset with a guaranteed teardown of the old connection, so the existing !isConnected() branch covers the return trip.

   private pauseConnection() {
     this.clearRetryTimer();
     this.retryAttempt = 0;
+    this.connectInFlight = false;
+    this.target.disconnect();
     this.notifyStateChange();
   }

Clearing connectInFlight in the same place is required, not cosmetic: a pause that lands mid-handshake would otherwise strand the flag forever and block every later attemptConnection, because MaterializeWebsocket.disconnect removes its own listeners before closing, so no open or close callback ever fires for that socket. Disconnecting on pause also stops the previous region's stream while the switch is parked, which is the residual half of finding 2.

2. MEDIUM -- useGlobalSubscribeCollection has no region reset, so the new reconnect replays the previous region's snapshot into the new region's collection and cache

console/src/api/materialize/useSubscribe.ts:215

useGlobalSubscribeCollection did not get the useRegionChangeReset treatment its sibling hook did, so its SubscribeManager still holds region A's completed snapshot when the cross-region reconnect this PR adds opens the new socket. SubscribeManager.onOpen re-emits that snapshot, which re-fills allNamespacesCollection with region A's databases and schemas as snapshotComplete: true after hydrate had just cleared them, and schedules a persist of those rows under region B's cache key — the same cache poisoning the third commit sets out to fix, reintroduced through the first commit's reconnect.

Details

SubscribeManager.onOpen (SubscribeManager.ts:232) re-emits snapshotState whenever snapshotComplete || error, deliberately, to hold data through a same-region resubscribe. useGlobalUpsertSubscribe is safe from it now only because subscribe.reset() empties snapshotState first. Ordering makes the replay land after the clear rather than before: hydrate(newScope) runs in a React effect immediately after the store write, while onOpen waits on a network handshake.

Reproduced against this branch by wiring a real SubscribeManager to a real createSubscribeCollection exactly as the hook does. After hydrating scope A and completing a snapshot, hydrating scope B correctly empties the collection and closes the gate; then reconnect("addr-b:6876") plus socket open leaves collection.has("region-a") === true, statusAtom.snapshotComplete === true, and mz-console:sync-engine:...|org|regionB|v1 holding region A's row once the throttle fires.

The localStorage write needs region B's first flush to take longer than PERSIST_THROTTLE_MS (1s), which is easy given the handshake plus ReadyForQuery plus planning plus snapshot round trip on mz_catalog_server; the object explorer rendering region A's namespaces as a complete tree happens unconditionally in the interim. Once poisoned, the next load in region B seeds those rows from hydrate and sets lastSnapshotComplete = true, serving them through the instant-load path.

…vers

Pausing on a non-healthy region left the socket connected to the
previous region's address, so returning to a healthy region matched
neither resume condition and the global subscribes never reconnected,
while the region reset had already stopped the manager consuming rows.
Disconnect on pause so the resume path reconnects, and clear
connectInFlight since a pause landing mid-handshake gets no open or
close callback for that socket.
useGlobalSubscribeCollection kept the previous region's completed
snapshot in its manager, and the cross-region reconnect's onOpen
replayed it into the new region's collection and scoped cache after
hydrate had cleared them. Reset the manager on region change so the
replay is an ignorable empty pre-snapshot.
@leedqin

leedqin commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

Both findings addressed:

  1. HIGH (unhealthy round trip strands the subscribes): pauseConnection now disconnects the target and clears connectInFlight, exactly as suggested — the return trip then takes the existing !isConnected() resume path. Regression test added for the A → crashed region → A sequence (verified red against the previous commit). Data held in the atoms still survives a same-region health blip: the manager keeps its snapshot and re-emits it on the resubscribe.

  2. MEDIUM (collection hook misses the region reset): useGlobalSubscribeCollection now resets its manager via the same useRegionChangeReset hook, so the reconnect's onOpen replay is an empty pre-snapshot that applySnapshot ignores instead of the previous region's completed snapshot re-filling the collection and its scoped cache.

The slt-5 failure is unrelated — this PR is TypeScript under console/src only.

The app-session subscribes accumulated one effect per concern across two
hooks: manager creation, connection lifecycle, snapshot bridging, a
collection keep-alive, and the region reset, with the ordering between
them carrying the correctness burden. Consolidate each into a session
object (createAtomSubscribeSession / createCollectionSubscribeSession)
that wires the SubscribeManager, WebsocketConnectionManager, sink writes
and region reset through direct store subscriptions, the same pattern
WebsocketConnectionManager already uses for health. The hooks keep one
effect that owns the session lifecycle, and no longer read the async
environment atom, so they cannot suspend. The allObjects-to-collection
bridge likewise moves to a store subscription, so atom ticks no longer
re-render the mounting component. Sessions are plain objects, so their
region behavior is tested headlessly and synchronously.
@leedqin

leedqin commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

Follow-up commit cf78475dd4 restructures how the fixes are wired, in response to a fair "why so many useEffects" review of my own: the global subscribe hooks had accumulated one effect per concern (manager creation, connection lifecycle, snapshot bridge, keep-alive, region reset), with effect ordering carrying the correctness burden — the same class of hazard that caused the #38609 bug.

The wiring now lives in plain session objects (subscribeSession.ts) that subscribe to the jotai store directly, the pattern WebsocketConnectionManager already uses for health/region. Each hook is down to a single lifecycle effect, no longer reads the async environment atom (so it can't suspend), and the allObjects→collection bridge feeds off a store subscription instead of re-rendering its mounting component on every catalog tick. Behavior is unchanged — the sessions carry the exact same guard/reset logic — and being plain objects, the region-switch behavior is now tested headlessly and synchronously instead of through jsdom. Full console suite passes (153 files / 1064 tests).

@def-

def- commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

QA LLM Review

1. MEDIUM -- the pause teardown kills every managed socket on a health-probe blip, including the SQL Shell's, and fires no close callback

console/src/api/materialize/WebsocketConnectionManager.ts:278

pauseConnection now disconnects unconditionally, so a single failed environment health probe tears down every socket a WebsocketConnectionManager owns — including the SQL Shell's, which this PR is not otherwise concerned with. Because MaterializeWebsocket.disconnect() removes its own close listener before closing, the Shell's onClose never runs: an in-flight command is never marked interrupted, and the session's SETs, temp tables and open transaction are silently replaced by a fresh session on resume.

Details

A health verdict is not a socket verdict. fetchEnvironmentHealth (console/src/store/environments.ts:465) issues a separate SELECT mz_version() over HTTP with a 10s abort; any error or timeout past maxBootDuration returns crashed, with no consecutive-failure debounce (updateEnviromentState smooths only booting/unknown). The websocket is unaffected by that request failing, but handleEnvironmentChange treats the verdict as authority to close it.

Verified against this branch: with a real MaterializeWebsocket behind the manager and one healthy region, flipping that same region's status.health to crashed leaves socket.isConnected() === false and the onClose spy uncalled. At a807e30b24 the socket stayed connected.

The Shell's onClose (console/src/platform/shell/ShellWebsocketProvider.tsx:313) is the only sender of CONNECTION_CLOSED, which sets interrupted and surfaces the Retry button (console/src/platform/shell/HistoryOutput.tsx:77); without it a streaming command stops mid-output with no marker and no error. handleTargetClose is skipped too, so no retry is scheduled and recovery waits on the next successful poll. The global subscribe atoms do survive (the manager re-emits its snapshot on resubscribe), but each blip now costs them a full snapshot re-read on mz_catalog_server.

Fix: scope the teardown to what it was added for, a socket pointing at a region the user has left, rather than to every unhealthy verdict. All nine WebsocketConnectionManager tests, including the new round-trip test, still pass with this, and an in-region blip keeps its socket.

   private attemptedHttpAddress?: string;
+  private attemptedRegionId?: string;
@@ private handleEnvironmentChange
+    const regionId = this.store.get(currentRegionIdSyncAtom);
     if (nowHealthy) {
       ...
     } else {
-      this.pauseConnection();
+      // Only tear down when the socket points at a region the user has left;
+      // a health blip in the current region leaves a working socket alone.
+      this.pauseConnection(regionId !== this.attemptedRegionId);
     }
@@ private attemptConnection
     this.attemptedHttpAddress = this.currentHttpAddress;
+    this.attemptedRegionId = this.store.get(currentRegionIdSyncAtom);
@@
-  private pauseConnection() {
+  private pauseConnection(disconnect: boolean) {
     this.clearRetryTimer();
     this.retryAttempt = 0;
-    this.connectInFlight = false;
-    this.target.disconnect();
+    if (disconnect) {
+      // Clear connectInFlight too: this socket gets no open/close callback.
+      this.connectInFlight = false;
+      this.target.disconnect();
+    }
     this.notifyStateChange();
   }

If you'd rather keep the unconditional teardown, the close notification is the part that has to change: an intentional disconnect() should still reach closeListeners/onClose, so consumers like the Shell can mark in-flight work interrupted instead of leaving it hanging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants