Skip to content

restart: stop hydration history flaking on worker-0 discovery - #38636

Merged
antiguru merged 1 commit into
MaterializeInc:mainfrom
antiguru:fix/hydration-history-mv-discovery-flake
Sep 3, 2026
Merged

restart: stop hydration history flaking on worker-0 discovery#38636
antiguru merged 1 commit into
MaterializeInc:mainfrom
antiguru:fix/hydration-history-mv-discovery-flake

Conversation

@antiguru

@antiguru antiguru commented Sep 3, 2026

Copy link
Copy Markdown
Member

The hydration-history-survives-restart workflow searched for a materialized view whose persist-sink worker sits off worker 0, so that the view's durable finish can be told apart from worker 0's earlier compute-only finish. That search is a dice roll: the sink worker is hash(sink_id) % workers, and a fast snapshot write can collapse both workers' stamps into one logging batch. When no trial separated the workers, the fixture aborted the run, which surfaced in CI as a flake with no product defect behind it.

builtins.AssertionError: test fixture did not produce an MV with its persist-sink worker off worker 0: [('hydration_history_mv_a', ...), ('hydration_history_mv_b', ...)]

Discovery now decides only which trial reports a regression, not whether the test passes. The loop still creates fresh views to roll the sink worker and the snapshot timing, and it still prefers a separating trial. Every fully hydrated trial is then checked against the all-worker maximum, which is the property under test whichever worker holds that maximum. A run where no trial separates the workers therefore still checks the property, it only loses the ability to tell a worker-0-only implementation apart on its own, and that query shape is pinned deterministically by collect_requires_every_worker.

The change is confined to test/restart/mzcompose.py.

🤖 Generated with Claude Code

The `hydration-history-survives-restart` workflow searched for a materialized
view whose persist-sink worker sits off worker 0, so that the view's durable
finish can be told apart from worker 0's earlier compute-only finish. That
search is a dice roll: the sink worker is `hash(sink_id) % workers`, and a fast
snapshot write can collapse both workers' stamps into one logging batch. When
no trial separated the workers, the fixture aborted the run, which showed up in
CI as a flake with no product defect behind it.

Discovery now decides only which trial reports a regression, not whether the
test passes. The loop still creates fresh views to roll the sink worker and the
snapshot timing, and it still prefers a separating trial. Every fully hydrated
trial is then checked against the all-worker maximum, which is the property
under test whichever worker holds that maximum. A run where no trial separates
the workers therefore still checks the property, it only loses the ability to
tell a worker-0-only implementation apart on its own, and that query shape is
pinned deterministically by `collect_requires_every_worker`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@antiguru
antiguru marked this pull request as ready for review September 3, 2026 08:34
@antiguru
antiguru requested a review from aljoscha September 3, 2026 08:34
@antiguru
antiguru merged commit 306f17a into MaterializeInc:main Sep 3, 2026
10 checks passed
@antiguru

antiguru commented Sep 3, 2026

Copy link
Copy Markdown
Member Author

Thanks for the review!

@antiguru
antiguru deleted the fix/hydration-history-mv-discovery-flake branch September 3, 2026 08:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants