DIAGNOSTICS ONLY — DO NOT MERGE: classify intermittent three-node sync divergence - #33
Closed
myobie wants to merge 3 commits into
Closed
DIAGNOSTICS ONLY — DO NOT MERGE: classify intermittent three-node sync divergence#33myobie wants to merge 3 commits into
myobie wants to merge 3 commits into
Conversation
…divergence three_node_2000_file_continuous_mutation_stays_bounded intermittently fails its convergence check on Linux with two nodes holding different content for the same key, observed as node B on c-219 while node A was still on c-154. Instruments the real scenario rather than a standalone reproduction, because the standalone attempt was an invalid instrument: with no reconcile happening during the burst it reported version=1 and converged in one round, exercising none of the behaviour under test. The real test reports version=35 for the same key, which is the difference. Per node per round it now reports the on-disk bytes, the winning manifest entry's version, author and content hash, and whether that node actually holds the bytes for the hash it believes in. Those separate propagation, content fetch and materialization. On divergence it then runs up to ten further rounds, so a case that converges later is classified as the fixed two rounds being too few rather than as an engine defect. Local macOS: converges in 2 rounds with all three nodes at version 35, holding their bytes. The failure is expected to need Linux, as the outbound attach release defect did.
…p in one job Two changes, both to make the instrument capable of observing the thing it is for. The fixed-count caps are asserted BEFORE the convergence check, so every mode-1 bound failure aborts the run and the mode-2 divergence can never be reached. Linux attempt 4 of run 30811632286 demonstrated it: 43 reconciles in 16.14s tripped the count cap of 36 while the rate was 2.66/s against a ceiling of 4, so the run died at the bound and produced no DISCRIMINATE lines at all. Those caps are now warnings here. The RATE ceiling stays a hard assertion because it is the invariant that actually expresses bounded reconcile amplification. And the scenario now repeats twelve times inside one job, stopping at the first divergence. Rerunning the whole workflow spends two to three minutes of the entire library suite to obtain one execution of this scenario; at roughly a one-in-five failure rate, twelve iterations in a single job is around a 93 percent chance of catching it against the 80 percent that eight whole-workflow attempts would give. Not a proposal for main. The mode-1 correction for main is still unwritten and still waiting to land with the mode-2 result in one review.
One line at the scan's republish branch, capturing the prior manifest entry, the scanned disk hash, the resulting entry, and whether the daemon-write journal recognises the file's current bytes and with what committed flag. The journal had to be threaded into the scan to observe this, which is itself the finding: in production the scan path has NO access to the daemon-write journal, so it structurally cannot distinguish bytes the engine wrote from bytes a user wrote. Healthy local run already separates the cases correctly: every republish there is a genuine writer edit and reports journal_match=false.
Collaborator
Author
|
Closing: this did its job. What it proved. Journal logging on CI run 30813115238 classified the intermittent three-node sync divergence as an engine defect — a permanent version leapfrog, not insufficient rounds. Attempt 1 iteration 9 showed node a holding What fixed it. Two follow-ups have since landed on top of that finding:
Superseded, not abandoned. Deleting the branch now. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Do not merge. Instrumentation only, to classify an intermittent Linux failure of
three_node_2000_file_continuous_mutation_stays_bounded.Observed on CI: the convergence check failed with node B on
c-219while node A heldc-154, with every bound comfortably inside its limit — so it is not the bound failure mode.Reports per node per round: on-disk bytes, the winning manifest entry (version, author, content hash), and whether that node holds the bytes for the hash it believes in. That separates propagation from content-fetch from materialization. On divergence it runs up to ten further rounds so a later convergence is classified as the fixed two rounds being too few rather than as an engine defect.
A standalone light reproduction was attempted first and discarded as an invalid instrument: with no reconcile during the burst it reported
version=1and converged in one round. The real test reportsversion=35for the same key.Expected to need several Linux runs; it failed roughly one run in five.