test(early-kickout): end-to-end reassignment + epoch-sync tests#16058
Draft
stedfn wants to merge 18 commits into
Draft
test(early-kickout): end-to-end reassignment + epoch-sync tests#16058stedfn wants to merge 18 commits into
stedfn wants to merge 18 commits into
Conversation
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…t-pr-6 # Conflicts: # CHANGELOG.md # chain/chain/src/store/mod.rs # chain/epoch-manager/src/epoch_info_aggregator.rs # chain/epoch-manager/src/lib.rs # test-loop-tests/src/tests/chunk_producers.rs
…sert Follow-up to the master merge, resolving the debated untrustworthy-sampling concern for the blacklist-aware writer. - seed_chunk_producer_rows: debug_assert that the blacklist-aware sampler yields a producer for every shard that has one. The safety valve in compute_chunk_producer_blacklist guarantees this; the assert enforces the coupling (a None only means an empty shard settlement, nothing to seed). - nonempty_blacklist_anchor_always_has_row: proves the missing-row region and the non-empty-blacklist region are disjoint, so the aggregator's lenient reader never height-samples (which would re-credit a blacklisted producer) while a blacklist is active. - seeded_rows_match_blacklist_aware_sampler: the seeded ChunkProducers row equals the plain height sampler while the blacklist is empty, and equals the blacklist-aware sampler (never the down node) once it is non-empty; the strict consensus reader returns that same row. Chose a writer-side assert over a reader-side one: at the aggregator miss site the per-shard stats are not in scope, and the partial aggregator there reflects a different point in the backward walk, so asserting against it would check the wrong time slice. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…t-pr-6 #15841 (PR5) landed on master independently; resolve the duplicate by taking master for the pure-PR5 files (epoch_info.rs adopts master's bounds-safe sampler, adapter.rs) and keeping the branch's PR6 payload (lib.rs seed rewrite, early_kickout.rs tests). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
test_finalize_epoch_large_epoch_length asserts the incremental aggregator visits each block exactly once (epoch_info_aggregator_loop_counter == N). With EarlyKickout active (nightly/spice builds), seed_chunk_producers reads get_epoch_info_aggregator_upto_last once per record_block_info, adding bounded extra walk iterations, so the exact count only holds with the feature compiled out. Gate the assertion with #[cfg(not(feature = "nightly"))] (the complement of the seed body's own cfg); the caching invariant stays covered on the stable build and the kickout seed path is covered by the early_kickout tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Remove stacked-PR references (#15908/PR5/PR6) and narration that just restates the code from the comments this branch added: revert the compute_chunk_producer_blacklist comment churn to master's wording, drop the seed_chunk_producers reader/activation paragraphs (readers and gating are visible in the code), and reword the test module doc. No behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…up blacklist helper Review response for the mid-epoch chunk-producer kickout writer: - Observability: emit a nightly counter (near_early_kickout_chunk_producer_reassigned_total, by shard) + info log when the seeder reassigns a slot away from a blacklisted producer. The kick detector is "the plain scheduled producer is blacklisted", not "plain != excluded" (the excluding sampler renormalizes, so that would false-fire). - DRY: extract blacklist_for_epoch, shared by the seeder and the get_chunk_producer_blacklist accessor, removing the duplicated epoch-reset check. - Tests: 2-shard per-shard blacklist isolation (+ seeding + metric delta), and a finality-stall regression guard pinning the per-block aggregator-walk cost. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
cspell (CI spellcheck) flagged 'unfinalized' in the stall-test comments; it is not in the project dictionary. Reword to 'not-yet-finalized'. Comment-only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add early_kickout_e2e.rs (nightly + test_features): - test_early_kickout_reassignment: induce a chunk producer below 80% via the adversarial StopProduce message; assert the offending slot is reassigned (DB-backed resolver returns a different validator for the target's own slots once the grandparent anchor blacklists it) while the shard keeps producing chunks (liveness). Condition-based waits, topology avoids the all-blacklisted safety valve. - test_early_kickout_epoch_sync_bootstrap: a fresh node epoch-syncs INTO a network that already has an active reassignment; assert the seeded first-block rows exist and match the source (no ChunkProducerNotInDB), then that after the node lives through a fresh epoch it reproduces the reassignment and agrees with the source (no synced-vs-source divergence). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Spice decouples chunk execution from block production, so the fresh-node epoch-sync path the test drives does not converge within the deadline. The reassignment test needs no epoch sync and stays enabled under spice. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #16058 +/- ##
=======================================
Coverage 73.27% 73.27%
=======================================
Files 857 857
Lines 188902 188902
Branches 188902 188902
=======================================
+ Hits 138412 138423 +11
+ Misses 46075 46065 -10
+ Partials 4415 4414 -1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…t-e2e-tests # Conflicts: # chain/epoch-manager/src/lib.rs # chain/epoch-manager/src/metrics.rs # chain/epoch-manager/src/tests/early_kickout.rs # chain/epoch-manager/src/tests/mod.rs
Drop an internal plan-label prefix from one test comment. Comments only, no code change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The changelog line will be added on stabilization, not on the nightly-gated feature PRs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Re-separate the gas-keys and promise-yield bullets that the previous changelog-drop commit accidentally merged onto one line. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
early_kickout_e2e.rs(nightly + test_features) exercising mid-epoch chunk-producer kickout end to end.test_early_kickout_reassignment: drive a chunk producer below 80% via the adversarial StopProduce message, assert its scheduled slots get reassigned once the grandparent anchor blacklists it, and the shard keeps producing chunks.test_early_kickout_epoch_sync_bootstrap: a fresh node epoch-syncs into a network with an active reassignment, assert the seeded first-block rows match the source, and after living through a fresh epoch it reproduces the reassignment without divergence.Both tests pass (2/2).
🤖 Generated with Claude Code