-
Notifications
You must be signed in to change notification settings - Fork 1.2k
test: intermittent error due to missing _other_ quorums #6671
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test: intermittent error due to missing _other_ quorums #6671
Conversation
… IS quorum 2025-05-15T19:50:51.884000Z TestFramework (INFO): Test no IS for asset unlock... 2025-05-15T19:50:52.892000Z TestFramework (INFO): Send tx with expected_error:'None'... 2025-05-15T19:51:54.214000Z TestFramework.utils (ERROR): wait_until() failed. Predicate: '''' def check_instantlock(): self.bump_mocktime(1) try: return node.getrawtransaction(txid, True)["instantlock"] except: return False ''' 2025-05-15T19:51:54.214000Z TestFramework (ERROR): Assertion failed Traceback (most recent call last): File "/home/knst/projects/dash-v21/test/functional/test_framework/test_framework.py", line 162, in main self.run_test() File "/home/knst/projects/dash-v21/test/functional/feature_asset_locks.py", line 266, in run_test self.test_asset_unlocks(node_wallet, node, pubkey) File "/home/knst/projects/dash-v21/test/functional/feature_asset_locks.py", line 373, in test_asset_unlocks self.wait_for_instantlock(is_id, node) File "/home/knst/projects/dash-v21/test/functional/test_framework/test_framework.py", line 1655, in wait_for_instantlock if self.wait_until(check_instantlock, timeout=timeout, sleep=1, do_assert=expected) and not expected: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/knst/projects/dash-v21/test/functional/test_framework/test_framework.py", line 902, in wait_until return wait_until_helper(test_function, timeout=timeout, lock=lock, timeout_factor=self.options.timeout_factor, sleep=sleep, do_assert=do_assert) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/knst/projects/dash-v21/test/functional/test_framework/util.py", line 287, in wait_until_helper raise AssertionError("Predicate {} not true after {} seconds".format(predicate_source, timeout)) AssertionError: Predicate '''' def check_instantlock(): self.bump_mocktime(1) try: return node.getrawtransaction(txid, True)["instantlock"] except: return False ''' not true after 60.0 seconds 2025-05-15T19:51:54.265000Z TestFramework (INFO): Stopping nodes node3 2025-05-15T19:50:54.054321Z (mocktime: 2014-12-04T18:19:47Z) [ scheduler] [llmq/signing.cpp:712] [AsyncSignIfMember] [llmq] CSigningManager::AsyncSignIfMember -- failed to select quorum. id=ce79d1f19020ec65caa8a81306362ddad6ffd0e6fc45e18c83630f401b38d54c, msgHash=83b6e8b1549ad9f2750e13e61aabdf08fd31eb1e80cbbd3729920c4c840318ae node2 2025-05-15T19:50:54.054326Z (mocktime: 2014-12-04T18:19:47Z) [ scheduler] [llmq/signing.cpp:712] [AsyncSignIfMember] [llmq] CSigningManager::AsyncSignIfMember -- failed to select quorum. id=ce79d1f19020ec65caa8a81306362ddad6ffd0e6fc45e18c83630f401b38d54c, msgHash=83b6e8b1549ad9f2750e13e61aabdf08fd31eb1e80cbbd3729920c4c840318ae
WalkthroughThe changes introduce conditional quorum mining and enhanced logging across several functional test files. Instead of unconditionally mining quorums, the tests now check if the required quorum types already exist on node 0 before proceeding to mine new ones, logging the outcome in each case. This logic is applied for InstantSend and ChainLocks quorums in 📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (4)
🧰 Additional context used🧬 Code Graph Analysis (2)test/functional/p2p_instantsend.py (1)
test/functional/feature_asset_locks.py (1)
⏰ Context from checks skipped due to timeout of 90000ms (10)
🔇 Additional comments (8)
✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
utACK b0dea8c
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
utACK b0dea8c
…xtra CL 21e6a4e fix: intermittent error in feature_notification due to extra CL (Konstantin Akimov) Pull request description: ## Issue being fixed or feature implemented It can happen if the best block get CL too fast. Locally it fails 1 time out of 10 with error: 2025-05-16T07:09:24.060000Z TestFramework (INFO): Mine single block, wait for chainlock 2025-05-16T07:09:24.275000Z TestFramework (ERROR): Assertion failed Traceback (most recent call last): File "DASH/test/functional/test_framework/test_framework.py", line 162, in main self.run_test() File "DASH/test/functional/feature_notifications.py", line 121, in run_test assert_equal([tip], sorted(os.listdir(self.chainlocknotify_dir))) File "DASH/test/functional/test_framework/util.py", line 69, in assert_equal raise AssertionError("not(%s)" % " == ".join(str(arg) for arg in (thing1, thing2) + args)) AssertionError: not(['21b88334ae6fc29ee3bb7c89dd760dad5dc8ce1064db370dc697b3da4c4315e0'] == ['21b88334ae6fc29ee3bb7c89dd760dad5dc8ce1064db370dc697b3da4c4315e0', '4fe83a613290bd755fba25da02f493376a96971b119884831a75cea33032718d']) 2025-05-16T07:09:24.326000Z TestFramework (INFO): Stopping nodes ## What was done? We should expect CL not only for tip but for block before tip, if CL has been formed really fast. ## How Has This Been Tested? Run feature_notifications.py multiple times with these extra changes #6671 ## Breaking Changes N/A ## Checklist: - [x] I have performed a self-review of my own code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have added or updated relevant unit/integration/functional/e2e tests - [ ] I have made corresponding changes to the documentation - [x] I have assigned this pull request to a milestone ACKs for top commit: UdjinM6: utACK 21e6a4e PastaPastaPasta: utACK 21e6a4e Tree-SHA512: 552ba9f25c7ef15ea7392b90f4d57577b7a9b2ab088325f7ba817d091be54e9d8b3d39129e73cff391ade5ddeb52af877575dfbc428b704931a412c619f08523
…ional tests 0d9418e test: reduce delay in wait_until from 0.5s to 0.05s (Konstantin Akimov) 876d6c8 test: enforce 1 second delay for wait_for_sporks helper (Konstantin Akimov) ec6e7bf test: enforce 1s delay for feature_mnehf test (Konstantin Akimov) 6ab3f7c test: reduce spamming quorum list to logs while waiting (Konstantin Akimov) 9d9975f test: simplify wait_for_quorum_list (Konstantin Akimov) Pull request description: ## Issue being fixed or feature implemented Waiting for 0.5s in functional test for every action is a bit excessive, especially for p2p tests that sending messages by localnetwork and waiting at least 0.5 seconds before checking if message is received. ## What was done? Decreasing default delay from 0.5s to 0.05s. It affects mostly p2p tests, but many other tests become faster too. For quorum formation; for sporks and some other dash specific features bigger delays (0.5s, 1s) are used. Further improvements are blocked by #6673, #6672, #6671 and are out of scope this PR. ## How Has This Been Tested? Speed up on CI for 30% and more. [develop] linux64-test https://gitlab.com/dashpay/dash/-/jobs/10049432489 ALL | ✓ Passed | 7241 s (accumulated) Runtime: 1272 s [PR] linux64-test https://gitlab.com/dashpay/dash/-/jobs/10067158169 ALL | ✓ Passed | 5421 s (accumulated) Runtime: 938 s **-25%** [develop] linux64-nowallet https://gitlab.com/dashpay/dash/-/jobs/10049432511 ALL | ✓ Passed | 2739 s (accumulated) Runtime: 488 s [PR] linux64-nowallet https://gitlab.com/dashpay/dash/-/jobs/10067158174 ALL | ✓ Passed | 1232 s (accumulated) Runtime: 243 s **-49%** [develop] linux64-tsan https://gitlab.com/dashpay/dash/-/jobs/10049432499 ALL | ✓ Passed | 10399 s (accumulated) Runtime: 2023 s [PR] linux64-tsan https://gitlab.com/dashpay/dash/-/jobs/10072993489 ALL | ✓ Passed | 8710 s (accumulated) Runtime: 1543 s **-25%** [develop] Functional tests on localhost (-O3, debug, no sanitizers, -j20): ALL | ✓ Passed | 6680 s (accumulated) Runtime: 372 s [PR] Functional tests on localhost (-O3, debug, no sanitizers, -j20): ALL | ✓ Passed | 4609 s (accumulated) Runtime: 365 s **Benefits of running locally in 20 parallel jobs are very slight. Accumulated time is decreased for 32% as expected, but total time is improved less than 2%.** It is because the slowest tests requires many quorums to be formed and they are still slow. ## Breaking Changes N/A ## Checklist: - [x] I have performed a self-review of my own code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have added or updated relevant unit/integration/functional/e2e tests - [ ] I have made corresponding changes to the documentation - [x] I have assigned this pull request to a milestone ACKs for top commit: PastaPastaPasta: utACK 0d9418e; hopefully it doesn't make tests flakey UdjinM6: utACK 0d9418e Tree-SHA512: 32405bd1f229af5146c96aea6031cee3f084d3ebfb3ec515ad743e79c3bc29a5c891d4330688d07b63b0e06ef7cd50240ab8b6d1a3939a56fe3e64a55918edd1
Issue being fixed or feature implemented
In functional test
feature_asset_locks.py
a quorum for InstantSend is not generated intentionally but usually it is enough time to be formed while other quorums are generated.Similar situation for
p2p_instantsend.py
andfeature_notifications.py
.This fix is one of the blockers for #6631 because with smaller delays extra quorums (beyond expected) almost never formed.
Example of logs:
Meanwhile, there's just no quorum for signing InstantSend, logs from nodes:
What was done?
How Has This Been Tested?
Run functional tests
Breaking Changes
N/A
Checklist: