test: intermittent error due to missing _other_ quorums #6671
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… IS quorum
2025-05-15T19:50:51.884000Z TestFramework (INFO): Test no IS for asset unlock...
2025-05-15T19:50:52.892000Z TestFramework (INFO): Send tx with expected_error:'None'...
2025-05-15T19:51:54.214000Z TestFramework.utils (ERROR): wait_until() failed. Predicate: ''''
def check_instantlock():
self.bump_mocktime(1)
try:
return node.getrawtransaction(txid, True)["instantlock"]
except:
return False
'''
2025-05-15T19:51:54.214000Z TestFramework (ERROR): Assertion failed
Traceback (most recent call last):
File "/home/knst/projects/dash-v21/test/functional/test_framework/test_framework.py", line 162, in main
self.run_test()
File "/home/knst/projects/dash-v21/test/functional/feature_asset_locks.py", line 266, in run_test
self.test_asset_unlocks(node_wallet, node, pubkey)
File "/home/knst/projects/dash-v21/test/functional/feature_asset_locks.py", line 373, in test_asset_unlocks
self.wait_for_instantlock(is_id, node)
File "/home/knst/projects/dash-v21/test/functional/test_framework/test_framework.py", line 1655, in wait_for_instantlock
if self.wait_until(check_instantlock, timeout=timeout, sleep=1, do_assert=expected) and not expected:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/knst/projects/dash-v21/test/functional/test_framework/test_framework.py", line 902, in wait_until
return wait_until_helper(test_function, timeout=timeout, lock=lock, timeout_factor=self.options.timeout_factor, sleep=sleep, do_assert=do_assert)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/knst/projects/dash-v21/test/functional/test_framework/util.py", line 287, in wait_until_helper
raise AssertionError("Predicate {} not true after {} seconds".format(predicate_source, timeout))
AssertionError: Predicate ''''
def check_instantlock():
self.bump_mocktime(1)
try:
return node.getrawtransaction(txid, True)["instantlock"]
except:
return False
''' not true after 60.0 seconds
2025-05-15T19:51:54.265000Z TestFramework (INFO): Stopping nodes
node3 2025-05-15T19:50:54.054321Z (mocktime: 2014-12-04T18:19:47Z) [ scheduler] [llmq/signing.cpp:712] [AsyncSignIfMember] [llmq] CSigningManager::AsyncSignIfMember -- failed to select quorum. id=ce79d1f19020ec65caa8a81306362ddad6ffd0e6fc45e18c83630f401b38d54c, msgHash=83b6e8b1549ad9f2750e13e61aabdf08fd31eb1e80cbbd3729920c4c840318ae
node2 2025-05-15T19:50:54.054326Z (mocktime: 2014-12-04T18:19:47Z) [ scheduler] [llmq/signing.cpp:712] [AsyncSignIfMember] [llmq] CSigningManager::AsyncSignIfMember -- failed to select quorum. id=ce79d1f19020ec65caa8a81306362ddad6ffd0e6fc45e18c83630f401b38d54c, msgHash=83b6e8b1549ad9f2750e13e61aabdf08fd31eb1e80cbbd3729920c4c840318ae
WalkthroughThe changes introduce conditional quorum mining and enhanced logging across several functional test files. Instead of unconditionally mining quorums, the tests now check if the required quorum types already exist on node 0 before proceeding to mine new ones, logging the outcome in each case. This logic is applied for InstantSend and ChainLocks quorums in 📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (4)
🧰 Additional context used🧬 Code Graph Analysis (2)test/functional/p2p_instantsend.py (1)
test/functional/feature_asset_locks.py (1)
⏰ Context from checks skipped due to timeout of 90000ms (10)
🔇 Additional comments (8)
✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
5 tasks
knst
changed the title
UdjinM6
approved these changes
May 19, 2025
5 tasks
PastaPastaPasta
added a commit
that referenced
this pull request
May 20, 2025
…xtra CL 21e6a4e fix: intermittent error in feature_notification due to extra CL (Konstantin Akimov) Pull request description: ## Issue being fixed or feature implemented It can happen if the best block get CL too fast. Locally it fails 1 time out of 10 with error: 2025-05-16T07:09:24.060000Z TestFramework (INFO): Mine single block, wait for chainlock 2025-05-16T07:09:24.275000Z TestFramework (ERROR): Assertion failed Traceback (most recent call last): File "DASH/test/functional/test_framework/test_framework.py", line 162, in main self.run_test() File "DASH/test/functional/feature_notifications.py", line 121, in run_test assert_equal([tip], sorted(os.listdir(self.chainlocknotify_dir))) File "DASH/test/functional/test_framework/util.py", line 69, in assert_equal raise AssertionError("not(%s)" % " == ".join(str(arg) for arg in (thing1, thing2) + args)) AssertionError: not(['21b88334ae6fc29ee3bb7c89dd760dad5dc8ce1064db370dc697b3da4c4315e0'] == ['21b88334ae6fc29ee3bb7c89dd760dad5dc8ce1064db370dc697b3da4c4315e0', '4fe83a613290bd755fba25da02f493376a96971b119884831a75cea33032718d']) 2025-05-16T07:09:24.326000Z TestFramework (INFO): Stopping nodes ## What was done? We should expect CL not only for tip but for block before tip, if CL has been formed really fast. ## How Has This Been Tested? Run feature_notifications.py multiple times with these extra changes #6671 ## Breaking Changes N/A ## Checklist: - [x] I have performed a self-review of my own code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have added or updated relevant unit/integration/functional/e2e tests - [ ] I have made corresponding changes to the documentation - [x] I have assigned this pull request to a milestone ACKs for top commit: UdjinM6: utACK 21e6a4e PastaPastaPasta: utACK 21e6a4e Tree-SHA512: 552ba9f25c7ef15ea7392b90f4d57577b7a9b2ab088325f7ba817d091be54e9d8b3d39129e73cff391ade5ddeb52af877575dfbc428b704931a412c619f08523
PastaPastaPasta
added a commit
that referenced
this pull request
Jun 18, 2025
…ional tests 0d9418e test: reduce delay in wait_until from 0.5s to 0.05s (Konstantin Akimov) 876d6c8 test: enforce 1 second delay for wait_for_sporks helper (Konstantin Akimov) ec6e7bf test: enforce 1s delay for feature_mnehf test (Konstantin Akimov) 6ab3f7c test: reduce spamming quorum list to logs while waiting (Konstantin Akimov) 9d9975f test: simplify wait_for_quorum_list (Konstantin Akimov) Pull request description: ## Issue being fixed or feature implemented Waiting for 0.5s in functional test for every action is a bit excessive, especially for p2p tests that sending messages by localnetwork and waiting at least 0.5 seconds before checking if message is received. ## What was done? Decreasing default delay from 0.5s to 0.05s. It affects mostly p2p tests, but many other tests become faster too. For quorum formation; for sporks and some other dash specific features bigger delays (0.5s, 1s) are used. Further improvements are blocked by #6673, #6672, #6671 and are out of scope this PR. ## How Has This Been Tested? Speed up on CI for 30% and more. [develop] linux64-test https://gitlab.com/dashpay/dash/-/jobs/10049432489 ALL | ✓ Passed | 7241 s (accumulated) Runtime: 1272 s [PR] linux64-test https://gitlab.com/dashpay/dash/-/jobs/10067158169 ALL | ✓ Passed | 5421 s (accumulated) Runtime: 938 s **-25%** [develop] linux64-nowallet https://gitlab.com/dashpay/dash/-/jobs/10049432511 ALL | ✓ Passed | 2739 s (accumulated) Runtime: 488 s [PR] linux64-nowallet https://gitlab.com/dashpay/dash/-/jobs/10067158174 ALL | ✓ Passed | 1232 s (accumulated) Runtime: 243 s **-49%** [develop] linux64-tsan https://gitlab.com/dashpay/dash/-/jobs/10049432499 ALL | ✓ Passed | 10399 s (accumulated) Runtime: 2023 s [PR] linux64-tsan https://gitlab.com/dashpay/dash/-/jobs/10072993489 ALL | ✓ Passed | 8710 s (accumulated) Runtime: 1543 s **-25%** [develop] Functional tests on localhost (-O3, debug, no sanitizers, -j20): ALL | ✓ Passed | 6680 s (accumulated) Runtime: 372 s [PR] Functional tests on localhost (-O3, debug, no sanitizers, -j20): ALL | ✓ Passed | 4609 s (accumulated) Runtime: 365 s **Benefits of running locally in 20 parallel jobs are very slight. Accumulated time is decreased for 32% as expected, but total time is improved less than 2%.** It is because the slowest tests requires many quorums to be formed and they are still slow. ## Breaking Changes N/A ## Checklist: - [x] I have performed a self-review of my own code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have added or updated relevant unit/integration/functional/e2e tests - [ ] I have made corresponding changes to the documentation - [x] I have assigned this pull request to a milestone ACKs for top commit: PastaPastaPasta: utACK 0d9418e; hopefully it doesn't make tests flakey UdjinM6: utACK 0d9418e Tree-SHA512: 32405bd1f229af5146c96aea6031cee3f084d3ebfb3ec515ad743e79c3bc29a5c891d4330688d07b63b0e06ef7cd50240ab8b6d1a3939a56fe3e64a55918edd1
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue being fixed or feature implemented
In functional test
feature_asset_locks.pya quorum for InstantSend is not generated intentionally but usually it is enough time to be formed while other quorums are generated.Similar situation for
p2p_instantsend.pyandfeature_notifications.py.This fix is one of the blockers for #6631 because with smaller delays extra quorums (beyond expected) almost never formed.
Example of logs:
Meanwhile, there's just no quorum for signing InstantSend, logs from nodes:
What was done?
How Has This Been Tested?
Run functional tests
Breaking Changes
N/A
Checklist: