phase out pseudorandom test-case generators

[skaunov]

(resolve #110)

Tests which require no structure in their input and intake only crypto randomness have no chance to have an advantage switching to proptest as it kind of searching an input space which feels pointless in such a case but bring overhead. Boundary tests seems to be a better thing (like trying identity elements, order, prime, maximal).

I've started trying to convert like <src/main_loop/proof_upgrader.rs> or <src/models/state/archival_state.rs> (<src/models/state/mod.rs>, <src/models/state/wallet/wallet_state.rs>), but just adapting it to helpers derandomized fits better. (The first approach was an attempt to do everything with a TestRunner drilling it down a call stack.)

helpers randomness

I feel that helpers should not generate more/own randomness as they can be reused in a proptest setting and only inspection of their calls can reveal if they do. =( I added Randomness (was Seeds) to pass a number of random values without cluttering arguments. Approach with pop helps to find mistakes in a test.

an awkward arbitrary test

block_hash_relates_to_predecessor_difficulty received a bit awkward arbitrary test instead of just random due to it's the only place using random_transaction_kernel helper and making no sense to be converted in a proptest

shamir strategies

I was considering to make a recursive strategy for shamir, but managed to go without that.

fn indices_strategy(n: usize, t: usize, oneshorter: bool) -> BoxedStrategy<Vec<usize>> {
    let mut indices = (0..n).collect_vec();
    indices.shuffle(&mut rng());
    indices.truncate(t);
    indices
}

Unsurprisingly a straightforward approach to generate a HashSet hits the local rejection limit,
#[strategy(proptest::collection::hash_set(0usize..#n, #t))] indices: HashSet<usize>,
so I relied on the arbitrariness of the HashSet itself. Also I've used that approach when modifying removal_record_missing_chunk_element_is_invalid_pbt. #arbitraryHashSetIterator

Also it could be an optimization to use lazy_static for the initial set generation, as it seems to me making a const for the initial set isn't possible. Note that it will significantly decrease the arbitrariness of the HashSet.

const EMPTY_SET: HashSet<usize, BuildHasherDefault<DefaultHasher>> = [2..20].into::<[usize; 18]>()
HashSet::with_hasher(BuildHasherDefault::new());

property coverage

For amount_scalar_mul_pbt a natural Strategy ends up in the rejection limit. I chose to go with an approach which better describes the properties of the method and tests the whole domain for the price of hitting the interesting part quite rarely. But I feel it's still often enough, and also a case is added to the proptest file. There still an option to generate the values for it via division.

(a, b) in ((0..42000000u32), (0..42000000u32)).prop_filter(
    "the product should be $\\leq 42000000$", |(a, b)| u64::from(*a) * u64::from(*b) <= 42000000
),

[aszepieniec]

Good call to pause and wait for feedback, because to be honest I do not like these changes.

Be careful to avoid changing tests that involve producing proofs. Because producing proofs is expensive and we would like to avoid doing it unless absolutely necessary. (This is why we derandomized so many of them.) Mind you, we will generate new proofs if there is a compelling reason to do so.

For test helper functions, replacing a seed or rng argument with a test runner does not constitute an improvement. Instead, the test helper function should be redesigned. And I'm not sure there is a single formula that applies to all. Perhaps it should return a BoxedStrategy instead whatever it returns now, and the derandomization should happen on the side of the caller? Come to think of it, it might be useful to have a helper function that takes a Strategy and returns a deterministic sample. In fact, given that prop_map and prop_compose exist, we probably should not be passing test runners as function arguments anywhere.

[skaunov]

@aszepieniec

Be careful to avoid changing tests that involve producing proofs. Because producing proofs is expensive and we would like to avoid doing it unless absolutely necessary. (This is why we derandomized so many of them.) Mind you, we will generate new proofs if there is a compelling reason to do so.

I understand the idea, but not sure I will apply it correctly on my own: could you add an example or two for the reference!

[skaunov]

@aszepieniec

Come to think of it, it might be useful to have a helper function that takes a Strategy and returns a deterministic sample. In fact, given that prop_map and prop_compose exist, we probably should not be passing test runners as function arguments anywhere.

I was considering to add a macros for this, but I felt that such a facility will hinder further switching to the actual proptest! -- I mean just visually this methods calls chain communicates that it'd be nice to deal with it. And I supposed after the current issue is done there should be some work converting appropriate tests into proptest!. Tell me if that not a thing, and macros is proffered.

[aszepieniec]

@aszepieniec

Be careful to avoid changing tests that involve producing proofs. Because producing proofs is expensive and we would like to avoid doing it unless absolutely necessary. (This is why we derandomized so many of them.) Mind you, we will generate new proofs if there is a compelling reason to do so.

I understand the idea, but not sure I will apply it correctly on my own: could you add an example or two for the reference!

For instance, in single_proof.rs there is a test called disallow_set_merge_bit_in_pc_path. It is a negative test that verifies that if you try to cheat at the end of a sequence of proofs, you fail. The test is entirely derandomized so that all the proofs leading up to that final step can be generated once and then saved for later. As a result, the second execution of the test is much faster. Also, CI passes only because the CI-runner obtains the proofs from a proof server. The entire trick works because everything is determinstic, so the proofs are valid for identical claims.

[aszepieniec]

@aszepieniec

Come to think of it, it might be useful to have a helper function that takes a Strategy and returns a deterministic sample. In fact, given that prop_map and prop_compose exist, we probably should not be passing test runners as function arguments anywhere.

I was considering to add a macros for this, but I felt that such a facility will hinder further switching to the actual proptest! -- I mean just visually this methods calls chain communicates that it'd be nice to deal with it. And I supposed after the current issue is done there should be some work converting appropriate tests into proptest!. Tell me if that not a thing, and macros is proffered.

Macros are okay as long as they are well documented. (Sometimes it can be tricky to discern how they work.)

[skaunov]

@aszepieniec
I committed a small example of the corrected approach, and will be moving in this vein. Again I'd appreciate early notes and questions!

[skaunov]

Be careful to avoid changing tests that involve producing proofs. Because producing proofs is expensive and we would like to avoid doing it unless absolutely necessary. (This is why we derandomized so many of them.) Mind you, we will generate new proofs if there is a compelling reason to do so.

For instance, in single_proof.rs there is a test called disallow_set_merge_bit_in_pc_path. It is a negative test that verifies that if you try to cheat at the end of a sequence of proofs, you fail. The test is entirely derandomized so that all the proofs leading up to that final step can be generated once and then saved for later. As a result, the second execution of the test is much faster. Also, CI passes only because the CI-runner obtains the proofs from a proof server. The entire trick works because everything is determinstic, so the proofs are valid for identical claims.

@aszepieniec
I just understood that I'm using TestRunner::deterministic() all over the place just because I didn't feel that thing and was taking the safe side relying on a reference. So for random test a default TestRunner should be ok? (It would be actually better because of more fuzzing.)

[skaunov]

@aszepieniec
I just added a commit where a proof generating test is impacted. A helper it uses along with other random tests was modified to accommodate both, so the expensive test received TestRunner::deterministic().
Is it ok to make such changes and regenerate the test proofs then. I mean does this qualify as compelling enough?

[aszepieniec]

(Side note: not all your tags are coming through. Feel free to tag me again if some time passes without sign of life from me.)

So for random test a default TestRunner should be ok?

Yes. Although if the object in question is not involved in any deterministic tests then it's preferable to use whatever shortcuts proptest or test_strategy provides to bypass the creation of a TestRunner in the first place.

I just added a commit where a proof generating test is impacted. A helper it uses along with other random tests was modified to accommodate both, so the expensive test received TestRunner::deterministic().
Is it ok to make such changes and regenerate the test proofs then. I mean does this qualify as compelling enough?

If I understand correctly (please correct any misunderstandings):

1. state_with_premine_and_self_mined_blocks.

• It used to take a &mut rng which is unfriendly to proptests.
• It now takes a [u8;32] which is friendly to both Rngs and proptests.

2. get_dummy_handshake_data_for_genesis.

• It previously used rand::random().
• Now it creates a new randomized TestRunner and samples a number from there.

3. mock_item_and_randomnesses.

• Previously it sampled randomness randomly.
• Now it no longer exists.

I would say (1) is good in principle: it makes the test helper usable in both contexts. But is it actually used on both contexts (or is that the plan)? If the answer is yes then let's go ahead with this change and regenerate proofs as necessary.

As for (2), the fact that it was not derandomizable suggests that it was not ever used in deterministic tests. So there is no need to worry about invalidating deterministic proofs because there are none. However, I might be missing something but I don't see how this change makes the function in question more friendly towards proptest. If the plan is to use this function in proptests -- I endorse this plan but I don't think it is currently the case -- then using rand::random() or sampling a new random tree and drawing the integer from there makes no difference for the surrounding proptest and both prohibit shrinking and derandomization. If for some test we decide we don't care about shrinking and rerandomization then we should not bother with proptest to begin with. Conversely, if we do use proptest then we had better make sure it is compatible with shrinking and derandomization -- and in this case, the function in question probably should return a Strategy<[..]> instead. So with an eye on those principles, this change is pointless.

I think (3) suffers from largely the same problem. Replacing rand::random() with [..] .new_tree(&mut TestRunner::default()).unwrap().current() is pointless. Either we make it into a Strategy or we keep it as is.

There is more to be said about (3), namely that I think this example (not your proposed changes but the thing being changed) is emblematic of a more profound deficiency, which I hope can be resolved (or at least identified) simultaneously with moving reliance more on proptest-related methods and less on Rng. The deficiency is that often times we wrote random generators (or arbitrary implementations) for structs or datatypes on a per-test basis, without understanding and factoring out common patterns. In the case of mock_item_and_randomnesses, the triple of Digests it returns ("item", "sender_randomness", "receiver_preimage") is actually strong pattern throughout the entire code base. If we had known then what we know now, we would probably have written a dedicated struct for this triple, and saved a lot of work. And accordingly, there would have been a proper Arbitrary implementation for this struct, not a function that samples a random triple of Digests 🤦‍♂️ In relation to this particular example I created Issue #546, but please make issues when you encounter similar deficiencies.

[aszepieniec]

Fabulous work, well done! By and large, this is a big step in what is absolutely the right direction.

I am requesting a few changes, but obliging them should take a negligible effort compared to what you've done already. Some change requests live inline. What follows here is general stuff.

I question the wisdom of including proptest failing instances for tests that were dramatically refactored since encountering that failure. If it's a small change, then the failing instances can catch regressions (which is good!) but if it's a dramatic change then catching regressions is unlikely and I wonder whether it makes sense to a) pollute the git environment, and b) reduce the speed for running the test suite.

You marked with #proofs_presaved_compat lines that look weird until you realize you want to reuse deterministic proofs. Thanks for the consideration, but it looks like you've already invalidated the existing proofs with changes further upstream in test_shared/. Going forward I suggest to fix these lines in the most natural way and I will find a way to re-generate all necessary proofs.

Some proptests use the proptest! macro, which takes some deciphering. I wonder if they can all be re-written to use the #[proptest] macro instead? If yes, that's preferable because it's a) consistent with other tests that are already present, and b) (IMHO) easier to read. But if not, it's okay to leave as-is.

There are a few proptest::prop_compose! functions. Please find a consistent naming pattern.

You marked with #arbitraryHashSetIterator lines that are equivalent obtaining an arbitrary HashSet from the proptest/test_strategy framework. Besides one comment I left inline about derandomizing it, I would suggest to factor out a helper struct and implement an appropriate arbitrary strategy for it.

Tests which require no structure in their input and intake only crypto randomness have no chance to have an advantage switching to proptest as it kind of searching an input space which feels pointless in such a case but bring overhead. Boundary tests seems to be a better thing (like trying identity elements, order, prime, maximal).

Agreed.

[aszepieniec]

is it ok to rely on the arbitrary nature of .iter() for HashSet? #arbitraryHashSetIterator

That's a great question, and sadly I think the answer is "no".

A LLM tells me:

the randomness [for HashSet] is not deterministic by default. Each time you run your program, the HashSet will use a new random seed. This means that the internal order of elements (as seen when iterating) and the hash values themselves will differ between runs, even if the same data is inserted in the same order.

which is great if we want to try a whole bunch of independently drawn samples from the same distribution. But that would undermine the other reason to use proptest, which is shrinking when inputs fail. To enable that, you need determinism.

You can use HashSet::with_hasher to derandomize the HashSet. However, I'm not sure how to use that in combination with .collect().

[skaunov]

Yes, it was the line of reasoning I was following too, and counter-arguments were quite weak. Take a look at a solution I found. Feels succinct and elegant, I guess even that additional helper for this would be excessive.

[skaunov]

I question the wisdom of including proptest failing instances for tests that were dramatically refactored since encountering that failure. If it's a small change, then the failing instances can catch regressions (which is good!) but if it's a dramatic change then catching regressions is unlikely and I wonder whether it makes sense to a) pollute the git environment, and b) reduce the speed for running the test suite.

That crossed my mind too, though I feel lack of knowledge on proptest internals to recommend anything on this from my side. I mean that sounds absolutely logical, but I'm not sure. X)

There are a few proptest::prop_compose! functions. Please find a consistent naming pattern.

I was moving to propcompose_ + name of the output type + depending on what's shorter: _of_ listing subjects to fuzzying, or _with_ fields which are fixed as arguments. Any suggestions are welcome!

You marked with #arbitraryHashSetIterator lines that are equivalent obtaining an arbitrary HashSet from the proptest/test_strategy framework. Besides one comment I left inline about derandomizing it, I would suggest to factor out a helper struct and implement an appropriate arbitrary strategy for it.

Yeah it's good for a helper, but the hashtag marks not an equivalent (as that hits the local rejection limit -- see the notes in the first comment for the PR, I put there a draft for the merge commit message).

Some proptests use the proptest! macro, which takes some deciphering. I wonder if they can all be re-written to use the #[proptest] macro instead? If yes, that's preferable because it's a) consistent with other tests that are already present, and b) (IMHO) easier to read. But if not, it's okay to leave as-is.

Will take another look, though it seems proptest! is used reasonably here. I guess you just not used to it yet. It's really like prop_compose! in structure. Also it reports errors better while running. Actually I feel that test_strategy only strength is their async support.

[aszepieniec]

I was moving to propcompose_ + name of the output type + depending on what's shorter: of listing subjects to fuzzying, or with fields which are fixed as arguments. Any suggestions are welcome!

I think with is the natural preposition that comes with compose. However, the thing that follows is not the thing being composed. What is being composed is strategies, and that part is left out. I'm just thinking out loud here.

This is how I might subvocalize things:

Prop-compose (a map of strategies resulting in) CompositeObject with (a strategy for) ArgumentObject.

Based on that I think with is the right choice.

[skaunov]

@aszepieniec
So this should become propcompose_transactionkernel_with_records_pa?

pub fn propcompose_transaction_kernel_with_nums_of_inputs_outputs_pa(
            num_inputs: usize,
            num_outputs: usize,
            num_public_announcements: usize,
        ) (
            inputs in collection::vec(crate::util_types::test_shared::mutator_set::propcompose_removal_record(), num_inputs),
            outputs in collection::vec(arb::<AdditionRecord>(), num_outputs),
            public_announcements in collection::vec(collection::vec(arb::<BFieldElement>(), 10..59), num_public_announcements).prop_map(
                |vecvec| vecvec.into_iter().map(|message| PublicAnnouncement { message }).collect_vec()
            ),
            fee in arb::<NativeCurrencyAmount>(),
            coinbase in arb::<Option<NativeCurrencyAmount>>(),
            timestamp in arb::<Timestamp>(),
            mutator_set_hash in arb::<Digest>(),
            merge_bit in any::<bool>(),
        ) -> TransactionKernel

I mean I was moving to what I described to have something easy to handle (and thank you for helping in this!). And my thought was like: if we will need a same Strategy that also takes a fixed fee or timestamp than it's easy to append those to the name or reverse and name the fields which are fuzzyied by the Strategy. So one would get maximum info from the name of the function and a peek to the output value type.

[aszepieniec]

So this should become propcompose_transactionkernel_with_records_pa?

That doesn't sit well either :/ Ideally there is a struct that holds inputs, outputs, and public announcements, and nothing else. But I think it's silly to introduce that struct in order to improve the names of test helper functions.

Come to think of it, it's not even the inputs, outputs, and public announcements per se -- it's their numbers. So maybe something like propcompose_transaction_kernel_with_lengths works best? I note that none of the other fields on TransactionKernel have a length.

[skaunov]

@aszepieniec
And then there is this.

pub fn propcompose_of_mutator(
            inputs: Vec<RemovalRecord>,
            outputs: Vec<AdditionRecord>,
            fee: NativeCurrencyAmount,
            timestamp: Timestamp,
        ) (mutator_set_hash in arb::<Digest>()) -> TransactionKernel {
            TransactionKernelProxy {
                inputs: inputs.clone(),
                outputs: outputs.clone(),
                public_announcements: vec![],
                fee,
                timestamp,
                coinbase: None,
                mutator_set_hash,
                merge_bit: false,
            }
            .into_kernel()
        }

I omitted the composition result in the name since it's in the vocally named module and transaction_kernel_tests::propcompose_of_mutator works great for me. X) The thing is that everything is fixed here - with or without args - but mutator_set_hash. And I'm really confused about a good name. That's basically how I found a need for a second preposition along "with".

[aszepieniec]

Following the same pattern it would be propcompose_transaction_kernel_with_mutator_set_hash, I think. That sounds good to my ears, and is appropriately descriptive and accurate. If this name is too long, we can use the abbreviation "ms" for mutator set, which is used elsewhere too.

[skaunov]

@aszepieniec Okay, will do this way. Though for me personally that is a confusing approach: in the previous case _lengths_ were fixed via the args, and here it's the thing which is being fuzzed. Also I remembered I was trying to be inline with that *arbitrary_with* method where "with" is again fixing the args from fuzzing. And I lack the feeling of the language which you have since mine English is meh. X)

…

On Fri, Apr 18 2025 at 09:10:19 AM -0700, aszepieniec ***@***.***> wrote: Following the same pattern it would be propcompose_transaction_kernel_with_mutator_set_hash, I think. That sounds good to my ears, and is appropriately descriptive and accurate. If this name is too long, we can use the abbreviation "ms" for mutator set, which is used elsewhere too. — Reply to this email directly, view it on GitHub <#543 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APXLOT7U6DE2LMBNQIGBM2322EPWXAVCNFSM6AAAAAB2LEUFPSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQMJVG4ZDIMJYHE>. You are receiving this because you authored the thread.Message ID: ***@***.***> *aszepieniec* left a comment (Neptune-Crypto/neptune-core#543) <#543 (comment)> Following the same pattern it would be propcompose_transaction_kernel_with_mutator_set_hash, I think. That sounds good to my ears, and is appropriately descriptive and accurate. If this name is too long, we can use the abbreviation "ms" for mutator set, which is used elsewhere too. — Reply to this email directly, view it on GitHub <#543 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APXLOT7U6DE2LMBNQIGBM2322EPWXAVCNFSM6AAAAAB2LEUFPSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQMJVG4ZDIMJYHE>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[aszepieniec]

@aszepieniec Okay, will do this way. Though for me personally that is a confusing approach: in the previous case lengths were fixed via the args, and here it's the thing which is being fuzzed. Also I remembered I was trying to be inline with that arbitrary_with method where "with" is again fixing the args from fuzzing. And I lack the feeling of the language which you have since mine English is meh. X)

Right, that's a great point that I missed. The roles of the argument have switched. So what I proposed is inconsistent. So let's go back to the drawing board.

The question revolves around the arguments/fields

            inputs: Vec<RemovalRecord>,
            outputs: Vec<AdditionRecord>,
            fee: NativeCurrencyAmount,
            timestamp: Timestamp,

which are fields of TransactionKernel but not the only fields -- in particular, TransactionKernel also has coinbase, merge_bit, and of course the thing being fuzzed, mutator_set_hash.

I can't think of a good name for a distinguisher between these two categories. How about "basic kernel info"? I would find it a little confusing that we introduce that term for just this function.

The feature that marks this collection of fields is that all transactions have them and for someone looking at the transaction graph these fields are the interesting ones. (Coinbase is also interesting but a) it has no origin and b) transactions with set coinbase fields are rare.) So in light of that, maybe something along the lines of "standard transaction data".

[aszepieniec]

I left two minor comments inline.

There are still some pseudorandom_* methods left. Are you planning to phase those out in a future PR, or is there a reason why they cannot be made into strategies?

Tomorrow I will have access to a big machine which will be able to run the test suite and generate proofs. Assuming everything is green, I'll be happy to merge.

[aszepieniec]

I think this comment is out of date.

And now that we are discussing comments on this struct anyway, how about adding a comment about the limitations of this strategy? Specifically, the chunk indices are independent of the absolute index set. This feature might cause some tests to fail if they assume these fields are in sync. I realize the original code had the same limitation and I do not think fixing it is within the scope of this PR -- but adding a comment to disclaim the limitation, is in scope ;-)

[skaunov]

There are still some pseudorandom_* methods left. Are you planning to phase those out in a future PR, or is there a reason why they cannot be made into strategies?

There are few ways to formulate the answers which boils down to the very end of #543 (review) line of discussion. I mean I guess they could be converted but what's the point if the decision is to keep the unstructured tests random.

[aszepieniec]

Unfortunately, the test suite is not passing. The following tests are stalling.

      SIGINT [6065.149s] neptune-cash peer_loop::peer_loop_tests::block_without_valid_pow_test
      SIGINT [6065.536s] neptune-cash peer_loop::peer_loop_tests::block_request_batch_in_order_test
      SIGINT [6073.161s] neptune-cash models::blockchain::block::block_tests::block_with_wrong_mmra_is_invalid
      SIGINT [6072.171s] neptune-cash models::peer::tests::sync_challenge_response_pow_witnesses_must_be_a_chain
      SIGINT [6063.567s] neptune-cash peer_loop::peer_loop_tests::test_peer_loop_receival_of_third_block_no_blocks_in_db
      SIGINT [6071.536s] neptune-cash models::state::global_state_tests::mock_global_state_is_valid
      SIGINT [6063.592s] neptune-cash peer_loop::peer_loop_tests::test_peer_loop_receival_of_fourth_block_one_block_in_db
      SIGINT [6072.079s] neptune-cash models::state::archival_state::archival_state_tests::block_hash_witness::stored_block_hash_witness_agrees_with_block_hash
      SIGINT [6073.167s] neptune-cash models::blockchain::block::block_tests::block_is_valid::blocks_with_0_to_10_inputs_and_successors_are_valid
      SIGINT [6065.097s] neptune-cash peer_loop::peer_loop_tests::different_genesis_test
      SIGINT [6063.905s] neptune-cash peer_loop::peer_loop_tests::test_peer_loop_block_with_block_in_db
      SIGINT [6065.314s] neptune-cash peer_loop::peer_loop_tests::block_request_batch_out_of_order_test
      SIGINT [6072.178s] neptune-cash models::peer::transfer_block::test::from_transfer_block
      SIGINT [6065.072s] neptune-cash peer_loop::peer_loop_tests::find_canonical_chain_when_multiple_blocks_at_same_height_test
      SIGINT [6063.596s] neptune-cash peer_loop::peer_loop_tests::test_peer_loop_receival_of_second_block_no_blocks_in_db
      SIGINT [6073.188s] neptune-cash models::blockchain::block::block_tests::block_is_valid::block_with_far_future_timestamp_is_invalid
      SIGINT [6065.570s] neptune-cash peer_loop::peer_loop_tests::block_proposals::accept_block_proposal_height_one
      SIGINT [6064.610s] neptune-cash peer_loop::peer_loop_tests::receive_block_request_by_height_block_7
      SIGINT [6064.766s] neptune-cash peer_loop::peer_loop_tests::receival_of_block_notification_height_1
      SIGINT [6063.924s] neptune-cash peer_loop::peer_loop_tests::test_block_reconciliation_interrupted_by_peer_list_request
      SIGINT [6064.188s] neptune-cash peer_loop::peer_loop_tests::sync_challenges::bad_sync_challenge_height_greater_than_tip
      SIGINT [6064.039s] neptune-cash peer_loop::peer_loop_tests::test_block_reconciliation_interrupted_by_block_notification
      SIGINT [6063.611s] neptune-cash peer_loop::peer_loop_tests::test_peer_loop_receival_of_first_block
      SIGINT [6064.890s] neptune-cash peer_loop::peer_loop_tests::prevent_ram_exhaustion_test
      SIGINT [6065.570s] neptune-cash peer_loop::peer_loop_tests::block_proposals::accept_block_proposal_notification_height_one
      SIGINT [6065.275s] neptune-cash peer_loop::peer_loop_tests::block_request_batch_simple
      SIGINT [6064.163s] neptune-cash peer_loop::peer_loop_tests::sync_challenges::sync_challenge_happy_path

[skaunov]

I took the random one (`block_with_wrong_mmra_is_invalid`) to have a look, and I see that it plainly wasn't affected by this PR.

…

On Wed, Apr 30 2025 at 08:36:08 AM -0700, aszepieniec ***@***.***> wrote: *aszepieniec* left a comment (Neptune-Crypto/neptune-core#543) <#543 (comment)> Unfortunately, the test suite is not passing. The following tests are stalling. SIGINT [6065.149s] neptune-cash peer_loop::peer_loop_tests::block_without_valid_pow_test SIGINT [6065.536s] neptune-cash peer_loop::peer_loop_tests::block_request_batch_in_order_test SIGINT [6073.161s] neptune-cash models::blockchain::block::block_tests::block_with_wrong_mmra_is_invalid SIGINT [6072.171s] neptune-cash models::peer::tests::sync_challenge_response_pow_witnesses_must_be_a_chain SIGINT [6063.567s] neptune-cash peer_loop::peer_loop_tests::test_peer_loop_receival_of_third_block_no_blocks_in_db SIGINT [6071.536s] neptune-cash models::state::global_state_tests::mock_global_state_is_valid SIGINT [6063.592s] neptune-cash peer_loop::peer_loop_tests::test_peer_loop_receival_of_fourth_block_one_block_in_db SIGINT [6072.079s] neptune-cash models::state::archival_state::archival_state_tests::block_hash_witness::stored_block_hash_witness_agrees_with_block_hash SIGINT [6073.167s] neptune-cash models::blockchain::block::block_tests::block_is_valid::blocks_with_0_to_10_inputs_and_successors_are_valid SIGINT [6065.097s] neptune-cash peer_loop::peer_loop_tests::different_genesis_test SIGINT [6063.905s] neptune-cash peer_loop::peer_loop_tests::test_peer_loop_block_with_block_in_db SIGINT [6065.314s] neptune-cash peer_loop::peer_loop_tests::block_request_batch_out_of_order_test SIGINT [6072.178s] neptune-cash models::peer::transfer_block::test::from_transfer_block SIGINT [6065.072s] neptune-cash peer_loop::peer_loop_tests::find_canonical_chain_when_multiple_blocks_at_same_height_test SIGINT [6063.596s] neptune-cash peer_loop::peer_loop_tests::test_peer_loop_receival_of_second_block_no_blocks_in_db SIGINT [6073.188s] neptune-cash models::blockchain::block::block_tests::block_is_valid::block_with_far_future_timestamp_is_invalid SIGINT [6065.570s] neptune-cash peer_loop::peer_loop_tests::block_proposals::accept_block_proposal_height_one SIGINT [6064.610s] neptune-cash peer_loop::peer_loop_tests::receive_block_request_by_height_block_7 SIGINT [6064.766s] neptune-cash peer_loop::peer_loop_tests::receival_of_block_notification_height_1 SIGINT [6063.924s] neptune-cash peer_loop::peer_loop_tests::test_block_reconciliation_interrupted_by_peer_list_request SIGINT [6064.188s] neptune-cash peer_loop::peer_loop_tests::sync_challenges::bad_sync_challenge_height_greater_than_tip SIGINT [6064.039s] neptune-cash peer_loop::peer_loop_tests::test_block_reconciliation_interrupted_by_block_notification SIGINT [6063.611s] neptune-cash peer_loop::peer_loop_tests::test_peer_loop_receival_of_first_block SIGINT [6064.890s] neptune-cash peer_loop::peer_loop_tests::prevent_ram_exhaustion_test SIGINT [6065.570s] neptune-cash peer_loop::peer_loop_tests::block_proposals::accept_block_proposal_notification_height_one SIGINT [6065.275s] neptune-cash peer_loop::peer_loop_tests::block_request_batch_simple SIGINT [6064.163s] neptune-cash peer_loop::peer_loop_tests::sync_challenges::sync_challenge_happy_path — Reply to this email directly, view it on GitHub <#543 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APXLOT4JBCYQKSV3W6TF7DT24DUWRAVCNFSM6AAAAAB2LEUFPSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQNBSGQYDENRQGY>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[aszepieniec]

I took the random one (block_with_wrong_mmra_is_invalid) to have a look, and I see that it plainly wasn't affected by this PR.

Okay. In this case I am profoundly confused by why the test behaves differently on master from on this branch, as observed by my machine:

alan@aravi:~/repos/neptune-core$ git checkout master
Already on 'master'
Your branch is up to date with 'origin/master'.
alan@aravi:~/repos/neptune-core$ cargo nextest run block_with_wrong_mmra_is_invalid
    Finished `test` profile [optimized + debuginfo] target(s) in 0.09s
------------
 Nextest run ID 1454da92-af0c-4ceb-9d03-e1baab253bd2 with nextest profile: default
    Starting 1 test across 6 binaries (662 tests skipped)
        PASS [   0.310s] neptune-cash models::blockchain::block::tests::block_with_wrong_mmra_is_invalid
------------
     Summary [   0.311s] 1 test run: 1 passed, 662 skipped
alan@aravi:~/repos/neptune-core$ git checkout 110-tests-random-phaseout 
branch '110-tests-random-phaseout' set up to track 'skaunov/110-tests-random-phaseout'.
Switched to a new branch '110-tests-random-phaseout'
alan@aravi:~/repos/neptune-core$ cargo nextest run block_with_wrong_mmra_is_invalid
    Blocking waiting for file lock on build directory
   Compiling neptune-cash v0.2.2 (/home/alan/repos/neptune-core)
    Finished `test` profile [optimized + debuginfo] target(s) in 2m 13s
------------
 Nextest run ID cc6017ae-e390-4aa8-8b23-cbf1560d7200 with nextest profile: default
    Starting 1 test across 6 binaries (653 tests skipped)
        SLOW [> 60.000s] neptune-cash models::blockchain::block::block_tests::block_with_wrong_mmra_is_invalid
        SLOW [>120.000s] neptune-cash models::blockchain::block::block_tests::block_with_wrong_mmra_is_invalid
        SLOW [>180.000s] neptune-cash models::blockchain::block::block_tests::block_with_wrong_mmra_is_invalid
        SLOW [>240.000s] neptune-cash models::blockchain::block::block_tests::block_with_wrong_mmra_is_invalid
        SLOW [>300.000s] neptune-cash models::blockchain::block::block_tests::block_with_wrong_mmra_is_invalid
        SLOW [>360.000s] neptune-cash models::blockchain::block::block_tests::block_with_wrong_mmra_is_invalid
     Running [ 00:06:06] [                                                                                                                                                                                    ]   0/1  : 1 running, 0 passed, 115 skippe        SLOW [>420.000s] neptune-cash models::blockchain::block::block_tests::block_with_wrong_mmra_is_invalid
        SLOW [>480.000s] neptune-cash models::blockchain::block::block_tests::block_with_wrong_mmra_is_invalid
        SLOW [>540.000s] neptune-cash models::blockchain::block::block_tests::block_with_wrong_mmra_is_invalid
        SLOW [>600.000s] neptune-cash models::blockchain::block::block_tests::block_with_wrong_mmra_is_invalid
     Running [ 00:10:01] [                                                                                                                                                                              ]   0/1  : 1 running, 0 passed, 115 skipped        Canceling due to interrupt: 1 test still running
      SIGINT [ 601.223s] neptune-cash models::blockchain::block::block_tests::block_with_wrong_mmra_is_invalid
------------
     Summary [ 601.224s] 1 test run: 0 passed, 1 failed, 653 skipped
error: test run failed

Did you observe zero (or rather: within error margin) time disparity for this test between the two branches?

[skaunov]

I'll try that on `master` too after the hackathon. IIRC I always observed tests working better on `master`, but after you noticed that there's cache for test cases I turned my eye blind to the difference. 🤷‍♂️

…

On 2025-05-02 13:02, aszepieniec wrote: aszepieniec left a comment (Neptune-Crypto/neptune-core#543) [1] > I took the random one (block_with_wrong_mmra_is_invalid) to have a > look, and I see that it plainly wasn't affected by this PR. Okay. In this case I am profoundly confused by why the test behaves differently on master from on this branch, as observed by my machine:

[dan-da]

   SLOW [>600.000s] neptune-cash models::blockchain::block::block_tests::block_with_wrong_mmra_is_invalid

At risk of stating the obvious, the test is most likely slow because it does not find a cached proof and is attempting to generate a new proof. So this implies that the code on the branch is doing something different in a way that causes proof inputs (program, claim, nondeterminism) to be different somehow.

[skaunov]

I'll try that on master too

I noticed some of those from your list was quite fast; overall is: "test result: ok. 646 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 197.28s".

[aszepieniec]

Sorry for the miscommunication. I thought the ball was in your court.

I've been running block_with_wrong_mmra_is_invalid for half an hour now on a machine capable of producing the proofs. There does not seem to be any progress and moreover the CPU and memory loads are negligible. This observation indicates that whatever is causing the stalling behavior, it is not proof production.

I'll leave the test running for a few more hours, just to make sure. But so far it looks as though the stall is not caused by absent proofs. So I would say the ball is back in your court not to figure out what is causing it.

If you are running into the problem of absent proofs, you can tell because you will start producing the proofs (or trying to). Unless you are running the test on a number cruncher, that task will eat all your RAM and CPU cores and will probably even be killed by the operating system.

[aszepieniec]

I ran the test suite. All tests pass, except for one: rpc_server::rpc_server_tests::public_announcements_in_block_test.

The behavior of this test is confusing. On the number cruncher that produced the proofs, it fails immediately. On my desktop machine, after lifting the proofs, the test runs seemingly forever and consumes negligible resources while doing so. I suspect that proptest is at play: it found a failing instance on the number cruncher and is running the test on that instance first thing every time. So forgive me for a rather lengthy error message as it includes the failing instance.

    thread 'rpc_server::rpc_server_tests::public_announcements_in_block_test' panicked at src/models/blockchain/block/mod.rs:1032:78:
    called `Option::unwrap()` on a `None` value
    stack backtrace:
       0: __rustc::rust_begin_unwind
                 at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:697:5
       1: core::panicking::panic_fmt
                 at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/panicking.rs:75:14
       2: core::panicking::panic
                 at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/panicking.rs:145:5
       3: core::option::unwrap_failed
                 at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/option.rs:2015:5
       4: core::option::Option<T>::unwrap
                 at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/option.rs:978:21
       5: neptune_cash::models::blockchain::block::Block::guesser_fee_utxos
                 at ./src/models/blockchain/block/mod.rs:1032:30
       6: neptune_cash::models::blockchain::block::Block::guesser_fee_addition_records
                 at ./src/models/blockchain/block/mod.rs:1049:9
       7: neptune_cash::models::blockchain::block::Block::mutator_set_update
                 at ./src/models/blockchain/block/mod.rs:1075:38
       8: neptune_cash::models::state::archival_state::ArchivalState::store_block::{{closure}}
                 at ./src/models/state/archival_state.rs:378:34
       9: neptune_cash::models::state::archival_state::ArchivalState::write_block_internal::{{closure}}
                 at ./src/models/state/archival_state.rs:413:37
      10: neptune_cash::models::state::archival_state::ArchivalState::write_block_as_tip::{{closure}}
                 at ./src/models/state/archival_state.rs:455:52
      11: neptune_cash::models::state::GlobalState::set_new_tip_internal::{{closure}}
                 at ./src/models/state/mod.rs:1414:14
      12: neptune_cash::models::state::GlobalState::set_new_tip::{{closure}}
                 at ./src/models/state/mod.rs:1375:46
      13: neptune_cash::models::state::GlobalStateLock::set_new_tip::{{closure}}
                 at ./src/models/state/mod.rs:248:60
      14: neptune_cash::rpc_server::rpc_server_tests::public_announcements_in_block_test::{{closure}}::{{closure}}
                 at ./src/rpc_server.rs:4331:54
      15: <core::pin::Pin<P> as core::future::future::Future>::poll
                 at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/future/future.rs:124:9
      16: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tracing-0.1.41/src/instrument.rs:321:9
      17: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/park.rs:284:63
      18: tokio::runtime::coop::with_budget
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/coop.rs:107:5
      19: tokio::runtime::coop::budget
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/coop.rs:73:5
      20: tokio::runtime::park::CachedParkThread::block_on
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/park.rs:284:31
      21: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/context/blocking.rs:66:9
      22: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/multi_thread/mod.rs:87:22
      23: tokio::runtime::context::runtime::enter_runtime
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/context/runtime.rs:65:16
      24: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/multi_thread/mod.rs:86:9
      25: tokio::runtime::runtime::Runtime::block_on_inner
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/runtime.rs:370:50
      26: tokio::runtime::runtime::Runtime::block_on
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/runtime.rs:340:13
      27: neptune_cash::rpc_server::rpc_server_tests::public_announcements_in_block_test::{{closure}}
                 at ./src/rpc_server.rs:4313:5
      28: core::ops::function::impls::<impl core::ops::function::Fn<A> for &F>::call
                 at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/ops/function.rs:262:13
      29: proptest::test_runner::runner::call_test::{{closure}}::{{closure}}
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/proptest-1.6.0/src/test_runner/runner.rs:258:56
      30: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
                 at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/panic/unwind_safe.rs:272:9
      31: std::panicking::try::do_call
                 at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:589:40
      32: std::panicking::try
                 at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:552:19
      33: std::panic::catch_unwind
                 at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panic.rs:359:14
      34: proptest::test_runner::runner::call_test::{{closure}}
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/proptest-1.6.0/src/test_runner/runner.rs:258:16
      35: proptest::test_runner::scoped_panic_hook::internal::with_hook
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/proptest-1.6.0/src/test_runner/scoped_panic_hook.rs:130:9
      36: proptest::test_runner::runner::call_test
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/proptest-1.6.0/src/test_runner/runner.rs:256:9
      37: proptest::test_runner::runner::TestRunner::shrink
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/proptest-1.6.0/src/test_runner/runner.rs:855:30
      38: proptest::test_runner::runner::TestRunner::run_one_with_replay
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/proptest-1.6.0/src/test_runner/runner.rs:741:27
      39: proptest::test_runner::runner::TestRunner::gen_and_run_case
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/proptest-1.6.0/src/test_runner/runner.rs:670:23
      40: proptest::test_runner::runner::TestRunner::run_in_process_with_replay
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/proptest-1.6.0/src/test_runner/runner.rs:604:13
      41: proptest::test_runner::runner::TestRunner::run_in_process
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/proptest-1.6.0/src/test_runner/runner.rs:574:9
      42: proptest::test_runner::runner::TestRunner::run
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/proptest-1.6.0/src/test_runner/runner.rs:417:13
      43: neptune_cash::rpc_server::rpc_server_tests::public_announcements_in_block_test
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/proptest-1.6.0/src/sugar.rs:178:17
      44: neptune_cash::rpc_server::rpc_server_tests::public_announcements_in_block_test::{{closure}}
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/proptest-1.6.0/src/sugar.rs:174:28
      45: core::ops::function::FnOnce::call_once
                 at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/ops/function.rs:250:5
      46: core::ops::function::FnOnce::call_once
                 at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/ops/function.rs:250:5
    note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

    thread 'rpc_server::rpc_server_tests::public_announcements_in_block_test' panicked at src/models/blockchain/block/mod.rs:1032:78:
    called `Option::unwrap()` on a `None` value
    stack backtrace:
       0: __rustc::rust_begin_unwind
                 at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:697:5
       1: core::panicking::panic_fmt
                 at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/panicking.rs:75:14
       2: core::panicking::panic
                 at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/panicking.rs:145:5
       3: core::option::unwrap_failed
                 at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/option.rs:2015:5
       4: core::option::Option<T>::unwrap
                 at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/option.rs:978:21
       5: neptune_cash::models::blockchain::block::Block::guesser_fee_utxos
                 at ./src/models/blockchain/block/mod.rs:1032:30
       6: neptune_cash::models::blockchain::block::Block::guesser_fee_addition_records
                 at ./src/models/blockchain/block/mod.rs:1049:9
       7: neptune_cash::models::blockchain::block::Block::mutator_set_update
                 at ./src/models/blockchain/block/mod.rs:1075:38
       8: neptune_cash::models::state::archival_state::ArchivalState::store_block::{{closure}}
                 at ./src/models/state/archival_state.rs:378:34
       9: neptune_cash::models::state::archival_state::ArchivalState::write_block_internal::{{closure}}
                 at ./src/models/state/archival_state.rs:413:37
      10: neptune_cash::models::state::archival_state::ArchivalState::write_block_as_tip::{{closure}}
                 at ./src/models/state/archival_state.rs:455:52
      11: neptune_cash::models::state::GlobalState::set_new_tip_internal::{{closure}}
                 at ./src/models/state/mod.rs:1414:14
      12: neptune_cash::models::state::GlobalState::set_new_tip::{{closure}}
                 at ./src/models/state/mod.rs:1375:46
      13: neptune_cash::models::state::GlobalStateLock::set_new_tip::{{closure}}
                 at ./src/models/state/mod.rs:248:60
      14: neptune_cash::rpc_server::rpc_server_tests::public_announcements_in_block_test::{{closure}}::{{closure}}
                 at ./src/rpc_server.rs:4331:54
      15: <core::pin::Pin<P> as core::future::future::Future>::poll
                 at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/future/future.rs:124:9
      16: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tracing-0.1.41/src/instrument.rs:321:9
      17: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/park.rs:284:63
      18: tokio::runtime::coop::with_budget
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/coop.rs:107:5
      19: tokio::runtime::coop::budget
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/coop.rs:73:5
      20: tokio::runtime::park::CachedParkThread::block_on
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/park.rs:284:31
      21: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/context/blocking.rs:66:9
      22: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/multi_thread/mod.rs:87:22
      23: tokio::runtime::context::runtime::enter_runtime
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/context/runtime.rs:65:16
      24: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/multi_thread/mod.rs:86:9
      25: tokio::runtime::runtime::Runtime::block_on_inner
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/runtime.rs:370:50
      26: tokio::runtime::runtime::Runtime::block_on
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/runtime.rs:340:13
      27: neptune_cash::rpc_server::rpc_server_tests::public_announcements_in_block_test::{{closure}}
                 at ./src/rpc_server.rs:4313:5
      28: core::ops::function::impls::<impl core::ops::function::Fn<A> for &F>::call
                 at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/ops/function.rs:262:13
      29: proptest::test_runner::runner::call_test::{{closure}}::{{closure}}
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/proptest-1.6.0/src/test_runner/runner.rs:258:56
      30: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
                 at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/panic/unwind_safe.rs:272:9
      31: std::panicking::try::do_call
                 at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:589:40
      32: std::panicking::try
                 at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:552:19
      33: std::panic::catch_unwind
                 at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panic.rs:359:14
      34: proptest::test_runner::runner::call_test::{{closure}}
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/proptest-1.6.0/src/test_runner/runner.rs:258:16
      35: proptest::test_runner::scoped_panic_hook::internal::with_hook
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/proptest-1.6.0/src/test_runner/scoped_panic_hook.rs:130:9
      36: proptest::test_runner::runner::call_test
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/proptest-1.6.0/src/test_runner/runner.rs:256:9
      37: proptest::test_runner::runner::TestRunner::shrink
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/proptest-1.6.0/src/test_runner/runner.rs:855:30
      38: proptest::test_runner::runner::TestRunner::run_one_with_replay
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/proptest-1.6.0/src/test_runner/runner.rs:741:27
      39: proptest::test_runner::runner::TestRunner::gen_and_run_case
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/proptest-1.6.0/src/test_runner/runner.rs:670:23
      40: proptest::test_runner::runner::TestRunner::run_in_process_with_replay
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/proptest-1.6.0/src/test_runner/runner.rs:604:13
      41: proptest::test_runner::runner::TestRunner::run_in_process
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/proptest-1.6.0/src/test_runner/runner.rs:574:9
      42: proptest::test_runner::runner::TestRunner::run
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/proptest-1.6.0/src/test_runner/runner.rs:417:13
      43: neptune_cash::rpc_server::rpc_server_tests::public_announcements_in_block_test
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/proptest-1.6.0/src/sugar.rs:178:17
      44: neptune_cash::rpc_server::rpc_server_tests::public_announcements_in_block_test::{{closure}}
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/proptest-1.6.0/src/sugar.rs:174:28
      45: core::ops::function::FnOnce::call_once
                 at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/ops/function.rs:250:5
      46: core::ops::function::FnOnce::call_once
                 at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/ops/function.rs:250:5
    note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
    proptest: Aborting shrinking after the PROPTEST_MAX_SHRINK_ITERS environment variable or ProptestConfig.max_shrink_iters iterations (set 1024 to a large(r) value to shrink more; current configuration: 1024 iterations)

    thread 'rpc_server::rpc_server_tests::public_announcements_in_block_test' panicked at src/rpc_server.rs:4313:5:
    Test failed: called `Option::unwrap()` on a `None` value.
    minimal failing input: input = _PublicAnnouncementsInBlockTestArgs {
        tx_block1: TransactionKernel {
            inputs: [],
            outputs: [
                AdditionRecord {
                    canonical_commitment: Digest(
                        [
                            BFieldElement(
                                0,
                            ),
                            BFieldElement(
                                0,
                            ),
                            BFieldElement(
                                0,
                            ),
                            BFieldElement(
                                0,
                            ),
                            BFieldElement(
                                0,
                            ),
                        ],
                    ),
                },
                AdditionRecord {
                    canonical_commitment: Digest(
                        [
                            BFieldElement(
                                0,
                            ),
                            BFieldElement(
                                0,
                            ),
                            BFieldElement(
                                0,
                            ),
                            BFieldElement(
                                0,
                            ),
                            BFieldElement(
                                0,
                            ),
                        ],
                    ),
                },
            ],
            public_announcements: [
                PublicAnnouncement {
                    message: [
                        BFieldElement(
                            0,
                        ),
                        BFieldElement(
                            9321591543486476998,
                        ),
                        BFieldElement(
                            14573288706066622957,
                        ),
                        BFieldElement(
                            9270886323901199519,
                        ),
                        BFieldElement(
                            6734302117330064099,
                        ),
                        BFieldElement(
                            16315627936601254197,
                        ),
                        BFieldElement(
                            1178352053662224171,
                        ),
                        BFieldElement(
                            8562213646120173239,
                        ),
                        BFieldElement(
                            14793640925920860846,
                        ),
                        BFieldElement(
                            7689510137104745238,
                        ),
                    ],
                },
                PublicAnnouncement {
                    message: [
                        BFieldElement(
                            5850826884696472421,
                        ),
                        BFieldElement(
                            10698350416477277912,
                        ),
                        BFieldElement(
                            13415571233392094868,
                        ),
                        BFieldElement(
                            9369934884966037034,
                        ),
                        BFieldElement(
                            1662925058444927082,
                        ),
                        BFieldElement(
                            7719516251173924890,
                        ),
                        BFieldElement(
                            5373964820120010726,
                        ),
                        BFieldElement(
                            10610712547096075267,
                        ),
                        BFieldElement(
                            11399396361836030022,
                        ),
                        BFieldElement(
                            13510348934276136326,
                        ),
                        BFieldElement(
                            9085767833742476303,
                        ),
                        BFieldElement(
                            5723913007932892529,
                        ),
                        BFieldElement(
                            14231465034676843008,
                        ),
                        BFieldElement(
                            6743161496293035502,
                        ),
                        BFieldElement(
                            2749227069055044627,
                        ),
                        BFieldElement(
                            18036348259610191480,
                        ),
                        BFieldElement(
                            5492289514897661826,
                        ),
                        BFieldElement(
                            1816896152465294170,
                        ),
                        BFieldElement(
                            11544661119781190865,
                        ),
                        BFieldElement(
                            12871712508422676972,
                        ),
                        BFieldElement(
                            5126909686560286199,
                        ),
                        BFieldElement(
                            4106022111572319076,
                        ),
                        BFieldElement(
                            4065245845049566976,
                        ),
                        BFieldElement(
                            8435945931257478577,
                        ),
                        BFieldElement(
                            581043917489909685,
                        ),
                        BFieldElement(
                            15460116745659416903,
                        ),
                        BFieldElement(
                            11595981507090494167,
                        ),
                        BFieldElement(
                            10449461704944053966,
                        ),
                        BFieldElement(
                            2348631313570448087,
                        ),
                        BFieldElement(
                            9927913115758177289,
                        ),
                        BFieldElement(
                            2633819268146899658,
                        ),
                        BFieldElement(
                            10669540681299957084,
                        ),
                        BFieldElement(
                            5013673220773961496,
                        ),
                        BFieldElement(
                            3948526355233602916,
                        ),
                        BFieldElement(
                            998937149810993859,
                        ),
                        BFieldElement(
                            3130813340322589295,
                        ),
                        BFieldElement(
                            16942730675331240903,
                        ),
                        BFieldElement(
                            6739880786431782644,
                        ),
                        BFieldElement(
                            1658810169485282839,
                        ),
                        BFieldElement(
                            4680543269938070822,
                        ),
                        BFieldElement(
                            4635403039071193656,
                        ),
                        BFieldElement(
                            9284534826280127145,
                        ),
                        BFieldElement(
                            9668509141778854314,
                        ),
                        BFieldElement(
                            18154100913628760140,
                        ),
                        BFieldElement(
                            6790729872447145021,
                        ),
                        BFieldElement(
                            11405845389050366412,
                        ),
                        BFieldElement(
                            2148834898663828983,
                        ),
                        BFieldElement(
                            10394265723755212783,
                        ),
                        BFieldElement(
                            6493915292875594243,
                        ),
                        BFieldElement(
                            12365884486086245291,
                        ),
                        BFieldElement(
                            17067169933775367752,
                        ),
                        BFieldElement(
                            5940587059403139278,
                        ),
                        BFieldElement(
                            5126795535245805706,
                        ),
                        BFieldElement(
                            968779048322462424,
                        ),
                        BFieldElement(
                            16406133146074705194,
                        ),
                        BFieldElement(
                            12848833384879852395,
                        ),
                        BFieldElement(
                            17404328651326943881,
                        ),
                    ],
                },
                PublicAnnouncement {
                    message: [
                        BFieldElement(
                            7594660911295056224,
                        ),
                        BFieldElement(
                            9005093803197862104,
                        ),
                        BFieldElement(
                            15025730599736576353,
                        ),
                        BFieldElement(
                            7914242664642492317,
                        ),
                        BFieldElement(
                            12018369970623897564,
                        ),
                        BFieldElement(
                            9872731862978059190,
                        ),
                        BFieldElement(
                            13894353222482954917,
                        ),
                        BFieldElement(
                            5477190326445056532,
                        ),
                        BFieldElement(
                            15805830104178530499,
                        ),
                        BFieldElement(
                            18290222969886878371,
                        ),
                        BFieldElement(
                            5809929793795126022,
                        ),
                        BFieldElement(
                            14375298667674961713,
                        ),
                        BFieldElement(
                            4708049240228224037,
                        ),
                        BFieldElement(
                            7418504464344747364,
                        ),
                        BFieldElement(
                            3933733290989638003,
                        ),
                        BFieldElement(
                            481998146018717127,
                        ),
                        BFieldElement(
                            12284253152957971365,
                        ),
                        BFieldElement(
                            15386012654265221971,
                        ),
                        BFieldElement(
                            13095150001978380178,
                        ),
                        BFieldElement(
                            295382796281590133,
                        ),
                        BFieldElement(
                            8620366749541598725,
                        ),
                        BFieldElement(
                            1304791959912836792,
                        ),
                        BFieldElement(
                            15032448768365563164,
                        ),
                        BFieldElement(
                            5739589016008141687,
                        ),
                        BFieldElement(
                            13327138790541052286,
                        ),
                        BFieldElement(
                            15815768633216926923,
                        ),
                        BFieldElement(
                            6870536856369804543,
                        ),
                        BFieldElement(
                            13847281322079990768,
                        ),
                        BFieldElement(
                            17867324372895401144,
                        ),
                        BFieldElement(
                            7460365984828993082,
                        ),
                        BFieldElement(
                            12571143625909199848,
                        ),
                        BFieldElement(
                            3658343088136045555,
                        ),
                        BFieldElement(
                            8268888862926619393,
                        ),
                        BFieldElement(
                            14224805658164422154,
                        ),
                        BFieldElement(
                            16285394648147870757,
                        ),
                    ],
                },
                PublicAnnouncement {
                    message: [
                        BFieldElement(
                            7439523716026445599,
                        ),
                        BFieldElement(
                            16506073198336684448,
                        ),
                        BFieldElement(
                            8705287737056517607,
                        ),
                        BFieldElement(
                            8231651265310199470,
                        ),
                        BFieldElement(
                            8882080563348816394,
                        ),
                        BFieldElement(
                            10981566394564037466,
                        ),
                        BFieldElement(
                            9944193038408968910,
                        ),
                        BFieldElement(
                            2673945369351400012,
                        ),
                        BFieldElement(
                            18410912251055652393,
                        ),
                        BFieldElement(
                            13529496378182984936,
                        ),
                        BFieldElement(
                            17507505312805448836,
                        ),
                        BFieldElement(
                            17093662154025067297,
                        ),
                        BFieldElement(
                            13577695661611463010,
                        ),
                        BFieldElement(
                            14027539052546300427,
                        ),
                        BFieldElement(
                            14320401750441668324,
                        ),
                        BFieldElement(
                            4642063197896523367,
                        ),
                        BFieldElement(
                            11435512758271904425,
                        ),
                        BFieldElement(
                            3594000353401027283,
                        ),
                        BFieldElement(
                            12948518030387709580,
                        ),
                        BFieldElement(
                            13872249565484404042,
                        ),
                        BFieldElement(
                            13711630440439798972,
                        ),
                        BFieldElement(
                            3055460397186942638,
                        ),
                        BFieldElement(
                            4471839359132047087,
                        ),
                        BFieldElement(
                            7252155648573431983,
                        ),
                        BFieldElement(
                            15488590433812695552,
                        ),
                        BFieldElement(
                            3073337756105039874,
                        ),
                        BFieldElement(
                            17524778332946899191,
                        ),
                        BFieldElement(
                            2992524196386455437,
                        ),
                        BFieldElement(
                            2680138085677036979,
                        ),
                        BFieldElement(
                            4833235556042958219,
                        ),
                        BFieldElement(
                            12699725177376044034,
                        ),
                        BFieldElement(
                            10023777893194050836,
                        ),
                        BFieldElement(
                            15269608515504270087,
                        ),
                        BFieldElement(
                            1068170828386830562,
                        ),
                        BFieldElement(
                            4028999331936546181,
                        ),
                        BFieldElement(
                            5058503403519597591,
                        ),
                        BFieldElement(
                            18155019592594960458,
                        ),
                        BFieldElement(
                            4673416853673096741,
                        ),
                        BFieldElement(
                            18193416338625244637,
                        ),
                        BFieldElement(
                            15033419402450607093,
                        ),
                        BFieldElement(
                            2820245543402743989,
                        ),
                        BFieldElement(
                            1287191723978980010,
                        ),
                        BFieldElement(
                            3163286325826709282,
                        ),
                        BFieldElement(
                            12820804990350095894,
                        ),
                        BFieldElement(
                            5974568893285562216,
                        ),
                        BFieldElement(
                            5229511236206493934,
                        ),
                        BFieldElement(
                            7531764728822743164,
                        ),
                        BFieldElement(
                            13540153938888160563,
                        ),
                        BFieldElement(
                            16591155923905484307,
                        ),
                        BFieldElement(
                            15548338335182666774,
                        ),
                        BFieldElement(
                            2907711232935622352,
                        ),
                        BFieldElement(
                            13364481892358474264,
                        ),
                        BFieldElement(
                            3324677873674596074,
                        ),
                        BFieldElement(
                            7924880984895465917,
                        ),
                        BFieldElement(
                            15828564237640882969,
                        ),
                    ],
                },
                PublicAnnouncement {
                    message: [
                        BFieldElement(
                            3165922451199185807,
                        ),
                        BFieldElement(
                            9189718261240953031,
                        ),
                        BFieldElement(
                            17776659556933552908,
                        ),
                        BFieldElement(
                            1311034704645877024,
                        ),
                        BFieldElement(
                            5202296038206071929,
                        ),
                        BFieldElement(
                            7411088882245642516,
                        ),
                        BFieldElement(
                            14558620641619267447,
                        ),
                        BFieldElement(
                            10675027321315435128,
                        ),
                        BFieldElement(
                            598665310789595969,
                        ),
                        BFieldElement(
                            7819087183872051628,
                        ),
                        BFieldElement(
                            12056578678418623332,
                        ),
                        BFieldElement(
                            6486259572712224753,
                        ),
                        BFieldElement(
                            826095746329047719,
                        ),
                    ],
                },
                PublicAnnouncement {
                    message: [
                        BFieldElement(
                            10724539382916705974,
                        ),
                        BFieldElement(
                            1303220643318753185,
                        ),
                        BFieldElement(
                            9473115488117201169,
                        ),
                        BFieldElement(
                            18414856417842107025,
                        ),
                        BFieldElement(
                            13025091628248206563,
                        ),
                        BFieldElement(
                            17528284268534257198,
                        ),
                        BFieldElement(
                            5429883350585155873,
                        ),
                        BFieldElement(
                            556130114172974692,
                        ),
                        BFieldElement(
                            961897794049784841,
                        ),
                        BFieldElement(
                            8967399253274453237,
                        ),
                        BFieldElement(
                            8717640694119776425,
                        ),
                        BFieldElement(
                            3407239960751345411,
                        ),
                        BFieldElement(
                            11910891741869293262,
                        ),
                        BFieldElement(
                            3089303601062184614,
                        ),
                        BFieldElement(
                            6412325466307238772,
                        ),
                        BFieldElement(
                            13520523155363096809,
                        ),
                        BFieldElement(
                            13210311521673699752,
                        ),
                        BFieldElement(
                            14313262878921039267,
                        ),
                        BFieldElement(
                            16375763654579367188,
                        ),
                        BFieldElement(
                            10723868226945010993,
                        ),
                        BFieldElement(
                            17899760067805144176,
                        ),
                        BFieldElement(
                            6796570027146899127,
                        ),
                        BFieldElement(
                            9843358327620200645,
                        ),
                        BFieldElement(
                            10878971082324431167,
                        ),
                        BFieldElement(
                            17633750723062498615,
                        ),
                        BFieldElement(
                            5423225153423790055,
                        ),
                    ],
                },
                PublicAnnouncement {
                    message: [
                        BFieldElement(
                            15063874826012908280,
                        ),
                        BFieldElement(
                            11794845301655914544,
                        ),
                        BFieldElement(
                            68103292217826961,
                        ),
                        BFieldElement(
                            7664254457830799794,
                        ),
                        BFieldElement(
                            9752717190870353867,
                        ),
                        BFieldElement(
                            7707108688065508978,
                        ),
                        BFieldElement(
                            11616340567884692049,
                        ),
                        BFieldElement(
                            8311508965875839190,
                        ),
                        BFieldElement(
                            4761161030291574899,
                        ),
                        BFieldElement(
                            1008348757178514264,
                        ),
                        BFieldElement(
                            9372206087471090780,
                        ),
                        BFieldElement(
                            2077983553569022931,
                        ),
                        BFieldElement(
                            7138630344912605983,
                        ),
                        BFieldElement(
                            8851221038629380645,
                        ),
                        BFieldElement(
                            13851104793546852152,
                        ),
                        BFieldElement(
                            12508746294651885015,
                        ),
                        BFieldElement(
                            3522013474798245655,
                        ),
                        BFieldElement(
                            12931322237949325250,
                        ),
                        BFieldElement(
                            10615197952485630066,
                        ),
                        BFieldElement(
                            14073208853836752003,
                        ),
                        BFieldElement(
                            17723075682634902071,
                        ),
                        BFieldElement(
                            5864566181164094734,
                        ),
                    ],
                },
            ],
            fee: NativeCurrencyAmount(
                -80189601208177427720452833441052689,
            ),
            coinbase: None,
            timestamp: Timestamp(
                BFieldElement(
                    81953083721190621,
                ),
            ),
            mutator_set_hash: Digest(
                [
                    BFieldElement(
                        1209579961828006965,
                    ),
                    BFieldElement(
                        17845347245891483320,
                    ),
                    BFieldElement(
                        8689404300665274380,
                    ),
                    BFieldElement(
                        12557353202409237661,
                    ),
                    BFieldElement(
                        10895426204171946523,
                    ),
                ],
            ),
            merge_bit: true,
            mast_sequences: OnceLock(
                <uninit>,
            ),
        },
    }
    	successes: 0
    	local rejects: 0
    	global rejects: 0

    stack backtrace:
       0: __rustc::rust_begin_unwind
                 at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:697:5
       1: core::panicking::panic_fmt
                 at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/panicking.rs:75:14
       2: neptune_cash::rpc_server::rpc_server_tests::public_announcements_in_block_test
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/proptest-1.6.0/src/sugar.rs:178:17
       3: neptune_cash::rpc_server::rpc_server_tests::public_announcements_in_block_test::{{closure}}
                 at /home/alan/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/proptest-1.6.0/src/sugar.rs:174:28
       4: core::ops::function::FnOnce::call_once
                 at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/ops/function.rs:250:5
       5: core::ops::function::FnOnce::call_once
                 at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/ops/function.rs:250:5
    note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

  Cancelling due to test failure

I used the opportunity to copy the proofs to one of the proof servers which serves them now. If all is well you should be able to query them and that should happen automatically under the hood when you run tests that require proofs. If proofs are needed after the point where the failure occurs then those have not been generated yet, but as it is a proptest I don't think there are any. So in summary, you should have access to all proofs you need now. (And if you don't -- please poke me.)

[skaunov]

@aszepieniec

(It's absolutely cool GH allow these long texts, but I suggest to move the output to a pastebin just for the scrolling comfort. Otherwise no problem with that.)

So, after the quick look the only Strategy this test is founded upon (propcompose_txkernel_with_lengths) produce --among the many arb() values -- fee in arb::<NativeCurrencyAmount>() which produce all the negative value, and which is then .unwrap() by let value_unlocked = total_guesser_reward.checked_sub(&value_locked).unwrap(); @ Block::guesser_fee_utxos. Which is kinda ironic in light of the discussion in my other PR. X)

I wonder what would you think of adding a type like NonNegativeNativeCurrencyAmount to leverage the type checking system and prevent the errors mentioned in the docs when a checked method is forgotten, by baking that into the types? Btw, is it possible to peer such an announcement? (I guess it's not economically viable, but have too limited outlook on the system to guess if that will be dropped on an earlier stage.)

[aszepieniec]

(It's absolutely cool GH allow these long texts, but I suggest to move the output to a pastebin just for the scrolling comfort. Otherwise no problem with that.)

Noted for next time.

The root cause of the issue is that Block::total_guesser_reward is undefined for un-validated blocks -- and indeed, if the fee happens to be negative, proceeding with this inconsistent return value will cause a panic somewhere. We (= @Sword-Smith and I) fixed this problem in https://github.com/Neptune-Crypto/neptune-core/tree/asz/110-tests-random-phaseout by modifying the return type to Result wrapper around what was there before. Unfortunately, this change has a lot of percolating effects elsewhere.

After that refactor, we modified the test to filter out blocks with negative transaction fees. The test in question passes now, along with all the other ones.

We also tried to merge into master but the resulting merge conflict led to head-scratching and guesswork and we think it's better if we pass the ball back to you because you have context readily at hand that we don't. So -- please rebase on top of master and if everything is still green then we'll merge right away.

Sorry for the very drawn-out review process. If it's any consolation, we are very happy with this PR.

[aszepieniec]

Can confirm that all tests pass. So I merged. Thank you for your contribution!

There is of course the decoration #[ignore = "TODO remove this when handled separately"] on top of test public_announcements_in_block_test, which cleanly separates this PR from the issue addressed by the top two commits on branch https://github.com/Neptune-Crypto/neptune-core/tree/asz/110-tests-random-phaseout. Rebasing those commits on top of now master falls on my and @Sword-Smith's plate. Tomorrow, probably.

[skaunov]

🎉
I would address the test actually to avoid filtering as we discussed today. Honestly, I feel this split into two merges is still much easier than packing all into one, as those additions to shared tests already managed to confuse me in the tracks.