Add bit reverse leaf index option

[veljkovranic]

Description

This PR introduces an option to select if your leaves are accessed by regular or a bit-reverse order of the index.
This feature is needed in certain cases, eg. committing to the results of NTT and the reordering of the leaves is too costly to perform before the commitment.

---

• Targeted PR against correct branch (main)
• Linked to Github issue with discussion and accepted design OR have an explanation in the PR that describes this work.
• Wrote unit tests
• Updated relevant documentation in the code
• Added a relevant changelog entry to the Pending section in CHANGELOG.md
• Re-reviewed Files changed in the Github PR explorer

[z-tech]

Hi, if you never reorder the vector v then what prevents you from leaving the MT library as lexicographic and handling the mapping just outside?

t.commit(v);
for i in SOME_ORDER {
   t.get_leaf(to_reverse_bit_index(i));
   or
   t.get_leaf(from_reverse_bit(i));
}

What is the difference between this pseudocode and the PR?

[veljkovranic]

So the data that we are commiting to is a polynomial.
This leaves us with two options:

1. Either reorded the NTT output data, so the rest of the methods stay the same (evaluation at a point, conversion to other coefficient/eval form, folding, etc). But this is costly if the polynomial is huge.
2. Or you treat the data as-is and now your whole codebase needs to support the bit-reverse ordering in all of the methods listed above.

Btw, nice to meet you, @WizardOfMenlo suggested for me to have a chat with you, multiple times, just didn't expect it to happen here 😄

[z-tech]

So you mean that using ordering BIT_REVERSED the leaves in physical memory locations corresponding to 0 and 4 would get hashed into an internal node (rather than 0 and 1) ?

If so, can you write tests that cover this?

[z-tech]

I think I understand the intent now. Personally, I think it would be more adaptable to open a new constructor for MerkleTree that accepts an Iterator over the vector you don't want to reorder. Something along those lines has been done in the EfficientSumcheck repo: https://github.com/compsec-epfl/efficient-sumcheck/blob/main/src/streams/stream_iterator.rs#L12

That way people can create their own orderings without modifying the actual MT Library code. It would also mean no breaking changes for the existing constructor.

What are your thoughts?

[z-tech]

Are theselexicographic and reverse_lexicographic ?

[veljkovranic]

I can close it in this case, since it will be an Iterator either way. The only thing that is important is the next() operation.