mpt: shorten, generalize proofs (API and wire format change) - #79
Merged
Conversation
As suggested in rsc/mpt#21 and FiloSottile#75, shorten proofs to remove the mptproof prefix, which costs 8 bytes per proof for little benefit. In fact, if we are going to strip non-essential information, then the single-byte prefix at the start of the proof is also unnecessary, so this commit removes that too. While we are changing the proof format, also use a varint-encoded length prefix ahead of keys and values, so that the wire format of proofs applies to any size keys and values, even though this API still assumes [32]byte. (Followup work will revise the package API to allow variable-size keys and values.) The handling of variable-length keys is slightly subtle since we have to pad keys of different lengths to make them comparable for insertion in the tree, but we want to avoid introducing any ambiguity where two different keys pad to the same bit sequence. The answer is to pad with a sequence that always differs from itself when not exactly aligned (so not all 0s or all 1s or any other purely repeating sequence). Working within that constraint, the chosen key padding is a 0x00 byte followed by as many 0xFF bytes as needed. The 0x00 ensures that NUL-free text keys sort in the usual order in the tree, while the 0xFFs that follow ensure that keys of other lengths (even keys ending in 0x00 or 0xFF) will not have the same padding at the same positions. The previous Prove and Verify signatures were: Prove(key Key) (proof Proof, err error) Verify(snap Snapshot, key Key, proof Proof) (val Val, ok bool, err error) Now they are: Prove(key Key) (val Val, ok bool, proof Proof, err error) Verify(snap Snapshot, key Key, val Val, ok bool, proof Proof) error The invariant maintained is that Verify is passed the snapshot plus all the results of Prove. The difference is that now the val, ok are returned by Prove instead of Verify. Of course, they should not be trusted until Prove has succeeded. This commit also adds VerifyPresent and VerifyNotPresent helpers wrapping Verify, which can improve clarity at some call sites. This commit also adds test vectors for Verify in testdata/verify.txt, for easier use by other implementations.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
As suggested in rsc/tmp#21 and #75, shorten proofs to remove the mptproof prefix, which costs 8 bytes per proof for little benefit. In fact, if we are going to strip non-essential information, then the single-byte prefix at the start of the proof is also unnecessary, so this commit removes that too.
While we are changing the proof format, also use a varint-encoded length prefix ahead of keys and values, so that the wire format of proofs applies to any size keys and values, even though this API still assumes [32]byte. (Followup work will revise the package API to allow variable-size keys and values.)
The handling of variable-length keys is slightly subtle since we have to pad keys of different lengths to make them comparable for insertion in the tree, but we want to avoid introducing any ambiguity where two different keys pad to the same bit sequence. The answer is to pad with a sequence that always differs from itself when not exactly aligned (so not all 0s or all 1s or any other purely repeating sequence). Working within that constraint, the chosen key padding is a 0x00 byte followed by as many 0xFF bytes as needed. The 0x00 ensures that NUL-free text keys sort in the usual order in the tree, while the 0xFFs that follow ensure that keys of other lengths (even keys ending in 0x00 or 0xFF) will not have the same padding at the same positions.
The previous Prove and Verify signatures were:
Now they are:
The invariant maintained is that Verify is passed the snapshot plus all the results of Prove. The difference is that now the val, ok are returned by Prove instead of Verify. Of course, they should not be trusted until Prove has succeeded.
This commit also adds VerifyPresent and VerifyNotPresent helpers wrapping Verify, which can improve clarity at some call sites.
This commit also adds test vectors for Verify in testdata/verify.txt, for easier use by other implementations.