mpt: shorten, generalize proofs (API and wire format change) (#79)

As suggested in rsc/tmp#21 and #75, shorten proofs
to remove the mptproof prefix, which costs 8 bytes per proof for little benefit.
In fact, if we are going to strip non-essential information, then the
single-byte prefix at the start of the proof is also unnecessary,
so this commit removes that too.

While we are changing the proof format, also use a varint-encoded length
prefix ahead of keys and values, so that the wire format of proofs applies
to any size keys and values, even though this API still assumes [32]byte.
(Followup work will revise the package API to allow variable-size keys and values.)

The handling of variable-length keys is slightly subtle since we have to
pad keys of different lengths to make them comparable for insertion in
the tree, but we want to avoid introducing any ambiguity where two different
keys pad to the same bit sequence. The answer is to pad with a sequence
that always differs from itself when not exactly aligned (so not all 0s or all 1s
or any other purely repeating sequence). Working within that constraint,
the chosen key padding is a 0x00 byte followed by as many 0xFF bytes as needed.
The 0x00 ensures that NUL-free text keys sort in the usual order in the tree,
while the 0xFFs that follow ensure that keys of other lengths
(even keys ending in 0x00 or 0xFF) will not have the same padding at the same
positions.

The previous Prove and Verify signatures were:

	Prove(key Key) (proof Proof, err error)
	Verify(snap Snapshot, key Key, proof Proof) (val Val, ok bool, err error)

Now they are:

	Prove(key Key) (val Val, ok bool, proof Proof, err error)
	Verify(snap Snapshot, key Key, val Val, ok bool, proof Proof) error

The invariant maintained is that Verify is passed the snapshot
plus all the results of Prove. The difference is that now the val, ok
are returned by Prove instead of Verify. Of course, they should
not be trusted until Prove has succeeded.

This commit also adds VerifyPresent and VerifyNotPresent helpers
wrapping Verify, which can improve clarity at some call sites.

This commit also adds test vectors for Verify in testdata/verify.txt,
for easier use by other implementations.

Author: rsc

mpt/DESIGN.md

@@ -8,7 +8,7 @@ Analogous to a [transparent log](https://research.swtch.com/tlog),
 an MPT can cryptographically prove that a given key-value pair exists
 (or that a key does not exist) in a given tree root.
 By recording the sequence of tree roots in a transparent log,
-a server can publish an record of the history of a key-value database,
+a server can publish a record of the history of a key-value database,
 in such a way that auditors can check that the database was correct
 at all times, and clients can be sure the responses they received
 came from the recorded database history.
@@ -67,80 +67,16 @@ After describing the fully in-memory version, we describe the hybrid version.
 
 ## Merkle Patricia Tree Overview {#mpt}
 
-An MPT starts with the concept of a binary tree of depth 256, where the key-value
-pairs are stored in the leaves at depth 256, and a lookup proceeds by walking
-left or right according to each of the 256 key bits. The root node represents
-the empty key prefix, its children represent key bit prefixes 0 and 1, their
-children represent key bit prefixes 00, 01, 10, 11, and so on: at depth d,
-the nodes represent key prefixes of d bits.
-The original Key Transparency system at Google used exactly this data structure,
-a [Merkle-hashed binary radix tree](https://github.com/google/keytransparency/blob/master/docs/overview.md).
-Since then, the transparency community has realized that it works better
- to apply Merkle hashing to a Patricia tree,
-which adds three optimizations to the binary radix tree.
-
-First, the tree is “path-compressed,” by removing inner nodes with a single child:
-a node that would have pointed at a single-child node is replaced by its child,
-recursively. Every node is therefore either a leaf or an inner node with two children.
-The path compression ensures that there are exactly _N_ inner nodes for a tree with _N_+1 leaf nodes.
-
-Second, unlike in a normal binary tree, an inner node stores only the bit position
-that determines whether a lookup should proceed to the left or right child.
-A lookup walks inner nodes down to some leaf, checking one bit at each step.
-Only upon reaching the leaf does it do a full key comparison.
-If it takes _O_(_K_) time to compare two keys, a normal binary tree would
-take _O_(_K_ log _N_) time for a walk; this optimization
-cuts the time to _O_(_K_ + log _N_).
-Furthermore, inner nodes need not store associated keys,
-cutting the number of stored keys by a factor of two.
-
-Third, nodes are “joined” by merging one inner node and one leaf node into
-a single stored node.
-(After joining _N_ inner nodes to _N_ leaf nodes, that leaves one “leaf-only”
-node not paired to an inner node,
-but the node is still stored using the joined representation.)
-Whether a stored node represents an inner node or leaf node depends
-on how it is reached
-while walking the tree.
-This trick is not essential, but it simplifies storage management
-to have only one type of stored node.
-
-The path-compression optimization implies that an inner node for key prefix _p_ exists
-if and only if the tree contains at least one key with prefix _p_0 and at least one key with prefix _p_1.
-That is, the specific inner nodes that exist in a Patricia depend only on which
-keys are present in the tree, not on their insertion order.
-This implies that we can batch or otherwise reorder insertions of distinct keys
-without affecting the final tree structure.
-
-For more about standard Patricia trees, see TODO REFERENCE.
-
-Cominbing the Merkle and Patricia pieces, a Merkle Patricia Tree provides the following operations:
+See the doc comment at the top of tree.go for details about the MPT data structure and proofs.
+
+A Merkle Patricia Tree provides the following operations:
 
  - Set(key, val): add a new key-value pair to the map.
- - Snap(version): set the tree's version and return the tree root's key prefix and hash.
- - Prove(key): return a proof of the result of looking up a given key in the current snapshot. A separate library function `Verify` verifies a proof and returns the lookup result (whether the key was found and, if so, its associated value).
+ - Snap(version): set the tree's version and return the tree root's hash.
+ - Prove(key): return a lookup result (a value and whether the key was found), along with proof of the result.
+   A separate library function `Verify` verifies the result using the proof.
  - Sync(): flush recent changes to disk.
-
-The recursive hash of an MPT is defined as follows:
-
- - The hash of a leaf node is the hash of its key and value.
- - The hash of an inner node is the hash of its bit position and its left and right children's hashes.
-
-A proof confirming that a key-value pair exists in an MPT with a given recursive hash
-is the value followed by the bit position and sibling hash for every inner node along the path
-back to the root.
-The key-value pair can be hashed to obtain the hash of the leaf node,
-and then the running hash can be hashed with the parent's bit position
-and sibling hash to obtain the hash of the next inner node toward the root.
-(Whether the running hash is the left or right child is determined by checking
-the specified bit of the key.)
-Recomputing the root's actual hash proves the lookup.
-
-A proof denying that a target key exists in an MPT is almost identical.
-It consists of the “other key” whose leaf would be found by looking up the key in the tree,
-followed by the proof that that other key is in the tree.
-The verification checks that the other key's proof is valid and also that
-the target key and other key agree at every relevant bit position.
+ - Predict(keyvals): return the snapshot hash that would result from adding all the keyvals to the map.
 
 An in-memory MPT implementation is in [mem.go](mem.go).
 It was useful to write and debug that version before adding the
@@ -150,112 +86,9 @@ that version first.
 It may also be useful read and understand that implementation
 before proceeding to the disk implementation.
 
-## Encoding Details {#encoding}
-
-Any MPT implementation must define the exact encodings it uses.
-The encodings used by this package are as follows.
-
-### Tree Hashes
-
-An empty tree is a special case that is otherwise independent
-of the tree hash definition. In this implementation,
-the hash of an empty tree is SHA256(_e_), the hash of the empty string (e3b0c442...7852b855).
-
-The hash of a leaf node is the hash of the concatenation of the key and value
-(both fixed-size 32-byte sequences).
-
-The hash of an inner node at bit position _b_ with left and right child hashes _left_ and _right_
-is SHA256(_left_ || _right_ || _b_) where _left_ and _right_ are 32-byte values and _b_ is a one-byte value.
-
-### Proofs
-
-Proofs are variable length strings beginning with the 8-byte sequence `mptproof`.
-
-In Go the verifier's signature is:
-
-	func Verify(snap Snapshot, key Key, proof Proof) (val Val, ok bool, err error)
-
-The verifier is given a tree hash (called a snapshot), a specific key, and a proof,
-and it returns three results: (1) the value associated with the key,
-if the proof proved the existence of the key in the tree,
-(2) whether the proved result confirms or denies the existence of the key,
-and (3) an error if the proof was invalid or did not match the tree hash.
-
-A proof of the empty tree is `mptproof` followed by a 0x00 byte.
-It only applies when the tree hash is the empty tree hash,
-and it disproves the existence of all possible keys.
-
-A proof confirming the existence of a key starts with `mptproof` followed by a 0x01 byte
-and then a 32-byte value _v_.
-The hash of the key's leaf node can be recomputed as _h_ = SHA256(_key_ || _v_).
-If the tree is a single node, that is the entire proof: the verifier must check that
-_h_ = _snap_.
-If the tree contains more than one node, the proof continues with
-one or more descriptions of sibling nodes along the path back to the tree root.
-Each sibling node is encoded as 33 bytes: a one-byte bit position _b_
-followed by a 32-byte sibling hash _sib_.
-The verifier must check whether the _b_'th bit of _key_ is 0 or 1
-and then update the running tree hash accordingly:
-
-   - _h_ = SHA256(_h_ || _sib_ || _b_) if bit _b_ of _key_ is 0, or
-   - _h_ = SHA256(_sib_ || _h_ || _b_) if bit _b_ of _key_ is 1.
-
-Then, as before, the recomputed tree hash _h_ can be
-compared against the actual tree hash.
-
-A proof denying the existence of a key starts with `mptproof` followed by a 0x02 byte
-and then a 32-byte key _k_ and 32-byte value _v_,
-describing a leaf node with hash _h_ = SHA256(_k_ || _v_).
-If the tree is a single node, that is the entire proof: the verifier
-must check that _h_ = _snap_ and that _k_ ≠ _key_.
-Otherwise the proof format contains one or more siblings
-encoded exactly as in the the existence proofs.
-Verification also proceeds as in the existence proofs,
-checking along the way that _k_ and _key_ agree on every bit _b_.
-(Otherwise the proof would not describe the path taken
-to walk through the tree in search of _key_.)
-At the end, the verifier must check that _h_ = _snap_ and that _k_ ≠ _key_.
-
-The worst case length of an existence proof is 8+1+32+33*256 = 8489 bytes,
-although random keys will never produce a path of length 256.
-
-The worst case length of a non-existence proof is 8+1+64+33*255 = 8488 bytes.
-A non-existence proof can only have 255 siblings because otherwise the proof
-would describe a key _k_ that agrees with _key_ at all 256 bit positions,
-but then _k_ ≠ _key_ could not be true.
-Again, random keys will never produce a path length of 256.
-
-Note: This encoding is considerably more compact than some others.
-For example, the Rust akd crate's [MembershipProof](https://docs.rs/akd/0.12.0/akd/struct.MembershipProof.html) is:
-
-	pub struct MembershipProof {
-	    pub label: NodeLabel,
-	    pub hash_val: AzksValue,
-	    pub sibling_proofs: Vec<SiblingProof>,
-	}
-
-[NodeLabel](https://docs.rs/akd/0.12.0/akd/struct.NodeLabel.html) is a key plus a bit length, 32+4 = 36 bytes. \
-[AzksValue](https://docs.rs/akd/0.12.0/akd/struct.AzksValue.html) is 32 bytes. \
-[SiblingProof](https://docs.rs/akd/0.12.0/akd/struct.SiblingProof.html) is:
-
-	pub struct SiblingProof {
-	    pub label: NodeLabel,
-	    pub siblings: [AzksElement; 1],
-	    pub direction: Direction,
-	}
-
-[AzksElement](https://docs.rs/akd/0.12.0/akd/struct.AzksElement.html) is a NodeLabel and AzksValue, 64 bytes. \
-[Direction](https://docs.rs/akd/0.12.0/akd/enum.Direction.html) is a single byte.
-
-So SiblingProof is 36+64+1 = 101 bytes, and the overall worst case MembershipProof, if there are 256 siblings,
-is 36+32+101*256 = 25,924 bytes.
-The largest contributor to the difference is that the siblings include two node labels
-when zero node labels suffice.
-The result is a factor of three in the size of the proofs generated (and sent over the network).
-
 ## Tree Algorithms
 
-There are two potentially important computations MPT hashes
+There are two potentially important computations on MPT hashes
 that can be done without creating an explicit tree representation.
 
 ### Whole Tree Hash

mpt/dmem.go

@@ -4,7 +4,10 @@
 
 package mpt
 
-import "fmt"
+import (
+	"encoding/binary"
+	"fmt"
+)
 
 // hash returns the hash for the given tree node.
 // pbit is the parent bit depth, controlling whether n is viewed as a leaf.
@@ -335,41 +338,43 @@ func (t *diskTree) predict(s []node, a addr, pbit int, list []KeyVal) ([]node, [
 }
 
 // Prove returns a proof of the presence or absence of key in t.
-func (t *diskTree) Prove(key Key) (Proof, error) {
+func (t *diskTree) Prove(key Key) (val Val, ok bool, proof Proof, err error) {
 	t.mmu.RLock()
 	defer t.mmu.RUnlock()
 
 	if t.err != nil {
-		return nil, t.err
+		return Val{}, false, nil, t.err
 	}
 	if t.hdr().dirty() {
-		return nil, ErrModifiedTree
+		return Val{}, false, nil, ErrModifiedTree
 	}
 	root, err := t.node(t.hdr().root())
 	if err != nil {
-		return nil, err
+		return Val{}, false, nil, err
 	}
 	if root == nil {
-		return Proof(proofEmpty), nil
+		return Val{}, false, Proof{}, nil
 	}
 	return root.prove(t, -1, key)
 }
 
-func (n *diskNode) prove(t *diskTree, pbit int, key Key) (Proof, error) {
+func (n *diskNode) prove(t *diskTree, pbit int, key Key) (val Val, ok bool, proof Proof, err error) {
 	nbit := n.bit()
 	if nbit <= pbit {
 		// view n as leaf
 		nkey, nval, err := n.keyVal(t)
 		if err != nil {
-			return nil, err
+			return Val{}, false, nil, err
 		}
-		var p Proof
 		if nkey == key {
-			p = Proof(proofConfirm)
-		} else {
-			p = append(Proof(proofDeny), nkey[:]...)
+			return nval, true, Proof{}, nil
 		}
-		return append(p, nval[:]...), nil
+		var p Proof
+		p = binary.AppendUvarint(p, uint64(len(nkey)))
+		p = append(p, nkey[:]...)
+		p = binary.AppendUvarint(p, uint64(len(nval)))
+		p = append(p, nval[:]...)
+		return Val{}, false, p, nil
 	}
 
 	childAddr, sibAddr := n.left(), n.right()
@@ -378,22 +383,24 @@ func (n *diskNode) prove(t *diskTree, pbit int, key Key) (Proof, error) {
 	}
 	child, err := t.node(childAddr)
 	if err != nil {
-		return nil, err
+		return Val{}, false, nil, err
 	}
 	sib, err := t.node(sibAddr)
 	if err != nil {
-		return nil, err
+		return Val{}, false, nil, err
 	}
 	sibHash, err := sib.hash(t, nbit)
 	if err != nil {
-		return nil, err
+		return Val{}, false, nil, err
 	}
 
-	p, err := child.prove(t, nbit, key)
+	val, ok, proof, err = child.prove(t, nbit, key)
 	if err != nil {
-		return nil, err
+		return
 	}
-	return append(append(p, byte(nbit)), sibHash[:]...), nil
+	proof = binary.AppendUvarint(proof, uint64(nbit))
+	proof = append(proof, sibHash[:]...)
+	return
 }
 
 func (t *diskTree) check() {

mpt/mem.go

@@ -5,6 +5,7 @@
 package mpt
 
 import (
+	"encoding/binary"
 	"errors"
 	"fmt"
 )
@@ -244,30 +245,32 @@ func (t *memTree) predict(s []node, n *memNode, pbit int, list []KeyVal) ([]node
 }
 
 // Prove returns a proof of the presence or absence of key in t.
-func (t *memTree) Prove(key Key) (Proof, error) {
+func (t *memTree) Prove(key Key) (val Val, ok bool, proof Proof, err error) {
 	if t.err != nil {
-		return nil, t.err
+		return Val{}, false, nil, t.err
 	}
 	if t.dirty {
-		return nil, ErrModifiedTree
+		return Val{}, false, nil, ErrModifiedTree
 	}
 	if t.root == nil {
-		return Proof(proofEmpty), nil
+		return Val{}, false, Proof{}, nil
 	}
-	return t.root.prove(-1, key), nil
+	return t.root.prove(-1, key)
 }
 
-func (n *memNode) prove(pbit int, key Key) Proof {
+func (n *memNode) prove(pbit int, key Key) (val Val, ok bool, proof Proof, err error) {
 	nbit := n.bit()
 	if nbit <= pbit {
 		// view n as leaf
-		var p Proof
 		if n.key == key {
-			p = Proof(proofConfirm)
-		} else {
-			p = append(Proof(proofDeny), n.key[:]...)
+			return n.val, true, Proof{}, nil
 		}
-		return append(p, n.val[:]...)
+		var p Proof
+		p = binary.AppendUvarint(p, uint64(len(n.key)))
+		p = append(p, n.key[:]...)
+		p = binary.AppendUvarint(p, uint64(len(n.val)))
+		p = append(p, n.val[:]...)
+		return Val{}, false, p, nil
 	}
 
 	var sib Hash
@@ -279,7 +282,11 @@ func (n *memNode) prove(pbit int, key Key) Proof {
 		child = n.right
 		sib = n.left.hash(nbit)
 	}
-	return append(append(child.prove(nbit, key), byte(nbit)), sib[:]...)
+
+	val, ok, proof, _ = child.prove(nbit, key)
+	proof = binary.AppendUvarint(proof, uint64(nbit))
+	proof = append(proof, sib[:]...)
+	return
 }
 
 // check checks all the tree invariants, walking the entire tree.