diff --git a/mpt/DESIGN.md b/mpt/DESIGN.md index 3848038..0d9215b 100644 --- a/mpt/DESIGN.md +++ b/mpt/DESIGN.md @@ -8,7 +8,7 @@ Analogous to a [transparent log](https://research.swtch.com/tlog), an MPT can cryptographically prove that a given key-value pair exists (or that a key does not exist) in a given tree root. By recording the sequence of tree roots in a transparent log, -a server can publish an record of the history of a key-value database, +a server can publish a record of the history of a key-value database, in such a way that auditors can check that the database was correct at all times, and clients can be sure the responses they received came from the recorded database history. @@ -67,80 +67,16 @@ After describing the fully in-memory version, we describe the hybrid version. ## Merkle Patricia Tree Overview {#mpt} -An MPT starts with the concept of a binary tree of depth 256, where the key-value -pairs are stored in the leaves at depth 256, and a lookup proceeds by walking -left or right according to each of the 256 key bits. The root node represents -the empty key prefix, its children represent key bit prefixes 0 and 1, their -children represent key bit prefixes 00, 01, 10, 11, and so on: at depth d, -the nodes represent key prefixes of d bits. -The original Key Transparency system at Google used exactly this data structure, -a [Merkle-hashed binary radix tree](https://github.com/google/keytransparency/blob/master/docs/overview.md). -Since then, the transparency community has realized that it works better - to apply Merkle hashing to a Patricia tree, -which adds three optimizations to the binary radix tree. - -First, the tree is “path-compressed,” by removing inner nodes with a single child: -a node that would have pointed at a single-child node is replaced by its child, -recursively. Every node is therefore either a leaf or an inner node with two children. -The path compression ensures that there are exactly _N_ inner nodes for a tree with _N_+1 leaf nodes. - -Second, unlike in a normal binary tree, an inner node stores only the bit position -that determines whether a lookup should proceed to the left or right child. -A lookup walks inner nodes down to some leaf, checking one bit at each step. -Only upon reaching the leaf does it do a full key comparison. -If it takes _O_(_K_) time to compare two keys, a normal binary tree would -take _O_(_K_ log _N_) time for a walk; this optimization -cuts the time to _O_(_K_ + log _N_). -Furthermore, inner nodes need not store associated keys, -cutting the number of stored keys by a factor of two. - -Third, nodes are “joined” by merging one inner node and one leaf node into -a single stored node. -(After joining _N_ inner nodes to _N_ leaf nodes, that leaves one “leaf-only” -node not paired to an inner node, -but the node is still stored using the joined representation.) -Whether a stored node represents an inner node or leaf node depends -on how it is reached -while walking the tree. -This trick is not essential, but it simplifies storage management -to have only one type of stored node. - -The path-compression optimization implies that an inner node for key prefix _p_ exists -if and only if the tree contains at least one key with prefix _p_0 and at least one key with prefix _p_1. -That is, the specific inner nodes that exist in a Patricia depend only on which -keys are present in the tree, not on their insertion order. -This implies that we can batch or otherwise reorder insertions of distinct keys -without affecting the final tree structure. - -For more about standard Patricia trees, see TODO REFERENCE. - -Cominbing the Merkle and Patricia pieces, a Merkle Patricia Tree provides the following operations: +See the doc comment at the top of tree.go for details about the MPT data structure and proofs. + +A Merkle Patricia Tree provides the following operations: - Set(key, val): add a new key-value pair to the map. - - Snap(version): set the tree's version and return the tree root's key prefix and hash. - - Prove(key): return a proof of the result of looking up a given key in the current snapshot. A separate library function `Verify` verifies a proof and returns the lookup result (whether the key was found and, if so, its associated value). + - Snap(version): set the tree's version and return the tree root's hash. + - Prove(key): return a lookup result (a value and whether the key was found), along with proof of the result. + A separate library function `Verify` verifies the result using the proof. - Sync(): flush recent changes to disk. - -The recursive hash of an MPT is defined as follows: - - - The hash of a leaf node is the hash of its key and value. - - The hash of an inner node is the hash of its bit position and its left and right children's hashes. - -A proof confirming that a key-value pair exists in an MPT with a given recursive hash -is the value followed by the bit position and sibling hash for every inner node along the path -back to the root. -The key-value pair can be hashed to obtain the hash of the leaf node, -and then the running hash can be hashed with the parent's bit position -and sibling hash to obtain the hash of the next inner node toward the root. -(Whether the running hash is the left or right child is determined by checking -the specified bit of the key.) -Recomputing the root's actual hash proves the lookup. - -A proof denying that a target key exists in an MPT is almost identical. -It consists of the “other key” whose leaf would be found by looking up the key in the tree, -followed by the proof that that other key is in the tree. -The verification checks that the other key's proof is valid and also that -the target key and other key agree at every relevant bit position. + - Predict(keyvals): return the snapshot hash that would result from adding all the keyvals to the map. An in-memory MPT implementation is in [mem.go](mem.go). It was useful to write and debug that version before adding the @@ -150,112 +86,9 @@ that version first. It may also be useful read and understand that implementation before proceeding to the disk implementation. -## Encoding Details {#encoding} - -Any MPT implementation must define the exact encodings it uses. -The encodings used by this package are as follows. - -### Tree Hashes - -An empty tree is a special case that is otherwise independent -of the tree hash definition. In this implementation, -the hash of an empty tree is SHA256(_e_), the hash of the empty string (e3b0c442...7852b855). - -The hash of a leaf node is the hash of the concatenation of the key and value -(both fixed-size 32-byte sequences). - -The hash of an inner node at bit position _b_ with left and right child hashes _left_ and _right_ -is SHA256(_left_ || _right_ || _b_) where _left_ and _right_ are 32-byte values and _b_ is a one-byte value. - -### Proofs - -Proofs are variable length strings beginning with the 8-byte sequence `mptproof`. - -In Go the verifier's signature is: - - func Verify(snap Snapshot, key Key, proof Proof) (val Val, ok bool, err error) - -The verifier is given a tree hash (called a snapshot), a specific key, and a proof, -and it returns three results: (1) the value associated with the key, -if the proof proved the existence of the key in the tree, -(2) whether the proved result confirms or denies the existence of the key, -and (3) an error if the proof was invalid or did not match the tree hash. - -A proof of the empty tree is `mptproof` followed by a 0x00 byte. -It only applies when the tree hash is the empty tree hash, -and it disproves the existence of all possible keys. - -A proof confirming the existence of a key starts with `mptproof` followed by a 0x01 byte -and then a 32-byte value _v_. -The hash of the key's leaf node can be recomputed as _h_ = SHA256(_key_ || _v_). -If the tree is a single node, that is the entire proof: the verifier must check that -_h_ = _snap_. -If the tree contains more than one node, the proof continues with -one or more descriptions of sibling nodes along the path back to the tree root. -Each sibling node is encoded as 33 bytes: a one-byte bit position _b_ -followed by a 32-byte sibling hash _sib_. -The verifier must check whether the _b_'th bit of _key_ is 0 or 1 -and then update the running tree hash accordingly: - - - _h_ = SHA256(_h_ || _sib_ || _b_) if bit _b_ of _key_ is 0, or - - _h_ = SHA256(_sib_ || _h_ || _b_) if bit _b_ of _key_ is 1. - -Then, as before, the recomputed tree hash _h_ can be -compared against the actual tree hash. - -A proof denying the existence of a key starts with `mptproof` followed by a 0x02 byte -and then a 32-byte key _k_ and 32-byte value _v_, -describing a leaf node with hash _h_ = SHA256(_k_ || _v_). -If the tree is a single node, that is the entire proof: the verifier -must check that _h_ = _snap_ and that _k_ ≠ _key_. -Otherwise the proof format contains one or more siblings -encoded exactly as in the the existence proofs. -Verification also proceeds as in the existence proofs, -checking along the way that _k_ and _key_ agree on every bit _b_. -(Otherwise the proof would not describe the path taken -to walk through the tree in search of _key_.) -At the end, the verifier must check that _h_ = _snap_ and that _k_ ≠ _key_. - -The worst case length of an existence proof is 8+1+32+33*256 = 8489 bytes, -although random keys will never produce a path of length 256. - -The worst case length of a non-existence proof is 8+1+64+33*255 = 8488 bytes. -A non-existence proof can only have 255 siblings because otherwise the proof -would describe a key _k_ that agrees with _key_ at all 256 bit positions, -but then _k_ ≠ _key_ could not be true. -Again, random keys will never produce a path length of 256. - -Note: This encoding is considerably more compact than some others. -For example, the Rust akd crate's [MembershipProof](https://docs.rs/akd/0.12.0/akd/struct.MembershipProof.html) is: - - pub struct MembershipProof { - pub label: NodeLabel, - pub hash_val: AzksValue, - pub sibling_proofs: Vec, - } - -[NodeLabel](https://docs.rs/akd/0.12.0/akd/struct.NodeLabel.html) is a key plus a bit length, 32+4 = 36 bytes. \ -[AzksValue](https://docs.rs/akd/0.12.0/akd/struct.AzksValue.html) is 32 bytes. \ -[SiblingProof](https://docs.rs/akd/0.12.0/akd/struct.SiblingProof.html) is: - - pub struct SiblingProof { - pub label: NodeLabel, - pub siblings: [AzksElement; 1], - pub direction: Direction, - } - -[AzksElement](https://docs.rs/akd/0.12.0/akd/struct.AzksElement.html) is a NodeLabel and AzksValue, 64 bytes. \ -[Direction](https://docs.rs/akd/0.12.0/akd/enum.Direction.html) is a single byte. - -So SiblingProof is 36+64+1 = 101 bytes, and the overall worst case MembershipProof, if there are 256 siblings, -is 36+32+101*256 = 25,924 bytes. -The largest contributor to the difference is that the siblings include two node labels -when zero node labels suffice. -The result is a factor of three in the size of the proofs generated (and sent over the network). - ## Tree Algorithms -There are two potentially important computations MPT hashes +There are two potentially important computations on MPT hashes that can be done without creating an explicit tree representation. ### Whole Tree Hash diff --git a/mpt/dmem.go b/mpt/dmem.go index df63cd5..ddeb4a0 100644 --- a/mpt/dmem.go +++ b/mpt/dmem.go @@ -4,7 +4,10 @@ package mpt -import "fmt" +import ( + "encoding/binary" + "fmt" +) // hash returns the hash for the given tree node. // pbit is the parent bit depth, controlling whether n is viewed as a leaf. @@ -335,41 +338,43 @@ func (t *diskTree) predict(s []node, a addr, pbit int, list []KeyVal) ([]node, [ } // Prove returns a proof of the presence or absence of key in t. -func (t *diskTree) Prove(key Key) (Proof, error) { +func (t *diskTree) Prove(key Key) (val Val, ok bool, proof Proof, err error) { t.mmu.RLock() defer t.mmu.RUnlock() if t.err != nil { - return nil, t.err + return Val{}, false, nil, t.err } if t.hdr().dirty() { - return nil, ErrModifiedTree + return Val{}, false, nil, ErrModifiedTree } root, err := t.node(t.hdr().root()) if err != nil { - return nil, err + return Val{}, false, nil, err } if root == nil { - return Proof(proofEmpty), nil + return Val{}, false, Proof{}, nil } return root.prove(t, -1, key) } -func (n *diskNode) prove(t *diskTree, pbit int, key Key) (Proof, error) { +func (n *diskNode) prove(t *diskTree, pbit int, key Key) (val Val, ok bool, proof Proof, err error) { nbit := n.bit() if nbit <= pbit { // view n as leaf nkey, nval, err := n.keyVal(t) if err != nil { - return nil, err + return Val{}, false, nil, err } - var p Proof if nkey == key { - p = Proof(proofConfirm) - } else { - p = append(Proof(proofDeny), nkey[:]...) + return nval, true, Proof{}, nil } - return append(p, nval[:]...), nil + var p Proof + p = binary.AppendUvarint(p, uint64(len(nkey))) + p = append(p, nkey[:]...) + p = binary.AppendUvarint(p, uint64(len(nval))) + p = append(p, nval[:]...) + return Val{}, false, p, nil } childAddr, sibAddr := n.left(), n.right() @@ -378,22 +383,24 @@ func (n *diskNode) prove(t *diskTree, pbit int, key Key) (Proof, error) { } child, err := t.node(childAddr) if err != nil { - return nil, err + return Val{}, false, nil, err } sib, err := t.node(sibAddr) if err != nil { - return nil, err + return Val{}, false, nil, err } sibHash, err := sib.hash(t, nbit) if err != nil { - return nil, err + return Val{}, false, nil, err } - p, err := child.prove(t, nbit, key) + val, ok, proof, err = child.prove(t, nbit, key) if err != nil { - return nil, err + return } - return append(append(p, byte(nbit)), sibHash[:]...), nil + proof = binary.AppendUvarint(proof, uint64(nbit)) + proof = append(proof, sibHash[:]...) + return } func (t *diskTree) check() { diff --git a/mpt/mem.go b/mpt/mem.go index 960356e..56fcd1a 100644 --- a/mpt/mem.go +++ b/mpt/mem.go @@ -5,6 +5,7 @@ package mpt import ( + "encoding/binary" "errors" "fmt" ) @@ -244,30 +245,32 @@ func (t *memTree) predict(s []node, n *memNode, pbit int, list []KeyVal) ([]node } // Prove returns a proof of the presence or absence of key in t. -func (t *memTree) Prove(key Key) (Proof, error) { +func (t *memTree) Prove(key Key) (val Val, ok bool, proof Proof, err error) { if t.err != nil { - return nil, t.err + return Val{}, false, nil, t.err } if t.dirty { - return nil, ErrModifiedTree + return Val{}, false, nil, ErrModifiedTree } if t.root == nil { - return Proof(proofEmpty), nil + return Val{}, false, Proof{}, nil } - return t.root.prove(-1, key), nil + return t.root.prove(-1, key) } -func (n *memNode) prove(pbit int, key Key) Proof { +func (n *memNode) prove(pbit int, key Key) (val Val, ok bool, proof Proof, err error) { nbit := n.bit() if nbit <= pbit { // view n as leaf - var p Proof if n.key == key { - p = Proof(proofConfirm) - } else { - p = append(Proof(proofDeny), n.key[:]...) + return n.val, true, Proof{}, nil } - return append(p, n.val[:]...) + var p Proof + p = binary.AppendUvarint(p, uint64(len(n.key))) + p = append(p, n.key[:]...) + p = binary.AppendUvarint(p, uint64(len(n.val))) + p = append(p, n.val[:]...) + return Val{}, false, p, nil } var sib Hash @@ -279,7 +282,11 @@ func (n *memNode) prove(pbit int, key Key) Proof { child = n.right sib = n.left.hash(nbit) } - return append(append(child.prove(nbit, key), byte(nbit)), sib[:]...) + + val, ok, proof, _ = child.prove(nbit, key) + proof = binary.AppendUvarint(proof, uint64(nbit)) + proof = append(proof, sib[:]...) + return } // check checks all the tree invariants, walking the entire tree. diff --git a/mpt/testdata/gen.go b/mpt/testdata/gen.go new file mode 100644 index 0000000..14cb6bd --- /dev/null +++ b/mpt/testdata/gen.go @@ -0,0 +1,1300 @@ +// Copyright 2025 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +// Gen generates testdata/verify.txt, a file of test vectors for [mpt.Verify]. +// +// Usage: +// +// go run testdata/gen.go > testdata/verify.txt +package main + +import ( + "bytes" + "crypto/sha256" + "encoding/binary" + "encoding/hex" + "fmt" + "os" + "slices" + "strings" + + "filippo.io/torchwood/mpt" +) + +func main() { + var g gen + g.header() + g.emptyTree() + g.singleLeaf() + g.twoLeaves() + g.threeLeaves() + g.corruption() + g.varLenKeys() + g.keyOverlap() + g.emptyKeyVal() + g.proofStructure() + if _, err := os.Stdout.WriteString(g.String()); err != nil { + fmt.Fprintln(os.Stderr, err) + os.Exit(1) + } +} + +// gen accumulates test output. +type gen struct { + buf bytes.Buffer +} + +func (g *gen) String() string { return g.buf.String() } + +// Output methods. + +func (g *gen) printf(format string, args ...any) { + fmt.Fprintf(&g.buf, format, args...) +} + +func (g *gen) comment(lines string) { + for _, line := range strings.Split(lines, "\n") { + if line == "" { + g.printf("#\n") + } else { + g.printf("# %s\n", line) + } + } +} + +func (g *gen) blank() { g.printf("\n") } + +func (g *gen) section(title string) { + g.printf("# === %s ===\n\n", title) +} + +func (g *gen) snap(h mpt.Hash) { + g.printf("snap %s\n", encHex(h[:])) +} + +func (g *gen) key(k []byte) { + if len(k) == 0 { + g.printf("key ''\n") + return + } + g.printf("key %s\n", encHex(k)) +} + +func (g *gen) val(v []byte) { + if len(v) == 0 { + g.printf("val ''\n") + return + } + g.printf("val %s\n", encHex(v)) +} + +func (g *gen) valAbsent() { + g.printf("val -\n") +} + +// proof writes a formatted proof line. +// ok indicates whether this is a presence proof (true) or absence proof (false), +// which affects how the proof bytes are parsed for display formatting. +func (g *gen) proof(p mpt.Proof, ok bool) { + if len(p) == 0 { + g.printf("proof ''\n") + return + } + groups := fmtGroups([]byte(p), ok) + if len(groups) == 1 { + g.printf("proof %s\n", groups[0]) + return + } + for i, gr := range groups { + switch { + case i == 0: + g.printf("proof %s \\\n", gr) + case i < len(groups)-1: + g.printf("\t%s \\\n", gr) + default: + g.printf("\t%s\n", gr) + } + } +} + +// rawProof writes a proof as unformatted hex on one line. +// Use for corrupted proofs that cannot be parsed structurally. +func (g *gen) rawProof(p []byte) { + if len(p) == 0 { + g.printf("proof ''\n") + } else { + g.printf("proof %s\n", encHex(p)) + } +} + +func (g *gen) verify(want bool) { + g.printf("verify %v\n", want) +} + +// encHex formats bytes as a hex string. +func encHex(b []byte) string { + return hex.EncodeToString(b) +} + +// fmtGroups splits raw proof bytes into display groups. +// Each group is a space-separated hex string that should appear on one line. +func fmtGroups(data []byte, ok bool) []string { + var groups []string + if !ok { + // altkey: varint(len) key + keyLen, n := binary.Uvarint(data) + groups = append(groups, encHex(data[:n])+" "+encHex(data[n:n+int(keyLen)])) + data = data[n+int(keyLen):] + // altval: varint(len) val + valLen, n := binary.Uvarint(data) + groups = append(groups, encHex(data[:n])+" "+encHex(data[n:n+int(valLen)])) + data = data[n+int(valLen):] + } + // path steps: varint(bit) + 32-byte hash + for len(data) > 0 { + _, n := binary.Uvarint(data) + groups = append(groups, encHex(data[:n])+" "+encHex(data[n:n+32])) + data = data[n+32:] + } + return groups +} + +// Tree helpers. + +func tkey(name string) mpt.Key { return sha256.Sum256([]byte("key:" + name)) } +func tval(name string) mpt.Val { return sha256.Sum256([]byte("val:" + name)) } + +// fixture is a tree with a snapshot, ready for proving. +type fixture struct { + tree mpt.Tree + snap mpt.Snapshot +} + +func newFixture(names ...string) *fixture { + t := mpt.NewMemTree() + for _, name := range names { + t.Set(tkey(name), tval(name)) + } + v := int64(1) + if len(names) == 0 { + v = 0 + } + snap, err := t.Snap(v) + if err != nil { + panic(err) + } + return &fixture{tree: t, snap: snap} +} + +func (f *fixture) prove(key mpt.Key) (mpt.Val, bool, mpt.Proof) { + v, ok, p, err := f.tree.Prove(key) + if err != nil { + panic(err) + } + return v, ok, p +} + +// Proof mutation helpers. + +func truncated(p mpt.Proof, n int) mpt.Proof { + return slices.Clone(p)[:n] +} + +func flipped(p mpt.Proof, byteIdx int) mpt.Proof { + q := slices.Clone(p) + q[byteIdx] ^= 0x80 + return q +} + +func appended(p mpt.Proof, b byte) mpt.Proof { + return append(slices.Clone(p), b) +} + +// Variable-length key helpers. + +func vhashLeaf(key, val []byte) mpt.Hash { + h := sha256.New() + h.Write([]byte{0}) + var buf [binary.MaxVarintLen64]byte + n := binary.PutUvarint(buf[:], uint64(len(key))) + h.Write(buf[:n]) + h.Write(key) + n = binary.PutUvarint(buf[:], uint64(len(val))) + h.Write(buf[:n]) + h.Write(val) + return mpt.Hash(h.Sum(nil)) +} + +func vhashInner(bit int, left, right mpt.Hash) mpt.Hash { + h := sha256.New() + h.Write([]byte{1}) + var buf [binary.MaxVarintLen64]byte + n := binary.PutUvarint(buf[:], uint64(bit)) + h.Write(buf[:n]) + h.Write(left[:]) + h.Write(right[:]) + return mpt.Hash(h.Sum(nil)) +} + +type pathStep struct { + bit int + hash mpt.Hash +} + +func buildDenyProof(altkey, altval []byte, steps []pathStep) mpt.Proof { + var p mpt.Proof + p = binary.AppendUvarint(p, uint64(len(altkey))) + p = append(p, altkey...) + p = binary.AppendUvarint(p, uint64(len(altval))) + p = append(p, altval...) + for _, s := range steps { + p = binary.AppendUvarint(p, uint64(s.bit)) + p = append(p, s.hash[:]...) + } + return p +} + +func buildConfirmProof(steps []pathStep) mpt.Proof { + var p mpt.Proof + for _, s := range steps { + p = binary.AppendUvarint(p, uint64(s.bit)) + p = append(p, s.hash[:]...) + } + return p +} + +// Test generation methods. + +func (g *gen) header() { + g.comment("Test vectors for mpt.Verify, generated by gen.go.") + g.comment("") + g.comment("Each test sets state with snap, key, val, and proof lines,") + g.comment("then calls verify to check the result. Blank lines and lines") + g.comment("beginning with # are ignored.") + g.comment("") + g.comment("Format:") + g.comment(" snap HEXHASH - set the snapshot hash") + g.comment(" key HEXKEY - set the lookup key") + g.comment(" val HEXVAL - set val and ok=true") + g.comment(" val - - set val to empty and ok=false") + g.comment(" proof HEXPROOF - set the proof (hex bytes, spaces ok)") + g.comment(" proof '' - set the proof to empty (zero-length)") + g.comment(" verify true - Verify should succeed (return nil)") + g.comment(" verify false - Verify should fail (return error)") + g.comment("") + g.comment("Proof lines may be continued with \\ at end of line;") + g.comment("continuation lines conventionally start with a tab.") + g.comment("The hex for snap, key, val, and proof may contain spaces.") + g.comment("") + g.comment("DO NOT EDIT. Generated by:") + g.comment(" go run testdata/gen.go > testdata/verify.txt") + g.blank() +} + +func (g *gen) emptyTree() { + g.section("Empty tree") + + f := newFixture() + k := tkey("missing") + + g.comment("Absent key: empty proof is valid for empty tree.") + g.snap(f.snap.Hash) + g.key(k[:]) + g.valAbsent() + g.proof(mpt.Proof{}, false) + g.verify(true) + g.blank() + + g.comment("Absent key: empty proof is invalid for non-empty snapshot.") + g.snap(sha256.Sum256([]byte("wrong"))) + g.key(k[:]) + g.valAbsent() + g.proof(mpt.Proof{}, false) + g.verify(false) + g.blank() +} + +func (g *gen) singleLeaf() { + g.section("Single-leaf tree") + + f := newFixture("a") + ka, va := tkey("a"), tval("a") + km := tkey("missing") + + g.comment("Present key: valid proof (no path steps for single leaf).") + _, _, pp := f.prove(ka) + g.snap(f.snap.Hash) + g.key(ka[:]) + g.val(va[:]) + g.proof(pp, true) + g.verify(true) + g.blank() + + g.comment("Present key: wrong value, same proof.") + g.snap(f.snap.Hash) + g.key(ka[:]) + wrongVal := tval("wrong") + g.val(wrongVal[:]) + g.proof(pp, true) + g.verify(false) + g.blank() + + g.comment("Absent key: valid non-existence proof.") + _, _, dp := f.prove(km) + g.snap(f.snap.Hash) + g.key(km[:]) + g.valAbsent() + g.proof(dp, false) + g.verify(true) + g.blank() + + g.comment("Absent key: valid proof but wrong snapshot.") + g.snap(sha256.Sum256([]byte("wrong"))) + g.key(km[:]) + g.valAbsent() + g.proof(dp, false) + g.verify(false) + g.blank() +} + +func (g *gen) twoLeaves() { + g.section("Two-leaf tree") + + f := newFixture("a", "b") + ka, va := tkey("a"), tval("a") + kb, vb := tkey("b"), tval("b") + km := tkey("missing") + + g.comment("Present key a: one path step.") + _, _, pa := f.prove(ka) + g.snap(f.snap.Hash) + g.key(ka[:]) + g.val(va[:]) + g.proof(pa, true) + g.verify(true) + g.blank() + + g.comment("Present key b: one path step.") + _, _, pb := f.prove(kb) + g.snap(f.snap.Hash) + g.key(kb[:]) + g.val(vb[:]) + g.proof(pb, true) + g.verify(true) + g.blank() + + g.comment("Absent key: non-existence proof with path.") + _, _, dp := f.prove(km) + g.snap(f.snap.Hash) + g.key(km[:]) + g.valAbsent() + g.proof(dp, false) + g.verify(true) + g.blank() +} + +func (g *gen) threeLeaves() { + g.section("Three-leaf tree") + + f := newFixture("a", "b", "c") + ka, va := tkey("a"), tval("a") + kb, vb := tkey("b"), tval("b") + kc, vc := tkey("c"), tval("c") + km := tkey("missing") + + g.comment("Present key a.") + _, _, pa := f.prove(ka) + g.snap(f.snap.Hash) + g.key(ka[:]) + g.val(va[:]) + g.proof(pa, true) + g.verify(true) + g.blank() + + g.comment("Present key b.") + _, _, pb := f.prove(kb) + g.snap(f.snap.Hash) + g.key(kb[:]) + g.val(vb[:]) + g.proof(pb, true) + g.verify(true) + g.blank() + + g.comment("Present key c.") + _, _, pc := f.prove(kc) + g.snap(f.snap.Hash) + g.key(kc[:]) + g.val(vc[:]) + g.proof(pc, true) + g.verify(true) + g.blank() + + g.comment("Absent key.") + _, _, dp := f.prove(km) + g.snap(f.snap.Hash) + g.key(km[:]) + g.valAbsent() + g.proof(dp, false) + g.verify(true) + g.blank() +} + +func (g *gen) corruption() { + g.section("Corrupted proofs") + + f2 := newFixture("a", "b") + ka, va := tkey("a"), tval("a") + km := tkey("missing") + + _, _, pa := f2.prove(ka) + _, _, dp := f2.prove(km) + + g.comment("Flipped bit in sibling hash.") + g.snap(f2.snap.Hash) + g.key(ka[:]) + g.val(va[:]) + g.rawProof(flipped(pa, len(pa)-1)) + g.verify(false) + g.blank() + + g.comment("Extra trailing byte.") + g.snap(f2.snap.Hash) + g.key(ka[:]) + g.val(va[:]) + g.rawProof(appended(pa, 0x00)) + g.verify(false) + g.blank() + + if len(pa) > 1 { + g.comment("Truncated proof: only varint, no hash.") + g.snap(f2.snap.Hash) + g.key(ka[:]) + g.val(va[:]) + g.rawProof(truncated(pa, 1)) + g.verify(false) + g.blank() + } + + g.comment("Empty proof for non-empty tree (presence claim).") + g.snap(f2.snap.Hash) + g.key(ka[:]) + g.val(va[:]) + g.proof(mpt.Proof{}, true) + g.verify(false) + g.blank() + + g.comment("Truncated non-existence proof: altkey but no altval.") + // Build a truncated deny proof: varint(32) + key only, missing val. + trunc := make([]byte, 0, 64) + trunc = binary.AppendUvarint(trunc, 32) + trunc = append(trunc, ka[:]...) + g.snap(f2.snap.Hash) + g.key(km[:]) + g.valAbsent() + g.rawProof(trunc) + g.verify(false) + g.blank() + + g.comment("Non-existence proof where altkey equals lookup key.") + fakeProof := buildDenyProof(km[:], va[:], nil) + g.snap(f2.snap.Hash) + g.key(km[:]) + g.valAbsent() + g.rawProof(fakeProof) + g.verify(false) + g.blank() + + g.comment("Flipped bit in altkey of non-existence proof.") + g.snap(f2.snap.Hash) + g.key(km[:]) + g.valAbsent() + g.rawProof(flipped(dp, 1)) // flip in altkey data + g.verify(false) + g.blank() + + f3 := newFixture("a", "b", "c") + ka3 := tkey("a") + va3 := tval("a") + _, _, pa3 := f3.prove(ka3) + + if len(pa3) > 33 { + g.comment("Truncated multi-step proof: cut after first path step.") + g.snap(f3.snap.Hash) + g.key(ka3[:]) + g.val(va3[:]) + g.rawProof(truncated(pa3, 33)) + g.verify(false) + g.blank() + } +} + +func (g *gen) varLenKeys() { + g.section("Variable-length keys") + + // Single leaf with 1-byte key. + shortKey := []byte{0xFF} + shortVal := []byte{0x42} + treeHash := vhashLeaf(shortKey, shortVal) + + g.comment("Short (1-byte) altkey: single-leaf tree, absent 32-byte key.") + var lk mpt.Key + lk[0] = 0xFE + dp := buildDenyProof(shortKey, shortVal, nil) + g.snap(treeHash) + g.key(lk[:]) + g.valAbsent() + g.proof(dp, false) + g.verify(true) + g.blank() + + // Two-leaf tree: short key (left) + normal key (right). + shortKey2 := []byte{0x00} + shortVal2 := []byte{0xAA} + var normalKey mpt.Key + normalKey[0] = 0x80 + var normalVal mpt.Val + for i := range normalVal { + normalVal[i] = 0xBB + } + lHash := vhashLeaf(shortKey2, shortVal2) + rHash := vhashLeaf(normalKey[:], normalVal[:]) + rootHash := vhashInner(0, lHash, rHash) + + g.comment("Short altkey in two-leaf tree: lookup on left side.") + var lk2 mpt.Key + lk2[0] = 0x01 // bit 0 = 0, same side as shortKey2 + dp2 := buildDenyProof(shortKey2, shortVal2, []pathStep{{0, rHash}}) + g.snap(rootHash) + g.key(lk2[:]) + g.valAbsent() + g.proof(dp2, false) + g.verify(true) + g.blank() + + g.comment("Normal-length altkey in two-leaf tree: lookup on right side.") + var lk3 mpt.Key + lk3[0] = 0xC0 // bit 0 = 1, same side as normalKey + dp3 := buildDenyProof(normalKey[:], normalVal[:], []pathStep{{0, lHash}}) + g.snap(rootHash) + g.key(lk3[:]) + g.valAbsent() + g.proof(dp3, false) + g.verify(true) + g.blank() + + // Single leaf with long key (64 bytes). + longKey := make([]byte, 64) + longKey[0] = 0x80 + for i := 1; i < len(longKey); i++ { + longKey[i] = 0xFF + } + longVal := []byte{0x99, 0x88} + longTreeHash := vhashLeaf(longKey, longVal) + + g.comment("Long (64-byte) altkey: single-leaf tree, absent 32-byte key.") + var lk4 mpt.Key + lk4[0] = 0x81 + dp4 := buildDenyProof(longKey, longVal, nil) + g.snap(longTreeHash) + g.key(lk4[:]) + g.valAbsent() + g.proof(dp4, false) + g.verify(true) + g.blank() + + // Variable-length value: short key AND short val. + shortKey3 := []byte{0xAB, 0xCD} + shortVal3 := []byte{0x01, 0x02, 0x03} + treeHash3 := vhashLeaf(shortKey3, shortVal3) + + g.comment("Short altkey and short altval (2-byte key, 3-byte val).") + var lk5 mpt.Key + lk5[0] = 0xAB + lk5[1] = 0xCE // differs from shortKey3 + dp5 := buildDenyProof(shortKey3, shortVal3, nil) + g.snap(treeHash3) + g.key(lk5[:]) + g.valAbsent() + g.proof(dp5, false) + g.verify(true) + g.blank() + + // Confirm proof with variable-length key (key present). + g.comment("Short key present in single-leaf tree (1-byte key, 1-byte val).") + shortKey4 := []byte{0xDD} + shortVal4 := []byte{0xEE} + treeHash4 := vhashLeaf(shortKey4, shortVal4) + g.snap(treeHash4) + g.key(shortKey4) + g.val(shortVal4) + g.proof(mpt.Proof{}, true) // single leaf, no path steps + g.verify(true) + g.blank() + + // Short key present in two-leaf tree. + leftKey := []byte{0x10} + leftVal := []byte{0xAA, 0xBB} + rightKey := []byte{0x90} + rightVal := []byte{0xCC} + lh := vhashLeaf(leftKey, leftVal) + rh := vhashLeaf(rightKey, rightVal) + root := vhashInner(0, lh, rh) + + g.comment("Short key present in two-leaf tree: left child (1-byte key, 2-byte val).") + g.snap(root) + g.key(leftKey) + g.val(leftVal) + g.proof(buildConfirmProof([]pathStep{{0, rh}}), true) + g.verify(true) + g.blank() + + g.comment("Short key present in two-leaf tree: right child (1-byte key, 1-byte val).") + g.snap(root) + g.key(rightKey) + g.val(rightVal) + g.proof(buildConfirmProof([]pathStep{{0, lh}}), true) + g.verify(true) + g.blank() + + // Invalid: wrong tree hash with short altkey. + g.comment("Short altkey: wrong tree hash.") + g.snap(sha256.Sum256([]byte("wrong"))) + g.key(lk[:]) + g.valAbsent() + g.proof(dp, false) + g.verify(false) + g.blank() + + // Invalid: bit mismatch between lookup key and short altkey. + g.comment("Short altkey: bit mismatch in path (altkey on wrong side).") + var lkWrong mpt.Key + lkWrong[0] = 0xC0 // bit 0 = 1, but shortKey2 bit 0 = 0 + dpWrong := buildDenyProof(shortKey2, shortVal2, []pathStep{{0, rHash}}) + g.snap(rootHash) + g.key(lkWrong[:]) + g.valAbsent() + g.proof(dpWrong, false) + g.verify(false) + g.blank() +} + +func (g *gen) keyOverlap() { + g.section("Key overlap (K vs K||0x00)") + g.comment("Keys are padded with a 0x00 byte followed by 0xFF bytes.") + g.comment("This means K and K||0x00 have different bit patterns") + g.comment("and can coexist in the same tree.") + g.blank() + + // K = {0xFF}, K0 = {0xFF, 0x00}. + // K pads to: FF 00 FF FF... (0x00 pad then 0xFF) + // K0 pads to: FF 00 00 FF... (0x00 pad then 0xFF) + // They agree through bit 15 and differ at bit 16. + K := []byte{0xFF} + K0 := []byte{0xFF, 0x00} + Kval := []byte{0x42} + K0val := []byte{0x99} + + // --- Tree containing K --- + Khash := vhashLeaf(K, Kval) + + g.comment("K={FF} stored. Lookup K: present.") + g.snap(Khash) + g.key(K) + g.val(Kval) + g.proof(mpt.Proof{}, true) + g.verify(true) + g.blank() + + g.comment("K={FF} stored. Lookup K0={FF00}: absent (lands at K's leaf).") + dpK0 := buildDenyProof(K, Kval, nil) + g.snap(Khash) + g.key(K0) + g.valAbsent() + g.proof(dpK0, false) + g.verify(true) + g.blank() + + g.comment("K={FF} stored. Existence proof for K must NOT verify K0 as present.") + g.snap(Khash) + g.key(K0) + g.val(Kval) + g.proof(mpt.Proof{}, true) + g.verify(false) + g.blank() + + // --- Tree containing K0 --- + K0hash := vhashLeaf(K0, K0val) + + g.comment("K0={FF00} stored. Lookup K0: present.") + g.snap(K0hash) + g.key(K0) + g.val(K0val) + g.proof(mpt.Proof{}, true) + g.verify(true) + g.blank() + + g.comment("K0={FF00} stored. Lookup K={FF}: absent (lands at K0's leaf).") + dpK := buildDenyProof(K0, K0val, nil) + g.snap(K0hash) + g.key(K) + g.valAbsent() + g.proof(dpK, false) + g.verify(true) + g.blank() + + g.comment("K0={FF00} stored. Existence proof for K0 must NOT verify K as present.") + g.snap(K0hash) + g.key(K) + g.val(K0val) + g.proof(mpt.Proof{}, true) + g.verify(false) + g.blank() + + // --- Same tests with a two-leaf tree to exercise path steps --- + // Left: K={0x10}, Right: {0x90, 0xAA} + Kp := []byte{0x10} + KpVal := []byte{0xBB} + Kp0 := []byte{0x10, 0x00} + other := []byte{0x90, 0xAA} + otherVal := []byte{0xCC, 0xDD} + + lh := vhashLeaf(Kp, KpVal) + rh := vhashLeaf(other, otherVal) + root := vhashInner(0, lh, rh) + + g.comment("Two-leaf tree with K={10}. Lookup K: present.") + g.snap(root) + g.key(Kp) + g.val(KpVal) + g.proof(buildConfirmProof([]pathStep{{0, rh}}), true) + g.verify(true) + g.blank() + + g.comment("Two-leaf tree with K={10}. Lookup K0={1000}: absent.") + dpKp0 := buildDenyProof(Kp, KpVal, []pathStep{{0, rh}}) + g.snap(root) + g.key(Kp0) + g.valAbsent() + g.proof(dpKp0, false) + g.verify(true) + g.blank() + + g.comment("Two-leaf tree with K={10}. Existence proof for K must NOT verify K0={1000}.") + g.snap(root) + g.key(Kp0) + g.val(KpVal) + g.proof(buildConfirmProof([]pathStep{{0, rh}}), true) + g.verify(false) + g.blank() + + // --- Padding allows K and K0 to coexist in the same tree --- + // With 0x00+0xFF padding, K={0xFF} pads to FF 00 FF FF..., + // while K0={0xFF,0x00} pads to FF 00 00 FF FF... + // They agree through bit 15 and split at bit 16: + // K has 1 (0xFF zone), K0 has 0 (0x00 padding byte). + Kboth := []byte{0xFF} + KbothVal := []byte{0x42} + K0both := []byte{0xFF, 0x00} + K0bothVal := []byte{0x99} + KbothLeaf := vhashLeaf(Kboth, KbothVal) + K0bothLeaf := vhashLeaf(K0both, K0bothVal) + // K has bit 16 = 1 (0xFF zone), so K is on the right. + // K0 has bit 16 = 0 (0x00 padding byte), so K0 is on the left. + bothRoot := vhashInner(16, K0bothLeaf, KbothLeaf) + + g.comment("K={FF} and K0={FF00} coexist: split at bit 16. Lookup K: present (right child).") + g.snap(bothRoot) + g.key(Kboth) + g.val(KbothVal) + g.proof(buildConfirmProof([]pathStep{{16, K0bothLeaf}}), true) + g.verify(true) + g.blank() + + g.comment("K={FF} and K0={FF00} coexist: lookup K0: present (left child).") + g.snap(bothRoot) + g.key(K0both) + g.val(K0bothVal) + g.proof(buildConfirmProof([]pathStep{{16, KbothLeaf}}), true) + g.verify(true) + g.blank() + + g.comment("K={FF} and K0={FF00} coexist: lookup {FE}: absent.") + var lookupFE [1]byte + lookupFE[0] = 0xFE + // {FE} pads to FE 00 FF FF..., K pads to FF 00 FF FF... + // At bit 16: {FE} has 1 (0xFF zone), K has 1 (0xFF zone). Same side (right). + // Deny proof has altkey=K, path step at bit 16. + dpFE := buildDenyProof(Kboth, KbothVal, []pathStep{{16, K0bothLeaf}}) + g.snap(bothRoot) + g.key(lookupFE[:]) + g.valAbsent() + g.proof(dpFE, false) + g.verify(true) + g.blank() + + // --- Empty key coexists with {0x00} --- + // Empty pads to 00 FF FF..., {0x00} pads to 00 00 FF FF... + // They agree at bits 0-7 (both 0x00) and split at bit 8: + // empty has 1 (0xFF zone), {0x00} has 0 (0x00 padding byte). + emptyKCoexist := []byte{} + emptyKVal := []byte{0x11} + zeroKCoexist := []byte{0x00} + zeroKVal := []byte{0x22} + emptyLeaf := vhashLeaf(emptyKCoexist, emptyKVal) + zeroLeaf := vhashLeaf(zeroKCoexist, zeroKVal) + // Empty has bit 8 = 1 (0xFF zone), so empty goes right. + // {0x00} has bit 8 = 0 (0x00 padding byte), so {0x00} goes left. + coexistRoot := vhashInner(8, zeroLeaf, emptyLeaf) + + g.comment("Empty key and {00} coexist: split at bit 8. Lookup empty: present (right child).") + g.snap(coexistRoot) + g.key(emptyKCoexist) + g.val(emptyKVal) + g.proof(buildConfirmProof([]pathStep{{8, zeroLeaf}}), true) + g.verify(true) + g.blank() + + g.comment("Empty key and {00} coexist: lookup {00}: present (left child).") + g.snap(coexistRoot) + g.key(zeroKCoexist) + g.val(zeroKVal) + g.proof(buildConfirmProof([]pathStep{{8, emptyLeaf}}), true) + g.verify(true) + g.blank() + + // --- Three-key prefix chain --- + // K={FF}, K0={FF,00}, K00={FF,00,00} all coexist. + // K pads to: FF 00 FF FF FF... + // K0 pads to: FF 00 00 FF FF... + // K00 pads to: FF 00 00 00 FF FF... + // All agree through byte 1 (00). At byte 2: + // K = FF (0xFF zone), K0 = 00 (padding), K00 = 00 (actual) → K differs at bit 16. + // K0 and K00 agree through byte 2 (00). At byte 3: + // K0 = FF (0xFF zone), K00 = 00 (padding) → differ at bit 24. + // Tree: inner(16, inner(24, K00leaf, K0leaf), Kleaf) + K3a := []byte{0xFF} + K3aVal := []byte{0x11} + K3b := []byte{0xFF, 0x00} + K3bVal := []byte{0x22} + K3c := []byte{0xFF, 0x00, 0x00} + K3cVal := []byte{0x33} + K3aLeaf := vhashLeaf(K3a, K3aVal) + K3bLeaf := vhashLeaf(K3b, K3bVal) + K3cLeaf := vhashLeaf(K3c, K3cVal) + // K0 has bit 24 = 1 (0xFF zone), K00 has bit 24 = 0 (padding byte). + K3inner := vhashInner(24, K3cLeaf, K3bLeaf) + // K has bit 16 = 1 (0xFF zone), K0 and K00 have bit 16 = 0. + K3root := vhashInner(16, K3inner, K3aLeaf) + + g.comment("Three-key prefix chain: K={FF}, K0={FF00}, K00={FF0000}. Lookup K: present.") + g.snap(K3root) + g.key(K3a) + g.val(K3aVal) + g.proof(buildConfirmProof([]pathStep{{16, K3inner}}), true) + g.verify(true) + g.blank() + + g.comment("Three-key prefix chain: lookup K0: present (two path steps).") + g.snap(K3root) + g.key(K3b) + g.val(K3bVal) + g.proof(buildConfirmProof([]pathStep{{24, K3cLeaf}, {16, K3aLeaf}}), true) + g.verify(true) + g.blank() + + g.comment("Three-key prefix chain: lookup K00: present (two path steps).") + g.snap(K3root) + g.key(K3c) + g.val(K3cVal) + g.proof(buildConfirmProof([]pathStep{{24, K3bLeaf}, {16, K3aLeaf}}), true) + g.verify(true) + g.blank() + + g.comment("Three-key prefix chain: lookup {FE}: absent (goes right with K at bit 16).") + dpChain := buildDenyProof(K3a, K3aVal, []pathStep{{16, K3inner}}) + g.snap(K3root) + g.key([]byte{0xFE}) + g.valAbsent() + g.proof(dpChain, false) + g.verify(true) + g.blank() + + g.comment("Three-key prefix chain: lookup {FF01}: absent (goes left at bit 16, right at bit 24 with K0).") + // {FF,01} pads to: FF 01 00 FF... At bit 16: byte 2 = 00 (padding) → 0 → left. + // At bit 24: byte 3 = FF (0xFF zone) → 1 → right with K0. + dpChain2 := buildDenyProof(K3b, K3bVal, []pathStep{{24, K3cLeaf}, {16, K3aLeaf}}) + g.snap(K3root) + g.key([]byte{0xFF, 0x01}) + g.valAbsent() + g.proof(dpChain2, false) + g.verify(true) + g.blank() + + // --- Wrong altkey across padding boundary --- + // In the K/K0 coexist tree (split at bit 16), a deny proof + // that claims altkey=K0 (left side) for a lookup key that goes right + // must be rejected because the bit agreement check fails at bit 16. + g.comment("Wrong altkey across padding boundary: lookup {FE} with altkey=K0 (wrong side), must reject.") + // {FE} at bit 16: byte 2 = FF (0xFF zone) → 1 (right side, like K). + // K0={FF,00} at bit 16: byte 2 = 00 (padding) → 0 (left side). + // They disagree at bit 16 → reject. + dpWrong := buildDenyProof(K0both, K0bothVal, []pathStep{{16, KbothLeaf}}) + g.snap(bothRoot) + g.key([]byte{0xFE}) + g.valAbsent() + g.proof(dpWrong, false) + g.verify(false) + g.blank() + + // --- 0xFF padding matching actual bytes --- + // K={FF} pads to: FF 00 FF FF FF... + // K'={FF,00,FF} pads to: FF 00 FF 00 FF FF... + // They agree through byte 2 (K has 0xFF from padding zone, + // K' has actual 0xFF). They differ at byte 3: + // K = FF (0xFF zone), K' = 00 (padding byte). + // Split at bit 24: K' goes left (0), K goes right (1). + KffPad := []byte{0xFF} + KffPadVal := []byte{0xAA} + KffActual := []byte{0xFF, 0x00, 0xFF} + KffActualVal := []byte{0xBB} + KffPadLeaf := vhashLeaf(KffPad, KffPadVal) + KffActualLeaf := vhashLeaf(KffActual, KffActualVal) + // K has bit 24 = 1 (0xFF zone), K' has bit 24 = 0 (padding). + KffRoot := vhashInner(24, KffActualLeaf, KffPadLeaf) + + g.comment("0xFF padding matches actual bytes: K={FF} and K'={FF00FF} split at byte 3 (bit 24).") + g.snap(KffRoot) + g.key(KffPad) + g.val(KffPadVal) + g.proof(buildConfirmProof([]pathStep{{24, KffActualLeaf}}), true) + g.verify(true) + g.blank() + + g.comment("0xFF padding matches actual bytes: K'={FF00FF} present (left child at bit 24).") + g.snap(KffRoot) + g.key(KffActual) + g.val(KffActualVal) + g.proof(buildConfirmProof([]pathStep{{24, KffPadLeaf}}), true) + g.verify(true) + g.blank() + + g.comment("0xFF padding matches actual bytes: lookup {FF00FF01}: absent (goes left at bit 24 with K').") + // {FF,00,FF,01} pads to: FF 00 FF 01 00 FF... + // At bit 24: byte 3 = 0x01 → bit 0 = 0 → left, same side as K' (bit 24 = 0 from padding). + dpFF := buildDenyProof(KffActual, KffActualVal, []pathStep{{24, KffPadLeaf}}) + g.snap(KffRoot) + g.key([]byte{0xFF, 0x00, 0xFF, 0x01}) + g.valAbsent() + g.proof(dpFF, false) + g.verify(true) + g.blank() +} + +func (g *gen) emptyKeyVal() { + g.section("Empty key and empty value") + + // Single leaf with empty key. + emptyK := []byte{} + kval := []byte{0xAA, 0xBB} + treeHash := vhashLeaf(emptyK, kval) + + g.comment("Empty key present in single-leaf tree.") + g.snap(treeHash) + g.key(emptyK) + g.val(kval) + g.proof(mpt.Proof{}, true) + g.verify(true) + g.blank() + + g.comment("Empty key stored. Lookup {00}: absent (different key).") + dpK0 := buildDenyProof(emptyK, kval, nil) + g.snap(treeHash) + g.key([]byte{0x00}) + g.valAbsent() + g.proof(dpK0, false) + g.verify(true) + g.blank() + + g.comment("Empty key stored. Existence proof must NOT verify {00} as present.") + g.snap(treeHash) + g.key([]byte{0x00}) + g.val(kval) + g.proof(mpt.Proof{}, true) + g.verify(false) + g.blank() + + // {00} stored, lookup empty key. + zeroKey := []byte{0x00} + zeroVal := []byte{0xCC} + treeHash2 := vhashLeaf(zeroKey, zeroVal) + + g.comment("{00} stored. Lookup empty key: absent (different key).") + dpEmpty := buildDenyProof(zeroKey, zeroVal, nil) + g.snap(treeHash2) + g.key(emptyK) + g.valAbsent() + g.proof(dpEmpty, false) + g.verify(true) + g.blank() + + g.comment("{00} stored. Existence proof must NOT verify empty key as present.") + g.snap(treeHash2) + g.key(emptyK) + g.val(zeroVal) + g.proof(mpt.Proof{}, true) + g.verify(false) + g.blank() + + // Key present with empty value. + someKey := []byte{0xCC} + emptyV := []byte{} + treeHash3 := vhashLeaf(someKey, emptyV) + + g.comment("Key present with empty value.") + g.snap(treeHash3) + g.key(someKey) + g.val(emptyV) + g.proof(mpt.Proof{}, true) + g.verify(true) + g.blank() + + // Both key and val empty. + treeHash4 := vhashLeaf(emptyK, emptyV) + g.comment("Both key and value are empty.") + g.snap(treeHash4) + g.key(emptyK) + g.val(emptyV) + g.proof(mpt.Proof{}, true) + g.verify(true) + g.blank() + + // Two-leaf tree with empty key. + // With 0x00+0xFF padding, empty key pads to 00 FF FF..., + // and {0x80} has bit 0 = 1 (actual byte). + // Empty key bit 0 = 0 (0x00 padding), so empty goes left. + // {0x80} bit 0 = 1, so {0x80} goes right. + emptyKval2 := []byte{0x11} + other := []byte{0x80} + otherVal := []byte{0x22} + lh := vhashLeaf(emptyK, emptyKval2) + rh := vhashLeaf(other, otherVal) + root := vhashInner(0, lh, rh) + + g.comment("Two-leaf tree: empty key on left, {80} on right. Lookup empty key: present.") + g.snap(root) + g.key(emptyK) + g.val(emptyKval2) + g.proof(buildConfirmProof([]pathStep{{0, rh}}), true) + g.verify(true) + g.blank() + + g.comment("Two-leaf tree: empty key on left. Lookup {01}: absent (bit 0 = 0, same side as empty).") + dpK0tree := buildDenyProof(emptyK, emptyKval2, []pathStep{{0, rh}}) + g.snap(root) + g.key([]byte{0x01}) + g.valAbsent() + g.proof(dpK0tree, false) + g.verify(true) + g.blank() +} + +func (g *gen) proofStructure() { + g.section("Proof structure and hash integrity") + + // --- Key/val boundary confusion --- + // If the leaf hash doesn't include key and val lengths, then + // key=AB,val=CD and key=A,val=BCD would collide. + k1 := []byte{0xAA, 0xBB} + v1 := []byte{0xCC} + th := vhashLeaf(k1, v1) + + g.comment("Key/val boundary confusion: key={AABB} val={CC} must NOT verify key={AA} val={BBCC}.") + g.snap(th) + g.key([]byte{0xAA}) + g.val([]byte{0xBB, 0xCC}) + g.proof(mpt.Proof{}, true) + g.verify(false) + g.blank() + + g.comment("Key/val boundary confusion: key={AABB} val={CC} must NOT verify key={AABBCC} val={}.") + g.snap(th) + g.key([]byte{0xAA, 0xBB, 0xCC}) + g.val([]byte{}) + g.proof(mpt.Proof{}, true) + g.verify(false) + g.blank() + + g.comment("Key/val boundary confusion: key={AABB} val={CC} must NOT verify key={} val={AABBCC}.") + g.snap(th) + g.key([]byte{}) + g.val([]byte{0xAA, 0xBB, 0xCC}) + g.proof(mpt.Proof{}, true) + g.verify(false) + g.blank() + + // --- Swapped key and val --- + g.comment("Swapped key and val of equal length: key={AA} val={BB} must NOT verify key={BB} val={AA}.") + k2 := []byte{0xAA} + v2 := []byte{0xBB} + th2 := vhashLeaf(k2, v2) + g.snap(th2) + g.key(v2) // swapped + g.val(k2) // swapped + g.proof(mpt.Proof{}, true) + g.verify(false) + g.blank() + + // --- Non-empty proof for empty tree --- + emptyHash := sha256.Sum256(nil) + fakeProof := buildDenyProof([]byte{0xFF}, []byte{0xAA}, nil) + + g.comment("Non-empty proof for empty tree (absent claim): must reject.") + g.snap(mpt.Hash(emptyHash)) + g.key([]byte{0x42}) + g.valAbsent() + g.rawProof(fakeProof) + g.verify(false) + g.blank() + + // --- Proof steps at various bit positions with wrong hash --- + // These steps are structurally valid but produce wrong hashes. + shortK := []byte{0xFF} + shortV := []byte{0x42} + shortHash := vhashLeaf(shortK, shortV) + lookupK := []byte{0xFE} + sib := sha256.Sum256([]byte("sibling")) + + g.comment("Proof step at bit 9 for 1-byte keys: wrong hash, must reject.") + g.snap(shortHash) + g.key(lookupK) + g.valAbsent() + g.rawProof(buildDenyProof(shortK, shortV, []pathStep{{9, mpt.Hash(sib)}})) + g.verify(false) + g.blank() + + g.comment("Proof step at bit 8 for 1-byte keys: wrong hash, must reject.") + g.snap(shortHash) + g.key(lookupK) + g.valAbsent() + g.rawProof(buildDenyProof(shortK, shortV, []pathStep{{8, mpt.Hash(sib)}})) + g.verify(false) + g.blank() + + g.comment("Proof step at bit 7 for 1-byte keys: wrong hash, must reject.") + g.snap(shortHash) + g.key(lookupK) + g.valAbsent() + g.rawProof(buildDenyProof(shortK, shortV, []pathStep{{7, mpt.Hash(sib)}})) + g.verify(false) + g.blank() + + // --- Non-decreasing bit positions --- + sib1 := sha256.Sum256([]byte("sib1")) + sib2 := sha256.Sum256([]byte("sib2")) + + g.comment("Equal bit positions in proof (bit 5, bit 5): must reject.") + g.snap(shortHash) + g.key(lookupK) + g.valAbsent() + g.rawProof(buildDenyProof(shortK, shortV, []pathStep{{5, mpt.Hash(sib1)}, {5, mpt.Hash(sib2)}})) + g.verify(false) + g.blank() + + g.comment("Increasing bit positions in proof (bit 3, bit 5): must reject.") + g.snap(shortHash) + g.key(lookupK) + g.valAbsent() + g.rawProof(buildDenyProof(shortK, shortV, []pathStep{{3, mpt.Hash(sib1)}, {5, mpt.Hash(sib2)}})) + g.verify(false) + g.blank() + + // --- Invalid varint --- + g.comment("Invalid varint in proof: unterminated continuation byte (0x80).") + g.snap(shortHash) + g.key(lookupK) + g.valAbsent() + g.rawProof([]byte{0x80}) + g.verify(false) + g.blank() + + g.comment("Invalid varint: five continuation bytes (overlong encoding).") + g.snap(shortHash) + g.key(lookupK) + g.valAbsent() + g.rawProof([]byte{0x80, 0x80, 0x80, 0x80, 0x80}) + g.verify(false) + g.blank() + + // --- Proof with huge varint for altkey length --- + g.comment("Huge varint for altkey length: must reject (not enough data).") + var hugeProof []byte + hugeProof = binary.AppendUvarint(hugeProof, 1<<32) + g.snap(shortHash) + g.key(lookupK) + g.valAbsent() + g.rawProof(hugeProof) + g.verify(false) + g.blank() + + // --- Altkey/altval swapped in non-existence proof --- + // If the proof has altkey and altval swapped (same total bytes), + // the leaf hash will differ and the proof should fail. + swappedKey := []byte{0x11, 0x22} + swappedVal := []byte{0x33, 0x44} + swappedHash := vhashLeaf(swappedKey, swappedVal) + + g.comment("Swapped altkey/altval in deny proof: must reject.") + g.snap(swappedHash) + g.key([]byte{0x11, 0x23}) // different from swappedKey + g.valAbsent() + // Build proof with altkey and altval swapped + g.proof(buildDenyProof(swappedVal, swappedKey, nil), false) + g.verify(false) + g.blank() + + // --- Existence proof reuse: proof for key K should not verify key K2 --- + // Two-leaf tree: K on left, K2 on right. Existence proof for K + // contains sibling hash. Using same proof with K2 should fail. + kLeft := []byte{0x10} + vLeft := []byte{0xAA} + kRight := []byte{0x90} + vRight := []byte{0xBB} + lh := vhashLeaf(kLeft, vLeft) + rh := vhashLeaf(kRight, vRight) + root := vhashInner(0, lh, rh) + + g.comment("Existence proof for K={10} must NOT verify K2={11} (different key, same side).") + g.snap(root) + g.key([]byte{0x11}) // same side as kLeft (bit 0 = 0) + g.val(vLeft) + g.proof(buildConfirmProof([]pathStep{{0, rh}}), true) + g.verify(false) + g.blank() + + g.comment("Existence proof for K={10} must NOT verify K2={90} (different key, other side).") + g.snap(root) + g.key(kRight) + g.val(vLeft) // right key but left val + g.proof(buildConfirmProof([]pathStep{{0, rh}}), true) + g.verify(false) + g.blank() + + // --- Cross-proof reuse: deny proof should not verify as existence --- + // A non-existence proof for key M contains altkey K and altval V. + // If someone strips the altkey/altval prefix and tries to use the + // path steps as an existence proof, it should fail. + g.comment("Path steps from deny proof reused as existence proof: must reject.") + // Extract just the path steps from what would be a deny proof. + pathOnly := buildConfirmProof([]pathStep{{0, rh}}) + // Use kLeft and vLeft as if present — hashLeafVar(kLeft,vLeft) = lh, + // and the path step hashes to root. So this SHOULD verify... + // unless the proof is for a different key. + // Actually this is valid! kLeft IS in the tree. Let me use a different key. + g.snap(root) + g.key([]byte{0x30}) // not in tree, same side as kLeft + g.val(vLeft) + g.proof(pathOnly, true) + g.verify(false) + g.blank() +} diff --git a/mpt/testdata/verify.txt b/mpt/testdata/verify.txt new file mode 100644 index 0000000..6ce8b4b --- /dev/null +++ b/mpt/testdata/verify.txt @@ -0,0 +1,639 @@ +# Test vectors for mpt.Verify, generated by gen.go. +# +# Each test sets state with snap, key, val, and proof lines, +# then calls verify to check the result. Blank lines and lines +# beginning with # are ignored. +# +# Format: +# snap HEXHASH - set the snapshot hash +# key HEXKEY - set the lookup key +# val HEXVAL - set val and ok=true +# val - - set val to empty and ok=false +# proof HEXPROOF - set the proof (hex bytes, spaces ok) +# proof '' - set the proof to empty (zero-length) +# verify true - Verify should succeed (return nil) +# verify false - Verify should fail (return error) +# +# Proof lines may be continued with \ at end of line; +# continuation lines conventionally start with a tab. +# The hex for snap, key, val, and proof may contain spaces. +# +# DO NOT EDIT. Generated by: +# go run testdata/gen.go > testdata/verify.txt + +# === Empty tree === + +# Absent key: empty proof is valid for empty tree. +snap e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 +key c51cbe39ee7ed29027d1c54fffb0b31d0475a502b811bd866b98b66bb0afd1f4 +val - +proof '' +verify true + +# Absent key: empty proof is invalid for non-empty snapshot. +snap 8810ad581e59f2bc3928b261707a71308f7e139eb04820366dc4d5c18d980225 +key c51cbe39ee7ed29027d1c54fffb0b31d0475a502b811bd866b98b66bb0afd1f4 +val - +proof '' +verify false + +# === Single-leaf tree === + +# Present key: valid proof (no path steps for single leaf). +snap 5586c5702eb4abc508d685b0833e8f83050c864796d18ae25dc3d22073038fb8 +key 682c42c526e3f5e87eecea25694c435c16a00bf316bd1a8dc6079bcd1d8f9e46 +val 6370040cdc5ae2cc980582fd877723e32bea9ed2844ff4ae7218711f161265b3 +proof '' +verify true + +# Present key: wrong value, same proof. +snap 5586c5702eb4abc508d685b0833e8f83050c864796d18ae25dc3d22073038fb8 +key 682c42c526e3f5e87eecea25694c435c16a00bf316bd1a8dc6079bcd1d8f9e46 +val 7c1961346483781a205da2a49d4e42a13d989e506f7e611d6f878ef09b5759a3 +proof '' +verify false + +# Absent key: valid non-existence proof. +snap 5586c5702eb4abc508d685b0833e8f83050c864796d18ae25dc3d22073038fb8 +key c51cbe39ee7ed29027d1c54fffb0b31d0475a502b811bd866b98b66bb0afd1f4 +val - +proof 20 682c42c526e3f5e87eecea25694c435c16a00bf316bd1a8dc6079bcd1d8f9e46 \ + 20 6370040cdc5ae2cc980582fd877723e32bea9ed2844ff4ae7218711f161265b3 +verify true + +# Absent key: valid proof but wrong snapshot. +snap 8810ad581e59f2bc3928b261707a71308f7e139eb04820366dc4d5c18d980225 +key c51cbe39ee7ed29027d1c54fffb0b31d0475a502b811bd866b98b66bb0afd1f4 +val - +proof 20 682c42c526e3f5e87eecea25694c435c16a00bf316bd1a8dc6079bcd1d8f9e46 \ + 20 6370040cdc5ae2cc980582fd877723e32bea9ed2844ff4ae7218711f161265b3 +verify false + +# === Two-leaf tree === + +# Present key a: one path step. +snap ce1e03bce2008026dd8f68f43b815f3f75bc14f3a2213e334c64230c6ebe8666 +key 682c42c526e3f5e87eecea25694c435c16a00bf316bd1a8dc6079bcd1d8f9e46 +val 6370040cdc5ae2cc980582fd877723e32bea9ed2844ff4ae7218711f161265b3 +proof 00 16a5fadb9ec081b9111a6e731c4a5cea7315ebc862fe48f75b0e658966e59df2 +verify true + +# Present key b: one path step. +snap ce1e03bce2008026dd8f68f43b815f3f75bc14f3a2213e334c64230c6ebe8666 +key e3ee6cc705f06140f832d62316f3cb1c7a324f04d4cba9dc8d0f8125e5f5393c +val 016a4fde9eff72a846636f8e5981cabc9e7d34041a6e9080691259fc9d2bffa8 +proof 00 5586c5702eb4abc508d685b0833e8f83050c864796d18ae25dc3d22073038fb8 +verify true + +# Absent key: non-existence proof with path. +snap ce1e03bce2008026dd8f68f43b815f3f75bc14f3a2213e334c64230c6ebe8666 +key c51cbe39ee7ed29027d1c54fffb0b31d0475a502b811bd866b98b66bb0afd1f4 +val - +proof 20 e3ee6cc705f06140f832d62316f3cb1c7a324f04d4cba9dc8d0f8125e5f5393c \ + 20 016a4fde9eff72a846636f8e5981cabc9e7d34041a6e9080691259fc9d2bffa8 \ + 00 5586c5702eb4abc508d685b0833e8f83050c864796d18ae25dc3d22073038fb8 +verify true + +# === Three-leaf tree === + +# Present key a. +snap 4a79f8c72c1e6aef51da667b958a4a60d9d54eeb4e71e169c97f6f0ef2462223 +key 682c42c526e3f5e87eecea25694c435c16a00bf316bd1a8dc6079bcd1d8f9e46 +val 6370040cdc5ae2cc980582fd877723e32bea9ed2844ff4ae7218711f161265b3 +proof 00 c8f7f7f5f65dd8ede617bb4e4d934c1012a1003c817a3d217e030fcc1b3ed1a0 +verify true + +# Present key b. +snap 4a79f8c72c1e6aef51da667b958a4a60d9d54eeb4e71e169c97f6f0ef2462223 +key e3ee6cc705f06140f832d62316f3cb1c7a324f04d4cba9dc8d0f8125e5f5393c +val 016a4fde9eff72a846636f8e5981cabc9e7d34041a6e9080691259fc9d2bffa8 +proof 02 65ccf55d89d2f4864ea1e8fbc0b1d74a29a67431d7210b0e82037ef2788f32c6 \ + 00 5586c5702eb4abc508d685b0833e8f83050c864796d18ae25dc3d22073038fb8 +verify true + +# Present key c. +snap 4a79f8c72c1e6aef51da667b958a4a60d9d54eeb4e71e169c97f6f0ef2462223 +key d1318ac2288d45fd05e788897f6672de3761c8e1565bef41c350e578283a9d6f +val c35b501388708557ba0bab3b2b62c2e14cc405b1415d565a62b3ff1bc59399e2 +proof 02 16a5fadb9ec081b9111a6e731c4a5cea7315ebc862fe48f75b0e658966e59df2 \ + 00 5586c5702eb4abc508d685b0833e8f83050c864796d18ae25dc3d22073038fb8 +verify true + +# Absent key. +snap 4a79f8c72c1e6aef51da667b958a4a60d9d54eeb4e71e169c97f6f0ef2462223 +key c51cbe39ee7ed29027d1c54fffb0b31d0475a502b811bd866b98b66bb0afd1f4 +val - +proof 20 d1318ac2288d45fd05e788897f6672de3761c8e1565bef41c350e578283a9d6f \ + 20 c35b501388708557ba0bab3b2b62c2e14cc405b1415d565a62b3ff1bc59399e2 \ + 02 16a5fadb9ec081b9111a6e731c4a5cea7315ebc862fe48f75b0e658966e59df2 \ + 00 5586c5702eb4abc508d685b0833e8f83050c864796d18ae25dc3d22073038fb8 +verify true + +# === Corrupted proofs === + +# Flipped bit in sibling hash. +snap ce1e03bce2008026dd8f68f43b815f3f75bc14f3a2213e334c64230c6ebe8666 +key 682c42c526e3f5e87eecea25694c435c16a00bf316bd1a8dc6079bcd1d8f9e46 +val 6370040cdc5ae2cc980582fd877723e32bea9ed2844ff4ae7218711f161265b3 +proof 0016a5fadb9ec081b9111a6e731c4a5cea7315ebc862fe48f75b0e658966e59d72 +verify false + +# Extra trailing byte. +snap ce1e03bce2008026dd8f68f43b815f3f75bc14f3a2213e334c64230c6ebe8666 +key 682c42c526e3f5e87eecea25694c435c16a00bf316bd1a8dc6079bcd1d8f9e46 +val 6370040cdc5ae2cc980582fd877723e32bea9ed2844ff4ae7218711f161265b3 +proof 0016a5fadb9ec081b9111a6e731c4a5cea7315ebc862fe48f75b0e658966e59df200 +verify false + +# Truncated proof: only varint, no hash. +snap ce1e03bce2008026dd8f68f43b815f3f75bc14f3a2213e334c64230c6ebe8666 +key 682c42c526e3f5e87eecea25694c435c16a00bf316bd1a8dc6079bcd1d8f9e46 +val 6370040cdc5ae2cc980582fd877723e32bea9ed2844ff4ae7218711f161265b3 +proof 00 +verify false + +# Empty proof for non-empty tree (presence claim). +snap ce1e03bce2008026dd8f68f43b815f3f75bc14f3a2213e334c64230c6ebe8666 +key 682c42c526e3f5e87eecea25694c435c16a00bf316bd1a8dc6079bcd1d8f9e46 +val 6370040cdc5ae2cc980582fd877723e32bea9ed2844ff4ae7218711f161265b3 +proof '' +verify false + +# Truncated non-existence proof: altkey but no altval. +snap ce1e03bce2008026dd8f68f43b815f3f75bc14f3a2213e334c64230c6ebe8666 +key c51cbe39ee7ed29027d1c54fffb0b31d0475a502b811bd866b98b66bb0afd1f4 +val - +proof 20682c42c526e3f5e87eecea25694c435c16a00bf316bd1a8dc6079bcd1d8f9e46 +verify false + +# Non-existence proof where altkey equals lookup key. +snap ce1e03bce2008026dd8f68f43b815f3f75bc14f3a2213e334c64230c6ebe8666 +key c51cbe39ee7ed29027d1c54fffb0b31d0475a502b811bd866b98b66bb0afd1f4 +val - +proof 20c51cbe39ee7ed29027d1c54fffb0b31d0475a502b811bd866b98b66bb0afd1f4206370040cdc5ae2cc980582fd877723e32bea9ed2844ff4ae7218711f161265b3 +verify false + +# Flipped bit in altkey of non-existence proof. +snap ce1e03bce2008026dd8f68f43b815f3f75bc14f3a2213e334c64230c6ebe8666 +key c51cbe39ee7ed29027d1c54fffb0b31d0475a502b811bd866b98b66bb0afd1f4 +val - +proof 2063ee6cc705f06140f832d62316f3cb1c7a324f04d4cba9dc8d0f8125e5f5393c20016a4fde9eff72a846636f8e5981cabc9e7d34041a6e9080691259fc9d2bffa8005586c5702eb4abc508d685b0833e8f83050c864796d18ae25dc3d22073038fb8 +verify false + +# === Variable-length keys === + +# Short (1-byte) altkey: single-leaf tree, absent 32-byte key. +snap d8d646b5e3b2bbf2c1c57c3287f0810ccc92065242eec4d45c7bff13fb47f104 +key fe00000000000000000000000000000000000000000000000000000000000000 +val - +proof 01 ff \ + 01 42 +verify true + +# Short altkey in two-leaf tree: lookup on left side. +snap 652af8899929dc42c34f0b98ad8e1fb6dc62528051cc45f0975cf3ea849ef39b +key 0100000000000000000000000000000000000000000000000000000000000000 +val - +proof 01 00 \ + 01 aa \ + 00 e9bbddf41645a9e33d10589174a7c6b5c7bc40c7853640b3ed95631906419cee +verify true + +# Normal-length altkey in two-leaf tree: lookup on right side. +snap 652af8899929dc42c34f0b98ad8e1fb6dc62528051cc45f0975cf3ea849ef39b +key c000000000000000000000000000000000000000000000000000000000000000 +val - +proof 20 8000000000000000000000000000000000000000000000000000000000000000 \ + 20 bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb \ + 00 70bbb047bac9a8daed58860c182e9104289ee8d954939a9c6b3a89f489c2f422 +verify true + +# Long (64-byte) altkey: single-leaf tree, absent 32-byte key. +snap 147e75f240d22122402bebca6208423b92a5eee49a11f229c42ebf2e68514e70 +key 8100000000000000000000000000000000000000000000000000000000000000 +val - +proof 40 80ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff \ + 02 9988 +verify true + +# Short altkey and short altval (2-byte key, 3-byte val). +snap 377c4371939c2ca7b322d821ad6d16e8d325a9c19227300e5417ee7ad3f9d766 +key abce000000000000000000000000000000000000000000000000000000000000 +val - +proof 02 abcd \ + 03 010203 +verify true + +# Short key present in single-leaf tree (1-byte key, 1-byte val). +snap 42d2b9000418448057e4c639f1c5d6a384df5afc123c65409eeb12e08a79dc30 +key dd +val ee +proof '' +verify true + +# Short key present in two-leaf tree: left child (1-byte key, 2-byte val). +snap 278ebf2a3f37f47f99da7f1dafadc8c109ca1c96fe8fb0097814547082d289c8 +key 10 +val aabb +proof 00 ea18b6eab7ac1357b71b1dffc58be8d1f5949d3d3d3b3949f67e2e561a7e92ba +verify true + +# Short key present in two-leaf tree: right child (1-byte key, 1-byte val). +snap 278ebf2a3f37f47f99da7f1dafadc8c109ca1c96fe8fb0097814547082d289c8 +key 90 +val cc +proof 00 458910339118ebaea20a8aff7fc345b4cae0fa9586188740fe06ab778c902261 +verify true + +# Short altkey: wrong tree hash. +snap 8810ad581e59f2bc3928b261707a71308f7e139eb04820366dc4d5c18d980225 +key fe00000000000000000000000000000000000000000000000000000000000000 +val - +proof 01 ff \ + 01 42 +verify false + +# Short altkey: bit mismatch in path (altkey on wrong side). +snap 652af8899929dc42c34f0b98ad8e1fb6dc62528051cc45f0975cf3ea849ef39b +key c000000000000000000000000000000000000000000000000000000000000000 +val - +proof 01 00 \ + 01 aa \ + 00 e9bbddf41645a9e33d10589174a7c6b5c7bc40c7853640b3ed95631906419cee +verify false + +# === Key overlap (K vs K||0x00) === + +# Keys are padded with a 0x00 byte followed by 0xFF bytes. +# This means K and K||0x00 have different bit patterns +# and can coexist in the same tree. + +# K={FF} stored. Lookup K: present. +snap d8d646b5e3b2bbf2c1c57c3287f0810ccc92065242eec4d45c7bff13fb47f104 +key ff +val 42 +proof '' +verify true + +# K={FF} stored. Lookup K0={FF00}: absent (lands at K's leaf). +snap d8d646b5e3b2bbf2c1c57c3287f0810ccc92065242eec4d45c7bff13fb47f104 +key ff00 +val - +proof 01 ff \ + 01 42 +verify true + +# K={FF} stored. Existence proof for K must NOT verify K0 as present. +snap d8d646b5e3b2bbf2c1c57c3287f0810ccc92065242eec4d45c7bff13fb47f104 +key ff00 +val 42 +proof '' +verify false + +# K0={FF00} stored. Lookup K0: present. +snap a6d4e31ef88824683f136dafbe8cb0bf786fde89a82088f8b9d09c6672c6894b +key ff00 +val 99 +proof '' +verify true + +# K0={FF00} stored. Lookup K={FF}: absent (lands at K0's leaf). +snap a6d4e31ef88824683f136dafbe8cb0bf786fde89a82088f8b9d09c6672c6894b +key ff +val - +proof 02 ff00 \ + 01 99 +verify true + +# K0={FF00} stored. Existence proof for K0 must NOT verify K as present. +snap a6d4e31ef88824683f136dafbe8cb0bf786fde89a82088f8b9d09c6672c6894b +key ff +val 99 +proof '' +verify false + +# Two-leaf tree with K={10}. Lookup K: present. +snap cc1bde60ce3b211f5ff73ef1d50819c699016eadc2ebe2c1c1657cb537a22d32 +key 10 +val bb +proof 00 969f212597e87fb2ccc93fd9f09762e8e3e037cc1bb6b34117b27f97072b255b +verify true + +# Two-leaf tree with K={10}. Lookup K0={1000}: absent. +snap cc1bde60ce3b211f5ff73ef1d50819c699016eadc2ebe2c1c1657cb537a22d32 +key 1000 +val - +proof 01 10 \ + 01 bb \ + 00 969f212597e87fb2ccc93fd9f09762e8e3e037cc1bb6b34117b27f97072b255b +verify true + +# Two-leaf tree with K={10}. Existence proof for K must NOT verify K0={1000}. +snap cc1bde60ce3b211f5ff73ef1d50819c699016eadc2ebe2c1c1657cb537a22d32 +key 1000 +val bb +proof 00 969f212597e87fb2ccc93fd9f09762e8e3e037cc1bb6b34117b27f97072b255b +verify false + +# K={FF} and K0={FF00} coexist: split at bit 16. Lookup K: present (right child). +snap 9de3a1dac7c1ce90c1bd63520099045f43769318cf67ac9d76316896e1978b9b +key ff +val 42 +proof 10 a6d4e31ef88824683f136dafbe8cb0bf786fde89a82088f8b9d09c6672c6894b +verify true + +# K={FF} and K0={FF00} coexist: lookup K0: present (left child). +snap 9de3a1dac7c1ce90c1bd63520099045f43769318cf67ac9d76316896e1978b9b +key ff00 +val 99 +proof 10 d8d646b5e3b2bbf2c1c57c3287f0810ccc92065242eec4d45c7bff13fb47f104 +verify true + +# K={FF} and K0={FF00} coexist: lookup {FE}: absent. +snap 9de3a1dac7c1ce90c1bd63520099045f43769318cf67ac9d76316896e1978b9b +key fe +val - +proof 01 ff \ + 01 42 \ + 10 a6d4e31ef88824683f136dafbe8cb0bf786fde89a82088f8b9d09c6672c6894b +verify true + +# Empty key and {00} coexist: split at bit 8. Lookup empty: present (right child). +snap f781fca5322ada0b26d8bc962e3cb149eae923072bf58c6426ec8a3fb3684a2f +key '' +val 11 +proof 08 43d32c743438ec8c3d23d199721e7999d8fc3124a4eebd6adfae42ef1474bb27 +verify true + +# Empty key and {00} coexist: lookup {00}: present (left child). +snap f781fca5322ada0b26d8bc962e3cb149eae923072bf58c6426ec8a3fb3684a2f +key 00 +val 22 +proof 08 c09a57053f599fc6ebf48e0605a17ea0c2ffe63fa9b48cec0ce289e5df60bcb6 +verify true + +# Three-key prefix chain: K={FF}, K0={FF00}, K00={FF0000}. Lookup K: present. +snap 4cf0e829df5d68543af1688dd858785a1403e0cea6d5cc0600833414b5c86b0f +key ff +val 11 +proof 10 b6abeec889bf995a472d9bbf379509a2b24f3ace0859b2b914b41545771f419a +verify true + +# Three-key prefix chain: lookup K0: present (two path steps). +snap 4cf0e829df5d68543af1688dd858785a1403e0cea6d5cc0600833414b5c86b0f +key ff00 +val 22 +proof 18 40006ed0e42658fbd3f01540eabb1177c7ad30842ef7d5486750c9a591753581 \ + 10 8baf0e4d45047f2d0ed95442bd339dcc51631dff62bdd537a14d6788d791862a +verify true + +# Three-key prefix chain: lookup K00: present (two path steps). +snap 4cf0e829df5d68543af1688dd858785a1403e0cea6d5cc0600833414b5c86b0f +key ff0000 +val 33 +proof 18 f26fc52fd34db084c15cd8cd5bcc0a8cd6c3e7aeb790aba879c11ee8fbf9c31f \ + 10 8baf0e4d45047f2d0ed95442bd339dcc51631dff62bdd537a14d6788d791862a +verify true + +# Three-key prefix chain: lookup {FE}: absent (goes right with K at bit 16). +snap 4cf0e829df5d68543af1688dd858785a1403e0cea6d5cc0600833414b5c86b0f +key fe +val - +proof 01 ff \ + 01 11 \ + 10 b6abeec889bf995a472d9bbf379509a2b24f3ace0859b2b914b41545771f419a +verify true + +# Three-key prefix chain: lookup {FF01}: absent (goes left at bit 16, right at bit 24 with K0). +snap 4cf0e829df5d68543af1688dd858785a1403e0cea6d5cc0600833414b5c86b0f +key ff01 +val - +proof 02 ff00 \ + 01 22 \ + 18 40006ed0e42658fbd3f01540eabb1177c7ad30842ef7d5486750c9a591753581 \ + 10 8baf0e4d45047f2d0ed95442bd339dcc51631dff62bdd537a14d6788d791862a +verify true + +# Wrong altkey across padding boundary: lookup {FE} with altkey=K0 (wrong side), must reject. +snap 9de3a1dac7c1ce90c1bd63520099045f43769318cf67ac9d76316896e1978b9b +key fe +val - +proof 02 ff00 \ + 01 99 \ + 10 d8d646b5e3b2bbf2c1c57c3287f0810ccc92065242eec4d45c7bff13fb47f104 +verify false + +# 0xFF padding matches actual bytes: K={FF} and K'={FF00FF} split at byte 3 (bit 24). +snap d59f32f32c09620cb2ca29629e1c91ad1e91735f4f811b6f46b95bc7060e10db +key ff +val aa +proof 18 a1e210e56d60f426e22e01cd5d8ecf1298fc4864db6a3a8e31dcf513aa30ef94 +verify true + +# 0xFF padding matches actual bytes: K'={FF00FF} present (left child at bit 24). +snap d59f32f32c09620cb2ca29629e1c91ad1e91735f4f811b6f46b95bc7060e10db +key ff00ff +val bb +proof 18 06ba466e2ba1b5cfbb73851b120447531ecb5bc1b62508774c622e57c48eade3 +verify true + +# 0xFF padding matches actual bytes: lookup {FF00FF01}: absent (goes left at bit 24 with K'). +snap d59f32f32c09620cb2ca29629e1c91ad1e91735f4f811b6f46b95bc7060e10db +key ff00ff01 +val - +proof 03 ff00ff \ + 01 bb \ + 18 06ba466e2ba1b5cfbb73851b120447531ecb5bc1b62508774c622e57c48eade3 +verify true + +# === Empty key and empty value === + +# Empty key present in single-leaf tree. +snap 3ef6294155d93a11692150d489e00c849a932b4bc7605c136b00de040377d5bc +key '' +val aabb +proof '' +verify true + +# Empty key stored. Lookup {00}: absent (different key). +snap 3ef6294155d93a11692150d489e00c849a932b4bc7605c136b00de040377d5bc +key 00 +val - +proof 00 \ + 02 aabb +verify true + +# Empty key stored. Existence proof must NOT verify {00} as present. +snap 3ef6294155d93a11692150d489e00c849a932b4bc7605c136b00de040377d5bc +key 00 +val aabb +proof '' +verify false + +# {00} stored. Lookup empty key: absent (different key). +snap a73cd958e4cebf777cfbdcd6147cd14efa444a55dd5a77db97615e866f890550 +key '' +val - +proof 01 00 \ + 01 cc +verify true + +# {00} stored. Existence proof must NOT verify empty key as present. +snap a73cd958e4cebf777cfbdcd6147cd14efa444a55dd5a77db97615e866f890550 +key '' +val cc +proof '' +verify false + +# Key present with empty value. +snap 629808407b1fa3203860db82a28eba54d0bcbc11997751f3137e2c2723bbf504 +key cc +val '' +proof '' +verify true + +# Both key and value are empty. +snap 709e80c88487a2411e1ee4dfb9f22a861492d20c4765150c0c794abd70f8147c +key '' +val '' +proof '' +verify true + +# Two-leaf tree: empty key on left, {80} on right. Lookup empty key: present. +snap 5fc6da224b95fa2e8069d72016bcdae9a35ff4878c0e2b204a1b9ee1a7a3f08e +key '' +val 11 +proof 00 cd7dcf45ead13419bd621041fd85a5f966309cdc2b5dd8b8246efdb50cdfbf4d +verify true + +# Two-leaf tree: empty key on left. Lookup {01}: absent (bit 0 = 0, same side as empty). +snap 5fc6da224b95fa2e8069d72016bcdae9a35ff4878c0e2b204a1b9ee1a7a3f08e +key 01 +val - +proof 00 \ + 01 11 \ + 00 cd7dcf45ead13419bd621041fd85a5f966309cdc2b5dd8b8246efdb50cdfbf4d +verify true + +# === Proof structure and hash integrity === + +# Key/val boundary confusion: key={AABB} val={CC} must NOT verify key={AA} val={BBCC}. +snap d537f29e12fc567e9f9813834a195e28a031fc39357d6ef4071f08a4f8ce48c6 +key aa +val bbcc +proof '' +verify false + +# Key/val boundary confusion: key={AABB} val={CC} must NOT verify key={AABBCC} val={}. +snap d537f29e12fc567e9f9813834a195e28a031fc39357d6ef4071f08a4f8ce48c6 +key aabbcc +val '' +proof '' +verify false + +# Key/val boundary confusion: key={AABB} val={CC} must NOT verify key={} val={AABBCC}. +snap d537f29e12fc567e9f9813834a195e28a031fc39357d6ef4071f08a4f8ce48c6 +key '' +val aabbcc +proof '' +verify false + +# Swapped key and val of equal length: key={AA} val={BB} must NOT verify key={BB} val={AA}. +snap 9b39e27e037c063ea6422a79d05f78b482fdbd33ffb65244c0828aa5d04714f3 +key bb +val aa +proof '' +verify false + +# Non-empty proof for empty tree (absent claim): must reject. +snap e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 +key 42 +val - +proof 01ff01aa +verify false + +# Proof step at bit 9 for 1-byte keys: wrong hash, must reject. +snap d8d646b5e3b2bbf2c1c57c3287f0810ccc92065242eec4d45c7bff13fb47f104 +key fe +val - +proof 01ff0142097d10de8554ed5ca40f9d0f0e0f4375b5b338af3fb96d33c9b2f53b5289b8f4fe +verify false + +# Proof step at bit 8 for 1-byte keys: wrong hash, must reject. +snap d8d646b5e3b2bbf2c1c57c3287f0810ccc92065242eec4d45c7bff13fb47f104 +key fe +val - +proof 01ff0142087d10de8554ed5ca40f9d0f0e0f4375b5b338af3fb96d33c9b2f53b5289b8f4fe +verify false + +# Proof step at bit 7 for 1-byte keys: wrong hash, must reject. +snap d8d646b5e3b2bbf2c1c57c3287f0810ccc92065242eec4d45c7bff13fb47f104 +key fe +val - +proof 01ff0142077d10de8554ed5ca40f9d0f0e0f4375b5b338af3fb96d33c9b2f53b5289b8f4fe +verify false + +# Equal bit positions in proof (bit 5, bit 5): must reject. +snap d8d646b5e3b2bbf2c1c57c3287f0810ccc92065242eec4d45c7bff13fb47f104 +key fe +val - +proof 01ff0142051283cbd3042c06ca007827821a45bcd9e2560f908609104b252ae1c3f30ae91d05954c4755fae8466b8fdbbd0299d73218a109bb2e98e107e1716b4f8303b420ec +verify false + +# Increasing bit positions in proof (bit 3, bit 5): must reject. +snap d8d646b5e3b2bbf2c1c57c3287f0810ccc92065242eec4d45c7bff13fb47f104 +key fe +val - +proof 01ff0142031283cbd3042c06ca007827821a45bcd9e2560f908609104b252ae1c3f30ae91d05954c4755fae8466b8fdbbd0299d73218a109bb2e98e107e1716b4f8303b420ec +verify false + +# Invalid varint in proof: unterminated continuation byte (0x80). +snap d8d646b5e3b2bbf2c1c57c3287f0810ccc92065242eec4d45c7bff13fb47f104 +key fe +val - +proof 80 +verify false + +# Invalid varint: five continuation bytes (overlong encoding). +snap d8d646b5e3b2bbf2c1c57c3287f0810ccc92065242eec4d45c7bff13fb47f104 +key fe +val - +proof 8080808080 +verify false + +# Huge varint for altkey length: must reject (not enough data). +snap d8d646b5e3b2bbf2c1c57c3287f0810ccc92065242eec4d45c7bff13fb47f104 +key fe +val - +proof 8080808010 +verify false + +# Swapped altkey/altval in deny proof: must reject. +snap 7f9d2d9a0fec1da6fbe83901a50fddd08e4bb5d1b352e8a2c33481dbf3ec34d1 +key 1123 +val - +proof 02 3344 \ + 02 1122 +verify false + +# Existence proof for K={10} must NOT verify K2={11} (different key, same side). +snap c00a1f56c8038d93c6fef4db76b1b64dbdbd9b0ffa0b28ea8cc6c469e02d9e73 +key 11 +val aa +proof 00 6ff5eff8d59bc26df2bbaeb7b9ccfbead59f32aa4499f1ced48245f05265c361 +verify false + +# Existence proof for K={10} must NOT verify K2={90} (different key, other side). +snap c00a1f56c8038d93c6fef4db76b1b64dbdbd9b0ffa0b28ea8cc6c469e02d9e73 +key 90 +val aa +proof 00 6ff5eff8d59bc26df2bbaeb7b9ccfbead59f32aa4499f1ced48245f05265c361 +verify false + +# Path steps from deny proof reused as existence proof: must reject. +snap c00a1f56c8038d93c6fef4db76b1b64dbdbd9b0ffa0b28ea8cc6c469e02d9e73 +key 30 +val aa +proof 00 6ff5eff8d59bc26df2bbaeb7b9ccfbead59f32aa4499f1ced48245f05265c361 +verify false + diff --git a/mpt/tree.go b/mpt/tree.go index bde42b2..1d8faf3 100644 --- a/mpt/tree.go +++ b/mpt/tree.go @@ -3,11 +3,152 @@ // license that can be found in the LICENSE file. // Package mpt implements a Merkle Patricia Tree. +// +// A Merkle Patricia Tree (MPT) is a map that stores key-value pairs, where +// each key and value is an opaque value (often a SHA256 hash). +// Analogous to a [transparent log], an MPT can cryptographically prove +// that a given key-value pair exists (or that a key does not exist) in a given tree snapshot. +// By recording the sequence of tree snapshots in a transparent log, +// a server can publish a record of the history of a key-value database, +// enabling auditors to check that the database was correct at all times, +// while allowing clients to be sure the responses they received +// came from the recorded database history. +// +// To use the package, see the [Tree] interface, the [New] and [Create] constructors, +// and the [Verify] function. +// The rest of this doc comment describes the tree and proof encodings +// in enough detail to build an alternate wire-compatible implementation. +// +// # Tree Format +// +// The tree format used in this package is as follows. +// +// Conceptually, start with a complete [binary trie] of arbitrary height H, +// where H is larger than the number of bits in any key to be stored. +// Each key-value pair is placed in the tree at the node reached by +// starting at the root of the tree and following a path making +// left or right turns according to successive key bits and ends with +// however many left turns are needed to reach the leaf level +// (more precisely, the number of trailing left turns is H minus the key size in bits). +// Now we apply two optimizations to that binary radix tree. +// +// First, the tree is “path-compressed,” by removing inner nodes with a single child: +// a node that would have pointed at a single-child node +// is replaced by its child, recursively. +// Every node is therefore either a leaf or an inner node with two children. +// The path compression ensures that there are exactly N inner nodes for a tree with N+1 leaf nodes, +// and it makes the height of the tree depend only on the specific set of keys, +// not on the arbitrary height H. +// +// Second, unlike in a normal binary tree, an inner node stores only the bit position +// that determines whether a lookup should proceed to the left or right child. +// A lookup walks inner nodes down to some leaf, checking one bit at each step. +// Only upon reaching the leaf does it do a full key comparison. +// If it takes O(K) time to compare two keys, a normal binary tree would +// take O(K log N) time for a walk; this optimization +// cuts the time to O(K + log N). +// Furthermore, inner nodes need not store associated keys, +// cutting the number of stored keys by a factor of two. +// +// The path-compression optimization implies that an inner node for key prefix P exists +// if and only if the tree contains at least one key with prefix P||0 and at least one key with prefix P||1. +// That is, the specific inner nodes that exist in a tree depend only on which +// keys are present in the tree, not on their insertion order. +// This implies that we can batch or otherwise reorder insertions of distinct keys +// without affecting the final tree structure. +// +// Although this package does not yet support them, the tree structure +// described here supports keys of varying length. +// However, in such a tree, a lookup for a short key may need to compare +// additional bits to distinguish the short key from a longer key with +// the short key as a prefix. In this case, we define that a key of L bytes +// is treated as if padded with a 0x00 byte at position L followed by 0xFF bytes +// in all subsequent positions. +// The 0x00 byte means that, for text keys without NULs, +// short keys sort before their longer extensions. +// The subsequent 0xFF bytes ensure that two distinct keys never have +// the same padded bit sequence, even when one key is a prefix of the other. +// +// # Tree Snapshots +// +// A tree snapshot is defined as the hash of a tree, defined as follows, where H = SHA256. +// +// - The hash of an empty tree is the hash of an empty (zero-length) input (e3b0c442...7852b855). +// - The hash of a leaf node is the hash of a zero byte followed by the length-prefixed key and length-prefixed value: H(0 || len(key) || key || len(val) || val). +// - The hash of an inner node is the hash a one byte followed by the node's bit position and its left and right children's hashes: H(1 || bit || left-hash || right-hash). +// +// The lengths and bit position are [varint-encoded], so that most are one byte. +// +// Notice that the hash of a node representing a subtree is the same +// as the hash of a tree containing only those nodes: the root node is not special. +// Although this package does not make use of that fact, it does mean that a +// large MPT could be split across multiple computers. +// +// # Proofs +// +// A proof cryptographically attests to a claim about the +// presence or absence of a specific key in a specific tree snapshot. +// The claim takes one of two forms: +// +// - The snapshot contains a specific key-value pair. +// - The snapshot does not contain a specific key. +// +// In this package, a claim and proof are returned by the [Tree.Prove] method, +// and the caller is expected to have already used the [Tree.Snapshot] method +// to obtain the snapshot. +// A verifier (possibly on another system) can then pass the snapshot, +// claim, and proof to [Verify] to cryptographically verify the claim. +// +// The proof only contains the supplemental information needed for verification. +// It does not include the snapshot or the claim, so the proof can only be checked +// with respect to a specific snapshot and claim. +// In fact, a single proof may be valid for many (snapshot, claim) pairs. +// +// The specific form of the proof depends on which of three cases is being proved. +// +// 1. If the snapshot is for an empty tree, the proof is empty (zero length). +// It proves any claim that a key is not present. +// +// 2. If the claim is that a specific key-value pair is present in a snapshot, +// then the proof is a concatenation of zero or more (bit, sibling hash) pairs +// giving the path from the key-value leaf node up to the root of the tree. +// The verifier computes the leaf hash from the key and value +// and then computes the hashes of successive parent nodes up to the root, +// checking that the final hash matches the snapshot. +// At each parent node, the key's specified bit position indicates whether +// the hash computed so far is the left or right child hash. +// The sibling hash provides the other. +// In the proof encoding, the bit positions are [varint-encoded]. +// +// 3. If the claim is that a specific key is not present in a non-empty snapshot, +// then a lookup for key in the tree must instead end at some pair altkey-altval, +// where altkey ≠ key. The proof consists of the altkey-altval pair, including +// varint-encoded length prefixes, followed by the proof that altkey-altval +// is in the tree (as in case 2). +// The verifier proceeds as in case 2 to confirm that altkey-altval is in the snapshot. +// Along the way, it must also check that key and altkey agree at every bit position +// in the path, confirming that the lookup for key would indeed end at altkey instead. +// +// For a tree storing N-bit keys, the longest existence proof is N-1 (bit, sibling hash) pairs, +// while the longest non-existence proof is a key, a value, and N (bit, sibling hash) pairs. +// In both cases the worst case length is dominated by the hashes, about 32N bytes. +// A tree using random keys (for example, using SHA256 hashes as keys) +// would of course never reach this maximum length for any sizable N. +// +// Note that this encoding is considerably more compact than some others. +// In particular, the [Rust akd crate's MembershipProof] includes the inner node key +// and sibling key for each step in the path, tripling the size of the proofs. +// +// [binary trie]: https://en.wikipedia.org/wiki/Trie +// [transparent log]: https://research.swtch.com/tlog +// [varint-encoded]: https://protobuf.dev/programming-guides/encoding/#varints +// [Rust akd crate's MembershipProof]: https://docs.rs/akd/0.12.0/akd/struct.MembershipProof.html package mpt import ( "bytes" "crypto/sha256" + "encoding/binary" "encoding/hex" "errors" "fmt" @@ -49,9 +190,16 @@ type Tree interface { // set the version. Snap(version int64) (Snapshot, error) - // Prove looks up key in the tree and returns a proof - // either of key's value or that key is not present. - // Use [Verify] to retrieve the lookup result. + // Prove looks up key in the tree and returns a claimed + // associated value (if any) and whether the key is present at all, + // along with a proof of those two claimed results. + // Use [Verify] to verify the proof before trusting the claims. + // + // If Prove returns normally (with err == nil), then proof is non-nil, + // although it may be empty. + // + // If Prove returns a non-nil error error, then val is Val{}, + // ok is false, and proof is nil. // // Prove is a read-only operation and can be called // concurrently with other calls to Prove, but not other @@ -60,7 +208,7 @@ type Tree interface { // It is an error to call Prove if Set has been called without // a subsequent call to Snap: in that case, the caller does not // know what the root hash is, so the proof will be unverifiable. - Prove(key Key) (Proof, error) + Prove(key Key) (val Val, ok bool, proof Proof, err error) // Sync flushes all changes from past Set and Snap calls to // the underlying files and then calls the files' Sync methods @@ -258,93 +406,119 @@ func TreeHash(seq iter.Seq[KeyVal]) Hash { // specific snapshot of a Tree. type Proof []byte -// Proof Format -// -// Proofs start with "mptproof", followed by a one-byte tag that determines -// the format of the additional data. The tags are: -// -// - 0: proof of empty tree; no data -// - 1: proof key is in tree; data is value and path -// - 2: proof key in not in tree; data is alt key, value, and path -// -// The proof of an empty tree carries no data; to verify the proof is to check that the -// tree snapshot is the empty tree hash. -// -// The proof of a key being in the tree is the key's value followed by the -// path from that key-value pair up to the tree root. -// For each node along the path, the data contains a one-byte overlap count -// (the number of bits shared by the left and right children of the node) -// and the 32-byte hash of the sibling not on the path. -// Verifying the proof requires computing the leaf hash corresponding to key-value -// and then combining that leaf hash with the overlap counts and sibling hashes, -// eventually producing a root tree hash that must match the tree snapshot. -// -// The proof of a key not being in the tree is an alternate key-value pair -// followed by the path from that key-value pair up to the tree root. -// Verifying the proof requires checking that the alt-key is not equal to the -// target key, then recomputing the tree hash from alt-key-value and path. -// During the recomputation, the verifier must check that for every overlap count -// in the path, the target key and the alt-key agree at that bit position, -// verifying that a search for the target would find the alt-key instead. -const ( - proofMagic = "mptproof" - proofEmpty = proofMagic + "\x00" - proofConfirm = proofMagic + "\x01" - proofDeny = proofMagic + "\x02" -) - var ( - // ErrMalformedProof indicates that a proof is not formatted correctly. - ErrMalformedProof = errors.New("malformed mpt proof") + // ErrInvalidProof indicates that a proof is not valid for the claimed result. + ErrInvalidProof = errors.New("invalid mpt proof") - // ErrMismatchedProof indicates that a proof does not match - // the snapshot and key passed to Verify. - ErrMismatchedProof = errors.New("mismatched mpt proof") + // ErrInvalidLookup indicates that ok is false but val is non-empty. + ErrInvalidLookup = errors.New("invalid mpt lookup result") ) -// Verify verifies that p is a valid proof of a lookup for key in snap, -// returning the proved lookup result (val, ok). -// If the proof is not valid for key in snap, Verify returns a non-nil error. -func Verify(snap Snapshot, key Key, proof Proof) (val Val, ok bool, err error) { - if string(proof) == proofEmpty { +// Verify verifies that p is a valid proof that a lookup for key in snap +// should return the result (val, ok). +// If the proof is not valid, Verify returns a non-nil error. +// +// [VerifyPresent] and [VerifyNotPresent] are convenience functions +// that wrap Verify. +func Verify(snap Snapshot, key, val []byte, ok bool, proof Proof) error { + if !ok && len(val) != 0 { + return ErrInvalidLookup + } + if !ok && len(proof) == 0 { if snap.Hash == emptyTreeHash() { - return Val{}, false, nil + return nil } - return Val{}, false, ErrMismatchedProof + return ErrInvalidProof } - var data []byte - var pkey Key - if data, ok = bytes.CutPrefix(proof, []byte(proofConfirm)); ok && len(data) >= 32 { + var pkey []byte + var h Hash + if ok { pkey = key - val, data = Val(data[:32]), data[32:] - } else if data, ok = bytes.CutPrefix(proof, []byte(proofDeny)); ok && len(data) >= 64 { - pkey, val, data = Key(data[:32]), Val(data[32:64]), data[64:] - if pkey == key { - return Val{}, false, ErrMalformedProof + h = hashLeafVar(key, val) + } else { + var pval []byte + var ok bool + pkey, proof, ok = cutVar(proof) + if !ok { + return ErrInvalidProof + } + pval, proof, ok = cutVar(proof) + if !ok { + return ErrInvalidProof } + if bytes.Equal(pkey, key) { + return ErrInvalidProof + } + h = hashLeafVar(pkey, pval) } - h := hashLeaf(pkey, val) - b := 256 - for len(data) >= 1+32 && int(data[0]) < b { + + b := 1 << 30 + for len(proof) > 0 { + ub, n := binary.Uvarint(proof) + if n <= 0 || ub >= uint64(b) { + break + } + b = int(ub) + proof = proof[n:] + if len(proof) < 32 { + return ErrInvalidProof + } var sib Hash - b, sib, data = int(data[0]), Hash(data[1:1+32]), data[1+32:] - if key.bit(b) != pkey.bit(b) { - return Val{}, false, ErrMalformedProof + sib, proof = Hash(proof[:32]), proof[32:] + if bit(key, b) != bit(pkey, b) { + return ErrInvalidProof } - if key.bit(b) == 0 { + if bit(key, b) == 0 { h = hashInner(b, h, sib) } else { h = hashInner(b, sib, h) } } - if len(data) != 0 || h != snap.Hash { - return Val{}, false, ErrMalformedProof + if len(proof) != 0 || h != snap.Hash { + return ErrInvalidProof + } + return nil +} + +// VerifyPresent is shorthand for [Verify](snap, key[:], val[:], true, proof). +func VerifyPresent(snap Snapshot, key Key, val Val, proof Proof) error { + return Verify(snap, key[:], val[:], true, proof) +} + +// VerifyNotPresent is shorthand for [Verify](snap, key[:], nil, false, proof). +func VerifyNotPresent(snap Snapshot, key Key, proof Proof) error { + return Verify(snap, key[:], nil, false, proof) +} + +// bit returns the n'th bit of the byte slice b, extended with padding. +// A key is padded with a 0x00 byte followed by arbitrarily many 0xFF bytes. +func bit(b []byte, n int) int { + i := n >> 3 + if i < len(b) { + return (int(b[i]) >> (7 - n&7)) & 1 } - if pkey == key { - return val, true, nil + if i == len(b) { + return 0 } - return Val{}, false, nil + return 1 +} + +// hashLeafVar returns the hash of a leaf with a given key and value, +// where key and val are variable-length byte slices. +// The hash is H(0 || len(key) || key || len(val) || val), +// where the lengths are varint-encoded. +func hashLeafVar(key, val []byte) Hash { + h := sha256.New() + h.Write([]byte{0}) + var buf [binary.MaxVarintLen64]byte + n := binary.PutUvarint(buf[:], uint64(len(key))) + h.Write(buf[:n]) + h.Write(key) + n = binary.PutUvarint(buf[:], uint64(len(val))) + h.Write(buf[:n]) + h.Write(val) + return Hash(h.Sum(nil)) } // emptyTreeHash returns the parent hash for a root no child nodes. @@ -354,22 +528,47 @@ func emptyTreeHash() Hash { } // hashLeaf returns the hash of a leaf with a given key and value. +// The hash is H(0 || len(key) || key || len(val) || val), +// where the lengths are varint-encoded. func hashLeaf(key Key, val Val) Hash { - var kv [64]byte - copy(kv[:32], key[:]) - copy(kv[32:64], val[:]) - h := sha256.Sum256(kv[:]) - return h + // 0 tag + varint(32) + 32 + varint(32) + 32 = 1+1+32+1+32 = 67 + var buf [67]byte + buf[0] = 0 + n := 1 + n += binary.PutUvarint(buf[n:], uint64(len(key))) + n += copy(buf[n:], key[:]) + n += binary.PutUvarint(buf[n:], uint64(len(val))) + n += copy(buf[n:], val[:]) + return sha256.Sum256(buf[:n]) } // hashInner returns the hash of an inner node // with the given bit position and left and right child hashes. +// The hash is H(1 || bit || left-hash || right-hash), +// where the bit is varint-encoded. func hashInner(b int, left, right Hash) Hash { - var enc [65]byte - copy(enc[:32], left[:]) - copy(enc[32:64], right[:]) - enc[64] = byte(b) - return sha256.Sum256(enc[:]) + // 1 tag + varint(bit) + 32 + 32; varint(bit) ≤ 2 bytes for bit ≤ 255 + var buf [67]byte + buf[0] = 1 + n := 1 + n += binary.PutUvarint(buf[n:], uint64(b)) + n += copy(buf[n:], left[:]) + n += copy(buf[n:], right[:]) + return sha256.Sum256(buf[:n]) +} + +// cutVar cuts a varint-length-prefixed value from the start of data, +// returning the value and the rest of the data. +func cutVar(data []byte) (value, rest []byte, ok bool) { + x, n := binary.Uvarint(data) + if n <= 0 { + return nil, nil, false + } + data = data[n:] + if uint64(len(data)) < x { + return nil, nil, false + } + return data[:x], data[x:], true } func reduce(s []node) []node { diff --git a/mpt/tree_test.go b/mpt/tree_test.go index b875b93..b9819a6 100644 --- a/mpt/tree_test.go +++ b/mpt/tree_test.go @@ -11,6 +11,7 @@ import ( "fmt" "io" "math/rand" + "os" "runtime/debug" "slices" "strings" @@ -28,38 +29,38 @@ var goldenTrees = []struct { }, { []Key{h("0...0")}, - sha(h("00...0"), h("420...0")), + sha("\x00\x20", h("00...0"), "\x20", h("420...0")), }, { []Key{h("80...0")}, - sha(h("80...0"), h("420...0")), + sha("\x00\x20", h("80...0"), "\x20", h("420...0")), }, { []Key{h("0...0"), h("80...0")}, sha( - sha(h("00...0"), h("420...0")), - sha(h("80...0"), h("420...01")), - "\x00", + "\x01\x00", + sha("\x00\x20", h("00...0"), "\x20", h("420...0")), + sha("\x00\x20", h("80...0"), "\x20", h("420...01")), ), }, { []Key{h("0...0"), h("0010...0")}, sha( - sha(h("0...0"), h("420...0")), - sha(h("0010...0"), h("420...01")), - "\x0b", + "\x01\x0b", + sha("\x00\x20", h("0...0"), "\x20", h("420...0")), + sha("\x00\x20", h("0010...0"), "\x20", h("420...01")), ), }, { []Key{h("0...0"), h("0010...0"), h("80...0")}, sha( + "\x01\x00", sha( - sha(h("0...0"), h("420...0")), - sha(h("0010...0"), h("420...01")), - "\x0b", + "\x01\x0b", + sha("\x00\x20", h("0...0"), "\x20", h("420...0")), + sha("\x00\x20", h("0010...0"), "\x20", h("420...01")), ), - sha(h("80...0"), h("420...02")), - "\x00", + sha("\x00\x20", h("80...0"), "\x20", h("420...02")), ), }, } @@ -336,12 +337,15 @@ func (tt *testTree) get(key Key, val Val, ok bool) { tt.t.Fatalf("Tree.Snap: %v\n\nLog:\n%s", err, &tt.log) } - proof, err := tt.tree.Prove(key) + v, o, proof, err := tt.tree.Prove(key) if err != nil { tt.t.Fatalf("Tree.Prove: %v\n\nLog:\n%s", err, &tt.log) } - - v, o, err := Verify(snap, key, proof) + var vb []byte + if o { + vb = v[:] + } + err = Verify(snap, key[:], vb, o, proof) if err != nil { tt.t.Fatalf("Verify %v: %v\nSnap: %v\nProof: %x\n\nLog:\n%s", key, err, snap, proof, &tt.log) } @@ -411,11 +415,10 @@ func benchmarkProof(b *testing.B, tree Tree, treeSize int) { b.ReportAllocs() for b.Loop() { - proof, err := tree.Prove(key) + _, _, _, err := tree.Prove(key) if err != nil { b.Fatal(err) } - _ = proof } } @@ -465,3 +468,83 @@ func TestPredict(t *testing.T) { }) } + +func TestVerify(t *testing.T) { + file := "testdata/verify.txt" + data, err := os.ReadFile(file) + if err != nil { + t.Fatal(err) + } + + var ( + snap Hash + key, val []byte + ok bool + proof Proof + ) + lines := strings.Split(string(data), "\n") + for i := 0; i < len(lines); i++ { + line := lines[i] + // Join continuation lines (trailing backslash). + for strings.HasSuffix(line, "\\") { + line = line[:len(line)-1] + i++ + if i < len(lines) { + line += lines[i] + } + } + line = strings.TrimSpace(line) + if line == "" || strings.HasPrefix(line, "#") { + continue + } + + cmd, arg, _ := strings.Cut(line, " ") + arg = strings.TrimSpace(arg) + + switch cmd { + case "snap": + b := decodeHex(t, file, i+1, arg) + if len(b) != 32 { + t.Fatalf("line %d: snap must be 32 bytes, got %d", i+1, len(b)) + } + snap = Hash(b) + case "key": + key = decodeHex(t, file, i+1, arg) + case "val": + if arg == "-" { + val = nil + ok = false + } else { + val = decodeHex(t, file, i+1, arg) + ok = true + } + case "proof": + proof = Proof(decodeHex(t, file, i+1, arg)) + case "verify": + want := arg == "true" + result := Verify(Snapshot{Version: 1, Hash: snap}, key, val, ok, proof) + if want && result != nil { + t.Errorf("%s:%d: Verify should succeed but got: %v", file, i+1, result) + } else if !want && result == nil { + t.Errorf("%s:%d: Verify should fail but succeeded", file, i+1) + } + default: + t.Fatalf("%s:%d: unknown directive %q", file, i+1, cmd) + } + } +} + +func decodeHex(t *testing.T, file string, lineno int, s string) []byte { + t.Helper() + if s == "''" { + return nil + } + // Remove spaces and tabs from hex string. + s = strings.ReplaceAll(s, " ", "") + s = strings.ReplaceAll(s, "\t", "") + b, err := hex.DecodeString(s) + if err != nil { + t.Fatalf("%s:%d: bad hex %q: %v", file, lineno, s, err) + } + return b +}