mpt: allow variable-width Key and Val (API and disk format change)

Allow variable-width Key and Val, so that applications are not
locked into 32-byte blobs as the only possible format.

Update disk format to support variable-width keys and values.
The first write of a key reserves space for the number of bytes
in the associated value. Future updates can overwrite that space
as long as the value does not grow in size. If the value does get
larger, then the existing space is abandoned (leaked) and new
space is allocated. That strategy should be fine for now,
since all of our applications have fixed-size values anyway.

Also create and use new testdata/treehash.txt, test vectors
for tree creation and snapshotting.

Author: rsc

mpt/DESIGN.md

@@ -245,14 +245,54 @@ Conceptually, we can stop updates,
 write the current tree memory to a new file,
 delete the old file, and then resume updates,
 now writing patches to the new file.
-It is worth introducing two complications.
+It is worth introducing three complications.
+
 First, we can reuse the old file as the output for the next compaction,
 alternating between a pair of files
 instead of continually deleting and recreating files.
+
 Second, we can let updates proceed concurrently
 with compaction, so that updates aren't blocked
 waiting to write a few hundred gigabytes to disk.
 
+Third, we can trade a few disk I/O per Set or Prove
+operation for reduced memory requirements.
+In this hybrid approach, the leaf nodes (meaning the key and value fields)
+are all stored in a “leaf file” and not stored in memory.
+Writes to the leaf file are still recorded in patch blocks,
+so that after recovery the leaf file is always at least as up to date as the main
+tree memory image. However, writes to the leaf file also happen immediately,
+so after recovery, the leaf file may also contain writes beyond those reflected
+in the main tree memory image. Having a leaf file that is “too new” cannot
+affect the structure of the overall tree, since keys are never changed after a
+node is allocated. However, the leaf file being too new can mean that values
+that are “too new” are recorded for leaf nodes, so the client must recover by
+replaying all the Set operations that may have happened after the point
+where the memory image was recovered. Once those are replayed, the
+memory image and the leaf file will be in sync.
+
+To support the recovery operation, there is a new method Tree.Version:
+
+	// Version returns the version number of the tree's last complete snapshot.
+	// All Set calls made prior to Snap(version) are guaranteed to be
+	// recorded in the tree. However, if exact is false, then the tree may
+	// include the effect of Set calls made after that snapshot.
+	// In that case, to bring the tree into a consistent state, the client is
+	// expected to replay all Set calls up to the next version.
+	Version() (version int64, exact bool)
+
+In this new approach, calling Prove requires around two disk I/Os:
+for a balanced tree, it would be one to read the leaf key and value
+at the end of the lookup, and one to read that node's sibling for inclusion in the proof.
+More precisely, Prove requires one disk I/O for the leaf and one disk I/O
+for each leaf sibling found along the path back to the root.
+Calling set requires around three disk I/Os: the same reads
+needed by Prove as well as one write to update or create a leaf.
+
+In exchange for these few disk I/Os per operation, the memory
+requirements are reduced to 48 bytes per record and become independent
+of key and value size.
+
 ## Memory Format {#mem}
 
 The memory format of the tree must be suitable for writing to disk
@@ -269,38 +309,39 @@ The tree memory starts with a header with the form:
 
 	version  [ 8 bytes]
 	dirty    [ 1 byte]
-	pad      [ 1 byte]
+	exact    [ 1 byte]
 	root     [ 6 bytes]
 	hash     [32 bytes]
-	nodes    [ 8 bytes]
 
 All numbers are stored in big-endian order
 for legibility when reading hex dumps.
 
  - “version” is a number for clients to use to match the
    tree contents to a position in the underlying transparent log.
+ - “exact” is a boolean indicating whether the tree includes
+   only the changes made before Snap(version).
+   If false, it may contain more changes made after that snapshot.
  - “root” is a pointer to the tree's root node,
    represented as a 48-bit byte offset within the
    tree memory.
- - “nodes” field counts the number of nodes (leaves)
-   stored in the tree.
  - “hash” is the Merkle hash of the tree root.
    When “dirty” is set, the hash is stale and needs to be recomputed.
- - “pad” pads “root” to a 16-bit boundary and “hash” and “nodes”
-   to a 64-bit boundary.
 
 The header is immediately followed by a sequence of Patricia nodes,
 each with the form:
 
-	key      [32 bytes]
-	val      [32 bytes]
 	bit      [ 1 byte]
 	dirty    [ 1 byte]
-	pad      [ 2 bytes]
 	left     [ 6 bytes]
 	right    [ 6 bytes]
+	leaf     [ 6 bytes]
 	ihash    [32 bytes]
 
+In a standard implementation, each Patricia node represents both one leaf node
+and one inner node. In this format, the leaf data is stored in a separate
+parallel file. The nodes we are considering only store inner node data,
+although each node still serves as both inner node and leaf in the tree structure.
+
 Remember that each Patricia node represents both one leaf node
 and one inner node.
 
@@ -343,10 +384,12 @@ Snapshots are still amortized O(1) but not an actual O(1).
 If the snapshot operations caused problematic latency hiccups,
 this lazy recomputation could be abandoned.
 
-Notice that a Patricia node takes 112 bytes,
-so a 2-billion node tree requires about 224 GB of memory,
+Notice that a Patricia node takes 52 bytes,
+so a 2-billion node tree requires about 104 GB of memory,
 well within the 512 GB we allotted ourselves on our
 “reasonably configured server”.
+(There is another 130 GB of disk for the leaf file,
+for the common case of 32-byte keys.)
 The actual memory for the tree is obtained directly
 using the operating system, not from the Go heap.
 Using _mmap_(2), we can reserve a very large amount
@@ -361,7 +404,7 @@ when loading a tree from disk.
 
 ## File Format {#format}
 
-A file consists of the magic string `"mpt tree\n\x00\x00\x00\x00\x00\x00\x00"`
+A file consists of the magic string `"mpt tree v2\n\x00\x00\x00\x00\x00"`
 followed by a sequence of variable-length frames.
 Each frame has the form:
 
@@ -394,12 +437,15 @@ a valid tree.
 The second and subsequent frames in the file each hold
 a patch block, which holds one or more mutations of the form:
 
-	offset   [varint]
-	N        [varint]
-	data     [N bytes]
+	offset+leaf [varint]
+	N           [varint]
+	data        [N bytes]
 
-That mutation says to write `data` of length `N` at
-`offset` in the tree memory.
+The offset+leaf field is a varint encoding offset<<1 + leaf,
+where leaf is a boolean indicating whether the update is for
+the tree memory or the leaf memory.
+The mutation says to write `data` of length `N` at
+`offset` in the specified memory.
 
 There is no guarantee that a file ends after a valid patch block frame.
 If a frame was only partially written before a process or system crash,
@@ -408,6 +454,13 @@ We do this by reading as many valid (checksum-matching) frames
 as possible from the file and stopping at EOF or when we reach
 a frame that is truncated or does not have a valid checksum.
 
+The leaf file data consists of a sequence of leaves.
+The leaf nodes are allocated in parallel with the Patricia nodes,
+so that Patricia node N corresponds to leaf N.
+Since each leaf may be variable length,
+the leaf pointer in the Patricia node records where
+the leaf begins on disk.
+
 ## Compaction {#compaction}
 
 When the current disk file holding a tree has grown too large,
@@ -428,9 +481,9 @@ being obsoleted.
 One approach would be to pause all tree updates,
 write the tree to the new file, and then continue
 updates, writing patches to the new file.
-If the tree is 224 GB,
+If the tree is 96 GB,
 then even if we can write at a relatively fast 10 GB/s,
-that would be a 22-second pause.
+that would be a 10-second pause.
 Instead, we can allow tree updates to proceed
 concurrently with compaction.
 
@@ -486,15 +539,22 @@ The improvement is possible because we keep all the data in memory at all times.
 ## Speed {#speed}
 
 On my circa-2023 home server with 128 GB of RAM
-and an NVMe disk using LVM encryption, storing 834 million hashes
-takes about 250 minutes, or about 55,000 Set operations per second.
-This is with constant disk compaction, and I suspect something in my
+and an NVMe disk using LVM encryption,
+using an earlier version of this format that did not have the leaf file
+and stored keys and values in the Patricia nodes,
+storing 834 million hashes takes about 250 minutes,
+or about 55,000 Set operations per second.
+This is with continuous disk compaction, and I suspect something in my
 kernel stack of slowing disk I/O.
 
+Prove operations run in microseconds.
+
+Snap is effectively free.
+
 A “lazy hash” optimization that delays recomputing all inner node hashes
-is delayed until the Sync operation can avoid spending time
+is delayed until the Snap operation can avoid spending time
 computing hashes that will be overwritten by a subsequent Set,
-but it dramatically increases the latency of Sync.
+but it dramatically increases the latency of Snap.
 More important than not computing the hashes is not writing
 them to disk, especially for the somewhat special case of writing all new entries
 when populating a new tree.
@@ -504,12 +564,9 @@ being written, so the disk file never reaches twice the memory size.
 In that case, the 834 million hashes can be written in 45 minutes,
 followed by a 7 minute sync, or about 260,000 Set operations second.
 
-A limited lazy hash that is lazy only up to a fixed number of
-Set operations may be the best of both worlds.
-
-Prove operations run in microseconds.
-
-Snap is effectively free.
+Although Snap with lazy hashes is still amortized O(1),
+callers that want Snap to run in limited time should limit
+the number of Set calls they make between Snaps.
 
 ## Recovery {#recovery}
 
@@ -525,42 +582,3 @@ all of the Set calls before Snap(V) has been retained, and some of the Set
 calls between Snap(V) and Snap(V+1) may also have been retained.
 It suffices to replay all the Set operations between Snap(V) and Snap(V+1)
 and then Snap(V+1) to get a consistent tree.
-
-## Hybrid Approach
-
-The approach described so far is the original in-memory approach.
-It is tagged as mpt v0.1.0.
-
-This section describes a hybrid approach implemented in later versions.
-The hybrid approach trades a constant number of disk I/O per Set or Prove
-operation for reduced memory requirements. In the hybrid approach, the
-leaf nodes (meaning the key and value fields) are all stored in a “leaf file”
-not stored in memory. Writes to the leaf file are still recorded in patch blocks,
-so that after recovery the leaf file is always at least as up to date as the main
-tree memory image. However, writes to the leaf file also happen immediately,
-so after recovery, the leaf file may also contain writes beyond those reflected
-in the main tree memory image. Having a leaf file that is “too new” cannot
-affect the structure of the overall tree, since keys are never changed after a
-node is allocated. However, the leaf file being too new can mean that values
-that are “too new” are recorded for leaf nodes, so the client must recover by
-replaying all the Set operations that may have happened after the point
-where the memory image was recovered. Once those are replayed, the
-memory image and the leaf file will be in sync.
-
-To support the recovery operation, there is a new method Tree.Version:
-
-	// Version returns the version number of the tree's last complete snapshot.
-	// All Set calls made prior to Snap(version) are guaranteed to be
-	// recorded in the tree. However, if exact is false, then the tree may
-	// include the effect of Set calls made after that snapshot.
-	// In that case, to bring the tree into a consistent state, the client is
-	// expected to replay all Set calls up to the next version.
-	Version() (version int64, exact bool)
-
-In this new approach, calling Prove requires two disk I/Os: one to read the
-leaf key and value at the end of the lookup, and one to read that node's sibling
-for inclusion in the proof. Calling set requires three disk I/Os: the same two reads
-needed by Prove as well as one write to update or create a leaf.
-
-In exchange for these two or three disk I/Os per operation, the memory
-requirements are reduced from 112 bytes per record to 48 bytes per record.