|
| 1 | +# Commutative Hasher |
| 2 | + |
| 3 | +An order-independent hasher that hashes a large byte stream in any order, from any number of threads, and always get the |
| 4 | +same digest. |
| 5 | + |
| 6 | +The stream is split into fixed-size blocks. Each block is hashed, mapped to a point on the Ristretto group over |
| 7 | +Curve25519, and all the points are summed. Because point addition is commutative and associative, blocks can be hashed |
| 8 | +in any order, in parallel, and combined later. |
| 9 | + |
| 10 | +To prevent collisions due to reordering, each block is hashed together with its byte offset and length, so a block only |
| 11 | +ever contributes the same point at the same position in the stream. This means the block size chosen is critical; for |
| 12 | +any data stream larger than one block, the hash value will be different for different block sizes. Each block must only |
| 13 | +be added once; adding a block multiple times will result in a different hash value. |
| 14 | + |
| 15 | +## Motivation |
| 16 | + |
| 17 | +Verifying a large object usually means a linear pass with a conventional hash: bytes must arrive in order, on one |
| 18 | +thread. That is a poor fit when data is uploaded or downloaded as ranged parts, sharded across workers, or written by a |
| 19 | +pipeline that finishes chunks out of order. |
| 20 | + |
| 21 | +`commutative_hasher` lets each worker hash the part it has access to, and lets the results be merged into a single |
| 22 | +digest for the whole object, without buffering the object or serializing the work. |
| 23 | + |
| 24 | +## Usage |
| 25 | + |
| 26 | +Add `commutative_hasher` and `sha2 = 0.11` to your `Cargo.toml`. |
| 27 | + |
| 28 | +The hashers are generic over any `digest::Digest` with a 64-byte output, since mapping into the Ristretto group requires |
| 29 | +64 bytes of hash input. `Sha512` is the usual choice, but others would work too. |
| 30 | + |
| 31 | +It is critical to make a future-proof choice of block size. The block size is difficult to change after the fact, as the |
| 32 | +same data hashed with different block sizes will produce different results if the data is larger than the block size. |
| 33 | + |
| 34 | +### Parallel |
| 35 | + |
| 36 | +Use `ParallelHasher` to hash a stream of data out-of-order from multiple threads. A block size must be provided at |
| 37 | +construction, and each block is hashed with something like `Sha512` first. All data must be provided to `update` in |
| 38 | +multiples of the block size. Larger blocks are more efficient for the computation but limit the granularity of the |
| 39 | +processed data. |
| 40 | + |
| 41 | +```rust |
| 42 | +use std::num::NonZeroUsize; |
| 43 | +use commutative_hasher::ParallelHasher; |
| 44 | +use rayon::prelude::*; |
| 45 | +use sha2::Sha512; |
| 46 | + |
| 47 | +const BLOCK_SIZE: NonZeroUsize = 4; |
| 48 | + |
| 49 | +fn hash_parallel() { |
| 50 | + let data = b"thisthatmoreverylong"; |
| 51 | + let hasher = ParallelHasher::<Sha512>::new(BLOCK_SIZE); |
| 52 | + |
| 53 | + // Parts may be hashed in any order, on any thread. |
| 54 | + data.par_chunks(8) |
| 55 | + .enumerate() |
| 56 | + .try_for_each(|(i, chunk)| hasher.update(i * 8, chunk)) |
| 57 | + .expect("hashing to succeed"); |
| 58 | + |
| 59 | + // Same digest as the sequential example. |
| 60 | + assert_eq!( |
| 61 | + hasher.finalize().hex_digest(), |
| 62 | + "9258f9de43401cd6e8f55545754b84ac58257ec7779723c9790a986daab18206" |
| 63 | + ); |
| 64 | +} |
| 65 | +``` |
| 66 | + |
| 67 | +### Sequential |
| 68 | + |
| 69 | +`SequentialHasher` generates the same hashes as `ParallelHasher` but does not require that the data be provided in |
| 70 | +multiples of the block size. |
| 71 | + |
| 72 | +```rust |
| 73 | +use std::num::NonZeroUsize; |
| 74 | +use commutative_hasher::SequentialHasher; |
| 75 | +use sha2::Sha512; |
| 76 | + |
| 77 | +const BLOCK_SIZE: NonZeroUsize = 4; |
| 78 | + |
| 79 | +fn hash_sequential() { |
| 80 | + let mut hasher = SequentialHasher::<Sha512>::new(BLOCK_SIZE); |
| 81 | + hasher.update(b"this"); |
| 82 | + hasher.update(b"that"); |
| 83 | + hasher.update(b"moreverylong"); |
| 84 | + let digest = hasher.finalize(); |
| 85 | + |
| 86 | + assert_eq!( |
| 87 | + digest.hex_digest(), |
| 88 | + "9258f9de43401cd6e8f55545754b84ac58257ec7779723c9790a986daab18206" |
| 89 | + ); |
| 90 | + |
| 91 | + // Or, for data you already have in memory: |
| 92 | + let digest = SequentialHasher::<Sha512>::digest_from_bytes(BLOCK_SIZE, b"thisthatmoreverylong"); |
| 93 | +} |
| 94 | +``` |
| 95 | + |
| 96 | +### Digests |
| 97 | + |
| 98 | +A `CommutativeHashDigest` is fundamentally a 32-byte compressed Ristretto point. It can be converted to bytes, a hex |
| 99 | +string, serialized/deserialized, and stored inline in packed on-disk structures. |
| 100 | + |
| 101 | +```rust |
| 102 | +use commutative_hasher::CommutativeHashDigest; |
| 103 | + |
| 104 | +fn digest() { |
| 105 | + ... |
| 106 | + let bytes: [u8; 32] = digest.to_bytes(); |
| 107 | + let hex: String = digest.hex_digest(); |
| 108 | + let parsed = CommutativeHashDigest::decode_hex_digest(&hex).expect("decode to succeed"); |
| 109 | + assert_eq!(parsed, digest); |
| 110 | +} |
| 111 | +``` |
0 commit comments