A commutative hasher that supports parallel and out-of-order computation

Makefile

@@ -38,6 +38,8 @@ test:
 	cargo check -p geo_filters --features test-support
 	cargo check -p geo_filters --features serde
 	cargo check -p geo_filters --features evaluation
+	# Check that commutative_hasher works with the serde feature
+	cargo test -p commutative_hasher --features serde
 
 .PHONY: test-ignored
 test-ignored:

README.md

@@ -5,6 +5,7 @@ A collection of useful algorithms written in Rust. Currently contains:
 - [`geo_filters`](crates/geo_filters): probabilistic data structures that solve the [Distinct Count Problem](https://en.wikipedia.org/wiki/Count-distinct_problem) using geometric filters.
 - [`bpe`](crates/bpe): fast, correct, and novel algorithms for the [Byte Pair Encoding Algorithm](https://en.wikipedia.org/wiki/Large_language_model#BPE) which are particularly useful for chunking of documents.
 - [`bpe-openai`](crates/bpe-openai): Fast tokenizers for OpenAI token sets based on the `bpe` crate.
+- [`commutative_hasher`](crates/commutative_hasher): hash function that handles receiving data out-of-order.
 - [`consistent-choose-k`](crates/consistent-choose-k): constant time consistent hashing algorithms with support for replication and bounded load.
 - [`hash-sorted-map`](crates/hash-sorted-map): a hash map whose groups are ordered by hash prefix, enabling efficient sorted-order iteration and linear-time merging.
 - [`sparse-ngrams`](crates/sparse-ngrams): fast sparse n-gram extraction from byte slices. Selects variable-length n-grams (2–8 bytes) deterministically using bigram frequency priorities, suitable for substring search indexes.

crates/commutative_hasher/Cargo.toml

@@ -0,0 +1,39 @@
+[package]
+name = "commutative_hasher"
+version = "0.1.0"
+edition = "2024"
+rust-version = "1.96.0"
+description = "Hashing function that supports processing data out-of-order"
+repository = "https://github.com/github/rust-gems"
+homepage = "https://github.com/github/rust-gems/tree/main/crates/commutative_hasher"
+documentation = "https://docs.rs/commutative_hasher"
+readme = "README.md"
+license = "MIT"
+keywords = ["hashing", "commutative", "digest", "elliptic-curve", "ristretto"]
+categories = ["algorithms", "concurrency"]
+
+[lib]
+crate-type = ["lib"]
+bench = false
+
+[features]
+default = []
+serde = ["dep:serde"]
+
+[dependencies]
+curve25519-dalek = { version = "5.0", features = ["digest"] }
+dataview = "1.0"
+digest = "0.11"
+hex = "0.4"
+parking_lot = "0.12"
+serde = { version = "1.0", features = ["derive"], optional = true }
+thiserror = "2.0"
+
+[dev-dependencies]
+itertools = "0.15"
+rand = "0.10"
+rayon = "1.12"
+rstest = "0.26"
+rstest_reuse = "0.7"
+serde_json = "1.0"
+sha2 = "0.11"

crates/commutative_hasher/README.md

@@ -0,0 +1,111 @@
+# Commutative Hasher
+
+An order-independent hasher that hashes a large byte stream in any order, from any number of threads, and always gets
+the same digest.
+
+The stream is split into fixed-size blocks. Each block is hashed, mapped to a point on the Ristretto group over
+Curve25519, and all the points are summed. Because point addition is commutative and associative, blocks can be hashed
+in any order, in parallel, and combined later.
+
+To prevent collisions due to reordering, each block is hashed together with its byte offset and length, so a block only
+ever contributes the same point at the same position in the stream. This means the block size chosen is critical; for
+any data stream larger than one block, the hash value will be different for different block sizes. Each block must only
+be added once; adding a block multiple times will result in a different hash value.
+
+## Motivation
+
+Verifying a large object usually means a linear pass with a conventional hash: bytes must arrive in order, on one
+thread. That is a poor fit when data is uploaded or downloaded as ranged parts, sharded across workers, or written by a
+pipeline that finishes chunks out of order.
+
+`commutative_hasher` lets each worker hash the part it has access to, and lets the results be merged into a single
+digest for the whole object, without buffering the object or serializing the work.
+
+## Usage
+
+Add `commutative_hasher` and `sha2 = 0.11` to your `Cargo.toml`.
+
+The hashers are generic over any `digest::Digest` with a 64-byte output, since mapping into the Ristretto group requires
+64 bytes of hash input. `Sha512` is the usual choice, but others would work too.
+
+It is critical to make a future-proof choice of block size. The block size is difficult to change after the fact, as the
+same data hashed with different block sizes will produce different results if the data is larger than the block size.
+
+### Parallel
+
+Use `ParallelHasher` to hash a stream of data out-of-order from multiple threads. A block size must be provided at
+construction, and each block is hashed with something like `Sha512` first. All data must be provided to `update` in
+multiples of the block size. Larger blocks are more efficient for the computation but limit the granularity of the
+processed data.
+
+```rust
+use std::num::NonZeroUsize;
+use commutative_hasher::ParallelHasher;
+use rayon::prelude::*;
+use sha2::Sha512;
+
+const BLOCK_SIZE: NonZeroUsize = NonZeroUsize::new(4).unwrap();
+
+fn hash_parallel() {
+    let data = b"thisthatmoreverylong";
+    let hasher = ParallelHasher::<Sha512>::new(BLOCK_SIZE);
+
+    // Parts may be hashed in any order, on any thread.
+    data.par_chunks(8)
+        .enumerate()
+        .try_for_each(|(i, chunk)| hasher.update(i * 8, chunk))
+        .expect("hashing to succeed");
+
+    // Same digest as the sequential example.
+    assert_eq!(
+        hasher.finalize().hex_digest(),
+        "9258f9de43401cd6e8f55545754b84ac58257ec7779723c9790a986daab18206"
+    );
+}
+```
+
+### Sequential
+
+`SequentialHasher` generates the same hashes as `ParallelHasher` but does not require that the data be provided in
+multiples of the block size.
+
+```rust
+use std::num::NonZeroUsize;
+use commutative_hasher::SequentialHasher;
+use sha2::Sha512;
+
+const BLOCK_SIZE: NonZeroUsize = NonZeroUsize::new(4).unwrap();
+
+fn hash_sequential() {
+    let mut hasher = SequentialHasher::<Sha512>::new(BLOCK_SIZE);
+    hasher.update(b"this");
+    hasher.update(b"that");
+    hasher.update(b"moreverylong");
+    let digest = hasher.finalize();
+
+    assert_eq!(
+        digest.hex_digest(),
+        "9258f9de43401cd6e8f55545754b84ac58257ec7779723c9790a986daab18206"
+    );
+
+    // Or, for data you already have in memory:
+    let digest = SequentialHasher::<Sha512>::digest_from_bytes(BLOCK_SIZE, b"thisthatmoreverylong");
+}
+```
+
+### Digests
+
+A `CommutativeHashDigest` is fundamentally a 32-byte compressed Ristretto point. It can be converted to bytes, a hex
+string, serialized/deserialized, and stored inline in packed on-disk structures.
+
+```rust
+use commutative_hasher::CommutativeHashDigest;
+
+fn digest() {
+    ...
+    let bytes: [u8; 32] = digest.to_bytes();
+    let hex: String = digest.hex_digest();
+    let parsed = CommutativeHashDigest::decode_hex_digest(&hex).expect("decode to succeed");
+    assert_eq!(parsed, digest);
+}
+```