Merge pull request #43 from hoxxep/v3-micro-mismatch

Rapidhash v4.0.0: V3 micro/nano mismatch, performance improvements, refactoring

Author: hoxxep

.github/workflows/rust.yaml

@@ -22,7 +22,7 @@ jobs:
         run: cargo test --no-fail-fast
 
   test-no-std:
-    name: "Test (no_std)"
+    name: "Test (no-std)"
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v5
@@ -43,33 +43,50 @@ jobs:
       - name: Compile and run tests
         run: cargo test -p rapidhash --no-fail-fast --lib --tests --features=unsafe
 
-  test-all-features:
-    name: "Test (all features)"
+  test-nightly-all-features:
+    name: "Test (nightly, all features)"
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v5
       - uses: dtolnay/rust-toolchain@stable
+        with:
+          toolchain: nightly
       - name: Rust dependency cache
         uses: Swatinem/rust-cache@v2
       - name: Compile and run tests
-        run: cargo test -p rapidhash --no-fail-fast --lib --tests --all-features
+        run: cargo +nightly test -p rapidhash --no-fail-fast --lib --tests --all-features
+
+  test-nightly-no-std:
+    name: "Test (nightly, no-std)"
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v5
+      - uses: dtolnay/rust-toolchain@stable
+        with:
+          toolchain: nightly
+      - name: Rust dependency cache
+        uses: Swatinem/rust-cache@v2
+      - name: Compile and run tests (no_std)
+        run: cargo +nightly test -p rapidhash --no-fail-fast --lib --tests --no-default-features
 
   test-msrv:
     name: "Test (MSRV)"
     runs-on: ubuntu-latest
     env:
-      RUST_VERSION: 1.77.0
+      RUST_VERSION: 1.71.0
     steps:
       - uses: actions/checkout@v5
       - name: Rust dependency cache
         uses: Swatinem/rust-cache@v2
       - name: Install Rust (rustup, ${{ env.RUST_VERSION }})
         shell: bash
         run: rustup update ${{ env.RUST_VERSION }} --no-self-update && rustup default ${{ env.RUST_VERSION }} && cargo -V && rustc -V
-      - name: Compile and run tests (std)
-        run: cargo test -p rapidhash --no-fail-fast --all-features
-      - name: Compile and run tests (no_std)
-        run: cargo test -p rapidhash --no-fail-fast --lib --tests --no-default-features
+      - name: Compile and run tests (no-std)
+        working-directory: rapidhash-msrv
+        run: cargo test --no-fail-fast --no-default-features
+      - name: Compile and run tests (all features)
+        working-directory: rapidhash-msrv
+        run: cargo test --no-fail-fast --all-features
 
   test-c:
     name: "Test (C implementations)"

CHANGELOG.md

@@ -1,11 +1,32 @@
 # Changelog
 
+## 4.0.0 (20250901)
+
+https://github.com/hoxxep/rapidhash/pull/43
+
+### Breaking changes
+- **`rapidhash::v3` micro/nano output change:** input lengths 5-7 were mismatching the intended C++ V3 output. The C++ rapidhash V3 has been [yanked and re-released as V3](https://github.com/Nicoshev/rapidhash/issues/33) to fix the bug, and this rust implementation will follow. This changes the hash outputs for `rapidhash_v3_micro_inline` and `rapidhash_v3_nano_inline` for inputs of size 5, 6, and 7 bytes.
+- **`RapidBuildHasher` renamed and refactored** to `SeedableState`.
+- **`RapidHasher<'s>` new lifetime parameter** added to support user-defined secrets via `SeedableState`.
+- **`RapidHashMap` and `RapidHashSet` moved to crate top level** for convenience. The top level uses the `fast::` variants, and the `quality::` and `inner::` hashmaps have been removed. They can still be built manually using `inner::RandomState` if required. The `fast::` collection variants have been deprecated to be removed in a future major release.
+
+### Additions
+- **`nightly` feature** which improves str hashing performance by omitting the `0xFF` suffix write and adds likely/unlikely hints.
+- **`SeedableState`**: a hasher builder which can be seeded with fixed or user-defined secrets. This replaces `RapidBuildHasher`, but still defaults to random seeds. It is slightly slower than `RandomState`.
+
+### Performance improvements
+- **Bounds check elision**: Improved `RapidHasher` by eliding extra bounds checks in some cases by using `assert_unchecked`.
+- **Likely/unlikely hints**: Added stable likely/unlikely hints in various places to ensure small inputs are favoured.
+
+### MSRV
+- **MSRV reduced to 1.71.0** from 1.77.0 by removing const usage of `first_chunk`.
+
 ## 3.1.0 (20250809)
 
-## Performance improvements
+### Performance improvements
 - Improved `RapidHasher` small string hashing performance by 1.5-15% depending on the benchmark, by reducing the small string hashing code size and allowing the compiler to inline more. Performance was also improved on big-endian platforms by reading native-endian bytes. The portable hashers (`rapidhash::v3` etc. modules) are unaffected by this change. [#37](https://github.com/hoxxep/rapidhash/pull/37)
 
-## Fixes
+### Fixes
 - Fixed compilation on targets without atomic pointers. [#38](https://github.com/hoxxep/rapidhash/issues/38), [#39](https://github.com/hoxxep/rapidhash/pull/39)
 
 ## 3.0.0 (20250730)

Cargo.toml

@@ -1,6 +1,7 @@
 [workspace]
 resolver = "2"
 members = [ "rapidhash", "rapidhash-bench", "rapidhash-bench-wasm", "rapidhash-c"]
+exclude = [ "rapidhash-msrv" ]
 
 [workspace.dependencies]
 rapidhash = { path = "rapidhash", default-features = false }

README.md

@@ -4,9 +4,8 @@ A rust implementation of [rapidhash](https://github.com/Nicoshev/rapidhash), the
 
 - **High quality**, the fastest hash passing all tests in the SMHasher and SMHasher3 benchmarks. Collision-based study showed a collision probability that's close to ideal.
 - **Very fast**, the fastest passing hash in SMHasher3. Significant peak throughput improvement over wyhash and foldhash. Fastest platform-independent hash. Fastest const hash.
-- **Platform independent**, works on all platforms, no dependency on machine-specific vectorized or cryptographic hardware instructions. Optimised for both AMD64 and AArch64.
+- **Platform independent and no-std compatible**, works on all platforms, no dependency on machine-specific vectorized or cryptographic hardware instructions. Optimised for both AMD64 and AArch64.
 - **Memory safe**, when the `unsafe` feature is disabled (default). This implementation has also been fuzz-tested with `cargo fuzz`.
-- **No dependencies and no-std compatible** when disabling default features.
 - **Official successor to wyhash** with improved speed, quality, and compatibility.
 - **Run-time and compile-time hashing** as the hash implementation is fully `const`.
 - **Idiomatic** `std::hash::Hasher` compatible hasher for `HashMap` and `HashSet`.
@@ -21,12 +20,12 @@ A rust implementation of [rapidhash](https://github.com/Nicoshev/rapidhash), the
 Following rust's `std::hash` traits, the underlying hash function may change between minor versions, and is only suitable for in-memory hashing. These types are optimised for speed and minimal DoS resistance, available in the `rapidhash::fast` and `rapidhash::quality` flavours.
 
 - `RapidHasher`: A `std::hash::Hasher` compatible hasher that uses the rapidhash algorithm.
-- `RapidHashBuilder`: A `std::hash::BuildHasher` for initialising the hasher with the default seed and secrets.
 - `RandomState`: A `std::hash::BuildHasher` for initialising the hasher with a random seed and secrets.
-- `RapidHashMap` and `RapidHashSet`: Helper types for using `RapidHasher` with `HashMap` and `HashSet`.
+- `SeedableState`: A `std::hash::BuildHasher` for initialising the hasher with the custom seed and secrets.
+- `RapidHashMap` and `RapidHashSet`: Helper types for using `fast::RandomState` with `HashMap` and `HashSet`.
 
 ```rust
-use rapidhash::fast::RapidHashMap;
+use rapidhash::RapidHashMap;
 
 // A HashMap using RapidHasher for fast in-memory hashing.
 let mut map = RapidHashMap::default();
@@ -35,11 +34,11 @@ map.insert("key", "value");
 
 ```rust
 use std::hash::BuildHasher;
-use rapidhash::quality::RapidBuildHasher;
+use rapidhash::quality::SeedableState;
 
 // Using the RapidHasher directly for in-memory hashing.
-let hasher = RapidBuildHasher::default();
-assert_eq!(hasher.hash_one(b"hello world"), 9938606849760368330);
+let hasher = SeedableState::fixed();
+assert_eq!(hasher.hash_one(b"hello world"), 3348275917668072623);
 ```
 
 ### Portable Hashing
@@ -94,6 +93,7 @@ echo "example" | rapidhash --v3
 - `rand`: Enables using the `rand` library to more securely initialise `RandomState`. Includes the `rand` crate dependency.
 - `rng`: Enables `RapidRng`, a fast, non-cryptographic PRNG based on rapidrng. Includes the `rand_core` crate dependency.
 - `unsafe`: Uses unsafe pointer arithmetic to skip some unnecessary bounds checks for a small 3-4% performance improvement.
+- `nightly`: Enable nightly-only features for even faster hashing, such as overriding `Hasher::write_str` and likely hints.
 
 ## Benchmarks
 
@@ -241,6 +241,13 @@ The benchmarks have been compiled with and without `-C target-cpu=native` on a v
 - **Comparison to gxhash**: gxhash achieves its high throughput by using AES instructions and consistently outperforms the other accelerated hashers (ahash, th1a, xxhash3_64). It's a great hash function, but is not a portable hash function, requiring `target-cpu=native` or specific feature flags to compile. Gxhash is a great choice for applications that can guarantee the availability of AES instructions and mostly hash strings, but rapidhash may be preferred for hashing tuples and structs, or by libraries that aim to support a wide range of platforms.
 - The default rust hasher (SipHasher) unexpectedly appears to run consistently faster _without_ `target-cpu=native` on various x86 and ARM chips.
 - Benchmark your own use case, with your real world dataset! We suggest experimenting with different hash functions to see which one works best for your use case. Rapidhash is great for fast general-purpose hashing in libraries and applications that only need minimal DoS resistance, but certain hashers will outperform for specific use cases.
+- We recommend using `lto = "fat"` and `codegen-units = 1` in your `Cargo.toml` release and bench profiles to ensure consistent inlining, application performance, and benchmarking results. For example:
+    ```toml
+    [profile.release]
+    opt-level = 3
+    lto = "fat"
+    codegen-units = 1
+    ```
 
 </details>
 
@@ -260,14 +267,14 @@ C++ compatibility is presented in `rapidhash::v1`, `rapidhash::v2`, and `rapidha
 Rapidhash V3 is the recommended, fastest, and most recent version of the hash. Streaming is only possible with the rapidhash V3 algorithm. Others are provided for backwards compatibility.
 
 ### In-Memory Hashing
-Rust hasing traits (`RapidHasher`, `RapidBuildHasher`, etc.) are implemented in `rapidhash::fast`, `rapidhash::quality`, and `rapidhash::inner` modules. These are not guaranteed to give a consistent hash output between platforms, compiler versions, or crate versions as the rust `Hasher` trait [is not suitable](https://github.com/hoxxep/portable-hash/?tab=readme-ov-file#whats-wrong-with-the-stdhash-traits) for portable hashing.
+Rust hashing traits (`RapidHasher`, `RandomState`, etc.) are implemented in `rapidhash::fast`, `rapidhash::quality`, and `rapidhash::inner` modules. These are not guaranteed to give a consistent hash output between platforms, compiler versions, or crate versions as the rust `Hasher` trait [is not suitable](https://github.com/hoxxep/portable-hash/?tab=readme-ov-file#whats-wrong-with-the-stdhash-traits) for portable hashing.
 
 - Use `rapidhash::fast` for optimal hashing speed with a slightly lower hash quality. Best for most datastructures such as HashMap and HashSet usage.
 - Use `rapidhash::quality` where statistical hash quality is the priority, such as HyperLogLog or MinHash algorithms.
 - Use `rapidhash::inner` to set advanced parameters to configure the hash function specifically to your use case.
 
 ## Crate Versioning
-The minimum supported Rust version (MSRV) is 1.77.0.
+The minimum supported Rust version (MSRV) is 1.71.0.
 
 The rapidhash crate follows the following versioning scheme:
 - Major for breaking API changes and MSRV version bumps or any changes to `rapidhash_v*` method output.

docs/CHEATSHEET.md

@@ -33,13 +33,13 @@ cd rapidhash
 ## Tests
 
 # Run tests
-cargo test --all-features
+cargo +nightly test --all-features
 
 # Run tests, for no_std with std = off and unsafe = off
 cargo test --no-default-features --lib
 
 # Check MSRV
-cargo +1.77.0 test --all-features
+cargo +1.77.0 test --features=std,rand,rng
 ```
 
 ## Running benchmarks
@@ -50,36 +50,42 @@ Benchmarks are run using `cargo-criterion` in the `rapidhash-bench` crate to sep
 cd rapidhash-bench
 
 # Run all benchmarks (assumes cargo-criterion is installed)
-RUSTFLAGS="-C target-cpu=native" cargo criterion --bench bench --all-features
+RUSTFLAGS="-C target-cpu=native" cargo criterion --bench bench --features bench
 
 # Run all benchmarks, but unsafe=disabled
 RUSTFLAGS="-C target-cpu=native" cargo criterion --bench bench --features bench
 
+# Run all benchmarks, with nightly
+RUSTFLAGS="-C target-cpu=native" cargo +nightly criterion --bench bench --features bench,nightly
+
 # Run the realworld benchmark, which is a modification of the foldhash benchmarks
-RUSTFLAGS="-C target-cpu=native" cargo criterion --bench realworld --all-features
+RUSTFLAGS="-C target-cpu=native" cargo criterion --bench realworld --features bench
 
 # Run quality tests across various hash functions
-RUSTFLAGS="-C target-cpu=native" cargo bench --bench quality --all-features
+RUSTFLAGS="-C target-cpu=native" cargo bench --bench quality --features bench
 
 # Run iai-callgrind to compare instruction counts and L1 cache misses
 # Requires: valgrind
-RUSTFLAGS="-C target-cpu=native" cargo bench --bench iai-callgrind --all-features
+RUSTFLAGS="-C target-cpu=native" cargo bench --bench iai-callgrind --features bench
 
 # Use cargo-instruments to diagnose performance
 # Requires: cargo-instruments and MacOS
-RUSTFLAGS="-C target-cpu=native" cargo instruments -t time --profile=bench --bench realworld --features bench,unsafe -- --bench hashonly-struuid-rapidhash-v2
+RUSTFLAGS="-C target-cpu=native" cargo instruments -t time --profile=bench --bench realworld --features bench -- --bench hashonly-struuid-rapidhash-v2
 
 # Benchmark WASM targets, which will automatically build the WASM target
-RUSTFLAGS="-C target-cpu=native" cargo criterion --bench wasm --all-features
+RUSTFLAGS="-C target-cpu=native" cargo criterion --bench wasm --features bench
+
+# Benchmark WASM targets, nightly
+RUSTFLAGS="-C target-cpu=native" cargo +nightly criterion --bench wasm --features nightly
 ```
 
 ## Fuzzing
 ```shell
 # fuzz the raw rapidhash method. (assumes cargo-fuzz is installed)
-cargo +nightly fuzz run --features unsafe rapidhash
+cargo +nightly fuzz run rapidhash
 
 # fuzz the RapidHasher struct with std::hash::Hasher write and finish calls.
-cargo +nightly fuzz run --features unsafe rapidhasher
+cargo +nightly fuzz run rapidhasher
 
 # use AFL fuzzing. (assumes cargo-afl is installed)
 cargo afl fuzz -i in -o out target/debug/afl_rapidhash

docs/generate_charts.py

@@ -66,18 +66,27 @@ def draw_hash():
             ("rapidhash_cc_v1", "r"),
         ]
 
-    if "--small" in sys.argv:
+    if "--small" in sys.argv or "--medium" in sys.argv:
         hash_settings = [
             ("rapidhash-f", "b"),
             ("foldhash-f", "y"),
+#             ("fxhash", "r"),
+        ]
+
+    if "--small" in sys.argv and "--raw" in sys.argv:
+        hash_settings = [
+            ("rapidhash_cc_v3", "y"),
+            ("rapidhash_cc_v3_1", "g"),
         ]
 
     hash_names = [hash_function for hash_function, _ in hash_settings]
 
     # also available: 65536, 524288000
     sizes = [2, 8, 16, 25, 50, 64, 80, 160, 256, 350, 1024, 4096, ]
     if "--small" in sys.argv:
-        sizes = list(range(0, 257))
+        sizes = list(range(0, 300))
+    if "--medium" in sys.argv:  # 50B increments up to 4kB
+        sizes = list(range(0, 40 * 100, 50))
 
     latency_data = []
     throughput_data = []
@@ -92,6 +101,8 @@ def draw_hash():
         prefix = "str"
         if "--small" in sys.argv:
             prefix = "small"
+        if "--medium" in sys.argv:
+            prefix = "medium"
 
         for size in sizes:
             latency, throughput = load_latest_measurement_file("hash", hash_function, f"{prefix}_{size}")
@@ -102,22 +113,31 @@ def draw_hash():
         throughput_data.append(throughput_row)
 
         u64_measurement = "u64"
+        s64k_measurement = "str_65536"
         if "--raw" in sys.argv:
             u64_measurement = "str_8"
 
+        if "--small" in sys.argv:
+            u64_measurement = "small_8"
+            s64k_measurement = "small_256"
+
+        if "--medium" in sys.argv:
+            u64_measurement = "medium_100"
+            s64k_measurement = "medium_19000"
+
         latency, throughput = load_latest_measurement_file("hash", hash_function, u64_measurement)
         latency_data_u64.append(latency)
         throughput_data_u64.append(throughput)
 
-        latency, throughput = load_latest_measurement_file("hash", hash_function, "str_65536")
+        latency, throughput = load_latest_measurement_file("hash", hash_function, s64k_measurement)
         latency_data_64k.append(latency)
         throughput_data_64k.append(throughput)
 
     fig, axs = plt.subplots(2, 2, figsize=(12, 8), dpi=300)
 
     for i, (hash_function, color) in reversed(list(enumerate(hash_settings))):
         linestyle = "--" if hash_function.endswith("-f") else "-"
-        linewidth = 1.0 if "--small" in sys.argv else 0.5
+        linewidth = 1.0 if "--small" in sys.argv or "--medium" in sys.argv else 0.5
 
         axs[0, 0].plot(sizes, latency_data[i], label=hash_function, color=color, linestyle=linestyle, linewidth=linewidth)
         axs[0, 1].plot(sizes, throughput_data[i], label=hash_function, color=color, linestyle=linestyle, linewidth=linewidth)
@@ -137,13 +157,13 @@ def draw_hash():
         axs[1, 1].bar(hash_function, throughput_data_64k[i], color=color, edgecolor=edgecolor, hatch=hatchstyle, zorder=3)
 
     labels = sizes
-    if "--small" in sys.argv:
+    if "--small" in sys.argv or "--medium" in sys.argv:
         labels = sizes[::20]
 
     axs[0, 0].set_title("Latency (byte stream)")
     axs[0, 0].set_xlabel("Input size (bytes)")
     axs[0, 0].set_ylabel("Latency (ns)")
-    if "--small" not in sys.argv:
+    if not ("--small" in sys.argv or "--medium" in sys.argv):
         axs[0, 0].set_xscale("log", base=2)
         axs[0, 0].set_yscale("log", base=10)
     axs[0, 0].set_xticks(labels)
@@ -152,19 +172,29 @@ def draw_hash():
     axs[0, 1].set_title("Throughput (byte stream)")
     axs[0, 1].set_xlabel("Input size (bytes)")
     axs[0, 1].set_ylabel("Throughput (GB/s)")
-    if "--small" not in sys.argv:
+    if not ("--small" in sys.argv or "--medium" in sys.argv):
         axs[0, 1].set_xscale("log", base=2)
         axs[0, 1].set_yscale("log", base=10)
     axs[0, 1].set_xticks(labels)
     axs[0, 1].set_xticklabels(labels, rotation=90, ha="right")
 
-    axs[1, 0].set_title("Throughput (u64)")
+    if "--small" in sys.argv:
+        axs[1, 0].set_title("Throughput (bytes, 8B)")
+    elif "--medium" in sys.argv:
+        axs[1, 0].set_title("Throughput (bytes, 100B)")
+    else:
+        axs[1, 0].set_title("Throughput (u64)")
     axs[1, 0].set_ylabel("Throughput (M Items/s)")
     axs[1, 0].set_xticks(range(len(hash_names)))
     axs[1, 0].set_xticklabels(hash_names, rotation=45, ha="right")
     axs[1, 0].grid(True, zorder=0, color="gainsboro")
 
-    axs[1, 1].set_title("Throughput (bytes, 64kB)")
+    if "--small" in sys.argv:
+        axs[1, 1].set_title("Throughput (bytes, 256B)")
+    if "--medium" in sys.argv:
+        axs[1, 1].set_title("Throughput (bytes, 19kB)")
+    else:
+        axs[1, 1].set_title("Throughput (bytes, 64kB)")
     axs[1, 1].set_ylabel("Throughput (GB/s)")
     axs[1, 1].set_xticks(range(len(hash_names)))
     axs[1, 1].set_xticklabels(hash_names, rotation=45, ha="right")