feat(segmenter): use k=2 blocked bloom filter to improve performance

Author: mhx

doc/mkdwarfs.md

@@ -176,13 +176,24 @@ Most other options are concerned with compression tuning:
   and `mkdwarfs` will be slightly slower and use more memory.
 
 - `--bloom-filter-size`=[*category*`::`]*value*:
-  The segmenting algorithm uses a bloom filter to determine quickly if
-  there is *no* match at a given position. This will filter out more than
-  90% of bad matches quickly with the default bloom filter size. The default
-  is pretty much where the sweet spot lies. If you have copious amounts of
-  RAM and CPU power, feel free to increase this by one or two and you *might*
-  be able to see some improvement. If your system is tight on memory, then
-  decreasing this will potentially save a few MiBs.
+  The segmenting algorithm uses a bloom filter to determine quickly if there
+  is *no* match at a given position. This will usually filter out more than
+  97% of bad matches quickly with the default bloom filter size. The default
+  of 4 is very close to the sweet spot for the default settings. Note that
+  the actual bloom filter memory size is a function of the block size (`-S`),
+  the window step size (`-w`), and the lookback size (`-B`). This option is
+  merely another 2^n multiplier on top of that. With every increase of this
+  option value by 1, the bloom filter memory size doubles, but the number of
+  false positives is reduced by a factor of around 3.5. However, increasing the
+  value also means the bloom filter is less likely to fit into the CPU cache.
+  It's a trade-off and it's hardware-dependent. The impact of the bloom filter
+  is more relevant as you increase the lookback size, as in addition to a
+  "global" bloom filter, each lookback block has its own "local" bloom filter,
+  so more filters need to be checked if a candidate isn't rejected by the
+  global filter. This means that as you increase the lookback size, it may
+  pay off to nudge this value up to 5 (if you have enough RAM to spare).
+  It is usually not recommended to drop this below 4 unless you're under
+  pressure to save memory, as it will noticeably hurt segmenter speed.
 
 - `-L`, `--memory-limit=auto|`*value*:
   Approximately how much memory you want `mkdwarfs` to use during filesystem

src/writer/segmenter.cpp

@@ -221,20 +221,19 @@ class alignas(64) bloom_filter {
     }
   }
 
-  DWARFS_FORCE_INLINE void add(size_t ix) {
+  DWARFS_FORCE_INLINE void add(size_t const ix) {
     assert(bits_);
     auto bits = bits_;
     BOOST_ALIGN_ASSUME_ALIGNED(bits, sizeof(bits_type));
-    bits[(ix >> index_shift) & index_mask_] |= static_cast<bits_type>(1)
-                                               << (ix & value_mask);
+    bits[(ix >> index_shift) & index_mask_] |= mask(ix);
   }
 
-  DWARFS_FORCE_INLINE bool test(size_t ix) const {
+  DWARFS_FORCE_INLINE bool test(size_t const ix) const {
     assert(bits_);
     auto bits = bits_;
     BOOST_ALIGN_ASSUME_ALIGNED(bits, sizeof(bits_type));
-    return bits[(ix >> index_shift) & index_mask_] &
-           (static_cast<bits_type>(1) << (ix & value_mask));
+    auto const m = mask(ix);
+    return (bits[(ix >> index_shift) & index_mask_] & m) == m;
   }
 
   // size in bits
@@ -258,6 +257,33 @@ class alignas(64) bloom_filter {
   uint64_t memory_usage() const { return size_ / 8; }
 
  private:
+  DWARFS_FORCE_INLINE static constexpr bits_type mask(size_t const ix) {
+    if constexpr (sizeof(bits_type) >= 8) {
+      //
+      // Make this a blocked bloom filter with k=2. This significantly
+      // increases the rejection rate and stat-sig improves performance
+      // of the segmenter:
+      //
+      //   ------------------------------------------------------------
+      //   block size | lookback | rejection rate | true positive rate
+      //              |          | k=1   | k=2    | k=1   | k=2
+      //   ------------------------------------------------------------
+      //     64 MiB   |     1    | 96.6% | 99.1%  | 0.48% | 1.89%
+      //     16 MiB   |     1    | 96.4% | 99.0%  | 0.35% | 1.26%
+      //     64 MiB   |     8    | 94.5% | 98.5%  | 1.05% | 3.88%
+      //   ------------------------------------------------------------
+      //
+      //   64 MiB, lookback=1:
+      //     k=1 -> 15.976s ± 0.021s
+      //     k=2 -> 15.277s ± 0.017s
+      //
+      return (static_cast<bits_type>(1) << (ix & value_mask)) |
+             (static_cast<bits_type>(1) << ((ix >> 26) & value_mask));
+    } else {
+      return static_cast<bits_type>(1) << (ix & value_mask);
+    }
+  }
+
   DWARFS_FORCE_INLINE bits_type const* cbegin() const { return bits_; }
   DWARFS_FORCE_INLINE bits_type const* cend() const {
     return bits_ + (size_ >> index_shift);