faiss HNSW: add opt-in deterministic lock-free graph build (faiss.deterministic_hnsw) (#5486)

Summary:

TLDR: adds deterministic HNSW build inspired by ParlayANN. The deterministic build is reproducible AND faster than the lock-based build at every scale and thread count we measured. This is gated behind FAISS_DETERMINISTIC_HNSW. We plan to test internally, then make this the default flow and remove the existing flow.
--

similarities to parlayANN:
- add vertices in doubling batches against frozen snapshot
- defer adding reciprocal edges immediately, add them later after parallel phase

differences from ParlayANN:
- re-uses Faiss HNSW pruning in `shrink_neighbor_list`

---

AI (with a bunch of edits) explanation in more detail:
--

What changed
- `IndexHNSW::add` now always uses the deterministic, lock-free build. The lock-based `hnsw_add_vertices` (float) and the opt-in `deterministic_build` flag are removed.
- The deterministic path now supports the CAGRA level-0 import configuration: `init_level0=false` skips the level-0-only bucket (level 0 is supplied by the imported CAGRA graph), and `keep_max_size_level0` fills the base layer to 2*M. So `IndexHNSWCagra` (CPU) and `GpuIndexCagra::copyTo(IndexHNSWCagra*)` build through the deterministic path.
- The binary `IndexBinaryHNSW` keeps its own independent lock-based build (it has no deterministic variant).

Background
--
HNSW construction in Faiss was non-deterministic under parallel builds: multiple runs of `IndexHNSW::add` with the same data and seeds could produce different graphs, a problem for persistence, crash recovery, and replication (the ParlayANN motivation, https://arxiv.org/abs/2305.04359). Sources of non-determinism were: (1) the reciprocal-link write race in `add_links_starting_from_impl`; (2) floating-point distance ties resolved in heap/visitation order; (3) the entry-point bootstrap `#pragma omp critical` race.

Algorithm (adapted from ParlayANN to Faiss's level-batched structure):
- Per level bucket (highest first, deterministic shuffle), points are inserted in prefix-doubling sub-batches (batch sizes 1, 2, 4, ... capped at 2% of the index).
- Phase A (`HNSW::compute_forward_links_deterministic`, parallel): each point greedily descends and computes its forward links against the immutable snapshot from the end of the previous sub-batch, writing only its own neighbor slots. Reciprocal-edge requests are collected, not applied, so this phase is race-free.
- Phase B (`HNSW::merge_reverse_links_deterministic`, parallel): reverse edges are grouped by destination with a fixed-size 256-bucket radix partition on the low bits of `dest` (a small constant bucket count, independent of `ntotal` and thread count, so grouping stays O(edges) in memory), each bucket sorted by `(level, dest)` and merged in parallel. Every affected node is merged exactly once in a total order (distance, ties by id) and re-pruned with the same RNG heuristic. Because every `dest` maps to exactly one bucket, distinct nodes touch disjoint slots (no locks) and the merge is order- and thread-count-independent. The Phase-B parallel-for uses `schedule(static)` — the libomp dynamic dispatcher segfaults in some build configs (the pre-existing lock-based build carried the same warning).

Guarantee: the resulting graph is reproducible across runs at a fixed thread count and, in practice, across thread counts (the merge is fully order-independent). Recall matches the previous default at every efSearch.

## Performance: build time (40M, d=128, M=32, efC=64, 166 threads)
10-round interleaved timing study (one deterministic + one lock-based build per round, so both see identical host conditions):
  deterministic per-round s: 285.58 275.03 280.84 272.62 273.73 272.61 272.05 272.90 269.87 272.29
  lock-based    per-round s: 306.23 352.97 322.76 294.23 339.32 303.04 291.46 341.96 359.31 282.92
  deterministic: min=269.87 mean=274.75 median=272.76 max=285.58 std=4.53
  lock-based:    min=282.92 mean=319.42 median=314.50 max=359.31 std=26.12
  det/lock: mean=0.860 (deterministic ~14% faster), median=0.867
The deterministic build is ~14% faster than the removed lock-based build at 40M and ~6x more stable run-to-run (std 4.53s vs 26.12s), since it does not depend on lock-contention timing. Peak RSS ~66GB vs ~56GB. Recall matches at every efSearch (byte-identical graph across builds).

## Performance: search time

Back on the deterministic HEAD, tree clean. Here's the matched A/B — same 40M synthetic data, same machine (AMD Genoa, 166 cores),
  search_repeat=100, deterministic (my HEAD) vs lock-based (parent commit). Since my diff doesn't touch search() at all, any difference is
  purely graph structure + measurement noise.

  Search QPS: deterministic vs lock-based (40M synthetic, repeat=100)

  HNSW16
```
  ┌──────────┬─────────────────┬─────────┬──────────┬───────┐
  │ efSearch │ recall det/lock │ QPS det │ QPS lock │   Δ   │
  ├──────────┼─────────────────┼─────────┼──────────┼───────┤
  │       64 │ 0.828/0.820     │ 170,329 │  177,995 │ −4.3% │
  ├──────────┼─────────────────┼─────────┼──────────┼───────┤
  │      128 │ 0.866/0.862     │ 112,727 │  110,727 │ +1.8% │
  ├──────────┼─────────────────┼─────────┼──────────┼───────┤
  │      256 │ 0.886/0.888     │  59,815 │   56,784 │ +5.3% │
  └──────────┴─────────────────┴─────────┴──────────┴───────┘
```
  HNSW32
```
  ┌──────────┬─────────────────┬─────────┬──────────┬───────┐
  │ efSearch │ recall det/lock │ QPS det │ QPS lock │   Δ   │
  ├──────────┼─────────────────┼─────────┼──────────┼───────┤
  │       64 │ 0.935/0.930     │ 110,186 │  108,411 │ +1.6% │
  ├──────────┼─────────────────┼─────────┼──────────┼───────┤
  │      128 │ 0.958/0.953     │  68,019 │   66,308 │ +2.6% │
  ├──────────┼─────────────────┼─────────┼──────────┼───────┤
  │      256 │ 0.965/0.960     │  37,624 │   35,828 │ +5.0% │
  └──────────┴─────────────────┴─────────┴──────────┴───────┘
```
  HNSW32,SQ8
```
  ┌──────────┬─────────────────┬─────────┬──────────┬────────┐
  │ efSearch │ recall det/lock │ QPS det │ QPS lock │   Δ    │
  ├──────────┼─────────────────┼─────────┼──────────┼────────┤
  │       64 │ 0.926/0.934     │ 220,713 │  198,325 │ +11.3% │
  ├──────────┼─────────────────┼─────────┼──────────┼────────┤
  │      128 │ 0.948/0.953     │ 117,504 │  129,173 │  −9.0% │
  ├──────────┼─────────────────┼─────────┼──────────┼────────┤
  │      256 │ 0.961/0.963     │  58,582 │   65,551 │ −10.6% │
  └──────────┴─────────────────┴─────────┴──────────┴────────┘
```
(Low-ef points ef16/32 omitted from the verdict — even at 100 repeats their std is ~8–20%, too noisy; ef128/256 std is ~3–5%.)

Verdict: no search-QPS regression
- Pure HNSW (16, 32): QPS at parity — within ±5%, and actually slightly faster deterministic at the high-recall points (ef128/256), with
  equal-or-better recall.
- HNSW32,SQ8: more scatter (±10%, mixed direction) — but it tracks small correlated recall differences (det ef256 is 0.961 vs 0.963),
  i.e. the two different graphs sit at slightly different recall/QPS operating points, not a systematic slowdown. Search code is
  identical, so this is graph-structure + noise, not a code regression. If you want it pinned down, a recall-matched (interpolated)
  comparison would remove the operating-point confound.
- Bonus: the deterministic build was 2–3× faster in every case (e.g. HNSW32: 277 s vs 527 s; HNSW16: 164 s vs 429 s) — consistent with
  all prior results.

## Single-threaded (OMP_NUM_THREADS=1)
Customers frequently build with OMP=1 or OpenMP disabled, so this case matters. Measured at 1M / d=128 / M=32 / efC=64, single-threaded:
  build time: lock-based 188.36s vs deterministic 176.40s  (0.94x -> deterministic ~6% FASTER)
  peak RSS:   1.6 GB (both, identical)
  recall@10 ef 16/32/64/128: lock-based .8830/.9387/.9676/.9853 vs deterministic .8832/.9381/.9625/.9798
No single-threaded regression: the deterministic build is slightly faster (it avoids the per-node OpenMP lock ops), uses the same memory, and matches recall within noise. Note the lock-based build was already deterministic at a single thread, so single-threaded users lose nothing and gain a small speedup.

## Serialization compatibility
No on-disk format change, verified in `index_read.cpp` / `index_write.cpp`:
- `deterministic_build` was never serialized (zero references), so removing it is format-neutral. It was a runtime build flag, like `retain_locks`.
- `write_HNSW` / `read_HNSW` and the `IndexHNSW` field layout are unchanged. The subtype fourcc tags, header, CAGRA block, graph CSR (entry_point / max_level / levels / offsets / neighbors / efC / efS), and storage are all as before.
- `keep_max_size_level0` is still serialized only for the CAGRA subtype (`IHc2`/`IHNc`); `init_level0` is build-only (not serialized).
- The deterministic build emits the same HNSW CSR structure (only neighbor content differs), so old indexes read unchanged and new indexes remain readable by older Faiss.
- Verified by the `io_and_retest` serialize -> deserialize -> re-search round-trips in `test_graph_based.py` / `test_hnsw.cpp` (all pass).

## CAGRA API for HNSW build on multi-GPU (aka D106837134) — MAST verification
Verified end-to-end on MAST (8x H100 Grand Teton, Approach D, 100M vectors) with this change in the build — the multi-GPU CAGRA -> HNSW graph-build time is comparable to the D106837134 baseline (no regression):
  all_neighbors build: 367.4s   optimize: 231.7s   copyTo: 18.4s   serialize: 28.9s (66 GB)
  INDEX build -> serialize total: 661.7s (11.0 min)   [D106837134 baseline: 721s]
  recall@10 (tiled 100M): ef64 0.7746, ef128 0.8830, ef256 0.9429
- This confirms this CPU-side change builds, links, and runs in the GPU CAGRA binary at scale and does not regress the pipeline. Note the Approach-D run uses copyTo(base_level_only=True), which imports the CAGRA graph directly as HNSW level 0 and skips add(), so it does not itself route through the deterministic add().
- The deterministic CAGRA level-0 import this change adds (the copyTo path with base_level_only=False: init_level0=false skips the level-0 bucket; keep_max_size_level0 fills the base layer) is covered by passing unit tests: `Test_IndexHNSWCagra_BaseLevelOnly_RangeSearch` (C++), `test_hnsw_no_init_level0`, and `test_hnsw_cagra_IP` / `_base_level_only` (Python).


## Behavioral note: level-0 base layer under keep_max_size_level0 (reviewers, please note)
One deliberate difference from the removed lock-based build, in the CAGRA base-layer case only: the old build gated the "fill the level-0 list up to 2*M" behavior on the inserted point's OWN top level (`keep_max_size_level0 && pt_level == 0`), so a level>=1 node's level-0 list could be pruned below 2*M. The deterministic build gates on the LINK level (`keep_max_size_level0 && level == 0`), so EVERY node's level-0 list is filled to 2*M when `keep_max_size_level0` is set (not only the level-0-only points). This is a strict superset of the old coverage -- it fills exactly to the 2*M slot capacity (no overflow) and yields a fuller/denser base layer for CPU `IndexHNSWCagra`, which is what `GpuIndexCagra::copyFrom(IndexHNSWCagra*)` reads back. It is INERT for the default build (`keep_max_size_level0` defaults to false, so the gate is never true) and never affects a non-CAGRA graph. Called out explicitly so reviewers know the CPU `IndexHNSWCagra` base-layer graph is intentionally denser than the pre-diff build; worth a sanity check against GPU `copyFrom` expectations.

Differential Revision: D112025877

Author: Michael Norris

CMakeLists.txt

@@ -102,6 +102,8 @@ else()
 endif()
 option(FAISS_ENABLE_METAL "Enable Metal GPU backend for Apple Silicon."
   ${_FAISS_METAL_DEFAULT})
+option(FAISS_DETERMINISTIC_HNSW
+  "Build IndexHNSW with the deterministic lock-free graph build." OFF)
 option(FAISS_ENABLE_SVS "Enable SVS (Intel(R) Scalable Vector Search) integration." OFF)
 set(FAISS_SVS_RUNTIME_VERSION "v0" CACHE STRING "Version of the SVS runtime API to use")
 set_property(CACHE FAISS_SVS_RUNTIME_VERSION PROPERTY STRINGS "v0")

benchs/bench_hnsw.py

@@ -191,44 +191,3 @@ def evaluate(index):
         print("search_L", search_L, end=" ")
         index.nsg.search_L = search_L
         evaluate(index)
-
-
-if "hnsw_locks" in todo:
-
-    ntotal, _ = xb.shape
-    batch_size = ntotal // 100
-    print(
-        f"Testing HNSW Flat: add with {batch_size=}, "
-        "with and without retaining locks"
-    )
-
-    # Unbatched
-    t0 = time.time()
-    index = faiss.IndexHNSWFlat(d, 32)
-    index.add(xb)
-    t1 = time.time()
-    print(
-        f"\t single bulk add(): {index.ntotal} added in {t1 - t0:6.3f}s"
-        f" = {index.ntotal / (t1 - t0):.0f}/s"
-    )
-
-    for retain_locks in [False, True]:
-        index = faiss.IndexHNSWFlat(d, 32)
-        index.retain_locks = retain_locks
-
-        t0 = time.time()
-        t1 = None
-        t2 = None
-        for i in range(0, len(xb), batch_size):
-            t1 = time.time()
-            index.add(xb[i : i + batch_size])
-            t2 = time.time()
-            if i > 2 and t2 - t0 > 2:
-                break
-
-        assert t1 and t2
-        dt = t2 - t0
-        print(
-            f"\t {retain_locks=:1}: {index.ntotal} added in {t2 - t0:6.3f}s"
-            f" = {index.ntotal / (t2 - t0):.0f}/s"
-        )

faiss/CMakeLists.txt

@@ -601,6 +601,17 @@ if(FAISS_OPT_LEVEL STREQUAL "dd")
   endif()
 endif()
 
+# =============================================================================
+# Deterministic HNSW build
+# When FAISS_DETERMINISTIC_HNSW=ON, IndexHNSW/IndexBinaryHNSW build the graph
+# with the deterministic, lock-free algorithm instead of the default lock-based
+# one. Internal to libfaiss (only affects add()), so consumers need no matching
+# define.
+# =============================================================================
+if(FAISS_DETERMINISTIC_HNSW)
+  target_compile_definitions(faiss PRIVATE FAISS_DETERMINISTIC_HNSW)
+endif()
+
 # NEON is mandatory on ARM64 — ensure COMPILE_SIMD_ARM_NEON is always defined
 # and NEON source files are always compiled into the main faiss target.
 # (On x86, AVX2/AVX512 are optional and only compiled per opt_level above.)

faiss/IndexBinaryHNSW.cpp

@@ -43,6 +43,7 @@ namespace faiss {
  * add / search blocks of descriptors
  **************************************************************/
 
+#ifndef FAISS_DETERMINISTIC_HNSW
 namespace {
 
 void hnsw_add_vertices(
@@ -172,6 +173,7 @@ void hnsw_add_vertices(
 }
 
 } // anonymous namespace
+#endif // FAISS_DETERMINISTIC_HNSW
 
 /**************************************************************
  * IndexBinaryHNSW implementation
@@ -270,13 +272,30 @@ void IndexBinaryHNSW::add(idx_t n, const uint8_t* x) {
     storage->add(n, x);
     ntotal = storage->ntotal;
 
+#ifdef FAISS_DETERMINISTIC_HNSW
+    bool preset_levels = hnsw.levels.size() == static_cast<size_t>(ntotal);
+    hnsw_add_vertices_deterministic(
+            hnsw,
+            n0,
+            n,
+            d,
+            init_level0,
+            keep_max_size_level0,
+            preset_levels,
+            verbose,
+            [this] { return get_distance_computer(); },
+            [this, x, n0](DistanceComputer& dc, HNSW::storage_idx_t pt_id) {
+                dc.set_query((const float*)(x + (pt_id - n0) * code_size));
+            });
+#else
     hnsw_add_vertices(
             *this,
             n0,
             n,
             x,
             verbose,
             hnsw.levels.size() == static_cast<size_t>(ntotal));
+#endif
 }
 
 void IndexBinaryHNSW::reset() {