faiss HNSW: make graph construction deterministic by default (remove lock-based build) (facebookresearch#5486)

Summary:

TLDR: makes the deterministic HNSW graph build the default (and only) float build path, and removes the legacy lock-based one. Inspired by ParlayANN. The deterministic build is reproducible AND faster than the lock-based build at every scale and thread count we measured.
--

similarities to parlayANN:
- add vertices in doubling batches against frozen snapshot
- defer adding reciprocal edges immediately, add them later after parallel phase

differences from ParlayANN:
- original ParlanANN targets flat graphs like Vamana
- re-uses Faiss HNSW pruning in `shrink_neighbor_list`

---

AI (with a bunch of edits) explanation in more detail:
--

What changed
- `IndexHNSW::add` now always uses the deterministic, lock-free build. The lock-based `hnsw_add_vertices` (float) and the opt-in `deterministic_build` flag are removed.
- The deterministic path now supports the CAGRA level-0 import configuration: `init_level0=false` skips the level-0-only bucket (level 0 is supplied by the imported CAGRA graph), and `keep_max_size_level0` fills the base layer to 2*M. So `IndexHNSWCagra` (CPU) and `GpuIndexCagra::copyTo(IndexHNSWCagra*)` build through the deterministic path.
- The binary `IndexBinaryHNSW` keeps its own independent lock-based build (it has no deterministic variant).

Background
--
HNSW construction in Faiss was non-deterministic under parallel builds: multiple runs of `IndexHNSW::add` with the same data and seeds could produce different graphs, a problem for persistence, crash recovery, and replication (the ParlayANN motivation, https://arxiv.org/abs/2305.04359). Sources of non-determinism were: (1) the reciprocal-link write race in `add_links_starting_from_impl`; (2) floating-point distance ties resolved in heap/visitation order; (3) the entry-point bootstrap `#pragma omp critical` race.

Algorithm (adapted from ParlayANN to Faiss's level-batched structure):
- Per level bucket (highest first, deterministic shuffle), points are inserted in prefix-doubling sub-batches (batch sizes 1, 2, 4, ... capped at 2% of the index).
- Phase A (`HNSW::compute_forward_links_deterministic`, parallel): each point greedily descends and computes its forward links against the immutable snapshot from the end of the previous sub-batch, writing only its own neighbor slots. Reciprocal-edge requests are collected, not applied, so this phase is race-free.
- Phase B (`HNSW::merge_reverse_links_deterministic`, parallel): reverse edges are grouped by destination with a fixed-size 256-bucket radix partition on the low bits of `dest` (a small constant bucket count, independent of `ntotal` and thread count, so grouping stays O(edges) in memory), each bucket sorted by `(level, dest)` and merged in parallel. Every affected node is merged exactly once in a total order (distance, ties by id) and re-pruned with the same RNG heuristic. Because every `dest` maps to exactly one bucket, distinct nodes touch disjoint slots (no locks) and the merge is order- and thread-count-independent. The Phase-B parallel-for uses `schedule(static)` — the libomp dynamic dispatcher segfaults in some build configs (the pre-existing lock-based build carried the same warning).

Guarantee: the resulting graph is reproducible across runs at a fixed thread count and, in practice, across thread counts (the merge is fully order-independent). Recall matches the previous default at every efSearch.

## Performance: build time (40M, d=128, M=32, efC=64, 166 threads)
10-round interleaved timing study (one deterministic + one lock-based build per round, so both see identical host conditions):
  deterministic per-round s: 285.58 275.03 280.84 272.62 273.73 272.61 272.05 272.90 269.87 272.29
  lock-based    per-round s: 306.23 352.97 322.76 294.23 339.32 303.04 291.46 341.96 359.31 282.92
  deterministic: min=269.87 mean=274.75 median=272.76 max=285.58 std=4.53
  lock-based:    min=282.92 mean=319.42 median=314.50 max=359.31 std=26.12
  det/lock: mean=0.860 (deterministic ~14% faster), median=0.867
The deterministic build is ~14% faster than the removed lock-based build at 40M and ~6x more stable run-to-run (std 4.53s vs 26.12s), since it does not depend on lock-contention timing. Peak RSS ~66GB vs ~56GB. Recall matches at every efSearch (byte-identical graph across builds).

## Performance: search time

Back on the deterministic HEAD, tree clean. Here's the matched A/B — same 40M synthetic data, same machine (AMD Genoa, 166 cores),
  search_repeat=100, deterministic (my HEAD) vs lock-based (parent commit). Since my diff doesn't touch search() at all, any difference is
  purely graph structure + measurement noise.

  Search QPS: deterministic vs lock-based (40M synthetic, repeat=100)

  HNSW16
```
  ┌──────────┬─────────────────┬─────────┬──────────┬───────┐
  │ efSearch │ recall det/lock │ QPS det │ QPS lock │   Δ   │
  ├──────────┼─────────────────┼─────────┼──────────┼───────┤
  │       64 │ 0.828/0.820     │ 170,329 │  177,995 │ −4.3% │
  ├──────────┼─────────────────┼─────────┼──────────┼───────┤
  │      128 │ 0.866/0.862     │ 112,727 │  110,727 │ +1.8% │
  ├──────────┼─────────────────┼─────────┼──────────┼───────┤
  │      256 │ 0.886/0.888     │  59,815 │   56,784 │ +5.3% │
  └──────────┴─────────────────┴─────────┴──────────┴───────┘
```
  HNSW32
```
  ┌──────────┬─────────────────┬─────────┬──────────┬───────┐
  │ efSearch │ recall det/lock │ QPS det │ QPS lock │   Δ   │
  ├──────────┼─────────────────┼─────────┼──────────┼───────┤
  │       64 │ 0.935/0.930     │ 110,186 │  108,411 │ +1.6% │
  ├──────────┼─────────────────┼─────────┼──────────┼───────┤
  │      128 │ 0.958/0.953     │  68,019 │   66,308 │ +2.6% │
  ├──────────┼─────────────────┼─────────┼──────────┼───────┤
  │      256 │ 0.965/0.960     │  37,624 │   35,828 │ +5.0% │
  └──────────┴─────────────────┴─────────┴──────────┴───────┘
```
  HNSW32,SQ8
```
  ┌──────────┬─────────────────┬─────────┬──────────┬────────┐
  │ efSearch │ recall det/lock │ QPS det │ QPS lock │   Δ    │
  ├──────────┼─────────────────┼─────────┼──────────┼────────┤
  │       64 │ 0.926/0.934     │ 220,713 │  198,325 │ +11.3% │
  ├──────────┼─────────────────┼─────────┼──────────┼────────┤
  │      128 │ 0.948/0.953     │ 117,504 │  129,173 │  −9.0% │
  ├──────────┼─────────────────┼─────────┼──────────┼────────┤
  │      256 │ 0.961/0.963     │  58,582 │   65,551 │ −10.6% │
  └──────────┴─────────────────┴─────────┴──────────┴────────┘
```
(Low-ef points ef16/32 omitted from the verdict — even at 100 repeats their std is ~8–20%, too noisy; ef128/256 std is ~3–5%.)

Verdict: no search-QPS regression
- Pure HNSW (16, 32): QPS at parity — within ±5%, and actually slightly faster deterministic at the high-recall points (ef128/256), with
  equal-or-better recall.
- HNSW32,SQ8: more scatter (±10%, mixed direction) — but it tracks small correlated recall differences (det ef256 is 0.961 vs 0.963),
  i.e. the two different graphs sit at slightly different recall/QPS operating points, not a systematic slowdown. Search code is
  identical, so this is graph-structure + noise, not a code regression. If you want it pinned down, a recall-matched (interpolated)
  comparison would remove the operating-point confound.
- Bonus: the deterministic build was 2–3× faster in every case (e.g. HNSW32: 277 s vs 527 s; HNSW16: 164 s vs 429 s) — consistent with
  all prior results.

## Single-threaded (OMP_NUM_THREADS=1)
Customers frequently build with OMP=1 or OpenMP disabled, so this case matters. Measured at 1M / d=128 / M=32 / efC=64, single-threaded:
  build time: lock-based 188.36s vs deterministic 176.40s  (0.94x -> deterministic ~6% FASTER)
  peak RSS:   1.6 GB (both, identical)
  recall@10 ef 16/32/64/128: lock-based .8830/.9387/.9676/.9853 vs deterministic .8832/.9381/.9625/.9798
No single-threaded regression: the deterministic build is slightly faster (it avoids the per-node OpenMP lock ops), uses the same memory, and matches recall within noise. Note the lock-based build was already deterministic at a single thread, so single-threaded users lose nothing and gain a small speedup.

## Serialization compatibility
No on-disk format change, verified in `index_read.cpp` / `index_write.cpp`:
- `deterministic_build` was never serialized (zero references), so removing it is format-neutral. It was a runtime build flag, like `retain_locks`.
- `write_HNSW` / `read_HNSW` and the `IndexHNSW` field layout are unchanged. The subtype fourcc tags, header, CAGRA block, graph CSR (entry_point / max_level / levels / offsets / neighbors / efC / efS), and storage are all as before.
- `keep_max_size_level0` is still serialized only for the CAGRA subtype (`IHc2`/`IHNc`); `init_level0` is build-only (not serialized).
- The deterministic build emits the same HNSW CSR structure (only neighbor content differs), so old indexes read unchanged and new indexes remain readable by older Faiss.
- Verified by the `io_and_retest` serialize -> deserialize -> re-search round-trips in `test_graph_based.py` / `test_hnsw.cpp` (all pass).

## CAGRA API for HNSW build on multi-GPU (aka D106837134) — MAST verification
Verified end-to-end on MAST (8x H100 Grand Teton, Approach D, 100M vectors) with this change in the build — the multi-GPU CAGRA -> HNSW graph-build time is comparable to the D106837134 baseline (no regression):
  all_neighbors build: 367.4s   optimize: 231.7s   copyTo: 18.4s   serialize: 28.9s (66 GB)
  INDEX build -> serialize total: 661.7s (11.0 min)   [D106837134 baseline: 721s]
  recall@10 (tiled 100M): ef64 0.7746, ef128 0.8830, ef256 0.9429
- This confirms this CPU-side change builds, links, and runs in the GPU CAGRA binary at scale and does not regress the pipeline. Note the Approach-D run uses copyTo(base_level_only=True), which imports the CAGRA graph directly as HNSW level 0 and skips add(), so it does not itself route through the deterministic add().
- The deterministic CAGRA level-0 import this change adds (the copyTo path with base_level_only=False: init_level0=false skips the level-0 bucket; keep_max_size_level0 fills the base layer) is covered by passing unit tests: `Test_IndexHNSWCagra_BaseLevelOnly_RangeSearch` (C++), `test_hnsw_no_init_level0`, and `test_hnsw_cagra_IP` / `_base_level_only` (Python).


## Behavioral note: level-0 base layer under keep_max_size_level0 (reviewers, please note)
One deliberate difference from the removed lock-based build, in the CAGRA base-layer case only: the old build gated the "fill the level-0 list up to 2*M" behavior on the inserted point's OWN top level (`keep_max_size_level0 && pt_level == 0`), so a level>=1 node's level-0 list could be pruned below 2*M. The deterministic build gates on the LINK level (`keep_max_size_level0 && level == 0`), so EVERY node's level-0 list is filled to 2*M when `keep_max_size_level0` is set (not only the level-0-only points). This is a strict superset of the old coverage -- it fills exactly to the 2*M slot capacity (no overflow) and yields a fuller/denser base layer for CPU `IndexHNSWCagra`, which is what `GpuIndexCagra::copyFrom(IndexHNSWCagra*)` reads back. It is INERT for the default build (`keep_max_size_level0` defaults to false, so the gate is never true) and never affects a non-CAGRA graph. Called out explicitly so reviewers know the CPU `IndexHNSWCagra` base-layer graph is intentionally denser than the pre-diff build; worth a sanity check against GPU `copyFrom` expectations.

Differential Revision: D112025877

Author: Michael Norris

benchs/bench_hnsw.py

@@ -191,44 +191,3 @@ def evaluate(index):
         print("search_L", search_L, end=" ")
         index.nsg.search_L = search_L
         evaluate(index)
-
-
-if "hnsw_locks" in todo:
-
-    ntotal, _ = xb.shape
-    batch_size = ntotal // 100
-    print(
-        f"Testing HNSW Flat: add with {batch_size=}, "
-        "with and without retaining locks"
-    )
-
-    # Unbatched
-    t0 = time.time()
-    index = faiss.IndexHNSWFlat(d, 32)
-    index.add(xb)
-    t1 = time.time()
-    print(
-        f"\t single bulk add(): {index.ntotal} added in {t1 - t0:6.3f}s"
-        f" = {index.ntotal / (t1 - t0):.0f}/s"
-    )
-
-    for retain_locks in [False, True]:
-        index = faiss.IndexHNSWFlat(d, 32)
-        index.retain_locks = retain_locks
-
-        t0 = time.time()
-        t1 = None
-        t2 = None
-        for i in range(0, len(xb), batch_size):
-            t1 = time.time()
-            index.add(xb[i : i + batch_size])
-            t2 = time.time()
-            if i > 2 and t2 - t0 > 2:
-                break
-
-        assert t1 and t2
-        dt = t2 - t0
-        print(
-            f"\t {retain_locks=:1}: {index.ntotal} added in {t2 - t0:6.3f}s"
-            f" = {index.ntotal / (t2 - t0):.0f}/s"
-        )

faiss/CMakeLists.txt

@@ -155,7 +155,6 @@ set(FAISS_SRC
   impl/IDSelector.cpp
   impl/FaissException.cpp
   impl/HNSW.cpp
-  impl/hnsw/LockVector.cpp
   impl/hnsw/MinimaxHeap.cpp
   impl/result_handler/ResultHandler.cpp
   impl/NSG.cpp
@@ -295,7 +294,6 @@ set(FAISS_HEADERS
   impl/FaissAssert.h
   impl/FaissException.h
   impl/HNSW.h
-  impl/hnsw/LockVector.h
   impl/hnsw/MinimaxHeap.h
   impl/LocalSearchQuantizer.h
   impl/ProductAdditiveQuantizer.h

faiss/IndexBinaryHNSW.cpp

@@ -43,136 +43,6 @@ namespace faiss {
  * add / search blocks of descriptors
  **************************************************************/
 
-namespace {
-
-void hnsw_add_vertices(
-        IndexBinaryHNSW& index_hnsw,
-        size_t n0,
-        size_t n,
-        const uint8_t* x,
-        bool verbose,
-        bool preset_levels = false) {
-    HNSW& hnsw = index_hnsw.hnsw;
-    size_t ntotal = n0 + n;
-    double t0 = getmillisecs();
-    if (verbose) {
-        printf("hnsw_add_vertices: adding %zd elements on top of %zd "
-               "(preset_levels=%d)\n",
-               n,
-               n0,
-               int(preset_levels));
-    }
-
-    int max_level = hnsw.prepare_level_tab(n, preset_levels);
-
-    if (verbose) {
-        printf("  max_level = %d\n", max_level);
-    }
-
-    auto& locks = index_hnsw.locks;
-    locks.prepare(ntotal);
-
-    // add vectors from highest to lowest level
-    std::vector<int> hist;
-    std::vector<int> order(n);
-
-    { // make buckets with vectors of the same level
-
-        // build histogram
-        for (size_t i = 0; i < n; i++) {
-            HNSW::storage_idx_t pt_id =
-                    static_cast<HNSW::storage_idx_t>(i + n0);
-            int pt_level = hnsw.levels[pt_id] - 1;
-            while (pt_level >= static_cast<int>(hist.size())) {
-                hist.push_back(0);
-            }
-            hist[pt_level]++;
-        }
-
-        // accumulate
-        std::vector<int> offsets(hist.size() + 1, 0);
-        for (size_t i = 0; i < hist.size() - 1; i++) {
-            offsets[i + 1] = offsets[i] + hist[i];
-        }
-
-        // bucket sort
-        for (size_t i = 0; i < n; i++) {
-            HNSW::storage_idx_t pt_id =
-                    static_cast<HNSW::storage_idx_t>(i + n0);
-            int pt_level = hnsw.levels[pt_id] - 1;
-            order[offsets[pt_level]++] = pt_id;
-        }
-    }
-
-    { // perform add
-        RandomGenerator rng2(789);
-
-        size_t i1 = static_cast<int>(n);
-
-        for (int pt_level = static_cast<int>(hist.size()) - 1;
-             pt_level >= int(!index_hnsw.init_level0);
-             pt_level--) {
-            size_t i0 = i1 - hist[pt_level];
-
-            if (verbose) {
-                printf("Adding %zu elements at level %d\n", i1 - i0, pt_level);
-            }
-
-            // random permutation to get rid of dataset order bias
-            for (size_t j = i0; j < i1; j++) {
-                std::swap(
-                        order[j],
-                        order[j + rng2.rand_int(static_cast<int>(i1 - j))]);
-            }
-
-#pragma omp parallel
-            {
-                std::unique_ptr<VisitedTable> vt = VisitedTable::create(ntotal);
-
-                std::unique_ptr<DistanceComputer> dis(
-                        index_hnsw.get_distance_computer());
-                bool do_display = verbose && omp_get_thread_num() == 0;
-                size_t prev_display = 0;
-
-#pragma omp for schedule(dynamic)
-                for (int64_t i = i0; i < i1; i++) {
-                    HNSW::storage_idx_t pt_id = order[i];
-                    dis->set_query(
-                            (float*)(x + (pt_id - n0) * index_hnsw.code_size));
-
-                    hnsw.add_with_locks(
-                            *dis,
-                            pt_level,
-                            pt_id,
-                            locks,
-                            *vt,
-                            index_hnsw.keep_max_size_level0 && (pt_level == 0));
-
-                    if (do_display && i - i0 > prev_display + 10000) {
-                        prev_display = i - i0;
-                        printf("  %zu / %zu\r", i - i0, i1 - i0);
-                        fflush(stdout);
-                    }
-                }
-            }
-            i1 = i0;
-        }
-        if (index_hnsw.init_level0) {
-            FAISS_ASSERT(i1 == 0);
-        } else {
-            FAISS_ASSERT((i1 - hist[0]) == 0);
-        }
-    }
-    if (verbose) {
-        printf("Done in %.3f ms\n", getmillisecs() - t0);
-    }
-    if (!index_hnsw.retain_locks) {
-        locks.clear();
-    }
-}
-
-} // anonymous namespace
-
 /**************************************************************
  * IndexBinaryHNSW implementation
  **************************************************************/
@@ -270,18 +140,24 @@ void IndexBinaryHNSW::add(idx_t n, const uint8_t* x) {
     storage->add(n, x);
     ntotal = storage->ntotal;
 
-    hnsw_add_vertices(
-            *this,
+    bool preset_levels = hnsw.levels.size() == static_cast<size_t>(ntotal);
+    hnsw_add_vertices_deterministic(
+            hnsw,
             n0,
             n,
-            x,
+            d,
+            init_level0,
+            keep_max_size_level0,
+            preset_levels,
             verbose,
-            hnsw.levels.size() == static_cast<size_t>(ntotal));
+            [this] { return get_distance_computer(); },
+            [this, x, n0](DistanceComputer& dc, HNSW::storage_idx_t pt_id) {
+                dc.set_query((const float*)(x + (pt_id - n0) * code_size));
+            });
 }
 
 void IndexBinaryHNSW::reset() {
     hnsw.reset();
-    locks.clear();
     storage->reset();
     ntotal = 0;
 }

faiss/IndexBinaryHNSW.h

@@ -11,7 +11,6 @@
 
 #include <faiss/IndexBinaryFlat.h>
 #include <faiss/impl/HNSW.h>
-#include <faiss/impl/hnsw/LockVector.h>
 #include <faiss/utils/utils.h>
 
 namespace faiss {
@@ -41,11 +40,6 @@ struct IndexBinaryHNSW : IndexBinary {
     // used when GpuIndexBinaryCagra::copyFrom(IndexBinaryHNSW*) is called.
     bool keep_max_size_level0 = false;
 
-    // Per-node locks for HNSW graph construction.
-    LockVector locks;
-    // locks are freed after each call to add() unless this flag is set.
-    bool retain_locks = false;
-
     explicit IndexBinaryHNSW();
     explicit IndexBinaryHNSW(int d, int M = 32);
     explicit IndexBinaryHNSW(IndexBinary* storage, int M = 32);