Raise HNSW visited-table hash-set threshold from 500k to 10M (#5446)

Summary:
Pull Request resolved: #5446

`HNSW` search maintains a per-thread visited set, and `faiss` chooses its implementation by index size via `visited_table_hashset_threshold`: below the threshold it uses a version-stamped byte array (`VisitedTableVector`, O(1) direct-index `get`/`set`, O(n) memory); at/above it, a hash set (`VisitedTableSet`, hash-and-probe per access, memory proportional to nodes visited).

The current 500k cutoff is too low. It forces million-scale in-memory indexes onto the hash set, whose hash-and-probe on every visited neighbor is markedly slower than the array's single indexed access - while the array's footprint (~1 byte/node, ~1 MB/thread at 1M) is entirely affordable for in-memory HNSW. **This raises the cutoff to 10M so ~1M–10M indexes get the faster array. Callers can still force either implementation per index via `HNSW::use_visited_hashset`**.

Note: 10M is a heuristic estimate, not a tuned optimum. The real array-vs-hash-set crossover is dataset-dependent - in particular it shrinks with vector dimension: at very high dimension the distance computation dominates per-node cost, so the array's access advantage vanishes and its larger cache footprint can even make it marginally slower (see test plan: dbpedia at 3072d is ~neutral, while 128d–960d gain clearly). A follow-up should make this smarter - e.g. tune the threshold per index accounting for dimension / vector byte-size, or use a lossy / more compact visited structure that trades a little accuracy for lower access + memory cost.

Reviewed By: mnorris11, pankajsingh88

Differential Revision: D112025912

fbshipit-source-id: 4789fa34a694a637dbea1871dd8c890ffb766dd4

Author: Lei Huang

faiss/impl/VisitedTable.cpp

@@ -15,8 +15,12 @@ namespace faiss {
 // advance() is O(1) except every 250 calls, which are O(size).
 // The hash set strategy is a constant factor slower for get()/set(),
 // but O(1) to construct and O(visits) to advance.
-// A size of ~1M seems to be the threshold where the hash set wins.
-size_t visited_table_hashset_threshold = 500000;
+// 10M is only a current estimated threshold, not a proven crossover: we are not
+// sure the array still wins at 10M. The point where the array stops paying off
+// varies by dataset (it shifts with dimension, working-set / cache pressure,
+// etc.), so this is a coarse default that should eventually be replaced by
+// smarter per-index tuning.
+size_t visited_table_hashset_threshold = 10000000;
 
 std::unique_ptr<VisitedTable> VisitedTable::create(
         size_t size,