increment version of jVector to support incremental construction (#167)

persist neighbors cache
add support for sorted index
searcher fixes for resolving the node -> docId
add incremental merge construction with leading segment
move additional tests to internal test for transparency
update documentation
add readme pictures
remove doc values by default
separate docIdtoOrdMap class and add tests

Signed-off-by: Samuel Herman <sherman8915@gmail.com>

Author: sam-herman

CHANGELOG.md

@@ -8,11 +8,18 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 ### Features
 ### Enhancements
 * PQ refinement during merge [109](https://github.com/opensearch-project/opensearch-jvector/issues/109)
+* Persistent Ordinal To docID Mapping [167](https://github.com/opensearch-project/opensearch-jvector/pull/167)
+* Incremental Insertion With Leading Segment [167](https://github.com/opensearch-project/opensearch-jvector/pull/167)
+* Remove Redundant FlatVectorFormat [167](https://github.com/opensearch-project/opensearch-jvector/pull/167)
+* Remove Redundant DocValuesFormat [167](https://github.com/opensearch-project/opensearch-jvector/pull/167)
 ### Bug Fixes
+* Fix for sorted indices [167](https://github.com/opensearch-project/opensearch-jvector/pull/167)
+* Fix for missing fields [167](https://github.com/opensearch-project/opensearch-jvector/pull/167)
 ### Infrastructure
 ### Documentation
 ### Maintenance
 ### Refactoring
+* Remove jVector Codec [167](https://github.com/opensearch-project/opensearch-jvector/pull/167)
 
 ## [Unreleased 2.x](https://github.com/opensearch-project/opensearch-jvector/compare/2.18...2.x)
 ### Features

README.md

@@ -27,6 +27,8 @@
 - _**DiskANN**_ - JVector is a pure Java implementation capable to perform vector ANN search in a way that is optimized for RAM bound environments with minimal additional overhead. No need involving native dependencies (FAISS) and cumbersome JNI mechanism.
 - _**Thread Safety**_ - JVector is a threadsafe index that supports concurrent modification and inserts with near perfect scalability as you add cores, Lucene is not threadsafe; This allows us to ingest much higher volume of vectors a lot faster without unnecessary merge operations to parallelize ingestion concurrency.
 - _**quantized index construction**_ - JVector can perform index construction w/ quantized vectors, saving memory = larger segments = fewer segments = faster searches
+- _**Quantization refinement**_ - JVector can refine the quantization codebooks during merge, this allows us to have a more accurate quantization and better recall without the penalty of full codebook recomputation
+- _**Incremental merges**_ - JVector plugin can perform incremental vector inserts into previously persisted indexes without the need for a full rebuild of the index. This is an enormous saving for updates (especially updates of large graphs!) which are a common operation in many search applications.
 - _**Quantized Disk ANN**_ - JVector supports DiskANN style quantization with rerank, it's quite easy (in principle) to demonstrate that this is a massive difference in performance for larger-than-memory indexes (in practice it takes days/weeks to insert enough vectors into Lucene to show this b/c of the single threaded problem, that's the only hard part)
 - _**PQ and BQ support**_  - As part of (3) JVector supports PQ as well as the BQ that Lucene offers, it seems that this is fairly rare (pgvector doesn't do PQ either) because (1) the code required to get high performance ADC with SIMD is a bit involved and (2) it requires a separate codebook which Lucene isn't set up to easily accommodate.  PQ at 64x compression gives you higher relevance than BQ at 32x
 - _**Fused ADC**_ - Features that nobody else has like Fused ADC and NVQ and Anisotropic PQ
@@ -132,6 +134,19 @@ For example if Lucene is doing 100x the number of disk reads compared to JVector
 ![latency.png](latency.png)
 ![recall.png](recall.png)
 
+### Incremental merges
+Incremental merges are a key differentiator between JVector and other KNN engines. The following graphs show the cost of merges as a function of the number of documents.
+You can see the stark differences when we leverage the incremental merge capabilities of JVector plugin compared to the full rebuild scenario on every merge.
+
+Without incremental merges:
+![merge_times_before_incremental.png](merge_times_before_incremental_smoothed_plot.png)
+
+With incremental merges:
+![merge_times_after_incremental.png](merge_times_plot_after_incremental.png)
+
+Comparison:
+![merge_times_comparison.png](merge_times_comparison.png)
+
 ## Credits and Acknowledgments
 
 This project uses two similarity search libraries to perform Approximate Nearest Neighbor Search: the Apache 2.0-licensed [Lucene](https://github.com/apache/lucene) and [jVector](https://github.com/jbellis/jvector).

benchmark-jmh/src/jmh/java/org/opensearch/knn/index/codec/jvector/BenchmarkCommon.java

@@ -11,12 +11,11 @@
 import org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat;
 import org.apache.lucene.index.VectorSimilarityFunction;
 import org.apache.lucene.search.TopDocs;
+import org.opensearch.knn.common.KNNConstants;
 import org.opensearch.knn.index.codec.KNNCodecVersion;
 
 import java.util.PriorityQueue;
 
-import static org.opensearch.knn.index.codec.jvector.JVectorFormat.DEFAULT_MINIMUM_BATCH_SIZE_FOR_QUANTIZATION;
-
 public class BenchmarkCommon {
     public static final String JVECTOR_NOT_QUANTIZED = "jvector_not_quantized";
     public static final String JVECTOR_QUANTIZED = "jvector_quantized";
@@ -26,7 +25,7 @@ public class BenchmarkCommon {
     public static Codec getCodec(String codecType) {
         return switch (codecType) {
             case JVECTOR_NOT_QUANTIZED -> getFilterJvectorCodec(Integer.MAX_VALUE);
-            case JVECTOR_QUANTIZED -> getFilterJvectorCodec(DEFAULT_MINIMUM_BATCH_SIZE_FOR_QUANTIZATION);
+            case JVECTOR_QUANTIZED -> getFilterJvectorCodec(KNNConstants.DEFAULT_MINIMUM_BATCH_SIZE_FOR_QUANTIZATION);
             case LUCENE101 -> new Lucene101Codec();
             default -> throw new IllegalStateException("Unexpected codec type: " + codecType);
         };
@@ -84,7 +83,7 @@ public KnnVectorsFormat knnVectorsFormat() {
 
                     @Override
                     public KnnVectorsFormat getKnnVectorsFormatForField(String field) {
-                        return new JVectorFormat(minBatchSizeForQuantization, true);
+                        return new JVectorFormat(minBatchSizeForQuantization);
                     }
                 };
             }

gradle.properties

@@ -5,7 +5,7 @@
 
 version=1.0.0
 systemProp.bwc.version=1.3.4
-jvector_version=4.0.0-rc.2
+jvector_version=4.0.0-rc.4
 java_release_version=21
 
 # org.gradle.jvmargs=--add-exports jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED \

merge_times_before_incremental_smoothed_plot.png

@@ -17,8 +17,9 @@ It's recommended to use a virtual environment to avoid conflicts with other Pyth
 
 1. Create a virtual environment:
    ```bash
+   sudo apt install python3.11-venv
    # Using venv (Python 3.3+)
-   python -m venv .venv
+   python3 -m venv .venv
 
    # Activate the virtual environment
    # On Windows:
@@ -138,3 +139,28 @@ python create_and_test_large_index.py --batch-size 1000 --force-merge-frequency
 # Generate plots from existing CSV
 python create_and_test_large_index.py --csv-output merge_times.csv --plot
 ```
+
+#### Important Note For Large Indices
+
+When working with large indices, it's important to consider the point at which we will require quantization.
+Quantization is becoming critical during index construction when we can't fit the full precision vectors in memory and are forced to use disk.
+Therefore, we want to set the `minimum_batch_size_for_quantization` to a value high enough so we can avoid quantization during index construction.
+Or alternatively, we can set it to a lower value and accept the additional compute cost of quantization during index construction, and thus avoid the disk access.
+
+```shell
+# Run with quantization disabled during index construction until we reach 10M documents
+python create_and_test_large_index.py --batch-size 1000 --force-merge-frequency 1000 --num-vectors 100000 --min-batch-size-for-quantization 10000000
+```
+
+For long running tests you would want to move the script to run in the background and redirect the output to a file:
+```shell
+nohup python create_and_test_large_index.py --batch-size 5000 --force-merge-frequency 100000 --num-vectors 10000000 --min-batch-size-for-quantization 10000000 > output.log 2>&1 &
+```
+
+You can also profile the java process while running the script:
+```shell
+# Get the process id of the opensearch java process
+PID=$(jps | grep OpenSearch | awk '{print $1}')
+# Start profiling
+jcmd $PID JFR.start name=OnDemand settings=profile duration=600s filename=/tmp/app_jfr_$(date +%s).jfr
+```

merge_times_comparison.png

@@ -11,14 +11,15 @@
 import matplotlib.pyplot as plt
 import os
 
-def create_index(host, index_name, dimension, shards=1):
+def create_index(host, index_name, dimension, shards=1, min_batch_size_for_quantization=1000000):
     """Create a knn index with jvector engine"""
     url = f"http://{host}/{index_name}"
 
     mapping = {
         "settings": {
             "index": {
                 "knn": True,
+                "knn.derived_source.enabled": True,
                 "number_of_shards": shards,
                 "number_of_replicas": 0
             }
@@ -32,7 +33,9 @@ def create_index(host, index_name, dimension, shards=1):
                         "name": "disk_ann",
                         "space_type": "l2",
                         "engine": "jvector",
-                        "parameters": {}
+                        "parameters": {
+                            "advanced.min_batch_size_for_quantization": min_batch_size_for_quantization
+                        }
                     }
                 },
                 "id": {"type": "keyword"}
@@ -422,6 +425,8 @@ def main():
                         help="Force merge after every N documents (0 to disable intermediate merges)")
     parser.add_argument("--csv-output", type=str, help="CSV file to save merge time data")
     parser.add_argument("--plot", action="store_true", help="Generate plots from CSV data")
+    parser.add_argument("--min-batch-size-for-quantization", type=int, default=1000000,
+                        help="Minimum batch size for quantization (default: 1M)")
 
     args = parser.parse_args()
 
@@ -434,7 +439,7 @@ def main():
         print(f"Estimated size: ~{args.num_vectors * args.dimension * 4 / (1024*1024*1024):.2f} GB (raw vectors only)")
 
         # Create index
-        create_index(args.host, args.index, args.dimension, args.shards)
+        create_index(args.host, args.index, args.dimension, args.shards, args.min_batch_size_for_quantization)
 
         # Index vectors
         index_vectors(args.host, args.index, args.num_vectors, args.dimension, args.batch_size, args.force_merge_frequency, args.csv_output)
@@ -467,4 +472,4 @@ def main():
         plot_merge_times(args.csv_output)
 
 if __name__ == "__main__":
-    main()
+    main()

merge_times_plot_after_incremental.png

@@ -124,11 +124,14 @@ public class KNNConstants {
     // Construction related params
     public static final String METHOD_PARAMETER_ALPHA = "advanced.alpha";
     public static final String METHOD_PARAMETER_NEIGHBOR_OVERFLOW = "advanced.neighbor_overflow";
-    public static final String METHOD_PARAMETER_HIERARCHY_ENABLED = "hierarchy_enabled"; // TODO: wire this after jvector upgrade
+    public static final String METHOD_PARAMETER_MIN_BATCH_SIZE_FOR_QUANTIZATION = "advanced.min_batch_size_for_quantization";
+    public static final String METHOD_PARAMETER_HIERARCHY_ENABLED = "advanced.hierarchy_enabled";
     public static final String METHOD_PARAMETER_NUM_PQ_SUBSPACES = "advanced.num_pq_subspaces";
     public static final Double DEFAULT_ALPHA_VALUE = 1.2;
     public static final Double DEFAULT_NEIGHBOR_OVERFLOW_VALUE = 1.2;
-    public static final Boolean DEFAULT_HIERARCHY_ENABLED = true; // TODO: wire this after jvector upgrade
+    public static final int DEFAULT_MINIMUM_BATCH_SIZE_FOR_QUANTIZATION = 1024; // above this batch size we will trigger quantization by
+                                                                                // default
+    public static final Boolean DEFAULT_HIERARCHY_ENABLED = false;
 
     // Parameter defaults/limits
     public static final Integer ENCODER_PARAMETER_PQ_CODE_COUNT_DEFAULT = 1;