3.1.0.0

Version 3.1.0.0 Release Notes

Compatible with OpenSearch 3.1.0.0

Features

• Introduce memory optimized search for Faiss binary index types #2735
• Create RescoreKnnVectorQuery to support rescore after executing inner vector search query #2709

Enhancements

• Removing redundant type conversions for script scoring for hamming space with binary vectors #2351
• Apply mask operation in preindex to optimize derived source #2704
• [Remote Vector Index Build] Add tuned repository upload/download configurations per benchmarking results #2662
• [Remote Vector Index Build] Add segment size upper bound setting and prepare other settings for GA #2734
• [Remote Vector Index Build] Make index.knn.remote_index_build.enabled default to true #2743
• Update rescore context for 4X Compression #2750

Bug Fixes

• [BUGFIX] Fix KNN Quantization state cache have an invalid weight threshold #2666
• [BUGFIX] Fix enable rescoring when dimensions > 1000. #2671
• [BUGFIX] Honors slice counts for non-quantization cases #2692
• [BUGFIX] Block derived source enable if index.knn is false #2702
• Block mode and compression for indices created before version 2.17.0 #2722
• [BUGFIX] Avoid opening of graph file if graph is already loaded in memory #2719
• [BUGFIX] [Remote Vector Index Build] End remote build metrics before falling back to CPU, exception logging #2693
• [BUGFIX] Fix RefCount and ClearCache in some race conditions #2728
• [BUGFIX] Fix nested vector query at efficient filter scenarios #2641
• [BUGFIX] Fix memory optimized searcher to use a sliced index input #2739

Infrastructure

• Add testing support to run all ITs with remote index builder #2659
• Fix KNNSettingsTests after change in MockNode constructor #2700

Refactoring

• Refactor Knn Search Results to use TopDocs #2727

3.0.0.0

Version 3.0.0.0 Release Notes

Compatible with OpenSearch 3.0.0

Breaking Changes

• Remove ef construction from Index Seeting #2564
• Remove m from Index Setting #2564
• Remove space type from index setting #2564
• Remove Knn Plugin enabled setting #2564

Features

• [Remote Vector Index Build] Client polling mechanism, encoder check, method parameter retrieval #2576
• [Remote Vector Index Build] Move client to separate module #2603
• Add filter function to KNNQueryBuilder with unit tests and integration tests #2599
• [Lucene On Faiss] Add a new mode, memory-optimized-search enable user to run vector search on FAISS index under memory constrained environment. #2630
• [Remote Vector Index Build] Add metric collection for remote build process #2615
• [Explain API Support] Added Explain API support for Exact/ANN/Radial/Disk based KNN search on Faiss Engine [#2403] (#2403)
• Introduce Remote Native Index Build feature flag, settings, and initial skeleton #2525
• Implement vector data upload and vector data size threshold setting #2550
• Implement data download and IndexOutput write functionality #2554
• Introduce Client Skeleton + basic Build Request implementation #2560
• Add concurrency optimizations with native memory graph loading and force eviction #2345

Enhancements

• Introduce node level circuit breakers for k-NN #2509
• Added more detailed error messages for KNN model training #2378

Bug Fixes

• Fixing bug to prevent NullPointerException while doing PUT mappings #2556
• Add index operation listener to update translog source #2629
• Add parent join support for faiss hnsw cagra #2647
• [Remote Vector Index Build] Fix bug to support COSINESIMIL space type #2627
• Disable doc value storage for vector field storage #2646
• Fix KNN Quantization state cache have an invalid weight threshold #2666
• Fix enable rescoring when dimensions > 1000. #2671
• Fix derived source for binary and byte vectors #2533
• Fix the put mapping issue for already created index with flat mapper #2542
• Fixing the bug to prevent index.knn setting from being modified or removed on restore snapshot #2445

Infrastructure

• Add github action to run ITs against remote index builder 2620
• Removed JDK 11 and 17 version from CI runs #1921
• Upgrade min JDK compatibility to JDK 21 #2422

Maintenance

• Update minimum required CMAKE version in NMSLIB #2635
• Revert CMake version bump, instead add CMake policy version flag to build task to support modern CMake builds #2645
• Update package name to fix compilation issue #2513
• Update gradle to 8.13 to fix command exec on java 21 #2571
• Add fix for nmslib pragma on arm #2574
• Removes Array based vector serialization #2587
• Enabled indices.breaker.total.use_real_memory setting via build.gradle for integTest Cluster to catch heap CB in local ITs and github CI actions #2395
• Fixing Lucene912Codec Issue with BWC for Lucene 10.0.1 upgrade#2429
• Enabled idempotency of local builds when using ./gradlew clean and nest jni/release directory under jni/build for easier cleanup #2516

Refactoring

• Switch derived source from field attributes to segment attribute #2606
• Migrate derived source from filter to mask #2612
• Consolidate MethodFieldMapper and LuceneFieldMapper into EngineFieldMapper #2646
• Small Refactor Post Lucene 10.0.1 upgrade #2541
• Refactor codec to leverage backwards_codecs #2546
• Blocking Index Creation using NMSLIB #2573
• Improve Streaming Compatibility Issue for MethodComponetContext and Remove OpenDistro URL #2575
• 3.0.0 Breaking Changes For KNN #2564

2.19.2.0

Version 2.19.2.0 Release Notes

Compatible with OpenSearch 2.19.2

Features

Enhancements

Bug Fixes

• Enables rescoring by default when dimensions > 1024. #2671

Infrastructure

Documentation

Maintenance

3.0.0.0-beta1

Version 3.0.0.0-beta1 Release Notes

Compatible with OpenSearch 3.0.0.beta1

Features

• [Remote Vector Index Build] Client polling mechanism, encoder check, method parameter retrieval #2576
• [Remote Vector Index Build] Move client to separate module #2603
• Add filter function to KNNQueryBuilder with unit tests and integration tests #2599
• [Lucene On Faiss] Add a new mode, memory-optimized-search enable user to run vector search on FAISS index under memory constrained environment. #2630
• [Remote Vector Index Build] Add metric collection for remote build process #2615
• [Explain API Support] Added Explain API support for Exact/ANN/Radial/Disk based KNN search on Faiss Engine [#2403] (#2403)

Enhancements

Bug Fixes

• Fixing bug to prevent NullPointerException while doing PUT mappings #2556
• Add index operation listener to update translog source #2629
• Add parent join support for faiss hnsw cagra #2647
• [Remote Vector Index Build] Fix bug to support COSINESIMIL space type #2627
• Disable doc value storage for vector field storage #2646

Infrastructure

• Add github action to run ITs against remote index builder 2620

Maintenance

• Update minimum required CMAKE version in NMSLIB #2635
• Revert CMake version bump, instead add CMake policy version flag to build task to support modern CMake builds #2645

Refactoring

• Switch derived source from field attributes to segment attribute #2606
• Migrate derived source from filter to mask #2612
• Consolidate MethodFieldMapper and LuceneFieldMapper into EngineFieldMapper #2646

3.0.0.0-alpha1

Version 3.0.0.0-alpha1 Release Notes

Compatible with OpenSearch 3.0.0.alpha1

Breaking Changes

• Remove ef construction from Index Seeting #2564
• Remove m from Index Setting #2564
• Remove space type from index setting #2564
• Remove Knn Plugin enabled setting #2564

Features

• Introduce Remote Native Index Build feature flag, settings, and initial skeleton #2525
• Implement vector data upload and vector data size threshold setting #2550
• Implement data download and IndexOutput write functionality #2554
• Introduce Client Skeleton + basic Build Request implementation #2560
• Add concurrency optimizations with native memory graph loading and force eviction #2345

Enhancements

• Introduce node level circuit breakers for k-NN #2509
• Added more detailed error messages for KNN model training #2378

Bug Fixes

• Fix derived source for binary and byte vectors #2533
• Fix the put mapping issue for already created index with flat mapper #2542
• Fixing the bug to prevent index.knn setting from being modified or removed on restore snapshot #2445

Infrastructure

• Removed JDK 11 and 17 version from CI runs #1921
• Upgrade min JDK compatibility to JDK 21 #2422

Maintenance

• Update package name to fix compilation issue #2513
• Update gradle to 8.13 to fix command exec on java 21 #2571
• Add fix for nmslib pragma on arm #2574
• Removes Array based vector serialization #2587
• Enabled indices.breaker.total.use_real_memory setting via build.gradle for integTest Cluster to catch heap CB in local ITs and github CI actions #2395
• Fixing Lucene912Codec Issue with BWC for Lucene 10.0.1 upgrade#2429
• Enabled idempotency of local builds when using ./gradlew clean and nest jni/release directory under jni/build for easier cleanup #2516

Refactoring

• Small Refactor Post Lucene 10.0.1 upgrade #2541
• Refactor codec to leverage backwards_codecs #2546
• Blocking Index Creation using NMSLIB #2573
• Improve Streaming Compatibility Issue for MethodComponetContext and Remove OpenDistro URL #2575
• 3.0.0 Breaking Changes For KNN #2564

2.19.0.0

Version 2.19.0.0 Release Notes

Compatible with OpenSearch 2.19.0

Features

• Add Support for Multi Values in innerHit for Nested k-NN Fields in Lucene and FAISS #2283
• Add binary index support for Lucene engine. #2292
• Add expand_nested_docs Parameter support to NMSLIB engine #2331
• Add a new build mode, FAISS_OPT_LEVEL=avx512_spr, which enables the use of advanced AVX-512 instructions introduced with Intel[R] Sapphire Rapids #2404
• Add cosine similarity support for faiss engine #2376
• Add concurrency optimizations with native memory graph loading and force eviction #2265
• Add derived source feature for vector fields #2449

Enhancements

• Introduced a writing layer in native engines where relies on the writing interface to process IO. #2241
• Allow method parameter override for training based indices #2290
• Optimizes lucene query execution to prevent unnecessary rewrites #2305
• Added more detailed error messages for KNN model training #2378
• Add check to directly use ANN Search when filters match all docs. #2320
• Use one formula to calculate cosine similarity #2357
• Make the build work for M series MacOS without manual code changes and local JAVA_HOME config #2397
• Remove DocsWithFieldSet reference from NativeEngineFieldVectorsWriter #2408
• Remove skip building graph check for quantization use case #2430
• Removing redundant type conversions for script scoring for hamming space with binary vectors #2351
• Update default to 0 to always build graph as default behavior #2452
• Enabled concurrent graph creation for Lucene engine with index thread qty settings#2480

Bug Fixes

• Fixing the bug when a segment has no vector field present for disk based vector search #2282
• Fixing the bug where search fails with "fields" parameter for an index with a knn_vector field #2314
• Fix for NPE while merging segments after all the vector fields docs are deleted #2365
• Allow validation for non knn index only after 2.17.0 #2315
• Fixing the bug to prevent updating the index.knn setting after index creation#2348
• Release query vector memory after execution #2346
• Fix shard level rescoring disabled setting flag #2352
• Fix filter rewrite logic which was resulting in getting inconsistent / incorrect results for cases where filter was getting rewritten for shards #2359
• Fixing it to retrieve space_type from index setting when both method and top level don't have the value. #2374
• Fixing the bug where setting rescore as false for on_disk knn_vector query is a no-op #2399
• Fixing the bug to prevent index.knn setting from being modified or removed on restore snapshot #2445
• Fix Faiss byte vector efficient filter bug #2448
• Fixing bug where mapping accepts both dimension and model-id #2410
• Add version check for full field name validation #2477
• Update engine for version 2.19 or above #2501

Infrastructure

• Updated C++ version in JNI from c++11 to c++17 #2259
• Upgrade bytebuddy and objenesis version to match OpenSearch core and, update github ci runner for macos #2279

Documentation

Maintenance

• Select index settings based on cluster version2236
• Added periodic cache maintenance for QuantizationStateCache and NativeMemoryCache #2308
• Added null checks for fieldInfo in ExactSearcher to avoid NPE while running exact search for segments with no vector field #2278
• Added Lucene BWC tests #2313
• Upgrade jsonpath from 2.8.0 to 2.9.02325
• Bump Faiss commit from 1f42e81 to 0cbc2a8 to accelerate hamming distance calculation using _mm512_popcnt_epi64 intrinsic and also add avx512-fp16 instructions to boost performance #2381
• Deprecate nmslib engine #2427
• Add spotless mirror repo for fixing builds #2453

2.18.0.0

Version 2.18.0.0 Release Notes

Compatible with OpenSearch 2.18.0

Features

• Add AVX512 support to k-NN for FAISS library #2069

Enhancements

• Introducing a loading layer in FAISS #2033
• Add short circuit if no live docs are in segments #2059
• Optimize reduceToTopK in ResultUtil by removing pre-filling and reducing peek calls #2146
• Update Default Rescore Context based on Dimension #2149
• KNNIterators should support with and without filters #2155
• Adding Support to Enable/Disble Share level Rescoring and Update Oversampling Factor#2172
• Add support to build vector data structures greedily and perform exact search when there are no engine files #1942
• Add CompressionLevel Calculation for PQ #2200
• Remove FSDirectory dependency from native engine constructing side and deprecated FileWatcher #2182
• Update approximate_threshold to 15K documents #2229
• Update default engine to FAISS #2221

Bug Fixes

• Add DocValuesProducers for releasing memory when close index #1946
• KNN80DocValues should only be considered for BinaryDocValues fields #2147
• Score Fix for Binary Quantized Vector and Setting Default value in case of shard level rescoring is disabled for oversampling factor#2183
• Java Docs Fix For 2.x#2190

Documentation

• Fix sed command in DEVELOPER_GUIDE.md to append a new line character '\n'. #2181

Maintenance

• Remove benchmarks folder from k-NN repo #2127
• Fix lucene codec after lucene version bumped to 9.12. #2195

Refactoring

• Does not create additional KNNVectorValues in NativeEngines990KNNVectorWriter when quantization is not needed #2133
• Minor refactoring and refactored some unit test #2167

2.17.0.0

Version 2.17.0.0 Release Notes

Compatible with OpenSearch 2.17.0

Features

• Integrate Lucene Vector field with native engines to use KNNVectorFormat during segment creation #1945
• k-NN query rescore support for native engines #1984
• Add support for byte vector with Faiss Engine HNSW algorithm #1823
• Add support for byte vector with Faiss Engine IVF algorithm #2002
• Add mode/compression configuration support for disk-based vector search #2034
• Add spaceType as a top level optional parameter while creating vector field. #2044

Enhancements

• Adds iterative graph build capability into a faiss index to improve the memory footprint during indexing and Integrates KNNVectorsFormat for native engines#1950
• Add model version to model metadata and change model metadata reads to be from cluster metadata #2005

Bug Fixes

• Corrected search logic for scenario with non-existent fields in filter #1874
• Add script_fields context to KNNAllowlist [#1917] (#1917)
• Fix graph merge stats size calculation #1844
• Disallow a vector field to have an invalid character for a physical file name. #1936
• Fix memory overflow caused by cache behavior #2015
• Use correct type for binary vector in ivf training #2086
• Switch MINGW32 to MINGW64 #2090

Infrastructure

• Parallelize make to reduce build time [#2006] (#2006)

Maintenance

• Fix a flaky unit test:testMultiFieldsKnnIndex, which was failing due to inconsistent merge behaviors #1924

Refactoring

• Introduce KNNVectorValues interface to iterate on different types of Vector values during indexing and search #1897
• Integrate KNNVectorValues with vector ANN Search flow #1952
• Clean up parsing for query #1824
• Refactor engine package structure #1913
• Refactor method structure and definitions #1920
• Refactor KNNVectorFieldType from KNNVectorFieldMapper to a separate class for better readability. #1931
• Generalize lib interface to return context objects #1925
• Restructure mappers to better handle null cases and avoid branching in parsing #1939
• Added Quantization Framework and implemented 1Bit and multibit quantizer#1889
• Encapsulate dimension, vector data type validation/processing inside Library #1957
• Add quantization state cache #1960
• Add quantization state reader and writer #1997

2.16.0.0

Version 2.16.0.0 Release Notes

Compatible with OpenSearch 2.16.0

Features

• Adds dynamic query parameter ef_search #1783
• Adds dynamic query parameter ef_search in radial search faiss engine #1790
• Adds dynamic query parameter nprobes #1792
• Add binary format support with HNSW method in Faiss Engine #1781
• Add script scoring support for knn field with binary data type #1826
• Add painless script support for hamming with binary vector data type #1839
• Add binary format support with IVF method in Faiss Engine #1784
• Add support for Lucene inbuilt Scalar Quantizer #1848

Enhancements

• Switch from byte stream to byte ref for serde #1825

Bug Fixes

• Fixing the arithmetic to find the number of vectors to stream from java to jni layer.#1804
• Fixed LeafReaders casting errors to SegmentReaders when segment replication is enabled during search.#1808
• Release memory properly for an array type #1820
• FIX Same Suffix Cause Recall Drop to zero #1802

Infrastructure

• Apply custom patch only once by comparing the last patch id #1833

Documentation

• Update dev guide to fix clang linking issue on arm #1746

Maintenance

• Bump faiss commit to 33c0ba5 #1796

2.15.0.0

Version 2.15.0.0 Release Notes

Compatible with OpenSearch 2.15.0

Features

• Use the Lucene Distance Calculation Function in Script Scoring for doing exact search #1699

Enhancements

• Add KnnCircuitBreakerException and modify exception message #1688
• Add stats for radial search #1684
• Support script score when doc value is disabled and fix misusing DISI #1696
• Add validation for pq m parameter before training starts #1713
• Block delete model requests if an index uses the model #1722

Bug Fixes

• Block commas in model description #1692
• Update threshold value after new result is added #1715