Skip to content

Commit 3737eba

Browse files
akash-shankaransam-hermanpeterzhuamazon
authored
changes to perform OpenSearch Jvector 3.2 upgrade (#165)
* changes to perform JVector 3.2 upgrade Signed-off-by: Akash Shankaran <akash.shankaran@ibm.com> * fix failing workflow, and spotless apply Signed-off-by: Akash Shankaran <akash.shankaran@ibm.com> * few more package upgrades for JDK24 Signed-off-by: Akash Shankaran <akash.shankaran@ibm.com> * add Akash as maintainer (#174) * add Akash as maintainer * change to IBM --------- Signed-off-by: Samuel Herman <sherman8915@gmail.com> * update akash in codeowners (#176) Signed-off-by: Samuel Herman <sherman8915@gmail.com> * Onboarding new maven snapshots publishing to s3 (jVector) (#178) Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> * increment version of jVector to support incremental construction (#167) persist neighbors cache add support for sorted index searcher fixes for resolving the node -> docId add incremental merge construction with leading segment move additional tests to internal test for transparency update documentation add readme pictures remove doc values by default separate docIdtoOrdMap class and add tests Signed-off-by: Samuel Herman <sherman8915@gmail.com> * update changelog Signed-off-by: Akash Shankaran <akash.shankaran@ibm.com> --------- Signed-off-by: Akash Shankaran <akash.shankaran@ibm.com> Signed-off-by: Samuel Herman <sherman8915@gmail.com> Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> Co-authored-by: sam-herman <97131656+sam-herman@users.noreply.github.com> Co-authored-by: Peter Zhu <zhujiaxi@amazon.com>
1 parent 998e932 commit 3737eba

11 files changed

Lines changed: 42 additions & 27 deletions

File tree

‎.github/workflows/CI.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ jobs:
3636
Build-k-NN-Linux:
3737
strategy:
3838
matrix:
39-
java: [21, 23]
39+
java: [21, 24]
4040

4141
env:
4242
CC: gcc10-gcc
@@ -81,7 +81,7 @@ jobs:
8181
Build-k-NN-MacOS:
8282
strategy:
8383
matrix:
84-
java: [21, 23]
84+
java: [21, 24]
8585

8686
name: Build and Test jVector k-NN Plugin on MacOS
8787
needs: Get-CI-Image-Tag
@@ -104,7 +104,7 @@ jobs:
104104
Build-k-NN-Windows:
105105
strategy:
106106
matrix:
107-
java: [21, 23]
107+
java: [21, 24]
108108

109109
name: Build and Test jVector k-NN Plugin on Windows
110110
needs: Get-CI-Image-Tag

‎.github/workflows/backwards_compatibility_tests_workflow.yml‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -35,8 +35,8 @@ jobs:
3535
matrix:
3636
java: [ 21 ]
3737
os: [ubuntu-latest]
38-
bwc_version : [ "2.0.1", "2.1.0", "2.2.1", "2.3.0", "2.4.1", "2.5.0", "2.6.0", "2.7.0", "2.8.0", "2.9.0", "2.10.0", "2.11.0", "2.12.0", "2.13.0", "2.14.0", "2.15.0", "2.16.0", "2.17.0", "2.18.0","2.19.0-SNAPSHOT", "2.20.0-SNAPSHOT"]
39-
opensearch_version : [ "3.0.0-SNAPSHOT" ]
38+
bwc_version : [ "2.0.1", "2.1.0", "2.2.1", "2.3.0", "2.4.1", "2.5.0", "2.6.0", "2.7.0", "2.8.0", "2.9.0", "2.10.0", "2.11.0", "2.12.0", "2.13.0", "2.14.0", "2.15.0", "2.16.0", "2.17.0", "2.18.0","2.19.0-SNAPSHOT", "2.20.0-SNAPSHOT", "3.0.0-SNAPSHOT"]
39+
opensearch_version : [ "3.2.0-SNAPSHOT" ]
4040
exclude:
4141
- os: windows-latest
4242
bwc_version: "2.0.1"
@@ -130,8 +130,8 @@ jobs:
130130
matrix:
131131
java: [ 21 ]
132132
os: [ubuntu-latest]
133-
bwc_version: [ "2.20.0-SNAPSHOT" ]
134-
opensearch_version: [ "3.0.0-SNAPSHOT" ]
133+
bwc_version: [ "3.0.0-SNAPSHOT" ]
134+
opensearch_version: [ "3.2.0-SNAPSHOT" ]
135135

136136
name: k-NN Rolling-Upgrade BWC Tests
137137
runs-on: ${{ matrix.os }}

‎CHANGELOG.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ All notable changes to this project are documented in this file.
44

55
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). See the [CONTRIBUTING guide](./CONTRIBUTING.md#Changelog) for instructions on how to add changelog entries.
66

7-
## [Unreleased 3.0](https://github.com/opensearch-project/opensearch-jvector/compare/2.x...HEAD)
7+
## [Unreleased 3.2](https://github.com/opensearch-project/opensearch-jvector/compare/2.x...HEAD)
88
### Features
99
### Enhancements
1010
* PQ refinement during merge [109](https://github.com/opensearch-project/opensearch-jvector/issues/109)
@@ -16,8 +16,12 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
1616
* Fix for sorted indices [167](https://github.com/opensearch-project/opensearch-jvector/pull/167)
1717
* Fix for missing fields [167](https://github.com/opensearch-project/opensearch-jvector/pull/167)
1818
### Infrastructure
19+
* Upgrade to JDK24 [165] (https://github.com/opensearch-project/opensearch-jvector/pull/165)
20+
* Upgrade Gradle to 8.14 [165] (https://github.com/opensearch-project/opensearch-jvector/pull/165)
1921
### Documentation
22+
* Add docker instructions [163] (https://github.com/opensearch-project/opensearch-jvector/pull/163)
2023
### Maintenance
24+
* Fix documentation bugs [161] (https://github.com/opensearch-project/opensearch-jvector/pull/161)
2125
### Refactoring
2226
* Remove jVector Codec [167](https://github.com/opensearch-project/opensearch-jvector/pull/167)
2327

‎build-tools/jvectorplugin-coverage.gradle‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
apply plugin: 'jacoco'
77

88
jacoco {
9-
toolVersion = "0.8.12"
9+
toolVersion = "0.8.13"
1010
}
1111

1212
/**

‎build.gradle‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ buildscript {
1616
ext {
1717
// build.version_qualifier parameter applies to knn plugin artifacts only. OpenSearch version must be set
1818
// explicitly as 'opensearch.version' property, for instance opensearch.version=2.0.0-rc1-SNAPSHOT
19-
opensearch_version = System.getProperty("opensearch.version", "3.0.0-SNAPSHOT")
19+
opensearch_version = System.getProperty("opensearch.version", "3.2.0-SNAPSHOT")
2020
version_qualifier = System.getProperty("build.version_qualifier", "")
2121
opensearch_group = "org.opensearch"
2222
isSnapshot = "true" == System.getProperty("build.snapshot", "true")
@@ -58,7 +58,7 @@ plugins {
5858
id 'java-test-fixtures'
5959
id 'idea'
6060
id "com.diffplug.spotless" version "6.25.0" apply false
61-
id 'io.freefair.lombok' version '8.13.1'
61+
id 'io.freefair.lombok' version '8.14'
6262
id "de.undercouch.download" version "5.3.0"
6363
id "me.champeau.jmh" version "0.7.1"
6464
}

‎gradle/wrapper/gradle-wrapper.properties‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
#Wed Feb 12 15:53:09 PST 2025
22
distributionBase=GRADLE_USER_HOME
33
distributionPath=wrapper/dists
4-
distributionUrl=https\://services.gradle.org/distributions/gradle-8.10-bin.zip
4+
distributionUrl=https\://services.gradle.org/distributions/gradle-8.14.3-all.zip
55
zipStoreBase=GRADLE_USER_HOME
66
zipStorePath=wrapper/dists

‎src/main/java/org/opensearch/knn/index/codec/KNN9120Codec/DerivedSourceStoredFieldsWriter.java‎

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,6 @@
99
import org.apache.lucene.codecs.StoredFieldsWriter;
1010
import org.apache.lucene.index.FieldInfo;
1111
import org.apache.lucene.index.MergeState;
12-
import org.apache.lucene.store.DataInput;
1312
import org.apache.lucene.util.BytesRef;
1413
import org.opensearch.common.collect.Tuple;
1514
import org.opensearch.common.io.stream.BytesStreamOutput;
@@ -58,11 +57,6 @@ public void writeField(FieldInfo fieldInfo, double v) throws IOException {
5857
delegate.writeField(fieldInfo, v);
5958
}
6059

61-
@Override
62-
public void writeField(FieldInfo info, DataInput value, int length) throws IOException {
63-
delegate.writeField(info, value, length);
64-
}
65-
6660
@Override
6761
public int merge(MergeState mergeState) throws IOException {
6862
// We have to wrap these here to avoid storing the vectors during merge

‎src/main/java/org/opensearch/knn/index/codec/KNN9120Codec/KNN9120BinaryVectorScorer.java‎

Lines changed: 12 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212
import org.apache.lucene.index.KnnVectorValues;
1313
import org.apache.lucene.util.hnsw.RandomVectorScorer;
1414
import org.apache.lucene.util.hnsw.RandomVectorScorerSupplier;
15+
import org.apache.lucene.util.hnsw.UpdateableRandomVectorScorer;
1516
import org.opensearch.knn.index.KNNVectorSimilarityFunction;
1617

1718
import java.io.IOException;
@@ -52,7 +53,7 @@ public RandomVectorScorer getRandomVectorScorer(
5253
throw new IllegalArgumentException("vectorValues must be an instance of RandomAccessVectorValues.Bytes");
5354
}
5455

55-
static class BinaryRandomVectorScorer implements RandomVectorScorer {
56+
static class BinaryRandomVectorScorer implements UpdateableRandomVectorScorer {
5657
private final ByteVectorValues vectorValues;
5758
private final byte[] queryVector;
5859

@@ -80,23 +81,26 @@ public int ordToDoc(int ord) {
8081
public Bits getAcceptOrds(Bits acceptDocs) {
8182
return vectorValues.getAcceptOrds(acceptDocs);
8283
}
84+
85+
@Override
86+
public void setScoringOrdinal(int node) throws IOException {
87+
System.arraycopy(vectorValues.vectorValue(node), 0, queryVector, 0, queryVector.length);
88+
}
8389
}
8490

8591
static class BinaryRandomVectorScorerSupplier implements RandomVectorScorerSupplier {
8692
protected final ByteVectorValues vectorValues;
87-
protected final ByteVectorValues vectorValues1;
88-
protected final ByteVectorValues vectorValues2;
93+
protected final ByteVectorValues targetVectors;
8994

9095
public BinaryRandomVectorScorerSupplier(ByteVectorValues vectorValues) throws IOException {
9196
this.vectorValues = vectorValues;
92-
this.vectorValues1 = vectorValues.copy();
93-
this.vectorValues2 = vectorValues.copy();
97+
this.targetVectors = vectorValues.copy();
9498
}
9599

96100
@Override
97-
public RandomVectorScorer scorer(int ord) throws IOException {
98-
byte[] queryVector = vectorValues1.vectorValue(ord);
99-
return new BinaryRandomVectorScorer(vectorValues2, queryVector);
101+
public UpdateableRandomVectorScorer scorer() throws IOException {
102+
byte[] queryVector = new byte[vectorValues.dimension()];
103+
return new BinaryRandomVectorScorer(vectorValues, queryVector);
100104
}
101105

102106
@Override

‎src/main/java/org/opensearch/knn/index/codec/KNN990Codec/QuantizationConfigKNNCollector.java‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77

88
import lombok.Getter;
99
import lombok.Setter;
10+
import org.apache.lucene.search.knn.KnnSearchStrategy;
1011
import org.apache.lucene.search.KnnCollector;
1112
import org.apache.lucene.search.TopDocs;
1213
import org.opensearch.knn.quantization.models.quantizationState.QuantizationState;
@@ -22,6 +23,11 @@ public class QuantizationConfigKNNCollector implements KnnCollector {
2223

2324
private final String NATIVE_ENGINE_SEARCH_ERROR_MESSAGE = "Search functionality using codec is not supported with Native Engine Reader";
2425

26+
@Override
27+
public KnnSearchStrategy getSearchStrategy() {
28+
throw new UnsupportedOperationException(NATIVE_ENGINE_SEARCH_ERROR_MESSAGE);
29+
}
30+
2531
@Override
2632
public boolean earlyTerminated() {
2733
throw new UnsupportedOperationException(NATIVE_ENGINE_SEARCH_ERROR_MESSAGE);

‎src/main/java/org/opensearch/knn/index/codec/jvector/JVectorKnnCollector.java‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@
66

77
import lombok.Value;
88
import org.apache.lucene.search.KnnCollector;
9+
import org.apache.lucene.search.knn.KnnSearchStrategy;
910
import org.apache.lucene.search.TopDocs;
1011

1112
/**
@@ -58,4 +59,9 @@ public float minCompetitiveSimilarity() {
5859
public TopDocs topDocs() {
5960
return delegate.topDocs();
6061
}
62+
63+
@Override
64+
public KnnSearchStrategy getSearchStrategy() {
65+
return delegate.getSearchStrategy();
66+
}
6167
}

0 commit comments

Comments
 (0)