Integrating Lucene's Better Binary Quantization #2838

adityamachiroutu · 2025-08-07T19:06:40Z

Description

Integrates Lucene’s Better Binary Quantization (BBQ) into the OpenSearch k-NN plugin, enabling a memory-efficient encoding option for high-dimensional vector search. Compared to Faiss binary quantization, BBQ has increased recall, while maintaining low memory usage and integrating well into existing query and rescoring pipelines.

This integration exposes BBQ through the Lucene engine with a new "encoder": "binary" parameter in index mappings, leveraging Lucene’s native vector quantization and storage formats. BBQ improves recall on large datasets with a modest trade-off in latency and throughput, as demonstrated in the benchmarks below. Users can adjust the oversample factor in queries to balance recall and performance. By default the oversample factor set is 3x for vectors with 1000+ dimensions, and 5x otherwise. This matches the FAISS 32x compression defaults set.

Users can additionally configure the BBQ encoder by setting their engine as Lucene and the compression level to 32x, from which BBQ is automatically enabled.

Benchmarking Results:

No Rescoring Tests Comparing FAISS BQ to Lucene BBQ

Metric	Sift-128, Lucene BBQ	Sift-128, FaissBQ	Cohere-1m, Lucene BBQ	Cohere-1m, FaissBQ	Unit
Mean recall@k	0.32	0.18	0.63	0.3
50th percentile service time	5.69515	5.71014	5.88795	5.58729	ms
90th percentile service time	8.88647	7.59109	7.74107	6.48138	ms
99th percentile service time	18.26953	17.57099	19.85139	17.39427	ms
Mean Throughput	6310.84	9569.38	3011.83	3112.96	docs/s
Median Throughput	6226.56	9582.13	3065.34	3152.94	docs/s

Config:
1 shard, 1 segments, 10 indexing clients, k=100, ef_search: 100, ef_construction: 100

Single Node Tests Comparing FAISS BQ (on disk) to Lucene BBQ

Dataset	32x Compression Technique	Recall@100	p90 ms latency (1 shard, 1 segment)
clip-flickr-image-text-queries	Lucene BBQ	0.91	96.1
	Faiss BQ (on disk)	0.81	58.24
cohere-v2-dbpedia	Lucene BBQ	0.75	52.4
	Faiss BQ (on disk)	0.76	12.29
cohere-v2-wiki	Lucene BBQ	0.9	58.9
	Faiss BQ (on disk)	0.9	15.74
cohere-v3-bioasq	Lucene BBQ	0.73	226.63
	Faiss BQ (on disk)	0.73	17.97
e5small-msmarco	Lucene BBQ	0.9	40.06
	Faiss BQ (on disk)	0.8	16.04
gist	Lucene BBQ	0.64	49.67
	Faiss BQ (on disk)	0.12	46.84
glove	Lucene BBQ	0.63	4.37
	Faiss BQ (on disk)	0.42	4.37
minilm-msmarco	Lucene BBQ	0.97	2.81
	Faiss BQ (on disk)	0.9	3.6
mpnet-msmarco	Lucene BBQ	0.98	4.98
	Faiss BQ (on disk)	0.95	3.68
mbread_marco	Lucene BBQ	0.96	5.93
	Faiss BQ (on disk)	0.92	4.96
sift	Lucene BBQ	0.69	5.23
	Faiss BQ (on disk)	0.39	3.72
snowflake-msmarco	Lucene BBQ	0.94	70.41
	Faiss BQ (on disk)	0.93	47.79
tasb-msmarco	Lucene BBQ	0.95	9.73
	Faiss BQ (on disk)	0.87	3.93

Config: ef_search: 256, ef_construction: 256, m: 16, 1 shard, 1 client, 1 segment

These tests compare Lucene’s better binary quantization with Faiss BQ (on disk), which is the current default quantization method used by OpenSearch when undergoing 32x compression.
Overall, we see that Lucene BBQ achieves higher recall compared to Faiss BQ on disk. This occurs for all nearly datasets, regardless of space type, dimensionality, dataset size, or modality of the data.
FAISS binary quantization significantly outperforms BBQ in p90 latency across all datasets.
Both models seems to struggle with image datasets compared to text datasets, although Lucene’s BBQ still outperformed FAISS
As dataset sizes increases, both quantization techniques scaled effectively, and the trade off recall to latency persisted
When accuracy is the focus, Lucene BBQ seems to be the use case fit, while Faiss BQ is more suitable for latency sensitive times.

Related Issues

Resolves #2805

Check List

New functionality includes testing.
New functionality has been documented.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

naveentatikonda · 2025-08-08T16:42:04Z

@adityamachiroutu as mentioned offline, can you double check the oversampling factor of Faiss BQ with a debugger(check the value of this firstPassK). I believe it's 5x not 3x if dimension of vector is less than 1000 - https://github.com/opensearch-project/k-NN/blob/main/src/main/java/org/opensearch/knn/index/mapper/CompressionLevel.java#L119

finnroblin · 2025-08-08T21:29:53Z

@naveentatikonda it's 5 for dimension < 1000 and 3 for >= 1000 as you mentioned. I checked with debugger last month. We also have a unit test confirming this: https://github.com/opensearch-project/k-NN/blob/b7fc5dd98072ea157a9b301e7f93d79e97[…]java/org/opensearch/knn/index/mapper/CompressionLevelTests.java

src/testFixtures/java/org/opensearch/knn/KNNRestTestCase.java

CHANGELOG.md

qa/restart-upgrade/build.gradle

src/main/java/org/opensearch/knn/index/codec/BasePerFieldKnnVectorsFormat.java

…which calls the lucene codec that implements better binary quantization Signed-off-by: Aditya Machiroutu <[email protected]>

…tless Signed-off-by: Aditya Machiroutu <[email protected]>

Signed-off-by: Aditya Machiroutu <[email protected]>

finnroblin

Few small comments and questions, thanks Aditya!

finnroblin · 2025-08-27T01:48:47Z

CHANGELOG.md

 ## [Unreleased 3.3](https://github.com/opensearch-project/k-NN/compare/main...HEAD)
+
+### Features
+* Integrated Lucene's better binary quantization [#2838](https://github.com/opensearch-project/k-NN/pull/2838)


Nit: present tense

finnroblin · 2025-08-27T01:49:41Z

qa/restart-upgrade/src/test/java/org/opensearch/knn/bwc/IndexingIT.java

        }
    }
+
+    @AwaitsFix(bugUrl = "https://github.com/opensearch-project/k-NN/issues/2805")


Is this meant to be here?

finnroblin · 2025-08-27T01:53:09Z

src/main/java/org/opensearch/knn/index/codec/BasePerFieldKnnVectorsFormat.java

        if (engine == KNNEngine.LUCENE) {
            if (params != null && params.containsKey(METHOD_ENCODER_PARAMETER)) {
+                KNNBBQVectorsFormatParams bbqParams = new KNNBBQVectorsFormatParams(params, defaultMaxConnections, defaultBeamWidth);
+                if (bbqParams.validate(params)) {


Can we please add debug log like the below?

finnroblin · 2025-08-27T01:54:38Z

...g/opensearch/knn/index/codec/backward_codecs/KNN990Codec/KNN990PerFieldKnnVectorsFormat.java

+            ),
+            knnBBQVectorsFormatParams -> new Lucene102HnswBinaryQuantizedVectorsFormat(
+                knnBBQVectorsFormatParams.getMaxConnections(),
+                knnBBQVectorsFormatParams.getBeamWidth(),


Is there no constructor parameter for bits like above?

No, Lucene does not have that constructor.

finnroblin · 2025-08-27T01:55:50Z

src/main/java/org/opensearch/knn/index/codec/params/KNNBBQVectorsFormatParams.java

+     * Check if BBQ is enabled
+     * @return true if BBQ is enabled, false otherwise
+     */
+    public boolean isBBQEnabled() {


What's the purpose of this method?

finnroblin · 2025-08-27T01:57:27Z

src/main/java/org/opensearch/knn/index/engine/lucene/LuceneMethodResolver.java

        MethodComponentContext methodComponentContext = resolvedKNNMethodContext.getMethodComponentContext();
-        MethodComponentContext encoderComponentContext = new MethodComponentContext(SQ_ENCODER.getName(), new HashMap<>());
+
+        String encoderName = (resolvedCompressionLevel == CompressionLevel.x32)


Nit: since there are multiple clauses let's make this one if check of x32 compression

finnroblin · 2025-08-27T01:58:15Z

src/main/java/org/opensearch/knn/index/mapper/CompressionLevel.java

     *                  is invalid.
     */
    public RescoreContext getDefaultRescoreContext(Mode mode, int dimension, Version version) {
-        // TODO move this to separate class called resolver to resolve rescore context


Does your PR address this TODO? If not can we please leave it in?

finnroblin · 2025-08-27T02:02:31Z

src/test/java/org/opensearch/knn/index/engine/lucene/LuceneMethodResolverTests.java

        );

-        // Invalid compression
-        expectThrows(


Let's keep this in and change to x16 to still throw with a comment explaining that x32 added in this PR

Signed-off-by: Aditya Machiroutu <[email protected]>

navneet1v · 2025-08-27T16:23:02Z

@adityamachiroutu during your tests on different datasets was the rescoring added? or this is just bare bone testing of the quantization techniques?

adityamachiroutu · 2025-08-27T17:00:20Z

@adityamachiroutu during your tests on different datasets was the rescoring added? or this is just bare bone testing of the quantization techniques?

Yes, during my tests, rescoring was added. For consistency, the same oversampling factor (5x for vectors with <1000 dimensions, 3x for vectors > 1000 dimensions) was kept for Faiss and Lucene runs.

finnroblin · 2025-08-27T19:03:16Z

src/main/java/org/opensearch/knn/index/codec/BasePerFieldKnnVectorsFormat.java

+                KNNBBQVectorsFormatParams bbqParams = new KNNBBQVectorsFormatParams(params, defaultMaxConnections, defaultBeamWidth);
+                if (bbqParams.validate(params)) {
+                    log.debug(
+                        "Initialize KNN vector format for field [{}] with params [{}] = \"{}\", [{}] = \"{}\"",


Will the log statement have BBQ in it? Idea is for the log to distinguish whether BBQ or ScalarQuantized format was instantiated.

kotwanikunal · 2025-08-27T19:47:58Z

src/main/java/org/opensearch/knn/index/codec/params/KNNBBQVectorsFormatParams.java

+            return false;
+        }
+
+        if (!(params.get(METHOD_ENCODER_PARAMETER) instanceof MethodComponentContext)) {


nit: use == false check. It's standard across OpenSearch

if ((params.get(METHOD_ENCODER_PARAMETER) instanceof MethodComponentContext) == false)

kotwanikunal · 2025-08-27T19:52:12Z

src/main/java/org/opensearch/knn/index/mapper/CompressionLevel.java

+    // Add new method signature with KNNEngine parameter
+    public RescoreContext getDefaultRescoreContext(Mode mode, int dimension, Version version, KNNEngine engine) {
        // TODO move this to separate class called resolver to resolve rescore context
        if (modesForRescore.contains(mode)) {


We would also need a version check similar to here - it could be a restore scenario across versions and the encoder might not be supported, unless we backport it.

thanks for reviewing, addressed your comments.

Signed-off-by: Aditya Machiroutu <[email protected]>

naveentatikonda · 2025-08-28T22:49:57Z

qa/restart-upgrade/build.gradle

                knn_bwc_version.startsWith("2.15.")) {
            filter {
                excludeTestsMatching "org.opensearch.knn.bwc.IndexingIT.testKNNIndexLuceneQuantization"
+                excludeTestsMatching "org.opensearch.knn.bwc.IndexingIT.testKNNIndexLuceneBBQ"


Why are you trying to exclude this test only until version 2.15 ? it won't work until version 3.2 ?

naveentatikonda · 2025-08-28T22:50:18Z

qa/restart-upgrade/src/test/java/org/opensearch/knn/bwc/IndexingIT.java

    private static final int NUM_DOCS = 10;
    private static int QUERY_COUNT = 0;

+    private static final String ALGO = "hnsw";


remove this constant

naveentatikonda · 2025-08-28T22:53:41Z

qa/restart-upgrade/src/test/java/org/opensearch/knn/bwc/IndexingIT.java

    }
+
+    public void testKNNIndexLuceneBBQ() throws Exception {
+        waitForClusterHealthGreen(NODES_BWC_CLUSTER);


We don't need to add the same condition to validate BWC version twice in if and else blocks, probably add it here after the cluster is green

if (!isBBQEncoderSupported(getBWCVersion())) { logger.info("Skipping testKNNIndexLuceneBBQ as BBQ encoder is not supported in version: {}", getBWCVersion()); return; }

naveentatikonda · 2025-08-28T23:12:02Z

src/main/java/org/opensearch/knn/index/engine/lucene/LuceneBBQEncoder.java

+    private static final Set<VectorDataType> SUPPORTED_DATA_TYPES = ImmutableSet.of(VectorDataType.FLOAT);
+
+    private final static MethodComponent METHOD_COMPONENT = MethodComponent.Builder.builder(ENCODER_BBQ)
+        .addSupportedDataTypes(SUPPORTED_DATA_TYPES)


As discussed offline in the past, can you add support for bits parameter and set default to 1 bit (32x compression) such that in the future if Lucene supports 2 and 4 bits we can use this parameter.

For the naming convention, pls keep it consistent with Faiss BQ

naveentatikonda · 2025-08-28T23:27:04Z

@adityamachiroutu can you also share the benchmarking results that you have without rescoring? Also, pls share your findings once you identify the hotspots for the reason behind this huge spike in latencies especially with cohere datasets (4x to 12x higher when compared to Faiss). Thanks!

naveentatikonda · 2025-08-28T23:40:39Z

Benchmarking Results:

Single Node Tests Comparing FAISS BQ (on disk) to Lucene BBQ

Dataset 32x Compression Technique Recall@100 p90 ms latency (1 shard, 1 segment)

@adityamachiroutu can you add the configuration used for running these tests specifically m, ef_construction ?

Signed-off-by: Aditya Machiroutu <[email protected]>

finnroblin · 2025-08-29T17:20:28Z

src/main/java/org/opensearch/knn/index/mapper/CompressionLevel.java

-    RescoreContext getDefaultRescoreContext(Mode mode, int dimension) {
-        return getDefaultRescoreContext(mode, dimension, Version.CURRENT);
+        // Special handling for Lucene BBQ (x32 compression)
+        if (this == x32 && engine == KNNEngine.LUCENE && version.onOrAfter(Version.V_3_2_0)) {


Should be after V_3_3_0, since this was not added as a part of 3.2.

no, version.onOrAfter(Version.V_3_3_0)

Signed-off-by: Aditya Machiroutu <[email protected]>

finnroblin reviewed Aug 8, 2025

View reviewed changes

src/testFixtures/java/org/opensearch/knn/KNNRestTestCase.java Outdated Show resolved Hide resolved

adityamachiroutu marked this pull request as ready for review August 8, 2025 23:22

adityamachiroutu requested review from 0ctopus13prime, VijayanB, Vikasht34, heemin32, jmazanec15, junqiu-lei, luyuncheng, martin-gaievski, naveentatikonda, navneet1v, ryanbogan, shatejas and vamshin as code owners August 8, 2025 23:22

kotwanikunal reviewed Aug 11, 2025

View reviewed changes

CHANGELOG.md Show resolved Hide resolved

qa/restart-upgrade/build.gradle Outdated Show resolved Hide resolved

src/main/java/org/opensearch/knn/index/codec/BasePerFieldKnnVectorsFormat.java Outdated Show resolved Hide resolved

adityamachiroutu changed the title ~~[DRAFT] Integrating Lucene's Better Binary Quantization~~ Integrating Lucene's Better Binary Quantization Aug 11, 2025

adityamachiroutu force-pushed the bbq-lucene-integration branch from 4094073 to 6471a07 Compare August 11, 2025 17:58

adityamachiroutu added 10 commits August 11, 2025 11:06

pre-design review PoC of my bbq lucine integration, added an encoder …

c181d5c

…which calls the lucene codec that implements better binary quantization Signed-off-by: Aditya Machiroutu <[email protected]>

renamed encoder from bbq to binary to match faiss implementation, spo…

f8b81e0

…tless Signed-off-by: Aditya Machiroutu <[email protected]>

added unit tests

b4c1d89

Signed-off-by: Aditya Machiroutu <[email protected]>

unit tests

d925212

Signed-off-by: Aditya Machiroutu <[email protected]>

starting BWC tests

5d15c2f

Signed-off-by: Aditya Machiroutu <[email protected]>

integration/bwc tests

3307128

Signed-off-by: Aditya Machiroutu <[email protected]>

spotless

f68cd43

Signed-off-by: Aditya Machiroutu <[email protected]>

removing comments, spotless, removing unneeded tests

011fe7d

Signed-off-by: Aditya Machiroutu <[email protected]>

removing comments, spotless, removing unneeded tests

ad6c48e

Signed-off-by: Aditya Machiroutu <[email protected]>

updated bwc test

f5d8a4a

Signed-off-by: Aditya Machiroutu <[email protected]>

adityamachiroutu and others added 2 commits August 21, 2025 14:46

Merge branch 'main' into bbq-lucene-integration

83a2aa2

Merge branch 'test-oversample' into bbq-lucene-integration

02b0cd7

adityamachiroutu force-pushed the bbq-lucene-integration branch 2 times, most recently from e9ee3b5 to b814e1f Compare August 25, 2025 17:12

upgrade to 3.3

6812655

Signed-off-by: Aditya Machiroutu <[email protected]>

adityamachiroutu force-pushed the bbq-lucene-integration branch from b814e1f to 6812655 Compare August 25, 2025 20:18

adityamachiroutu added 4 commits August 25, 2025 15:43

fixed defaulting rescoring functionality

79cdd79

Signed-off-by: Aditya Machiroutu <[email protected]>

spotless

0abc8dc

Signed-off-by: Aditya Machiroutu <[email protected]>

added unit tests for compression level change

2a85559

Signed-off-by: Aditya Machiroutu <[email protected]>

spotless

379520b

Signed-off-by: Aditya Machiroutu <[email protected]>

adityamachiroutu force-pushed the bbq-lucene-integration branch 2 times, most recently from 988d959 to f8aea0c Compare August 26, 2025 17:59

Merge branch 'main' into bbq-lucene-integration

a8a31c7

adityamachiroutu force-pushed the bbq-lucene-integration branch from f8aea0c to a8a31c7 Compare August 26, 2025 20:43

finnroblin suggested changes Aug 27, 2025

View reviewed changes

addressed comments

aa29c46

Signed-off-by: Aditya Machiroutu <[email protected]>

adityamachiroutu changed the base branch from main to feature/lucene-bbq August 27, 2025 17:33

finnroblin reviewed Aug 27, 2025

View reviewed changes

kotwanikunal reviewed Aug 27, 2025

View reviewed changes

addressed further comments

6e783da

Signed-off-by: Aditya Machiroutu <[email protected]>

naveentatikonda reviewed Aug 28, 2025

View reviewed changes

resolved comments

e3bc2dc

Signed-off-by: Aditya Machiroutu <[email protected]>

adityamachiroutu force-pushed the bbq-lucene-integration branch from 3994def to e3bc2dc Compare August 29, 2025 17:04

finnroblin reviewed Aug 29, 2025

View reviewed changes

fixed versioning for compression level

ca8cb09

Signed-off-by: Aditya Machiroutu <[email protected]>

adityamachiroutu force-pushed the bbq-lucene-integration branch from f2ad1d6 to ca8cb09 Compare August 29, 2025 17:36

Integrating Lucene's Better Binary Quantization #2838

Are you sure you want to change the base?

Integrating Lucene's Better Binary Quantization #2838

Conversation

adityamachiroutu commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Benchmarking Results:

No Rescoring Tests Comparing FAISS BQ to Lucene BBQ

Single Node Tests Comparing FAISS BQ (on disk) to Lucene BBQ

Related Issues

Check List

Uh oh!

naveentatikonda commented Aug 8, 2025

Uh oh!

finnroblin commented Aug 8, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

finnroblin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

navneet1v commented Aug 27, 2025

Uh oh!

adityamachiroutu commented Aug 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

naveentatikonda commented Aug 28, 2025

Uh oh!

naveentatikonda commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarking Results:

Single Node Tests Comparing FAISS BQ (on disk) to Lucene BBQ

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

adityamachiroutu commented Aug 7, 2025 •

edited

Loading

naveentatikonda commented Aug 28, 2025 •

edited

Loading