[FSTORE-1970][APPEND] Fix similarity-search find_neighbors on OpenSearch 2.19.5 (k-too-large parsing + faiss efficient filtering) (#1008)

* [FSTORE-1970][APPEND] Parse OpenSearch 2.19.5 k-NN "k too large" error

OpenSearch 2.19.5 (k-NN plugin) reports an out-of-range k via
KNNQueryBuilder.Builder.validate() as
"[knn] requires k to be in the range (0, N]", whereas 1.3.6 reported
"[knn] requires k <= N". The vector-DB error parser only matched the
old form, so the max-k discovery probe in
VectorDbClient._find_neighbors could not extract the limit, left the
exception info empty, and re-raised instead of caching the limit.
find_neighbors() on a project index therefore failed with
"Requested k is too large".

Match both message forms (the new (0, N] range and the retained <= N
form, both present in the 2.19.5 source). Also tighten the guard so it
only classifies genuine upper-bound violations as REQUESTED_K_TOO_LARGE:
"[knn] requires k > 0" and "[knn] requires exactly one of k, distance
or score to be set" now fall through to OTHERS. The "Result window is
too large" parser is unchanged; its bracketed-number format is verified
against DefaultSearchContext.java at tag 2.19.5.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* [FSTORE-1970][APPEND] Use faiss efficient k-NN filtering so find_neighbors returns k

The ee OpenSearch 2.19.5 upgrade switched the embedding-index engine from
nmslib to faiss. PR #951 had moved the similarity-search query to faiss
efficient filtering (the filter nested inside the knn clause), but #1005
reverted it to a bool/must post-filter after hitting "[knn] unknown token
[START_OBJECT] after [filter]" — the signature of an engine that does not
support in-knn filtering (nmslib), most likely seen against an index not
yet recreated under faiss.

Post-filtering retrieves the k nearest first and prunes with the filter
afterwards, so a selective filter (the per-feature-group exists clause on
a shared project index) returns fewer than k results — e.g. find_neighbors
k=10 returning 7. faiss supports efficient filtering since OpenSearch 2.9
(GA in 2.19) and applies the filter during graph traversal, guaranteeing k
results when at least k exist.

Re-apply the efficient-filter query form (reverting #1005). The explicit
oversized-k path (find_neighbors k=2**31-1) still raises
VectorDatabaseException via the first search, and the parser fix in this
branch keeps the "k too large" message parseable on 2.19.5.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: robzor92

python/hopsworks_common/core/opensearch.py

@@ -231,9 +231,19 @@ def _close(self):
 
     def _create_vector_database_exception(self, message):
         """Create appropriate VectorDatabaseException based on error message."""
-        if "[knn] requires k" in message:
-            pattern = r"\[knn\] requires k <= (\d+)"
-            match = re.search(pattern, message)
+        # Only an upper-bound violation means "k too large". Older OpenSearch
+        # reported "[knn] requires k <= N"; newer versions (2.x) report
+        # "[knn] requires k to be in the range (0, N]". The upper bound N is the
+        # inclusive max in both forms. Other "[knn] requires k ..." messages
+        # (e.g. "requires k > 0") are not too-large errors and fall through to
+        # OTHERS.
+        if (
+            "[knn] requires k <=" in message
+            or "[knn] requires k to be in the range" in message
+        ):
+            match = re.search(r"\[knn\] requires k <= (\d+)", message) or re.search(
+                r"\[knn\] requires k to be in the range \(\d+, (\d+)\]", message
+            )
             if match:
                 k = match.group(1)
                 reason = VectorDatabaseException.REQUESTED_K_TOO_LARGE

python/hsfs/core/vector_db_client.py

@@ -133,17 +133,18 @@ def _find_neighbors(
                 )
         self._check_filter(filter, embedding_feature.feature_group)
         col_name = embedding_feature.embedding_index.col_prefix + embedding_feature.name
+        filter_clauses = [
+            {"exists": {"field": col_name}},
+        ] + self._get_query_filter(filter, embedding_feature.embedding_index.col_prefix)
         query = {
             "size": k,
             "query": {
-                "bool": {
-                    "must": [
-                        {"knn": {col_name: {"vector": embedding, "k": k}}},
-                        {"exists": {"field": col_name}},
-                    ]
-                    + self._get_query_filter(
-                        filter, embedding_feature.embedding_index.col_prefix
-                    )
+                "knn": {
+                    col_name: {
+                        "vector": embedding,
+                        "k": k,
+                        "filter": {"bool": {"must": filter_clauses}},
+                    }
                 }
             },
             "_source": list(
@@ -170,7 +171,7 @@ def _find_neighbors(
             # Get the max number of results allowed to request if it is not available.
             # This is expected to be executed once only.
             if not VectorDbClient._index_result_limit_k.get(index_name):
-                query["query"]["bool"]["must"][0]["knn"][col_name]["k"] = 2**31 - 1
+                query["query"]["knn"][col_name]["k"] = 2**31 - 1
                 try:
                     # It is expected that this request ALWAYS fails because requested k is too large.
                     # The purpose here is to get the max k allowed from the vector database, and cache it.
@@ -189,7 +190,7 @@ def _find_neighbors(
                         )
                     else:
                         raise e
-            query["query"]["bool"]["must"][0]["knn"][col_name]["k"] = min(
+            query["query"]["knn"][col_name]["k"] = min(
                 VectorDbClient._index_result_limit_k.get(index_name, k), 3 * k
             )
             results = opensearch_client._search(

python/tests/core/test_opensearch.py

@@ -36,6 +36,12 @@ class TestOpenSearchClientSingleton:
                 VectorDatabaseException.REQUESTED_K_TOO_LARGE,
                 {},
             ),
+            (
+                # Newer OpenSearch k-NN range format
+                "[knn] requires k to be in the range (0, 10000]",
+                VectorDatabaseException.REQUESTED_K_TOO_LARGE,
+                {VectorDatabaseException.REQUESTED_K_TOO_LARGE_INFO_K: 10000},
+            ),
             (
                 "Result window is too large, from + size must be less than or equal to: [10000] but was [80000]",
                 VectorDatabaseException.REQUESTED_NUM_RESULT_TOO_LARGE,
@@ -47,6 +53,18 @@ class TestOpenSearchClientSingleton:
                 VectorDatabaseException.REQUESTED_NUM_RESULT_TOO_LARGE,
                 {},
             ),
+            (
+                # Lower-bound violation is not a "too large" error
+                "[knn] requires k > 0",
+                VectorDatabaseException.OTHERS,
+                {},
+            ),
+            (
+                # Unrelated knn validation must not be misclassified as too large
+                "[knn] requires exactly one of k, distance or score to be set",
+                VectorDatabaseException.OTHERS,
+                {},
+            ),
             (
                 "Some other error message",
                 VectorDatabaseException.OTHERS,

python/tests/core/test_vector_db_client.py

@@ -494,25 +494,25 @@ def test_find_neighbors_builds_knn_query_without_filter(self):
 
         body = self.mock_os_wrapper._search.call_args.kwargs["body"]
         assert body["size"] == 5
-        # The knn clause and filters are combined in a bool query. OpenSearch
-        # 2.19 rejects a `filter` nested inside the knn clause.
-        must = body["query"]["bool"]["must"]
-        knn = must[0]["knn"]["f2"]
+        knn = body["query"]["knn"]["f2"]
         assert knn["vector"] == [1.0, 2.0, 3.0]
         assert knn["k"] == 5
-        assert "filter" not in knn
-        assert must[1] == {"exists": {"field": "f2"}}
+        # The filter lives inside the knn clause (OpenSearch 2.x KNN syntax),
+        # not as a sibling bool/post_filter.
+        assert knn["filter"] == {"bool": {"must": [{"exists": {"field": "f2"}}]}}
 
     def test_find_neighbors_builds_knn_query_with_filter(self):
         self.target._find_neighbors(
             [1.0, 2.0, 3.0], feature=self.f2, k=5, filter=self.f3 > 10
         )
 
         body = self.mock_os_wrapper._search.call_args.kwargs["body"]
-        must = body["query"]["bool"]["must"]
-        knn = must[0]["knn"]["f2"]
-        assert "filter" not in knn
-        assert must[1:] == [
-            {"exists": {"field": "f2"}},
-            {"range": {"f3": {"gt": 10}}},
-        ]
+        knn = body["query"]["knn"]["f2"]
+        assert knn["filter"] == {
+            "bool": {
+                "must": [
+                    {"exists": {"field": "f2"}},
+                    {"range": {"f3": {"gt": 10}}},
+                ]
+            }
+        }