[BUG] neural query yields inconsistent results when searching against aliases/many indexes

[swolehee]

Describe the bug

When specifying search_pipeline query parameter in a search against an alias, the result comes varies greatly. For example, in our issue, the same query can return 5117 hits, 294 hits, or 0 hits.

However, when I perform the same query on individual indexes that is in the alias, the result does not vary.

Related component

Search

To Reproduce

• Create a search pipeline - for us, it was a pipeline with neural_query_enricher

{
  "default_model_pipeline": {
    "request_processors": [
      {
        "neural_query_enricher": {
          "neural_field_default_id": {
            "searchTextEmbedding_nomic-embed-text-v1-5": "sdBJwJQBMYQjvGaa1r_u"
          }
        }
      }
    ]
  }
}

• Create indexes with a mapping that contains the field searchTextEmbedding_nomic-embed-text-v1-5. Add the index in an alias

        "searchTextEmbedding_nomic-embed-text-v1-5": {
          "type": "knn_vector",
          "dimension": 768,
          "method": {
            "engine": "faiss",
            "space_type": "l2",
            "name": "hnsw",
            "parameters": {
              "ef_construction": 100,
              "m": 16
            }
          }
        },

• Add events into the indexes

• Perform a search with the pipeline against the alias

GET routing-events-alias/_search?routing=customer123&search_pipeline=default_model_pipeline
{
  "size": 20,
  "query": {
    "neural": {
      "searchTextEmbedding_nomic-embed-text-v1-5": {
        "query_text": "dinner",
        "k": 300,
        "filter": {
          "bool": {
            "must": [
              {
                "range": {
                  "eventHeader.timestamp": {
                    "gte": 1743546731000,
                    "lte": 1746138731000
                  }
                }
              },
              {
                "term": {
                  "eventHeader.customerId": "customer123"
                }
              }
              }
            ]
          }
        }
      }
    }
  },
  "sort": [
    {
      "_score": {
        "order": "desc"
      }
    },
    {
      "eventHeader.timestamp": {
        "order": "desc"
      }
    }
  ],
  "_source": {
    "excludes": [
      "searchTextEmbedding_nomic-embed-text-v1-5"
    ]
  }
}

• Perform the same search with the pipeline against an index in the alias

GET routing-events-2025-05-01/_search?search_pipeline=default_model_pipeline
{
   ...
}

Expected behavior

Expected the results when searching against the alias to yield consistent result.

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: N/A
• AWS OpenSearch
• Version: OpenSearch 2.17

Additional context
Add any other context about the problem here.

[swolehee]

Original issue cut on neural side: opensearch-project/neural-search#1310

[msfroh]

@swolehee -- based on the discussion on the neural-search issue, where it sounds like things work when fanning out over a smaller set of indices, can you please confirm if there were any shard failures in the response? It's possible that some shards failed to respond. (Unfortunately, due to a bad decision made a long time ago, partial results are almost indistinguishable from complete results.)

[swolehee]

@msfroh thanks for getting back. There were no shard failures in the response - I'm looking at

"_shards": {
  "total": 21,
  "successful": 21,
  "skipped": 0,
  "failed": 0
}

[swolehee]

Another discovery I had is when testing on a different alias, which has just as many indexes (if not more), but the indexes are smaller (1/4 number of documents, 25% smaller shards), it happens much less frequently.

	Problematic Alias	Less problematic Alias
Number of Indexes	30	39
Avg number of documents per index	21.5 M	5.8 M
Avg shards size	40GB	27GB
Reproducibility	> 50%	~25%

(Reproducibility is purely based on manually testing and observing; we don't have exact metrics on it)

[swolehee]

Observed that passing model_id directly, instead of using search_pipeline faces the same issue. Updated the issue title accordingly.

GET routing-events-alias/_search?routing=customer123&preference=_replica_first
{
  "explain": true, 
  "size": 20,
  "query": {
    "neural": {
      "searchTextEmbedding_nomic-embed-text-v1-5": {
        "model_id": "sdBJwJQBMYQjvGaa1r_u",
        "query_text": "dinner",
        "k": 300,

[heemin32]

The issue is confirmed as a bug in k-nn and the fix is released in 2.18. See the detail in opensearch-project/neural-search#1310 (comment)