Criteria.or() with .exists().not() generates incorrect query - negation is lost

[diegourban]

What is the bug?

When combining .or() with .exists().not() in the Criteria API, the negation is lost or misapplied, producing incorrect query results.
Standalone .exists().not() works.

How can one reproduce the bug?

Given three documents in an index:

ID	name	status
product-1	Widget	ACTIVE
product-2	Gadget	INACTIVE
product-3	Gizmo	(field missing)

Execute the following Criteria query:

Criteria criteria = new Criteria("status").is("ACTIVE").or(new Criteria("status").exists().not());
CriteriaQuery query = new CriteriaQuery(criteria);
SearchHits<Product> hits = operations.search(query, Product.class);

What is the expected behavior?

Expected: Returns product-1 (status = ACTIVE) and product-3 (status field does not exist).
Actual: Does not return the expected results, the .exists().not() negation inside .or() is lost.

What is your host/environment?

• OpenSearch version: 3.5.0
• Spring Boot version: 4.0.2
• spring-data-opensearch version: 3.0.2
• Java version: 21

Do you have any additional context?

Minimal reproducible example:
https://github.com/diegourban/spring-data-opensearch-criteria-or-not-exists-issue

[sothawo]

I cannot reproduce this error with the Spring-Data-Elasticsearch setup using Elasticsearch 9.2.5.

I copied the Product class and the test code for the testCriteriaIsOrExistsNot method into the spring-data-elasticsearch project and ran it - no error here.

I noticed that you seem to expect a different query than what is built, the query that Spring Data Elasticsearch creates is a query_string query, not a term query:

{
  "bool": {
    "should": [
      {
        "query_string": {
          "default_operator": "and",
          "fields": [
            "status"
          ],
          "query": "ACTIVE"
        }
      },
      {
        "bool": {
          "must_not": [
            {
              "exists": {
                "field": "status"
              }
            }
          ]
        }
      }
    ]
  }
}

In the built query there is no problem with the negation, but it has a different query for the status field. The CriteriaQueryProcessor that builds this query has no information about the types of that are used, it always builds a query_string query on the field for .is().

To make sure that the index is setup in the correct way and to check the query and responses I ran the test while monitoring the network traffic with an intercepting proxy.
These are the requests and responses:

create index

{"aliases":{},"mappings":{"properties":{"name":{"type":"keyword"},"_class":{"type":"keyword","doc_values":false,"index":false},"status":{"type":"keyword"}}},"settings":{"index":{"number_of_shards":"1","number_of_replicas":"1","refresh_interval":"1s"}}}

{"acknowledged":true,"shards_acknowledged":true,"index":"criteria-1-315609-product"}

save documents (omitting the responses here)

{"_class":"org.springframework.data.elasticsearch.core.query.CriteriaQueryIntegrationTests$Product","id":"product-1","name":"Widget","status":"ACTIVE"}
{"_class":"org.springframework.data.elasticsearch.core.query.CriteriaQueryIntegrationTests$Product","id":"product-2","name":"Gadget","status":"INACTIVE"}
{"_class":"org.springframework.data.elasticsearch.core.query.CriteriaQueryIntegrationTests$Product","id":"product-3","name":"Gizmo"}

search

{
  "from": 0,
  "query": {
    "bool": {
      "should": [
        {
          "query_string": {
            "default_operator": "and",
            "fields": [
              "status"
            ],
            "query": "ACTIVE"
          }
        },
        {
          "bool": {
            "must_not": [
              {
                "exists": {
                  "field": "status"
                }
              }
            ]
          }
        }
      ]
    }
  },
  "size": 10000,
  "track_scores": false,
  "version": true
}

{
  "took": 45,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 2,
      "relation": "eq"
    },
    "max_score": 0.6931471,
    "hits": [
      {
        "_index": "criteria-1-315609-product",
        "_id": "product-1",
        "_version": 1,
        "_score": 0.6931471,
        "_source": {
          "_class": "org.springframework.data.elasticsearch.core.query.CriteriaQueryIntegrationTests$Product",
          "id": "product-1",
          "name": "Widget",
          "status": "ACTIVE"
        }
      },
      {
        "_index": "criteria-1-315609-product",
        "_id": "product-3",
        "_version": 1,
        "_score": 0.0,
        "_source": {
          "_class": "org.springframework.data.elasticsearch.core.query.CriteriaQueryIntegrationTests$Product",
          "id": "product-3",
          "name": "Gizmo"
        }
      }
    ]
  }
}

It seems that Opensearch handles this query on a field of type keyword differently that Elasticsearch does.
You should take this problem to the Opensearch teams then.