Filters using CMIP6 properties appear to have no effect

[nocollier]

Apologies @rhysrevans3 if this issue is misplaced.

I am trying to learn how to do more complex searches with the STAC API and this extension through the API you all have at CEDA. I can do a simple query:

client = pystac_client.Client.open("https://api.stac.ceda.ac.uk")
results = client.search(
    collections="cmip6",
    max_items=None,
    limit=100,
    query=[
        "cmip6:variable_id=rsus",
        "cmip6:experiment_id=historical",
    ],
) # This works!

But I could not figure out how to use query to do more complex searches. Say I wanted all records where the variable was rsus OR rsds. I read on the STAC extensions documentation for query that filter was preferred--more feature rich and used a standard language. However I find this does not work:

results = client.search(
    collections="cmip6",
    max_items=10,
    filter={"op": "=", "args": [{"property": "cmip6:variable_id"}, "rsus"]},
) # Returns 0 matched items

I suspect that it has something to do with the CMIP6 extension, but I am not sure. I am fairly certain it is not user error. Any thoughts?

[rhysrevans3]

Sorry @nocollier I hadn't had time to respond to this. You're right the filter extension is the correct way to do more complex queries. I think the property you're searching on is properties.cmip6:variable_id I believe this is to allow you to include other non properties variables (collection, id, etc.) in your query. But if this isn't desired behaviour let me know and I can raise it with in the API.

So a search for rsus OR rsds would look something like:

results = client.search(
    collections="cmip6",
    max_items=10,
    filter={
      "op": "or",
      "args": [
          {
            "args": ["property": "cmip6:variable_id"}, "rsus"],
            "op": "="
          },
          {
            "args": ["property": "cmip6:variable_id"}, "rsds"],
            "op": "="
          }
      ]
    }
)

I've tested that filter query using a post search here and am returned 806 records is that about how many you expected?

[nocollier]

@rhysrevans3 Thanks so much for taking a minute to look. I am relieved that it is user error, but I am still not getting results. What am I doing wrong? Here is your query with a fixed typo (I think { is needed before "property").

import pystac_client

client = pystac_client.Client.open("https://api.stac.ceda.ac.uk")
results = client.search(
    collections="cmip6",
    max_items=10,
    filter={
        "op": "or",
        "args": [
            {"args": [{"property": "cmip6:variable_id"}, "rsus"], "op": "="},
            {"args": [{"property": "cmip6:variable_id"}, "rsds"], "op": "="},
        ],
    },
)
print(results.item_collection_as_dict())
# {'type': 'FeatureCollection', 'features': []}
print(list(results.items_as_dicts()))
# []

It must be a stupid mistake, but I cannot see what I am missing?

[rhysrevans3]

Sorry @nocollier I coppied and pasted your search and then didn't make the change I recommended 🤦 so you need to include properties. before the variable name your property args

import pystac_client

client = pystac_client.Client.open("https://api.stac.ceda.ac.uk")
results = client.search(
    collections="cmip6",
    max_items=10,
    filter={
        "op": "or",
        "args": [
            {"args": [{"property": "properties.cmip6:variable_id"}, "rsus"], "op": "="},
            {"args": [{"property": "properties.cmip6:variable_id"}, "rsds"], "op": "="},
        ],
    },
)
print(results.item_collection_as_dict())
print(list(results.items_as_dicts()))

[nocollier]

Thanks so much Rhys, I am not sure I would have ever thought to add properties..

I wonder why it is needed here? When I look at the filter examples, I see one using the eo STAC extension which adds cloud_cover:

https://github.com/stac-api-extensions/filter?tab=readme-ov-file#example-2-post-with-cql2-json

      {
        "op": "<=",
        "args": [ { "property": "eo:cloud_cover" }, 10 ]
      }

This is what led me to think that something was off about the extension. Maybe there is a way to implement it that circumvents the need for properties.. After all, the cmip6 namespace should be enough?

This is sufficient for me, we just need to communicate it to the userbase somehow. Thanks again!

[rhysrevans3]

From the stac-api-extensions

Fields in Item Properties should be exposed with their un-prefixed names, and not require
expressions to prefix them with properties, e.g., eo:cloud_cover instead of properties.eo:cloud_cover.

So it looks like our API is not behaving correctly. I'll let you know once I've found a solution to this and updated the API.

[nocollier]

What I noticed in the eo extension is that they define all their properties in definitions at the top level and then refer to them later in definitions:fields with a reference "$ref": "#/definitions/eo:cloud_cover". Could it be that making them available at that top level allows their discovery without the properties. prefix? Just a thought, I don't know how to dig in much deeper.