Advanced Filtering on Array Fields in Milvus Lite gives "Invalid expr" error

[MohmedMonsef]

I am using pymilvus version: 2.4.4

When trying to re-produce the code in documentations for advanced filtering on array fields https://milvus.io/docs/array_data_type.md#Advanced-filtering:

res = client.query(
    collection_name="test_collection",
    filter="ARRAY_CONTAINS_ANY(color_coord, [7, 8, 9])",
    output_fields=["id", "color", "color_tag", "color_coord"],
    limit=3
)

It gives the following error:

MilvusException: <MilvusException: (code=1100, message=Invalid expr: ARRAY_CONTAINS_ANY(color_coord, [7, 8, 9]): invalid parameter)>

So, how to perform this operation correctly?

[yhmo]

ARRAY_CONTAINS_ANY is parsed in server side. It requires the milvus server version greater or equal than v2.3.2.

Double check your milvus server version.

[MohmedMonsef]

Thanks for your quick response.

In the documentation, they show the following example:

import random, time
from pymilvus import MilvusClient, DataType

CLUSTER_ENDPOINT = "http://localhost:19530"

# 1. Set up a Milvus client
client = MilvusClient(
    uri=CLUSTER_ENDPOINT
)
...

However, when I try this in Google Colab, I encounter the following error:

MilvusException: <MilvusException: (code=2, message=Fail connecting to server on localhost:19530, illegal connection params or server unavailable)>

ERROR:pymilvus.milvus_client.milvus_client:Failed to create new connection using: 4489a36b942a47618597a06f06ff55f0

Instead, I use this approach:

client = MilvusClient(
    "Test.db"
)

and it connects to the database, but the filter operation does not work.

According to your response, I think this could be the problem but I am new to Milvus and don't fully understand the difference between these methods. How can I resolve this issue?

[yhmo]

client = MilvusClient("Test.db") ------------------- This is a milvus lite, which is a lightweight milvus, it only supports part of functions. Array type is not supported by milvus lite.

Follow this document to deploy a milvus standalone server in local:
https://milvus.io/docs/install_standalone-docker-compose.md

If the deployment is correct, you will see 3 containers:

Then use this script to test:

import random
import time

from pymilvus import (
    MilvusClient, Index,
    FieldSchema, CollectionSchema, DataType,
    utility,
)


client = MilvusClient("http://localhost:19530")

collection_name = "AAA"
dim = 128
metric_type = "COSINE"

schema = CollectionSchema(
    fields=[
        FieldSchema(name="id", dtype=DataType.INT64, is_primary=True, auto_id=False),
        FieldSchema(name="vector", dtype = DataType.FLOAT_VECTOR, dim=dim),
        FieldSchema(name="color_coord", dtype=DataType.ARRAY, element_type=DataType.INT64, max_capacity=5)
    ])

client.create_collection(collection_name=collection_name, schema=schema)
print(f"Collection '{collection_name}' created")

index_params = client.prepare_index_params()
index_params.add_index(field_name = "vector", metric_type="L2")
client.create_index(collection_name, index_params)

client.insert(collection_name=collection_name, data={"id": 1, "vector": [random.random() for _ in range(dim)], "color_coord": [1, 2]})
client.insert(collection_name=collection_name, data={"id": 2, "vector": [random.random() for _ in range(dim)], "color_coord": [4, 8]})
client.insert(collection_name=collection_name, data={"id": 3, "vector": [random.random() for _ in range(dim)], "color_coord": [7, 9]})

client.load_collection(collection_name=collection_name)

results = client.query(collection_name=collection_name, filter='ARRAY_CONTAINS_ANY(color_coord, [7, 8, 9])',
                       output_fields=["id"], consistency_level="Strong")
ids = [res["id"] for res in results]
print(ids)

[MohmedMonsef]

Thanks, this is really helpful.
But it should be clearly stated in the documentation.

[ifrancke-pryoninc]

Could you please add this to the filter row of this table on the Milvus Lite page? https://milvus.io/docs/milvus_lite.md#:~:text=N-,Insert%20%26%20Search,-Method%20/%20Parameter

[xiaofan-luan]

@junjiejiangjjj
please help on support array types.

[junjiejiangjjj]

@presburger is currently working on supporting the array type and will release a new version once it is ready.

[junjiejiangjjj]

Array-related operations are supported in milvus-lite 2.4.12

[ifrancke-pryoninc]

That's amazing, thank you!