Merge pull request #907 from greengori11a/new-master

Implement FulltextFilter

Author: mariusconjeaud

doc/source/semantic_indexes.rst

@@ -1,20 +1,20 @@
-.. _Semantic Indexes: 
+.. _Semantic Indexes:
 
 ==================================
 Semantic Indexes
 ==================================
 
 Full Text Index
 ----------------
-From version x.x (version number tbc) neomodel provides a way to interact with neo4j `Full Text indexing <https://neo4j.com/docs/cypher-manual/current/indexes/semantic-indexes/full-text-indexes/>`_. 
-The Full Text Index can be be created for both node and relationship properties. Only available for Neo4j version 5.16 or higher.
+From version 6.0.0 neomodel provides a way to interact with neo4j `Full Text indexing <https://neo4j.com/docs/cypher-manual/current/indexes/semantic-indexes/full-text-indexes/>`_. 
+The Full Text Index can be created for both node and relationship properties. Only available for Neo4j version 5.16 or higher.
 
 Defining a Full Text Index on a Property
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 Within neomodel, indexing is a decision that is made at class definition time as the index needs to be built. A Full Text index is defined using :class:`~neomodel.properties.FulltextIndex`
-To define a property with a full text index we use the following symantics::
+To define a property with a full text index we use the following syntax::
     
-    StringProperty(fulltext_index=FulltextIndex(analyzer="standard-no-stop-words", eventually_consistent=False)
+    StringProperty(fulltext_index=FulltextIndex(analyzer="standard-no-stop-words", eventually_consistent=False))
 
 Where,
     - ``analyzer``: The analyzer to use. The default is ``standard-no-stop-words``.
@@ -27,34 +27,67 @@ Please refer to the `Neo4j documentation <https://neo4j.com/docs/cypher-manual/c
 Querying a Full Text Index on a Property
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-This is not currently implemented as a native neomodel query type. If you would like this please submit a github issue highlighting your useage pattern
+Node Property 
+^^^^^^^^^^^^^
+The following Fulltext Index property::
+
+    class Product(StructuredNode):
+            name = StringProperty()
+            description = StringProperty(
+                fulltext_index=FulltextIndex(
+                    analyzer="standard-no-stop-words", eventually_consistent=False
+                )
+            )
+
+Can be queried using :class:`~neomodel.semantic_filters.FulltextFilter`. Such as::
+
+    from neomodel.semantic_filters import FulltextFilter
+    result = Product.nodes.filter(
+        fulltext_filter=FulltextFilter(
+            topk=10,
+            fulltext_attribute_name="description",
+            query_string="product")).all()
+
+Where the result will be a list of length topk of nodes with the form (ProductNode, score).
+
+The :class:`~neomodel.semantic_filters.FulltextFilter` can be used in conjunction with the normal filter types.
+
+.. attention:: 
+    If you use FulltextFilter in conjunction with normal filter types, only nodes that fit the filters will return thus, you may get less than the topk specified.
+    Furthermore, all node filters **should** work with FulltextFilter, relationship filters will also work but WILL NOT return the fulltext similarity score alongside the relationship filter, instead the topk nodes and their appropriate relationships will be returned.
+
+RelationshipProperty
+^^^^^^^^^^^^^^^^^^^^
 
-Alternatively, whilst this has not bbeen implemetned yet you can still leverage `db.cypher_query` with the correct syntax to perform your required query.
+Currently neomodel has not implemented an OGM method for querying full text indexes on relationships.
+If this is something that you would like, please submit a GitHub issue with requirements highlighting your usage pattern. 
+
+Alternatively, whilst this has not been implemented yet you can still leverage `db.cypher_query` with the correct syntax to perform your required query. 
 
 Vector Index 
 ------------
-From version x.x (version number tbc) neomodel provides a way to interact with neo4j `vector indexing <https://neo4j.com/docs/cypher-manual/current/indexes/semantic-indexes/vector-indexes/>`_.
+From version 5.5.0 neomodel provides a way to interact with neo4j `vector indexing <https://neo4j.com/docs/cypher-manual/current/indexes/semantic-indexes/vector-indexes/>`_.
 
 The Vector Index can be created on both node and relationship properties. Only available for Neo4j version 5.15 (node) and 5.18 (relationship) or higher. 
 
 Defining a Vector Index on a Property 
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Within neomodel, indexing is a decision that is made at class definition time as the index needs to be built. A vector index is defined using :class:`~neomodel.properties.VectorIndex`.
-To define a property with a vector index we use the following symantics::
+To define a property with a vector index we use the following syntax::
 
-    ArrayProperty(base_property=FloatProperty(), vector_index=VectorIndex(dimensions=512, similarity_function="cosine")
+    ArrayProperty(base_property=FloatProperty(), vector_index=VectorIndex(dimensions=512, similarity_function="cosine"))
     
 Where,
     - ``dimensions``: The dimension of the vector. The default is 1536.
     - ``similarity_function``: The similarity algorithm to use. The default is ``cosine``.
 
-The index must then be built, this occurs when the function :func:`~neomodel.sync_.core.install_all_labels` is run
+The index must then be built, this occurs when the function :func:`~neomodel.sync_.core.install_all_labels` is run.
 
 The vector indexes will then have the name "vector_index_{node.__label__}_{propertyname_with_vector_index}".
 
 .. attention:: 
-   Neomodel creates a new vectorindex for each specified property, thus you cannot have two distinct properties being placed into the same index. 
+   Neomodel creates a new vector index for each specified property, thus you cannot have two distinct properties being placed into the same index. 
 
 Querying a Vector Index on a Property 
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -64,10 +97,10 @@ Node Property
 The following node vector index property::
 
     class someNode(StructuredNode):
-        vector = ArrayProperty(base_property=FloatProperty(), vector_index=VectorIndex(dimensions=512, similarity_function="cosine")
+        vector = ArrayProperty(base_property=FloatProperty(), vector_index=VectorIndex(dimensions=512, similarity_function="cosine"))
         name = StringProperty()
 
-Can be queried using :class:`~neomodel.sematic_filters.VectorFilter`. Such as::
+Can be queried using :class:`~neomodel.semantic_filters.VectorFilter`. Such as::
 
     from neomodel.semantic_filters import VectorFilter
     result = someNode.nodes.filter(vector_filter=VectorFilter(topk=3, vector_attribute_name="vector")).all()
@@ -78,12 +111,11 @@ The :class:`~neomodel.semantic_filters.VectorFilter` can be used in conjunction
 
 .. attention:: 
     If you use VectorFilter in conjunction with normal filter types, only nodes that fit the filters will return thus, you may get less than the topk specified.
-   Furthermore, all node filters **should** work with VectorFilter, relationship filters will also work but WILL NOT return the vector similiarty score alongside the relationship filter, instead the topk nodes and their appropriate relationships will be returned.
+    Furthermore, all node filters **should** work with VectorFilter, relationship filters will also work but WILL NOT return the vector similarity score alongside the relationship filter, instead the topk nodes and their appropriate relationships will be returned.
 
 RelationshipProperty
 ^^^^^^^^^^^^^^^^^^^^
 Currently neomodel has not implemented an OGM method for querying vector indexes on relationships.
-If this is something that you like please submit a github issue requirements highlighting your usage pattern. 
+If this is something that you would like, please submit a GitHub issue with requirements highlighting your usage pattern. 
 
 Alternatively, whilst this has not been implemented yet you can still leverage `db.cypher_query` with the correct syntax to perform your required query. 
-

neomodel/async_/match.py

@@ -12,7 +12,7 @@
 from neomodel.exceptions import MultipleNodesReturned
 from neomodel.match_q import Q, QBase
 from neomodel.properties import AliasProperty, ArrayProperty, Property
-from neomodel.semantic_filters import VectorFilter
+from neomodel.semantic_filters import FulltextFilter, VectorFilter
 from neomodel.typing import Subquery, Transformation
 from neomodel.util import RelationshipDirection
 
@@ -403,6 +403,7 @@ class QueryAST:
     additional_return: list[str] | None
     is_count: bool | None
     vector_index_query: type | None
+    fulltext_index_query: type | None
 
     def __init__(
         self,
@@ -420,6 +421,7 @@ def __init__(
         additional_return: list[str] | None = None,
         is_count: bool | None = False,
         vector_index_query: type | None = None,
+        fulltext_index_query: type | None = None,
     ) -> None:
         self.match = match if match else []
         self.optional_match = optional_match if optional_match else []
@@ -437,6 +439,7 @@ def __init__(
         )
         self.is_count = is_count
         self.vector_index_query = vector_index_query
+        self.fulltext_index_query = fulltext_index_query
         self.subgraph: dict = {}
         self.mixed_filters: bool = False
 
@@ -467,6 +470,15 @@ async def build_ast(self) -> "AsyncQueryBuilder":
         ):
             self.build_vector_query(self.node_set.vector_query, self.node_set.source)
 
+        if (
+            isinstance(self.node_set, AsyncNodeSet)
+            and hasattr(self.node_set, "fulltext_query")
+            and self.node_set.fulltext_query
+        ):
+            self.build_fulltext_query(
+                self.node_set.fulltext_query, self.node_set.source
+            )
+
         await self.build_source(self.node_set)
 
         if hasattr(self.node_set, "skip"):
@@ -573,6 +585,31 @@ def build_vector_query(self, vectorfilter: "VectorFilter", source: "AsyncNodeSet
         self._ast.return_clause = f"{vectorfilter.node_set_label}, score"
         self._ast.result_class = source.__class__
 
+    def build_fulltext_query(self, fulltextquery: "FulltextFilter", source: "NodeSet"):
+        """
+        Query a free text indexed property on the node.
+        """
+        try:
+            attribute = getattr(source, fulltextquery.fulltext_attribute_name)
+        except AttributeError as e:
+            raise AttributeError(
+                f"Atribute '{fulltextquery.fulltext_attribute_name}' not found on '{type(source).__name__}'."
+            ) from e
+
+        if not attribute.fulltext_index:
+            raise AttributeError(
+                f"Attribute {fulltextquery.fulltext_attribute_name} is not declared with a full text index."
+            )
+
+        fulltextquery.index_name = (
+            f"fulltext_index_{source.__label__}_{fulltextquery.fulltext_attribute_name}"
+        )
+        fulltextquery.node_set_label = source.__label__.lower()
+
+        self._ast.fulltext_index_query = fulltextquery
+        self._ast.return_clause = f"{fulltextquery.node_set_label}, score"
+        self._ast.result_class = source.__class__
+
     async def build_traversal(self, traversal: "AsyncTraversal") -> str:
         """
         traverse a relationship from a node to a set of nodes
@@ -974,6 +1011,16 @@ def build_query(self) -> str:
             # This ensures that we bring the context of the new nodeSet and score along with us for metadata filtering
             query += f""" WITH {self._ast.vector_index_query.node_set_label}, score"""
 
+        if self._ast.fulltext_index_query:
+            query += f"""CALL () {{
+                CALL db.index.fulltext.queryNodes("{self._ast.fulltext_index_query.index_name}", "{self._ast.fulltext_index_query.query_string}")
+                YIELD node AS {self._ast.fulltext_index_query.node_set_label}, score
+                RETURN {self._ast.fulltext_index_query.node_set_label}, score LIMIT {self._ast.fulltext_index_query.topk}
+                }}
+                """
+            # This ensures that we bring the context of the new nodeSet and score along with us for metadata filtering
+            query += f""" WITH {self._ast.fulltext_index_query.node_set_label}, score"""
+
         # Instead of using only one MATCH statement for every relation
         # to follow, we use one MATCH per relation (to avoid cartesian
         # product issues...).
@@ -1446,6 +1493,7 @@ def __init__(self, source: Any) -> None:
         self._intermediate_transforms: list = []
         self._unique_variables: list[str] = []
         self.vector_query: str | None = None
+        self.fulltext_query: str | None = None
 
     def __await__(self) -> Any:
         return self.all().__await__()  # type: ignore[attr-defined]
@@ -1555,6 +1603,10 @@ def filter(self, *args: Any, **kwargs: Any) -> "AsyncBaseSet":
             for arg in args:
                 if isinstance(arg, VectorFilter) and (not self.vector_query):
                     self.vector_query = arg
+
+                if isinstance(arg, FulltextFilter) and (not self.fulltext_query):
+                    self.fulltext_query = arg
+
                 new_args.append(arg)
 
             new_args = tuple(new_args)
@@ -1566,6 +1618,13 @@ def filter(self, *args: Any, **kwargs: Any) -> "AsyncBaseSet":
             ):
                 self.vector_query = kwargs.pop("vector_filter")
 
+            if (
+                kwargs.get("fulltext_filter")
+                and isinstance(kwargs["fulltext_filter"], FulltextFilter)
+                and not self.fulltext_query
+            ):
+                self.fulltext_query = kwargs.pop("fulltext_filter")
+
             self.q_filters = Q(self.q_filters & Q(*new_args, **kwargs))
 
         return self

neomodel/semantic_filters.py

@@ -19,3 +19,22 @@ def __init__(
         self.index_name = None
         self.node_set_label = None
         self.vector = candidate_vector
+
+class FulltextFilter(object):
+    """
+    Represents a CALL db.index.fulltext.query* neo function call within the OGM.
+    :param query_strng: The string you are finding the nearest
+    :type query_string: str
+    :param freetext_attribute_name: The property name for the free text indexed property.
+    :type fulltext_attribute_name: str
+    :param topk: Amount to nodes to return
+    :type topk: int
+
+    """
+
+    def __init__(self, query_string: str, fulltext_attribute_name: str, topk: int):
+        self.query_string = query_string
+        self.fulltext_attribute_name = fulltext_attribute_name
+        self.index_name = None
+        self.node_set_label = None
+        self.topk = topk