Merge branch 'rc/5.5.1' of https://github.com/neo4j-contrib/neomodel into rc/5.5.1

Author: mariusconjeaud

.gitignore

@@ -22,4 +22,4 @@ pyvenv.cfg
 coverage_report/
 .coverage*
 .DS_STORE
-cov.xml
+cov.xml

doc/source/index.rst

@@ -77,6 +77,7 @@ Contents
    filtering_ordering
    traversal
    advanced_query_operations
+   semantic_indexes
    cypher
    transactions
    hooks

doc/source/schema_management.rst

@@ -46,38 +46,11 @@ Indexes
 The following indexes are supported:
 
 - ``index=True``: This will create the default Neo4j index on the property (currently RANGE).
-- ``fulltext_index=FulltextIndex()``: This will create a FULLTEXT index on the property. Only available for Neo4j version 5.16 or higher. With this one, you can define the following options:
-    - ``analyzer``: The analyzer to use. The default is ``standard-no-stop-words``.
-    - ``eventually_consistent``: Whether the index should be eventually consistent. The default is ``False``.
-  
-Please refer to the `Neo4j documentation <https://neo4j.com/docs/cypher-manual/current/indexes/semantic-indexes/full-text-indexes/#configuration-settings>`_. for more information on fulltext indexes.
-
-- ``vector_index=VectorIndex()``: This will create a VECTOR index on the property. Only available for Neo4j version 5.15 (node) and 5.18 (relationship) or higher. With this one, you can define the following options:
-    - ``dimensions``: The dimension of the vector. The default is 1536.
-    - ``similarity_function``: The similarity algorithm to use. The default is ``cosine``.
-
-Those indexes are available for both node- and relationship properties.
+- :ref:`Semantic Indexes`
 
 .. note:: 
     Yes, you can create multiple indexes of a different type on the same property. For example, a default index and a fulltext index.
 
-.. note:: 
-    For the semantic indexes (fulltext and vector), this allows you to create indexes, but searching those indexes require using Cypher queries.
-    This is because Cypher only supports querying those indexes through a specific procedure for now.
-
-Full example: ::
-
-    from neomodel import StructuredNode, StringProperty, FulltextIndex, VectorIndex
-    class VeryIndexedNode(StructuredNode):
-        name = StringProperty(
-            index=True,
-            fulltext_index=FulltextIndex(analyzer='english', eventually_consistent=True)
-        )
-        name_embedding = ArrayProperty(
-            FloatProperty(),
-            vector_index=VectorIndex(dimensions=512, similarity_function='euclidean')
-        )
-
 Constraints
 ===========
 
@@ -93,4 +66,4 @@ Extracting the schema from a database
 =====================================
 
 You can extract the schema from an existing database using the ``neomodel_inspect_database`` script (:ref:`inspect_database_doc`).
-This script will output the schema in the neomodel format, including indexes and constraints.
+This script will output the schema in the neomodel format, including indexes and constraints.

doc/source/semantic_indexes.rst

@@ -0,0 +1,89 @@
+.. _Semantic Indexes: 
+
+==================================
+Semantic Indexes
+==================================
+
+Full Text Index
+----------------
+From version x.x (version number tbc) neomodel provides a way to interact with neo4j `Full Text indexing <https://neo4j.com/docs/cypher-manual/current/indexes/semantic-indexes/full-text-indexes/>`_. 
+The Full Text Index can be be created for both node and relationship properties. Only available for Neo4j version 5.16 or higher.
+
+Defining a Full Text Index on a Property
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Within neomodel, indexing is a decision that is made at class definition time as the index needs to be built. A Full Text index is defined using :class:`~neomodel.properties.FulltextIndex`
+To define a property with a full text index we use the following symantics::
+    
+    StringProperty(fulltext_index=FulltextIndex(analyzer="standard-no-stop-words", eventually_consistent=False)
+
+Where,
+    - ``analyzer``: The analyzer to use. The default is ``standard-no-stop-words``.
+    - ``eventually_consistent``: Whether the index should be eventually consistent. The default is ``False``.
+
+The index must then be built, this occurs when the function :func:`~neomodel.sync_.core.install_all_labels` is run. 
+
+Please refer to the `Neo4j documentation <https://neo4j.com/docs/cypher-manual/current/indexes/semantic-indexes/full-text-indexes/#configuration-settings>`_ for more information on fulltext indexes.
+
+Querying a Full Text Index on a Property
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This is not currently implemented as a native neomodel query type. If you would like this please submit a github issue highlighting your useage pattern
+
+Alternatively, whilst this has not bbeen implemetned yet you can still leverage `db.cypher_query` with the correct syntax to perform your required query.
+
+Vector Index 
+------------
+From version x.x (version number tbc) neomodel provides a way to interact with neo4j `vector indexing <https://neo4j.com/docs/cypher-manual/current/indexes/semantic-indexes/vector-indexes/>`_.
+
+The Vector Index can be created on both node and relationship properties. Only available for Neo4j version 5.15 (node) and 5.18 (relationship) or higher. 
+
+Defining a Vector Index on a Property 
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Within neomodel, indexing is a decision that is made at class definition time as the index needs to be built. A vector index is defined using :class:`~neomodel.properties.VectorIndex`.
+To define a property with a vector index we use the following symantics::
+
+    ArrayProperty(base_property=FloatProperty(), vector_index=VectorIndex(dimensions=512, similarity_function="cosine")
+    
+Where,
+    - ``dimensions``: The dimension of the vector. The default is 1536.
+    - ``similarity_function``: The similarity algorithm to use. The default is ``cosine``.
+
+The index must then be built, this occurs when the function :func:`~neomodel.sync_.core.install_all_labels` is run
+
+The vector indexes will then have the name "vector_index_{node.__label__}_{propertyname_with_vector_index}".
+
+.. attention:: 
+   Neomodel creates a new vectorindex for each specified property, thus you cannot have two distinct properties being placed into the same index. 
+
+Querying a Vector Index on a Property 
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Node Property
+^^^^^^^^^^^^^
+The following node vector index property::
+
+    class someNode(StructuredNode):
+        vector = ArrayProperty(base_property=FloatProperty(), vector_index=VectorIndex(dimensions=512, similarity_function="cosine")
+        name = StringProperty()
+
+Can be queried using :class:`~neomodel.sematic_filters.VectorFilter`. Such as::
+
+    from neomodel.semantic_filters import VectorFilter
+    result = someNode.nodes.filter(vector_filter=VectorFilter(topk=3, vector_attribute_name="vector")).all()
+
+Where the result will be a list of length topk of tuples having the form (someNode, score). 
+
+The :class:`~neomodel.semantic_filters.VectorFilter` can be used in conjunction with the normal filter types.
+
+.. attention:: 
+    If you use VectorFilter in conjunction with normal filter types, only nodes that fit the filters will return thus, you may get less than the topk specified.
+   Furthermore, all node filters **should** work with VectorFilter, relationship filters will also work but WILL NOT return the vector similiarty score alongside the relationship filter, instead the topk nodes and their appropriate relationships will be returned.
+
+RelationshipProperty
+^^^^^^^^^^^^^^^^^^^^
+Currently neomodel has not implemented an OGM method for querying vector indexes on relationships.
+If this is something that you like please submit a github issue requirements highlighting your usage pattern. 
+
+Alternatively, whilst this has not been implemented yet you can still leverage `db.cypher_query` with the correct syntax to perform your required query. 
+

neomodel/async_/match.py

@@ -10,6 +10,7 @@
 from neomodel.async_ import relationship_manager
 from neomodel.async_.core import AsyncStructuredNode, adb
 from neomodel.async_.relationship import AsyncStructuredRel
+from neomodel.semantic_filters import VectorFilter
 from neomodel.exceptions import MultipleNodesReturned
 from neomodel.match_q import Q, QBase
 from neomodel.properties import AliasProperty, ArrayProperty, Property
@@ -404,7 +405,7 @@ class QueryAST:
     lookup: TOptional[str]
     additional_return: TOptional[list[str]]
     is_count: TOptional[bool]
-
+    vector_index_query: TOptional[type]
     def __init__(
         self,
         match: TOptional[list[str]] = None,
@@ -420,6 +421,7 @@ def __init__(
         lookup: TOptional[str] = None,
         additional_return: TOptional[list[str]] = None,
         is_count: TOptional[bool] = False,
+        vector_index_query: TOptional[type] = None,
     ) -> None:
         self.match = match if match else []
         self.optional_match = optional_match if optional_match else []
@@ -436,6 +438,7 @@ def __init__(
             additional_return if additional_return else []
         )
         self.is_count = is_count
+        self.vector_index_query = vector_index_query
         self.subgraph: dict = {}
         self.mixed_filters: bool = False
 
@@ -458,6 +461,10 @@ async def build_ast(self) -> "AsyncQueryBuilder":
         ):
             for relation in self.node_set.relations_to_fetch:
                 self.build_traversal_from_path(relation, self.node_set.source)
+        
+        if isinstance(self.node_set, AsyncNodeSet) and hasattr(self.node_set, "_vector_query"):
+            if self.node_set._vector_query:
+                self.build_vector_query(self.node_set._vector_query, self.node_set.source)
 
         await self.build_source(self.node_set)
 
@@ -540,6 +547,27 @@ def build_order_by(self, ident: str, source: "AsyncNodeSet") -> None:
                         order_by.append(f"{result[0]}.{prop}")
             self._ast.order_by = order_by
 
+
+    def build_vector_query(self, vectorfilter: "VectorFilter", source: "NodeSet"):
+        """
+        Query a vector indexed property on the node. 
+        """
+        try:
+            attribute = getattr(source, vectorfilter.vector_attribute_name)
+        except AttributeError:
+            raise # This raises the base AttributeError and provides potential correction
+
+        if not attribute.vector_index:
+            raise AttributeError(f"Attribute {vectorfilter.vector_attribute_name} is not declared with a vector index.")
+
+        vectorfilter.index_name = f"vector_index_{source.__label__}_{vectorfilter.vector_attribute_name}"
+        vectorfilter.nodeSetLabel = source.__label__.lower()
+
+        self._ast.vector_index_query = vectorfilter
+        self._ast.return_clause = f"{vectorfilter.nodeSetLabel}, score"
+        self._ast.result_class = source.__class__
+
+
     async def build_traversal(self, traversal: "AsyncTraversal") -> str:
         """
         traverse a relationship from a node to a set of nodes
@@ -933,6 +961,17 @@ def build_query(self) -> str:
         if self._ast.lookup:
             query += self._ast.lookup
 
+        if self._ast.vector_index_query:
+
+            query += f"""CALL () {{ 
+                CALL db.index.vector.queryNodes("{self._ast.vector_index_query.index_name}", {self._ast.vector_index_query.topk}, {self._ast.vector_index_query.vector}) 
+                YIELD node AS {self._ast.vector_index_query.nodeSetLabel}, score 
+                RETURN {self._ast.vector_index_query.nodeSetLabel}, score 
+                }}"""
+
+            # This ensures that we bring the context of the new nodeSet and score along with us for metadata filtering
+            query += f""" WITH {self._ast.vector_index_query.nodeSetLabel}, score"""
+
         # Instead of using only one MATCH statement for every relation
         # to follow, we use one MATCH per relation (to avoid cartesian
         # product issues...).
@@ -1404,6 +1443,7 @@ def __init__(self, source: Any) -> None:
         self._subqueries: list[Subquery] = []
         self._intermediate_transforms: list = []
         self._unique_variables: list[str] = []
+        self._vector_query: str = None
 
     def __await__(self) -> Any:
         return self.all().__await__()  # type: ignore[attr-defined]
@@ -1506,6 +1546,129 @@ def filter(self, *args: Any, **kwargs: Any) -> "AsyncBaseSet":
         :return: self
         """
         if args or kwargs:
+            # Need to grab and remove the VectorFilter from both args and kwargs
+            new_args = [] # As args are a tuple, theyre immutable. But we need to remove the vectorfilter from the arguments so they dont go into Q. 
+            for arg in args:
+                if isinstance(arg, VectorFilter) and (not self._vector_query):
+                    self._vector_query = arg
+                new_args.append(arg)
+
+            new_args = tuple(new_args)
+
+            if kwargs.get("vector_filter"):
+                if isinstance(kwargs["vector_filter"], VectorFilter) and (not self._vector_query):
+                    self._vector_query = kwargs.pop("vector_filter")
+                    
+
+            self.q_filters = Q(self.q_filters & Q(*new_args, **kwargs))
+
+        return self
+
+    def exclude(self, *args: Any, **kwargs: Any) -> "BaseSet":
+        """
+        Exclude nodes from the NodeSet via filters.
+
+        :param kwargs: filter parameters see syntax for the filter method
+        :return: self
+        """
+        if args or kwargs:
+            self.q_filters = Q(self.q_filters & ~Q(*args, **kwargs))
+        return self
+
+    def has(self, **kwargs: Any) -> "BaseSet":
+        must_match, dont_match = process_has_args(self.source_class, kwargs)
+        self.must_match.update(must_match)
+        self.dont_match.update(dont_match)
+        return self
+
+    def order_by(self, *props: Any) -> "BaseSet":
+        """
+        Order by properties. Prepend with minus to do descending. Pass None to
+        remove ordering.
+        """
+        should_remove = len(props) == 1 and props[0] is None
+        if not hasattr(self, "order_by_elements") or should_remove:
+            self.order_by_elements = []
+            if should_remove:
+                return self
+        if "?" in props:
+            self.order_by_elements.append("?")
+        else:
+            for prop in props:
+                if isinstance(prop, RawCypher):
+                    self.order_by_elements.append(prop)
+                    continue
+                prop = prop.strip()
+                if prop.startswith("-"):
+                    prop = prop[1:]
+                    desc = True
+                else:
+                    desc = False
+
+                if prop in self.source_class.defined_properties(rels=False):
+                    property_obj = getattr(self.source_class, prop)
+                    if isinstance(property_obj, AliasProperty):
+                        prop = property_obj.aliased_to()
+
+                self.order_by_elements.append(prop + (" DESC" if desc else ""))
+
+        return self
+
+    def _register_relation_to_fetch(
+        self, relation_def: Any, alias: TOptional[str] = None
+    ) -> "Path":
+        if isinstance(relation_def, Path):
+            item = relation_def
+        else:
+            item = Path(
+                value=relation_def,
+            )
+        if alias:
+            item.alias = alias
+        return item
+
+    def unique_variables(self, *paths: tuple[str, ...]) -> "NodeSet":
+        """Generate unique variable names for the given paths."""
+        self._unique_variables = paths
+        return self
+
+    def traverse(self, *paths: tuple[str, ...], **aliased_paths: dict) -> "NodeSet":
+        """Specify a set of paths to traverse."""
+        relations = []
+        for path in paths:
+            relations.append(self._register_relation_to_fetch(path))
+        for alias, aliased_path in aliased_paths.items():
+            relations.append(
+                self._register_relation_to_fetch(aliased_path, alias=alias)
+            )
+        self.relations_to_fetch = relations
+        return self
+
+    def fetch_relations(self, *relation_names: tuple[str, ...]) -> "NodeSet":
+        """Specify a set of relations to traverse and return."""
+        warnings.warn(
+            "fetch_relations() will be deprecated in version 6, use traverse() instead.",
+            DeprecationWarning,
+        )
+        relations = []
+        for relation_name in relation_names:
+            if isinstance(relation_name, Optional):
+                relation_name = Path(value=relation_name.relation, optional=True)
+            relations.append(self._register_relation_to_fetch(relation_name))
+        self.relations_to_fetch = relations
+        return self
+
+    def traverse_relations(
+        self, *relation_names: tuple[str, ...], **aliased_relation_names: dict
+    ) -> "NodeSet":
+        """Specify a set of relations to traverse only."""
+
+        warnings.warn(
+            "traverse_relations() will be deprecated in version 6, use traverse() instead.",
+            DeprecationWarning,
+        )
+
+        def convert_to_path(input: Union[str, Optional]) -> Path:
             self.q_filters = Q(self.q_filters & Q(*args, **kwargs))
         return self