Skip to content

add disk storage graph filter tests #2093

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 136 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
136 commits
Select commit Hold shift + click to select a range
3a10189
restruct filters package, impl node filter graph
shivamka1 Mar 31, 2025
40b44fb
fix bench
shivamka1 Mar 31, 2025
7d4dff6
impl and/or as left and right composition instead of vec, fix tests
shivamka1 Mar 31, 2025
03efde7
replace struct based filter expr with trait based abstraction, fix al…
shivamka1 Apr 1, 2025
43acbe5
impl node field filtered graph
shivamka1 Apr 1, 2025
9d1f73a
impl edge field filtered graph
shivamka1 Apr 1, 2025
847e5ea
ref edge property filter
shivamka1 Apr 1, 2025
0bab9d6
impl and/or filtered graph
shivamka1 Apr 2, 2025
48a3a7d
Merge branch 'master' into features/new-filter-apis
shivamka1 Apr 2, 2025
67fe9a5
fix bench
shivamka1 Apr 2, 2025
5271646
ref
shivamka1 Apr 2, 2025
2432809
fix graphql for new filter abs
shivamka1 Apr 2, 2025
864027c
impl filter abs in python
shivamka1 Apr 2, 2025
1ce7ea3
impl python for new filter expr abstraction
shivamka1 Apr 2, 2025
32931f1
make filter expr work with filter apis
shivamka1 Apr 3, 2025
2cbe89f
fix infinite recursion
ljeub-pometry Apr 3, 2025
f85981d
fix/add py tests
shivamka1 Apr 4, 2025
733318c
more python tests
shivamka1 Apr 4, 2025
3827c90
fmt
shivamka1 Apr 4, 2025
f782b8c
fix tests
shivamka1 Apr 4, 2025
ce93d7b
add filter api tests for rest WIP
shivamka1 Apr 4, 2025
119546d
move search tests and ref
shivamka1 Apr 6, 2025
2d78d03
ref, fix tests
shivamka1 Apr 6, 2025
4b325f1
Merge branch 'master' into features/new-filter-apis
shivamka1 Apr 6, 2025
db2fa3c
impl filter tests for cached view graph
shivamka1 Apr 7, 2025
40df3e9
ref persistent graph
shivamka1 Apr 7, 2025
c046d2e
impl filter tests for layered graph
shivamka1 Apr 7, 2025
2eb7aa1
impl filter tests for node subgraph
shivamka1 Apr 7, 2025
1a655be
impl tests for node type filtered subgraph view
shivamka1 Apr 7, 2025
95d079c
impl filter tests for window graph
shivamka1 Apr 8, 2025
3772d4c
ref tests
shivamka1 Apr 8, 2025
3bb8be8
fix tests
shivamka1 Apr 8, 2025
9038605
fix tests
shivamka1 Apr 8, 2025
9611982
fix tests
shivamka1 Apr 8, 2025
49025dc
fix semantics
shivamka1 Apr 10, 2025
f2dc3eb
impl name, type builder in python
shivamka1 Apr 11, 2025
5511068
fix tests
shivamka1 Apr 11, 2025
ed208e7
fix tests
shivamka1 Apr 11, 2025
92c0fb8
change includes/excludes to is_in/is_not_in
shivamka1 Apr 11, 2025
07ada34
ref
shivamka1 Apr 13, 2025
29c111a
add fuzzy search filter test
shivamka1 Apr 13, 2025
30ed02f
impl contains and contains_not for filter
shivamka1 Apr 13, 2025
b5f10cf
support both tokenized and non-tokenized search queries
shivamka1 Apr 14, 2025
60a3a69
impl filter_expr for graphql fitler apis, fix all warnings, impl/fix …
shivamka1 Apr 15, 2025
84d12cd
merge from master
shivamka1 Apr 15, 2025
947268b
fixed bug where graphql property model for list was not getting trans…
shivamka1 Apr 15, 2025
6d041f8
ref, fix tests, rid duplicate tests
shivamka1 Apr 15, 2025
3401e12
Merge branch 'master' into features/new-filter-apis
shivamka1 Apr 15, 2025
d5c67b6
merge from master
shivamka1 Apr 16, 2025
67481fb
Merge branch 'master' into features/new-filter-apis
shivamka1 Apr 16, 2025
a0921e5
into path to index
shivamka1 Apr 17, 2025
462e776
Merge branch 'master' into features/new-filter-apis
shivamka1 Apr 17, 2025
d0c3db1
create indexes
shivamka1 Apr 17, 2025
e8c4667
persist index upon encode graph
shivamka1 Apr 22, 2025
2c24446
ref
shivamka1 Apr 22, 2025
b9b29d1
impl load graph index
shivamka1 Apr 23, 2025
dc72e44
Merge branch 'master' into features/new-filter-apis
shivamka1 Apr 24, 2025
f5fa3af
impl encode/decode indexes, impl tests
shivamka1 Apr 24, 2025
cfb0be8
Merge branch 'features/new-filter-apis' into features/index-disk
shivamka1 Apr 24, 2025
db7d2df
encode/decode index as zip
shivamka1 Apr 24, 2025
b755b5e
del index safely
shivamka1 Apr 25, 2025
7437a58
ref
shivamka1 Apr 25, 2025
6b4a11f
refactor cached_view tests
shivamka1 Apr 25, 2025
10916e4
ref layered_graph tests
shivamka1 Apr 25, 2025
e2f1f9b
ref node_subgraph tests
shivamka1 Apr 25, 2025
e117de2
ref window_graph tests
shivamka1 Apr 25, 2025
1240524
ref
shivamka1 Apr 25, 2025
e62598e
ref tests
shivamka1 Apr 25, 2025
e8410cf
ref test filters
shivamka1 Apr 25, 2025
91a3c0e
ref macros
shivamka1 Apr 25, 2025
1fdb417
add fuzzy search test, ref, tests
shivamka1 Apr 25, 2025
7d32c14
add tests
shivamka1 Apr 25, 2025
36c218b
disable tantivy fuzzy search
shivamka1 Apr 28, 2025
ad28aa2
fix/ add tests
shivamka1 Apr 28, 2025
1c4ed8e
add tests
shivamka1 Apr 28, 2025
43d4ea1
ref/add tests
shivamka1 Apr 28, 2025
fe00fb5
contains_not => not_contains
shivamka1 Apr 28, 2025
f61e3df
resurrect node property filter pytests
shivamka1 Apr 29, 2025
cd74647
add tests
shivamka1 Apr 29, 2025
b9d8b49
ref tests
shivamka1 Apr 29, 2025
3db6d4f
ref/add tests
shivamka1 Apr 29, 2025
db5e8d0
add tests
shivamka1 Apr 29, 2025
2441a1b
impl partialeq, eq
shivamka1 Apr 29, 2025
0a9f996
restruct filter expr
shivamka1 Apr 29, 2025
6a009fe
Merge branch 'master' into features/new-filter-apis
shivamka1 Apr 29, 2025
c48bb7b
skip exploded edges tests
shivamka1 Apr 29, 2025
5d9f11d
change src/dst to src/dst name
shivamka1 Apr 30, 2025
757dd84
fix warnings
shivamka1 Apr 30, 2025
0d26ebe
add test
shivamka1 Apr 30, 2025
bb55283
fix benches
shivamka1 Apr 30, 2025
e549c56
Merge branch 'features/new-filter-apis' into features/index-disk
shivamka1 Apr 30, 2025
0cc32cc
add suggestions
shivamka1 Apr 30, 2025
ca898cf
ref
shivamka1 Apr 30, 2025
dbfb321
ref recursive copy
shivamka1 May 1, 2025
79d8114
ref graph_index
shivamka1 May 1, 2025
9509246
fix issue: re-register tokenizer after loading the index from disk.
shivamka1 May 1, 2025
4f9d05b
add test: persisting an updated index should be loaded
shivamka1 May 1, 2025
e8eecd2
impl create_index_in_ram for tests
shivamka1 May 1, 2025
e768630
skip unzip_index if there is no index/ dir
shivamka1 May 1, 2025
71938a1
ref
shivamka1 May 1, 2025
798bcc2
ref
shivamka1 May 1, 2025
01a3bec
fix build
shivamka1 May 2, 2025
031209f
fix build
shivamka1 May 2, 2025
0121693
add tests
shivamka1 May 2, 2025
1fc3de9
gql filter error handling
shivamka1 May 2, 2025
ffe62df
add create_index_in_ram python
shivamka1 May 2, 2025
88b0e1b
Features/index disk (#2030)
shivamka1 May 2, 2025
46df48e
fix for windows
shivamka1 May 5, 2025
7722108
fix bench
shivamka1 May 5, 2025
85edac7
ref grapherrors
shivamka1 May 6, 2025
4272c7a
suggestions on PR
shivamka1 May 7, 2025
190eb07
impl cache index
shivamka1 May 7, 2025
0d14222
create index in cache graph tmp dir
shivamka1 May 7, 2025
371f1fe
Merge branch 'master' into features/new-filter-apis
shivamka1 May 7, 2025
2f83e27
fix bench
shivamka1 May 7, 2025
5a37e5d
Merge branch 'features/new-filter-apis' of github.com:Pometry/Raphtor…
shivamka1 May 7, 2025
7634dd2
fix build
shivamka1 May 7, 2025
485b628
disable exploded edges filtering
shivamka1 May 8, 2025
99be976
disable exploded edges filtering (#2087)
shivamka1 May 8, 2025
c2d0038
impl filter negation
shivamka1 May 8, 2025
0ca21c2
impl filter negation for python
shivamka1 May 8, 2025
226e269
impl filter negation for gql
shivamka1 May 8, 2025
e29e6ab
Merge branch 'features/new-filter-apis' into features/filter-negation
shivamka1 May 8, 2025
50d1a70
fix test
shivamka1 May 9, 2025
c0ca76b
Merge branch 'features/new-filter-apis' into features/filter-negation
shivamka1 May 9, 2025
053296f
not in GraphQL no longer requires a list as input
ljeub-pometry May 9, 2025
0e6ee80
check if node and edges is present
shivamka1 May 9, 2025
f7dc987
ref test
shivamka1 May 9, 2025
b05a103
remove dead code
shivamka1 May 12, 2025
cd7e776
impl filter negation (#2088)
shivamka1 May 12, 2025
aaf1ccf
fmt
shivamka1 May 12, 2025
16b553e
Merge from master
shivamka1 May 12, 2025
ce4c4c9
add disk storage graph filter tests
shivamka1 May 12, 2025
0699e76
impl diskstoragegraph tests
shivamka1 May 13, 2025
c30564a
add more tests
shivamka1 May 13, 2025
e59c1ab
ref/fix tests
shivamka1 May 13, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,7 @@ moka = { version = "0.12.7", features = ["sync"] }
indexmap = { version = "2.7.0", features = ["rayon"] }
fake = { version = "3.1.0", features = ["chrono"] }
strsim = { version = "0.11.1" }
uuid = { version = "1.16.0", features = ["v4"] }

# Make sure that transitive dependencies stick to disk_graph 50
[patch.crates-io]
Expand Down
20 changes: 19 additions & 1 deletion python/test_utils/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
import tempfile
import time
from typing import TypeVar, Callable

import os
import pytest

from raphtory.graphql import GraphServer
Expand All @@ -13,6 +13,24 @@
PORT = 1737


if "DISK_TEST_MARK" in os.environ:
def with_disk_graph(func):
def inner(graph):
def inner2(graph, tmpdirname):
g = graph.to_disk_graph(tmpdirname)
func(g)

func(graph)
with tempfile.TemporaryDirectory() as tmpdirname:
inner2(graph, tmpdirname)

return inner

else:
def with_disk_graph(func):
return func


def measure(name: str, f: Callable[..., B], *args, print_result: bool = True) -> B:
start_time = time.time()
result = f(*args)
Expand Down
54 changes: 54 additions & 0 deletions python/tests/test_base_install/test_filters/conftest.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
from raphtory import Graph, PersistentGraph
from raphtory import DiskGraphStorage
import tempfile
import pytest
import shutil
import atexit

def default_init_graph(graph):
nodes = [
(1, 1, {"p1": "shivam_kapoor", "p9": 5, "p10": "Paper_airplane"}, "fire_nation"),
(2, 2, {"p1": "prop12", "p2": 2, "p10": "Paper_ship"}, "air_nomads"),
(3, 1, {"p1": "shivam_kapoor", "p9": 5}, "fire_nation"),
(3, 3, {"p2": 6, "p3": 1, "p10": "Paper_airplane"}, "fire_nation"),
(4, 1, {"p1": "shivam_kapoor", "p9": 5}, "fire_nation"),
(3, 4, {"p4": "pometry"}, None),
(4, 4, {"p5": 12}, None),
]

for time, id, props, node_type in nodes:
graph.add_node(time, str(id), props, node_type)

edge_data = [
(1, "1", "2", {"p1": "shivam_kapoor", "p10": "Paper_airplane"}, "fire_nation"),
(2, "1", "2", {"p1": "shivam_kapoor", "p2": 4}, "fire_nation"),
(2, "2", "3", {"p1": "prop12", "p2": 2, "p10": "Paper_ship"}, "air_nomads"),
(3, "3", "1", {"p2": 6, "p3": 1}, "fire_nation"),
(3, "2", "1", {"p2": 6, "p3": 1, "p10": "Paper_airplane"}, None),
(4, "David Gilmour", "John Mayer", {"p2": 6, "p3": 1}, None),
(4, "John Mayer", "Jimmy Page", {"p2": 6, "p3": 1}, None),
]

for time, src, dst, props, edge_type in edge_data:
graph.add_edge(time, src, dst, props, edge_type)

return graph

def generate_graph_variants(init_graph=None):
init_graph = init_graph or default_init_graph
graph = init_graph(Graph())
persistent_graph = init_graph(PersistentGraph())

tmpdir = tempfile.TemporaryDirectory()
atexit.register(tmpdir.cleanup)
path = f"{tmpdir.name}/graph"

graph.to_disk_graph(path)
disk_graph = DiskGraphStorage.load_from_dir(path)

return [
graph,
persistent_graph,
disk_graph.to_events(),
disk_graph.to_persistent(),
]
244 changes: 244 additions & 0 deletions python/tests/test_base_install/test_filters/semantics/conftest.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,244 @@
from raphtory import Graph, PersistentGraph
from raphtory import DiskGraphStorage
import tempfile
import pytest
import shutil
import atexit

def init_nodes_graph(graph):
nodes = [
(6, "N1", {"p1": 2}),
(7, "N1", {"p1": 1}),
(6, "N2", {"p1": 1}),
(7, "N2", {"p1": 2}),
(8, "N3", {"p1": 1}),
(9, "N4", {"p1": 1}),
(5, "N5", {"p1": 1}),
(6, "N5", {"p1": 2}),
(5, "N6", {"p1": 1}),
(6, "N6", {"p1": 1}),
(3, "N7", {"p1": 1}),
(5, "N7", {"p1": 1}),
(3, "N8", {"p1": 1}),
(4, "N8", {"p1": 2}),
(2, "N9", {"p1": 2}),
(2, "N10", {"q1": 0}),
(2, "N10", {"p1": 3}),
(2, "N11", {"p1": 3}),
(2, "N11", {"q1": 0}),
(2, "N12", {"q1": 0}),
(3, "N12", {"p1": 3}),
(2, "N13", {"q1": 0}),
(3, "N13", {"p1": 3}),
(2, "N14", {"q1": 0}),
(2, "N15", {}),
]

for time, label, props in nodes:
graph.add_node(time, label, props)

constant_properties = {
"N1": {"p1": 1},
"N4": {"p1": 2},
"N9": {"p1": 1},
"N10": {"p1": 1},
"N11": {"p1": 1},
"N12": {"p1": 1},
"N13": {"p1": 1},
"N14": {"p1": 1},
"N15": {"p1": 1},
}

for label, props in constant_properties.items():
graph.node(label).add_constant_properties(props)

edges = [
(6, "N1", "N2", {"p1": 2}),
(7, "N1", "N2", {"p1": 1}),
(6, "N2", "N3", {"p1": 1}),
(7, "N2", "N3", {"p1": 2}),
(8, "N3", "N4", {"p1": 1}),
(9, "N4", "N5", {"p1": 1}),
(5, "N5", "N6", {"p1": 1}),
(6, "N5", "N6", {"p1": 2}),
(5, "N6", "N7", {"p1": 1}),
(6, "N6", "N7", {"p1": 1}),
(3, "N7", "N8", {"p1": 1}),
(5, "N7", "N8", {"p1": 1}),
(3, "N8", "N9", {"p1": 1}),
(4, "N8", "N9", {"p1": 2}),
(2, "N9", "N10", {"p1": 2}),
(2, "N10", "N11", {"q1": 0}),
(2, "N10", "N11", {"p1": 3}),
(2, "N11", "N12", {"p1": 3}),
(2, "N11", "N12", {"q1": 0}),
(2, "N12", "N13", {"q1": 0}),
(3, "N12", "N13", {"p1": 3}),
(2, "N13", "N14", {"q1": 0}),
(3, "N13", "N14", {"p1": 3}),
(2, "N14", "N15", {"q1": 0}),
(2, "N15", "N1", {}),
]

for time, src, dst, props in edges:
graph.add_edge(time, src, dst, props)

constant_properties = [
("N1", "N2", {"p1": 1}),
("N4", "N5", {"p1": 2}),
("N9", "N10", {"p1": 1}),
("N10", "N11", {"p1": 1}),
("N11", "N12", {"p1": 1}),
("N12", "N13", {"p1": 1}),
("N13", "N14", {"p1": 1}),
("N14", "N15", {"p1": 1}),
("N15", "N1", {"p1": 1}),
]

for src, dst, props in constant_properties:
graph.edge(src, dst).add_constant_properties(props)

return graph

# For this graph there won't be any temporal property index for property name "p1".
def init_nodes_graph1(graph):
nodes = [
(2, "N1", {"q1": 0}),
(2, "N2", {}),
]

for time, label, props in nodes:
graph.add_node(time, label, props)

constant_properties = {
"N1": {"p1": 1},
"N2": {"p1": 1},
}

for label, props in constant_properties.items():
graph.node(label).add_constant_properties(props)

return graph

# For this graph there won't be any constant property index for property name "p1".
def init_nodes_graph2(graph):
nodes = [
(1, "N1", {"p1": 1}),

(2, "N2", {"p1": 1}),
(3, "N2", {"p1": 2}),

(2, "N3", {"p1": 2}),
(3, "N3", {"p1": 1}),

(2, "N4", {}),
]

for time, label, props in nodes:
graph.add_node(time, label, props)

constant_properties = {
"N1": {"p2": 1},
"N2": {"p1": 1},
}

for label, props in constant_properties.items():
graph.node(label).add_constant_properties(props)

return graph


def init_edges_graph(graph):
edges = [
(6, "N1", "N2", {"p1": 2}),
(7, "N1", "N2", {"p1": 1}),
(6, "N2", "N3", {"p1": 1}),
(7, "N2", "N3", {"p1": 2}),
(8, "N3", "N4", {"p1": 1}),
(9, "N4", "N5", {"p1": 1}),
(5, "N5", "N6", {"p1": 1}),
(6, "N5", "N6", {"p1": 2}),
(5, "N6", "N7", {"p1": 1}),
(6, "N6", "N7", {"p1": 1}),
(3, "N7", "N8", {"p1": 1}),
(5, "N7", "N8", {"p1": 1}),
(3, "N8", "N9", {"p1": 1}),
(4, "N8", "N9", {"p1": 2}),
(2, "N9", "N10", {"p1": 2}),
(2, "N10", "N11", {"q1": 0}),
(2, "N10", "N11", {"p1": 3}),
(2, "N11", "N12", {"p1": 3}),
(2, "N11", "N12", {"q1": 0}),
(2, "N12", "N13", {"q1": 0}),
(3, "N12", "N13", {"p1": 3}),
(2, "N13", "N14", {"q1": 0}),
(3, "N13", "N14", {"p1": 3}),
(2, "N14", "N15", {"q1": 0}),
(2, "N15", "N1", {}),
]

for time, src, dst, props in edges:
graph.add_edge(time, src, dst, props)

constant_properties = [
("N1", "N2", {"p1": 1}),
("N4", "N5", {"p1": 2}),
("N9", "N10", {"p1": 1}),
("N10", "N11", {"p1": 1}),
("N11", "N12", {"p1": 1}),
("N12", "N13", {"p1": 1}),
("N13", "N14", {"p1": 1}),
("N14", "N15", {"p1": 1}),
("N15", "N1", {"p1": 1}),
]

for src, dst, props in constant_properties:
graph.edge(src, dst).add_constant_properties(props)

return graph


# For this graph there won't be any constant property index for property name "p1".
def init_edges_graph1(graph):
edges = [
(2, "N1", "N2", {"q1": 0}),
(2, "N2", "N3", {}),
]

for time, src, dst, props in edges:
graph.add_edge(time, src, dst, props)

constant_properties = [
("N1", "N2", {"p1": 1}),
("N2", "N3", {"p1": 1}),
]

for src, dst, props in constant_properties:
graph.edge(src, dst).add_constant_properties(props)

return graph


# For this graph there won't be any constant property index for property name "p1".
def init_edges_graph2(graph):
edges = [
(2, "N1", "N2", {"p1": 1}),
(2, "N2", "N3", {"p1": 1}),
(2, "N2", "N3", {"p1": 2}),
(2, "N3", "N4", {"p1": 2}),
(2, "N3", "N4", {"p1": 1}),
(2, "N4", "N5", {}),
]

for time, src, dst, props in edges:
graph.add_edge(time, src, dst, props)

constant_properties = [
("N1", "N2", {"p2": 1}),
]

for src, dst, props in constant_properties:
graph.edge(src, dst).add_constant_properties(props)

return graph

Loading