Skip to content

Commit 140e3f7

Browse files
committed
Add performance evaluation results in README
1 parent 815baf9 commit 140e3f7

File tree

3 files changed

+31
-5
lines changed

3 files changed

+31
-5
lines changed

README.md

+29-4
Original file line numberDiff line numberDiff line change
@@ -11,17 +11,42 @@ and work with them in terms of linear algebra.
1111
For the installation instructions, refer to [docs/install.md](docs/install.md).
1212

1313
## CLI
14-
CFPQ_PyAlgo provides a command line interface for running
14+
The CFPQ_PyAlgo project includes a command line interface for running
1515
all-pairs CFPQ solver with relation query semantics.
1616

1717
For more details, refer to [docs/cli.md](docs/cli.md).
1818

1919
## Evaluation
2020

21-
CFPQ_PyAlgo provides scripts for evaluating performance
22-
of various CFPQ solvers (including third-party ones).
21+
The CFPQ_PyAlgo project includes a [CFPQ evaluator](cfpq_eval) tool for evaluating the performance
22+
of various CFPQ solvers.
2323

24-
For more details, refer to [docs/eval.md](docs/eval.md).
24+
For more details on [CFPQ evaluator](cfpq_eval) usage, refer to [docs/eval.md](docs/eval.md).
25+
26+
We used the [CFPQ evaluator](cfpq_eval) to compare our solver, FastMatrixCFPQ, with five
27+
state-of-the-art competitors:
28+
[PEARL](https://figshare.com/articles/dataset/ASE_2023_artifact/23702271),
29+
[POCR](https://github.com/kisslune/POCR),
30+
[KotGLL](https://github.com/vadyushkins/kotgll),
31+
[Graspan](https://github.com/Graspan/Graspan-C), and
32+
[Gigascale](https://bitbucket.org/jensdietrich/gigascale-pointsto-oopsla2015/src),
33+
as well as with the previous version of our solver, MatrixCFPQ.
34+
The input data was provided by the
35+
[CFPQ_Data](https://github.com/FormalLanguageConstrainedPathQuerying/CFPQ_Data),
36+
[CFPQ_JavaGraphMiner](https://github.com/FormalLanguageConstrainedPathQuerying/CFPQ_JavaGraphMiner), and
37+
[CPU17-graphs](https://github.com/kisslune/CPU17-graphs)
38+
projects and is included in the [CFPQ evaluator distribution](docs/eval_install.md).
39+
We ran the evaluation five times on a Ryzen 9 7900X CPU with 128 GB of DDR5 RAM,
40+
with a timeout of 10,000 seconds for each solver invocation.
41+
42+
As shown in the bar chart below, FastMatrixCFPQ consistently outperforms other universal solvers
43+
but is sometimes slower than the specialized solver
44+
[Gigascale](https://bitbucket.org/jensdietrich/gigascale-pointsto-oopsla2015/src),
45+
which is optimized for one specific context-free grammar.
46+
Therefore, generalizing the ideas of [Gigascale](https://bitbucket.org/jensdietrich/gigascale-pointsto-oopsla2015/src) (i.e., recognizing and exploiting answer patterns)
47+
is one of our future research directions.
48+
49+
![Performance of All-Pairs CFPQ Solvers With Relation Query Semantics](performance.png)
2550

2651
## Project structure
2752
The global project structure is the following.

docs/eval.md

+2-1
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,8 @@
33
This document explains how to use `CFPQ_eval` to evaluate the performance of various Context-Free Path Querying (CFPQ)
44
solvers, including both the `CFPQ_PyAlgo` solver and third-party tools.
55

6-
To install the `CFPQ_eval` tool, refer to [eval_install.md](eval_install.md).
6+
To install the `CFPQ_eval` tool, refer to [eval_install.md](eval_install.md).
7+
For our evaluation results, refer to [main project README](../README.md#evaluation).
78

89
## Running the Tool
910

performance.png

139 KB
Loading

0 commit comments

Comments
 (0)