What's in MO-Tester?

MO-Tester is a java-based tester suite for MatrixOne. It has built a whole toolchain to run automatic SQL tests. It contains the test cases and results. Once launched, MO-Tester runs all SQL test cases with MatrixOne, and compares all output SQL results with expected results. All successful and failed cases will be logged into reports.

MO-Tester content locations:

Cases: https://github.com/matrixorigin/matrixone/tree/main/test/cases
Result: https://github.com/matrixorigin/mo-tester/tree/main/result
- Result can also generated in the path of /cases, for example, /cases/auto_increment.
Report: once finished running, a mo-tester/report will be generated in the local directory.

The Cases and Results are 1-1 correspondence, and they are actually git submodules from MatrixOne repository. Adding new cases and results should be in MatrixOne repo: https://github.com/matrixorigin/matrixone/tree/main/test

How to use MO-Tester?

1. Prepare the testing environment

Make sure you have installed jdk8.
Launch MatrixOne or other database instance. Please refer to more information about how to install and launch MatrixOne.

Clone mo-tester repository.

git clone https://github.com/matrixorigin/mo-tester.git

Clone matrixOne repository.

git clone https://github.com/matrixorigin/matrixone.git

2. Configure `mo-tester`

In mo.yml file, configure the server address, default database name, username, and password, etc. MO-tester is based on java, so these parameters are required for the JDBC(JDBC，Java Database Connectivity) driver. Below is a default example for a local standalone version MatrixOne.

#jdbc
jdbc:
  driver: "com.mysql.cj.jdbc.Driver"
  server:
  - addr: "127.0.0.1:6001"
  database:
    default: "test"
  paremeter:
    characterSetResults: "utf8"
    continueBatchOnError: "false"
    useServerPrepStmts: "true"
    alwaysSendSetIsolation: "false"
    useLocalSessionState: "true"
    zeroDateTimeBehavior: "CONVERT_TO_NULL"
    failoverReadOnly: "false"
    serverTimezone: "Asia/Shanghai"
    socketTimeout: 30000
#users
user:
  name: "dump"
  passwrod: "111"

In kafka.yml file, configure the kafka server address, only for the cases that need to produce or consume messages from kafka server.

3. Run mo-tester

With the simple below command, all the SQL test cases will automatically run and generate reports and error messages to report/report.txt and report/error.txt.

> ./run.sh -p {path_name}/matrixone/test/cases

If you'd like to adjust the test range, you can just change the path parameter of run.yml. And you can also specify some parameters when executing the command ./run.sh, parameters are as followings:

Parameters	Description
-p	set the path of test cases needed to be executed by mo-tester, the default value is configured by the `path` in `run.yaml`
-m	set the method that mo-tester will run with, the default value is configured by the `method` in `run.yaml`
-t	set the times that mo-tester will execute cases for, must be numeric, default is 1
-r	set The success rate that test cases should reach, the default value is configured by the `rate` in `run.yaml`
-i	set the including list, and only script files in the path whose name contains one of the lists will be executed, if more than one, separated by `,`, if not specified, refers to all cases included
-e	set the excluding list, and script files in the path whose name contains one of the lists will not be executed, if more than one, separated by `,`, if not specified, refers to none of the cases excluded
-g	means SQL commands which is marked with [bvt:issue] flag will not be executed,this flag starts with [-- @bvt:issue#{issueNO.}],and ends with [-- @bvt:issue],eg: -- @bvt:issue#3236 select date_add("1997-12-31 23:59:59",INTERVAL "-10000:1" HOUR_MINUTE); select date_add("1997-12-31 23:59:59",INTERVAL "-100 1" YEAR_MONTH); -- @bvt:issue Those two sql commands are associated with issue#3236, and they will not be executed in bvt test, until the flag is removed when issue#3236 is fixed.
-n	means the metadata of the resultset will be ignored when comparing the result
-c	only check whether the case file matches the related result file
-s	set the resource path that mo-tester use to store resources, and can be refered to in test file

Examples:

./run.sh -p {path_name}/matrixone/test/cases -m run -t script -r 100 -i select,subquery -e substring -g

If you want to automatically generate SQL results for the new SQL cases, you can just change the method parameter of run.yml file to genrs, or you can just change the command -m run to -m genrs, then running the ./run.sh scripts will directly record test results in the same path of the new SQL case file. For more information on example, see Example 4.

!!! note Every time running run.sh will overwrite the report of the error.txt file, report.txt file, and success.txt file in the mo-tester repository.

4. Set tags in case scripts

Sometimes, to achieve some specific purposes, such as pausing or creating a new connection, you can add some special tags to the script file. The mo tester provides the following tags for use:

Tags	Description
-- @skip:issue#{IssueNo.}	If set, the whole script file will be skipped, and not be executed any more for issue{IssueNo.}
-- @bvt:issue#{IssueNo.} -- @bvt:issue	The sql statements between those two tags will be not executed for issue{IssueNo.}
-- @sleep:{time}	The mo-tester will wait for {time} s
-- @session:id=2&user=root&password=111 -- @session	The mo-tester will create a new connetion to execute sql statements between those two tags. Default value of id is 1, max is 10. Defualt value of user and password is configured in `mo.yml`.
-- @sortkey:	If the result is sorted, need set this tag for the sql statement. e.g. -- @sortkey:0,1: means sort keys are first column and second colum.
-- @delimiter {C}	Set new delimeter to String C, C can any string expect that starts with [/,#,--]
-- @system {C}	Set System Command that will be executed by the runner system
-- @wait:{D}:[commit or wait]	means this command will be blocked until the connection[id={D}] commit or rollback
-- @ignore:{num},...{num}	means the designated columns which index are in {num}s will not be check.
-- @kafka:produce:{topic}} JSON ARRAY -- @bvt:produce	means the mo-tester will send all the items in this json array to the designated topic of kafka server. The kafka server is configured in `kafka.yml`

5. Check the report

Once the test is finished, mo-tester generates error.txt file, report.txt file and success.txt file reports.
An example of report.txt file looks like this:

[SUMMARY] COST : 98s, TOTAL :12702, SUCCESS : 11851, FAILED :13, IGNORED :838, ABNORAML :0, SUCCESS RATE : 99%
[{path_name}/matrixone/test/cases/auto_increment/auto_increment_columns.sql] COST : 2.159s, TOTAL :185, SUCCESS :163, FAILED :0, IGNORED :22, ABNORAML :0, SUCCESS RATE : 100%
[{path_name}/matrixone/test/cases/benchmark/tpch/01_DDL/01_create_table.sql] COST : 0.226s, TOTAL :11, SUCCESS :11, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%
[{path_name}/matrixone/test/cases/benchmark/tpch/02_LOAD/02_insert_customer.sql] COST : 0.357s, TOTAL :16, SUCCESS :16, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%

Report Keywords	Description
TOTAL	the total number of executed test cases (SQL)
SUCCESS	The total number of successfully executed test cases(SQL)
FAILED	the total number of failed executed test case(SQL)
IGNORED	the total number of ignored executed test cases (SQL), especially with the `--bvt:issue` tag test cases (SQL)
ABNORAML	the total number of abnormal executed test cases (SQL), such as the execution of MatrixOne can't determine the actual result is a system exception or .result file parsing error, etc
SUCCESS RATE	success rate: SUCCESS/(TOTAL - IGNORED)

An example of error.txt file looks like this:

[ERROR]
[SCRIPT   FILE]: cases/transaction/atomicity.sql
[ROW    NUMBER]: 14
[SQL STATEMENT]: select * from test_11 ;
[EXPECT RESULT]:
c	d
1	1
2 2
[ACTUAL RESULT]:
c	d
1	1

6. Test Examples

Example 1

Example Description: Run all test cases in the /cases path of the matrixone repository.

Steps:

Get the latest matrixone code.

cd matrixone
git pull https://github.com/matrixorigin/matrixone.git

To run all the test cases of the matrixone repository, you need switch into the mo-tester repository first, see the following commands:
```
cd mo-tester
./run.sh -p {path_name}/matrixone/test/cases
```
Check the result reports in the error.txt file, report.txt file, and success.txt file in the mo-tester/report/ path.

Example 2

Example Description: Run the test cases in the /cases/transaction/ path of the matrixone repository.

Steps:

Get the latest matrixone code.

cd matrixone
git pull https://github.com/matrixorigin/matrixone.git

To run the test cases in the cases/transaction/ path of the matrixone repository, you need switch into the mo-tester repository first, see the following commands:
```
cd mo-tester
./run.sh -p {path_name}/matrixone/test/cases/transaction/
```

Check the result reports in the error.txt file, report.txt file, and success.txt file in the mo-tester/report/ path. The example of the expected report.txt looks like this:

[SUMMARY] COST : 5s, TOTAL :1362, SUCCESS : 1354, FAILED :0, IGNORED :8, ABNORAML :0, SUCCESS RATE : 100%
[{path_name}/matrixone/test/cases/transaction/atomicity.sql] COST : 0.575s, TOTAL :66, SUCCESS :66, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%
[{path_name}/matrixone/test/cases/transaction/autocommit.test] COST : 0.175s, TOTAL :50, SUCCESS :50, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%
[{path_name}/matrixone/test/cases/transaction/autocommit_1.sql] COST : 1.141s, TOTAL :296, SUCCESS :288, FAILED :0, IGNORED :8, ABNORAML :0, SUCCESS RATE : 100%
[{path_name}/matrixone/test/cases/transaction/autocommit_atomicity.sql] COST : 0.52s, TOTAL :75, SUCCESS :75, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%
[{path_name}/matrixone/test/cases/transaction/autocommit_isolation.sql] COST : 1.607s, TOTAL :215, SUCCESS :215, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%
[{path_name}/matrixone/test/cases/transaction/autocommit_isolation_1.sql] COST : 1.438s, TOTAL :241, SUCCESS :241, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%
[{path_name}/matrixone/test/cases/transaction/isolation.sql] COST : 1.632s, TOTAL :202, SUCCESS :202, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%
[{path_name}/matrixone/test/cases/transaction/isolation_1.sql] COST : 1.512s, TOTAL :217, SUCCESS :217, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%

Example 3

Example Description: Run the single test case cases/transaction/atomicity.sql.

Steps:

Get the latest matrixone code.

cd matrixone
git pull https://github.com/matrixorigin/matrixone.git

To run the test cases cases/transaction/atomicity.sql, you need switch into the mo-tester repository first, see the following commands:
```
cd mo-tester
./run.sh -p {path_name}/matrixone/test/cases/transaction/atomicity.sql
```

Check the result reports in the error.txt file, report.txt file, and success.txt file in the mo-tester/report/ path. The example of the expected report.txt looks like this:

[SUMMARY] COST : 0s, TOTAL :66, SUCCESS : 66, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%
[{path_name}/matrixone/test/cases/transaction/atomicity.sql] COST : 0.56s, TOTAL :66, SUCCESS :66, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%

Example 4

Example Description:

Create a new folder named local_test and place it in {path_name}/matrixone/test/cases
Add a test file named new_test.sql to {path_name}/matrixone/test/cases/local_test/
Only run the single test case new_test.sql*

Steps

Get the latest matrixone code.

cd matrixone
git pull https://github.com/matrixorigin/matrixone.git

Generate test results:
- Method 1: To generate the test result, you need switch into the mo-tester repository first, then, run the following command.
```
cd mo-tester
./run.sh -p {path_name}/matrixone/test/cases/local_test/new_test.sql -m genrs -g
```
- Method 2: Open the run.yml file in the mo-tester repository, change the method parameter from the default run to genrs, and run the following command to generate the test result.
```
cd mo-tester
./run.sh -p {path_name}/matrixone/test/cases/local_test/new_test.sql
```
Check the result file in the test/cases、result/ path of the matrixone repository.

Check the result reports in the error.txt file, report.txt file, and success.txt file in the mo-tester/report/ path. The example of the expected report.txt looks like this:

[SUMMARY] COST : 0s, TOTAL :66, SUCCESS : 66, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%
[{path_name}/matrixone/test/cases/transaction/atomicity.sql] COST : 0.56s, TOTAL :66, SUCCESS :66, FAILED :0, IGNORED :0, ABNORAML :0, SUCCESS RATE : 100%

Reference

For more information on the annotations of MO-Tester, see MO-Tester Annotations.

Name		Name	Last commit message	Last commit date
Latest commit History 241 Commits
.github		.github
cases		cases
lib		lib
resources/data		resources/data
src		src
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
assembly.xml		assembly.xml
kafka.yml		kafka.yml
log4j.properties		log4j.properties
mo.yml		mo.yml
pom.xml		pom.xml
pprof.sh		pprof.sh
run.sh		run.sh
run.yml		run.yml

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Repository files navigation

What's in MO-Tester?

How to use MO-Tester?

1. Prepare the testing environment

2. Configure `mo-tester`

3. Run mo-tester

4. Set tags in case scripts

5. Check the report

6. Test Examples

Example 1

Example 2

Example 3

Example 4

Reference

About

Uh oh!

Releases

Packages

Uh oh!

Contributors 11

Uh oh!

Languages

License

matrixorigin/mo-tester

Folders and files

Latest commit

History

Repository files navigation

What's in MO-Tester?

How to use MO-Tester?

1. Prepare the testing environment

2. Configure mo-tester

3. Run mo-tester

4. Set tags in case scripts

5. Check the report

6. Test Examples

Example 1

Example 2

Example 3

Example 4

Reference

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors 11

Uh oh!

Languages

2. Configure `mo-tester`

Packages