Skip to content

Commit f855f65

Browse files
sarda-deveshsarda-devesh
authored
Merge pull request #70 from sirius-db/devesh/clickbench_testing
ClickBench Test
2 parents 9613851 + 53840c0 commit f855f65

75 files changed

Lines changed: 1007 additions & 27 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,3 +20,4 @@ cufile.log
2020
4stock_5day_avg_close.csv
2121
*result.csv
2222
*output.txt
23+
*.tsv

‎clickbench_load_duckdb.sql‎

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
DROP TABLE IF EXISTS hits;
2+
3+
CREATE TABLE hits
4+
(
5+
WatchID BIGINT NOT NULL,
6+
JavaEnable SMALLINT NOT NULL,
7+
Title TEXT,
8+
GoodEvent SMALLINT NOT NULL,
9+
EventTime TIMESTAMP NOT NULL,
10+
EventDate Date NOT NULL,
11+
CounterID INTEGER NOT NULL,
12+
ClientIP INTEGER NOT NULL,
13+
RegionID INTEGER NOT NULL,
14+
UserID BIGINT NOT NULL,
15+
CounterClass SMALLINT NOT NULL,
16+
OS SMALLINT NOT NULL,
17+
UserAgent SMALLINT NOT NULL,
18+
URL TEXT,
19+
Referer TEXT,
20+
IsRefresh SMALLINT NOT NULL,
21+
RefererCategoryID SMALLINT NOT NULL,
22+
RefererRegionID INTEGER NOT NULL,
23+
URLCategoryID SMALLINT NOT NULL,
24+
URLRegionID INTEGER NOT NULL,
25+
ResolutionWidth SMALLINT NOT NULL,
26+
ResolutionHeight SMALLINT NOT NULL,
27+
ResolutionDepth SMALLINT NOT NULL,
28+
FlashMajor SMALLINT NOT NULL,
29+
FlashMinor SMALLINT NOT NULL,
30+
FlashMinor2 TEXT,
31+
NetMajor SMALLINT NOT NULL,
32+
NetMinor SMALLINT NOT NULL,
33+
UserAgentMajor SMALLINT NOT NULL,
34+
UserAgentMinor VARCHAR(255) NOT NULL,
35+
CookieEnable SMALLINT NOT NULL,
36+
JavascriptEnable SMALLINT NOT NULL,
37+
IsMobile SMALLINT NOT NULL,
38+
MobilePhone SMALLINT NOT NULL,
39+
MobilePhoneModel TEXT,
40+
Params TEXT,
41+
IPNetworkID INTEGER NOT NULL,
42+
TraficSourceID SMALLINT NOT NULL,
43+
SearchEngineID SMALLINT NOT NULL,
44+
SearchPhrase TEXT,
45+
AdvEngineID SMALLINT NOT NULL,
46+
IsArtifical SMALLINT NOT NULL,
47+
WindowClientWidth SMALLINT NOT NULL,
48+
WindowClientHeight SMALLINT NOT NULL,
49+
ClientTimeZone SMALLINT NOT NULL,
50+
ClientEventTime TIMESTAMP NOT NULL,
51+
SilverlightVersion1 SMALLINT NOT NULL,
52+
SilverlightVersion2 SMALLINT NOT NULL,
53+
SilverlightVersion3 INTEGER NOT NULL,
54+
SilverlightVersion4 SMALLINT NOT NULL,
55+
PageCharset TEXT,
56+
CodeVersion INTEGER NOT NULL,
57+
IsLink SMALLINT NOT NULL,
58+
IsDownload SMALLINT NOT NULL,
59+
IsNotBounce SMALLINT NOT NULL,
60+
FUniqID BIGINT NOT NULL,
61+
OriginalURL TEXT,
62+
HID INTEGER NOT NULL,
63+
IsOldCounter SMALLINT NOT NULL,
64+
IsEvent SMALLINT NOT NULL,
65+
IsParameter SMALLINT NOT NULL,
66+
DontCountHits SMALLINT NOT NULL,
67+
WithHash SMALLINT NOT NULL,
68+
HitColor CHAR NOT NULL,
69+
LocalEventTime TIMESTAMP NOT NULL,
70+
Age SMALLINT NOT NULL,
71+
Sex SMALLINT NOT NULL,
72+
Income SMALLINT NOT NULL,
73+
Interests SMALLINT NOT NULL,
74+
Robotness SMALLINT NOT NULL,
75+
RemoteIP INTEGER NOT NULL,
76+
WindowName INTEGER NOT NULL,
77+
OpenerName INTEGER NOT NULL,
78+
HistoryLength SMALLINT NOT NULL,
79+
BrowserLanguage TEXT,
80+
BrowserCountry TEXT,
81+
SocialNetwork TEXT,
82+
SocialAction TEXT,
83+
HTTPError SMALLINT NOT NULL,
84+
SendTiming INTEGER NOT NULL,
85+
DNSTiming INTEGER NOT NULL,
86+
ConnectTiming INTEGER NOT NULL,
87+
ResponseStartTiming INTEGER NOT NULL,
88+
ResponseEndTiming INTEGER NOT NULL,
89+
FetchTiming INTEGER NOT NULL,
90+
SocialSourceNetworkID SMALLINT NOT NULL,
91+
SocialSourcePage TEXT,
92+
ParamPrice BIGINT NOT NULL,
93+
ParamOrderID TEXT,
94+
ParamCurrency TEXT,
95+
ParamCurrencyID SMALLINT NOT NULL,
96+
OpenstatServiceName TEXT,
97+
OpenstatCampaignID TEXT,
98+
OpenstatAdID TEXT,
99+
OpenstatSourceID TEXT,
100+
UTMSource TEXT,
101+
UTMMedium TEXT,
102+
UTMCampaign TEXT,
103+
UTMContent TEXT,
104+
UTMTerm TEXT,
105+
FromTag TEXT,
106+
HasGCLID SMALLINT NOT NULL,
107+
RefererHash BIGINT NOT NULL,
108+
URLHash BIGINT NOT NULL,
109+
CLID INTEGER NOT NULL
110+
);
111+
112+
COPY hits FROM 'test_datasets/test_hits.tsv' (QUOTE '');

‎docs/README.md‎

Lines changed: 28 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -151,10 +151,13 @@ python3 -m venv --prompt duckdb .venv
151151
source .venv/bin/activate
152152
```
153153

154-
## Generating and Loading TPC-H dataset
154+
## Generating and Loading test dataset
155+
156+
### TPC-H Datasets
157+
155158
To generate the TPC-H dataset
156159
```
157-
unzip dbgen.zip
160+
unzip test_datasets/dbgen.zip
158161
cd dbgen
159162
./dbgen -s 1 && mkdir s1 && mv *.tbl s1 # this generates dataset of SF1
160163
cd ..
@@ -166,6 +169,22 @@ To load the TPC-H dataset to duckdb:
166169
.read tpch_load_duckdb.sql
167170
```
168171

172+
### ClickBench Datasets
173+
174+
To download the dataset run:
175+
```
176+
cd test_datasets
177+
wget https://sirius-datasets.s3.us-east-2.amazonaws.com/test_hits.tsv.gz
178+
gzip -d test_hits.tsv.gz
179+
cd ..
180+
```
181+
182+
To load the dataset to duckdb run:
183+
```
184+
./build/release/duckdb {DATABASE_NAME}.duckdb
185+
.read clickbench_load_duckdb.sql
186+
```
187+
169188
## Running Sirius: CLI
170189
To run Sirius CLI, simply start the shell with `./build/release/duckdb {DATABASE_NAME}.duckdb`.
171190
From the duckdb shell, initialize the Sirius buffer manager with `call gpu_buffer_init`. This API accepts 2 parameters, the GPU caching region size and the GPU processing region size. The GPU caching region is a memory region where the raw data is stored in GPUs, whereas the GPU processing region is where intermediate results are stored in GPUs (hash tables, join results .etc).
@@ -248,11 +267,17 @@ con.execute('''
248267
```
249268

250269
## Correctness Testing
251-
Sirius provides a unit test that compares Sirius against DuckDB for correctness across all 22 TPC-H queries. To run the unittest, generate SF=1 TPC-H dataset using the method described [here](https://github.com/sirius-db/sirius?tab=readme-ov-file#generating-tpc-h-dataset) and run the unittest using the following command:
270+
Sirius provides a unit test that compares Sirius against DuckDB for correctness across many test queries. To run the unittest, generate the datasets using the method described [here](https://github.com/sirius-db/sirius?tab=readme-ov-file#generating-tpc-h-dataset) and run the unittest using the following command:
252271
```
253272
make test
254273
```
255274

275+
To run a specific test run the command from the root directory:
276+
```
277+
make -j {nproc}
278+
build/release/test/unittest --test-dir . test/sql/clickbench_queries.test
279+
```
280+
256281
## Performance Testing
257282
Make sure to build the duckdb-python package before running this test using the method described [here](https://github.com/sirius-db/sirius?tab=readme-ov-file#building-sirius). To test Sirius performance against DuckDB across all 22 TPC-H queries, run the following command (replace {SF} with the desired scale factor):
258283
```

‎setup_sirius.sh‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,8 @@
11
#!/bin/bash
22

3+
echo -e "\nexport SIRIUS_HOME_PATH=\"$(pwd)\"" >> ~/.bashrc
4+
source ~/.bashrc
35
git submodule update --init --recursive
4-
export SIRIUS_HOME_PATH=`pwd`
56
cd duckdb
67
mkdir -p extension_external && cd extension_external
78
git clone https://github.com/duckdb/substrait.git

‎src/operator/gpu_physical_table_scan.cpp‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1358,6 +1358,11 @@ GPUPhysicalTableScan::GetData(GPUIntermediateRelation &output_relation) const {
13581358

13591359
//Find table name in the buffer manager
13601360
auto gpuBufferManager = &(GPUBufferManager::GetInstance());
1361+
SIRIUS_LOG_DEBUG("Buffer Manager has {} cached tables", gpuBufferManager->tables.size());
1362+
for (const auto& curr_pair : gpuBufferManager->tables) {
1363+
SIRIUS_LOG_DEBUG("Buffer Manager Cached Table {}", curr_pair.first);
1364+
}
1365+
13611366
transform(table_name.begin(), table_name.end(), table_name.begin(), ::toupper);
13621367
SIRIUS_LOG_DEBUG("Table Scanning {}", table_name);
13631368
auto it = gpuBufferManager->tables.find(table_name);

‎test/answers/clickbench/q1.csv‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
total
2+
1428535

‎test/answers/clickbench/q10.csv‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
RegionID|sum|c|avg|c_dist
2+
229|38831|530550|1563.1453680143247|37579
3+
2|12922|177044|1544.206479745148|13612
4+
158|143|96366|288.79302866156115|22725
5+
208|2724|40767|1313.277184978046|4942
6+
1|1869|33759|1543.1276696584614|2370
7+
34|549|22019|1446.801989191153|2354
8+
42|642|16748|1458.2838547886315|1515
9+
47|1043|16520|1521.673426150121|1388
10+
184|30|15560|1425.5323264781491|1663
11+
169|21|14309|1022.6057027045915|2561

‎test/answers/clickbench/q11.csv‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
MobilePhoneModel|u
2+
iPad|4362
3+
iPhone|221
4+
A500|78
5+
iPho|18
6+
GT-P7300B|18
7+
N8-00|14
8+
3110000|13
9+
eagle75|10
10+
ONE TOUCH 6030A|9
11+
308|7

‎test/answers/clickbench/q12.csv‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
MobilePhone|MobilePhoneModel|u
2+
1|iPad|3663
3+
5|iPad|203
4+
7|iPad|157
5+
6|iPad|112
6+
118|A500|78
7+
6|iPhone|75
8+
60|iPad|67
9+
26|iPhone|66
10+
32|iPad|52
11+
13|iPad|32

‎test/answers/clickbench/q13.csv‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
SearchPhrase|c
2+
ведомосквы вместу|4943
3+
ведомосквы вы из|2471
4+
ведомосквиталия страции|2026
5+
ведомосковский|1686
6+
рецепты сбербан|1244
7+
покеты рецепт засня|961
8+
авторий|705
9+
инстанец жизнь|596
10+
тип туроп|524
11+
ведомосква|446

0 commit comments

Comments
 (0)