Skip to content

Commit 16e1610

Browse files
andygroveandygrove
committed
4.0
1 parent 17b8a8e commit 16e1610

405 files changed

Lines changed: 68576 additions & 21562 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_0/q1.native_datafusion/explain.txt‎

Lines changed: 160 additions & 108 deletions
Large diffs are not rendered by default.

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_0/q1.native_datafusion/extended.txt‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
TakeOrderedAndProject
2+
+- Project
3+
+- BroadcastHashJoin
4+
:- Project
5+
: +- BroadcastHashJoin
6+
: :- Project
7+
: : +- BroadcastHashJoin
8+
: : :- Filter
9+
: : : +- HashAggregate
10+
: : : +- CometColumnarToRow
11+
: : : +- CometColumnarExchange
12+
: : : +- HashAggregate
13+
: : : +- Project
14+
: : : +- BroadcastHashJoin
15+
: : : :- Filter
16+
: : : : +- ColumnarToRow
17+
: : : : +- Scan parquet spark_catalog.default.store_returns [COMET: Native DataFusion scan does not support subqueries/dynamic pruning]
18+
: : : : +- SubqueryBroadcast
19+
: : : : +- BroadcastExchange
20+
: : : : +- CometColumnarToRow
21+
: : : : +- CometProject
22+
: : : : +- CometFilter
23+
: : : : +- CometNativeScan parquet spark_catalog.default.date_dim
24+
: : : +- BroadcastExchange
25+
: : : +- CometColumnarToRow
26+
: : : +- CometProject
27+
: : : +- CometFilter
28+
: : : +- CometNativeScan parquet spark_catalog.default.date_dim
29+
: : +- BroadcastExchange
30+
: : +- Filter
31+
: : +- HashAggregate
32+
: : +- CometColumnarToRow
33+
: : +- CometColumnarExchange
34+
: : +- HashAggregate
35+
: : +- HashAggregate
36+
: : +- CometColumnarToRow
37+
: : +- CometColumnarExchange
38+
: : +- HashAggregate
39+
: : +- Project
40+
: : +- BroadcastHashJoin
41+
: : :- Filter
42+
: : : +- ColumnarToRow
43+
: : : +- Scan parquet spark_catalog.default.store_returns [COMET: Native DataFusion scan does not support subqueries/dynamic pruning]
44+
: : : +- ReusedSubquery
45+
: : +- BroadcastExchange
46+
: : +- CometColumnarToRow
47+
: : +- CometProject
48+
: : +- CometFilter
49+
: : +- CometNativeScan parquet spark_catalog.default.date_dim
50+
: +- BroadcastExchange
51+
: +- CometColumnarToRow
52+
: +- CometProject
53+
: +- CometFilter
54+
: +- CometNativeScan parquet spark_catalog.default.store
55+
+- BroadcastExchange
56+
+- CometColumnarToRow
57+
+- CometProject
58+
+- CometFilter
59+
+- CometNativeScan parquet spark_catalog.default.customer
60+
61+
Comet accelerated 18 out of 49 eligible operators (36%). Final plan contains 10 transitions between Spark and Comet.

‎spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_0/q1.native_datafusion/simplified.txt‎

Lines changed: 51 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
TakeOrderedAndProject [c_customer_id]
2-
WholeStageCodegen (7)
2+
WholeStageCodegen (9)
33
Project [c_customer_id]
44
BroadcastHashJoin [ctr_customer_sk,c_customer_sk]
55
Project [ctr_customer_sk]
@@ -8,53 +8,65 @@ TakeOrderedAndProject [c_customer_id]
88
BroadcastHashJoin [ctr_store_sk,ctr_store_sk,ctr_total_return,(avg(ctr_total_return) * 1.2)]
99
Filter [ctr_total_return]
1010
HashAggregate [sr_customer_sk,sr_store_sk,sum] [sum(UnscaledValue(sr_return_amt)),ctr_customer_sk,ctr_store_sk,ctr_total_return,sum]
11-
InputAdapter
12-
Exchange [sr_customer_sk,sr_store_sk] #1
13-
WholeStageCodegen (1)
14-
HashAggregate [sr_customer_sk,sr_store_sk,sr_return_amt] [sum,sum]
15-
ColumnarToRow
16-
InputAdapter
17-
CometProject [sr_customer_sk,sr_store_sk,sr_return_amt]
18-
CometBroadcastHashJoin [sr_customer_sk,sr_store_sk,sr_return_amt,sr_returned_date_sk,d_date_sk]
19-
CometFilter [sr_customer_sk,sr_store_sk,sr_return_amt,sr_returned_date_sk]
20-
CometNativeScan: `spark_catalog`.`default`.`store_returns` [sr_customer_sk,sr_store_sk,sr_return_amt,sr_returned_date_sk]
21-
CometBroadcastExchange [d_date_sk] #2
22-
CometProject [d_date_sk]
23-
CometFilter [d_date_sk,d_year]
24-
CometNativeScan: `spark_catalog`.`default`.`date_dim` [d_date_sk,d_year]
11+
CometColumnarToRow
12+
InputAdapter
13+
CometColumnarExchange [sr_customer_sk,sr_store_sk] #1
14+
WholeStageCodegen (2)
15+
HashAggregate [sr_customer_sk,sr_store_sk,sr_return_amt] [sum,sum]
16+
Project [sr_customer_sk,sr_store_sk,sr_return_amt]
17+
BroadcastHashJoin [sr_returned_date_sk,d_date_sk]
18+
Filter [sr_store_sk,sr_customer_sk]
19+
ColumnarToRow
20+
InputAdapter
21+
Scan parquet spark_catalog.default.store_returns [sr_customer_sk,sr_store_sk,sr_return_amt,sr_returned_date_sk]
22+
SubqueryBroadcast [d_date_sk] #1
23+
BroadcastExchange #2
24+
WholeStageCodegen (1)
25+
CometColumnarToRow
26+
InputAdapter
27+
CometProject [d_date_sk]
28+
CometFilter [d_date_sk,d_year]
29+
CometNativeScan parquet spark_catalog.default.date_dim [d_date_sk,d_year]
30+
InputAdapter
31+
ReusedExchange [d_date_sk] #2
2532
InputAdapter
2633
BroadcastExchange #3
27-
WholeStageCodegen (4)
34+
WholeStageCodegen (6)
2835
Filter [(avg(ctr_total_return) * 1.2)]
2936
HashAggregate [ctr_store_sk,sum,count] [avg(ctr_total_return),(avg(ctr_total_return) * 1.2),sum,count]
30-
InputAdapter
31-
Exchange [ctr_store_sk] #4
32-
WholeStageCodegen (3)
33-
HashAggregate [ctr_store_sk,ctr_total_return] [sum,count,sum,count]
34-
HashAggregate [sr_customer_sk,sr_store_sk,sum] [sum(UnscaledValue(sr_return_amt)),ctr_store_sk,ctr_total_return,sum]
35-
InputAdapter
36-
Exchange [sr_customer_sk,sr_store_sk] #5
37-
WholeStageCodegen (2)
38-
HashAggregate [sr_customer_sk,sr_store_sk,sr_return_amt] [sum,sum]
39-
ColumnarToRow
40-
InputAdapter
41-
CometProject [sr_customer_sk,sr_store_sk,sr_return_amt]
42-
CometBroadcastHashJoin [sr_customer_sk,sr_store_sk,sr_return_amt,sr_returned_date_sk,d_date_sk]
43-
CometFilter [sr_customer_sk,sr_store_sk,sr_return_amt,sr_returned_date_sk]
44-
CometNativeScan: `spark_catalog`.`default`.`store_returns` [sr_customer_sk,sr_store_sk,sr_return_amt,sr_returned_date_sk]
45-
ReusedExchange [d_date_sk] #2
37+
CometColumnarToRow
38+
InputAdapter
39+
CometColumnarExchange [ctr_store_sk] #4
40+
WholeStageCodegen (5)
41+
HashAggregate [ctr_store_sk,ctr_total_return] [sum,count,sum,count]
42+
HashAggregate [sr_customer_sk,sr_store_sk,sum] [sum(UnscaledValue(sr_return_amt)),ctr_store_sk,ctr_total_return,sum]
43+
CometColumnarToRow
44+
InputAdapter
45+
CometColumnarExchange [sr_customer_sk,sr_store_sk] #5
46+
WholeStageCodegen (4)
47+
HashAggregate [sr_customer_sk,sr_store_sk,sr_return_amt] [sum,sum]
48+
Project [sr_customer_sk,sr_store_sk,sr_return_amt]
49+
BroadcastHashJoin [sr_returned_date_sk,d_date_sk]
50+
Filter [sr_store_sk]
51+
ColumnarToRow
52+
InputAdapter
53+
Scan parquet spark_catalog.default.store_returns [sr_customer_sk,sr_store_sk,sr_return_amt,sr_returned_date_sk]
54+
ReusedSubquery [d_date_sk] #1
55+
InputAdapter
56+
ReusedExchange [d_date_sk] #2
4657
InputAdapter
4758
BroadcastExchange #6
48-
WholeStageCodegen (5)
49-
ColumnarToRow
59+
WholeStageCodegen (7)
60+
CometColumnarToRow
5061
InputAdapter
5162
CometProject [s_store_sk]
5263
CometFilter [s_store_sk,s_state]
53-
CometNativeScan: `spark_catalog`.`default`.`store` [s_store_sk,s_state]
64+
CometNativeScan parquet spark_catalog.default.store [s_store_sk,s_state]
5465
InputAdapter
5566
BroadcastExchange #7
56-
WholeStageCodegen (6)
57-
ColumnarToRow
67+
WholeStageCodegen (8)
68+
CometColumnarToRow
5869
InputAdapter
59-
CometFilter [c_customer_sk,c_customer_id]
60-
CometNativeScan: `spark_catalog`.`default`.`customer` [c_customer_sk,c_customer_id]
70+
CometProject [c_customer_id] [c_customer_sk,c_customer_id]
71+
CometFilter [c_customer_sk,c_customer_id]
72+
CometNativeScan parquet spark_catalog.default.customer [c_customer_sk,c_customer_id]

0 commit comments

Comments
 (0)