Skip to content

Commit 5bf04b9

Browse files
committed
perf(test): Split HashJoinTest into two files to reduce test bottleneck
HashJoinTest.cpp (9,625 lines, ~807s) was the single heaviest test file, dominating group0's execution time at ~13.6 min. Split it into: - HashJoinTest.cpp (3,656 lines): MultiThreadedHashJoinTest (67 tests × 4 params) plus interleaved HashJoinTest cases, ~525s estimated - HashJoinTestSpill.cpp (6,041 lines): Remaining HashJoinTest cases including memory, lazy vectors, dynamic filters, spill, and memory reclaim tests (71 tests × 2 params), ~282s estimated Both files share the HashJoinTestBase from utils/HashJoinTestBase.h. Also rebalanced all 8 test groups with the new file. Max group time drops from ~816s (13.6 min) to ~534s (8.9 min). All groups are now balanced at ~461-534s, which should reduce test phase wall time from ~20 min to ~9 min with ctest -j 24.
1 parent 32669f3 commit 5bf04b9

File tree

3 files changed

+6101
-6030
lines changed

3 files changed

+6101
-6030
lines changed

velox/exec/tests/CMakeLists.txt

Lines changed: 61 additions & 61 deletions
Original file line numberDiff line numberDiff line change
@@ -40,13 +40,12 @@ target_link_libraries(
4040
# bin-packing by estimated execution time. With VELOX_TESTS_PER_GROUP=10,
4141
# CMake takes files positionally in batches of 10 (files 1-10 = group0,
4242
# 11-20 = group1, etc.), so each batch of 10 must be independently balanced.
43-
# Group0 is dominated by HashJoinTest (~13.6 min); groups 1-7 are ~7 min each.
44-
# The previous alphabetical order concentrated slow tests (HashJoinTest,
45-
# IndexLookupJoinTest, HashTableTest) into the same groups, creating a
46-
# worst-case of ~18.8 min.
43+
# HashJoinTest was split into HashJoinTest (multi-threaded tests, ~525s) and
44+
# HashJoinTestSpill (single-threaded/spill tests, ~282s) to reduce the
45+
# bottleneck group from ~816s to ~534s. All groups are now ~462-534s (~8-9 min).
4746
set(
4847
VELOX_EXEC_TEST_SOURCES
49-
# group0 (~816s): HashJoinTest is the single heaviest test; pad with light tests.
48+
# group0 (~534s)
5049
HashJoinTest.cpp
5150
PrintPlanWithStatsTest.cpp
5251
ThreadDebugInfoTest.cpp
@@ -57,81 +56,82 @@ set(
5756
ValuesTest.cpp
5857
ParallelProjectTest.cpp
5958
AsyncConnectorTest.cpp
60-
# group1 (~424s)
59+
# group1 (~463s)
6160
IndexLookupJoinTest.cpp
62-
PrefixSortTest.cpp
63-
HashPartitionFunctionTest.cpp
64-
ExpandTest.cpp
65-
SplitListenerTest.cpp
61+
MergeTest.cpp
62+
FilterProjectTest.cpp
63+
ScaleWriterLocalPartitionTest.cpp
64+
CountingJoinTest.cpp
65+
CustomJoinTest.cpp
6666
AssignUniqueIdTest.cpp
67-
HashBitRangeTest.cpp
67+
TaskListenerTest.cpp
6868
SplitTest.cpp
69-
ArrowStreamTest.cpp
7069
SqlTest.cpp
71-
# group2 (~423s)
70+
# group2 (~462s)
71+
HashJoinTestSpill.cpp
72+
OperatorTraceTest.cpp
73+
OrderByTest.cpp
74+
PrefixSortTest.cpp
75+
MarkDistinctTest.cpp
76+
MergerTest.cpp
77+
StreamingEnforceDistinctTest.cpp
78+
MixedUnionWithTableScanTest.cpp
79+
HashBitRangeTest.cpp
80+
WindowFunctionRegistryTest.cpp
81+
# group3 (~462s)
7282
TableScanTest.cpp
73-
AggregationTest.cpp
74-
MergeTest.cpp
75-
LimitTest.cpp
76-
TopNTest.cpp
77-
SpatialIndexTest.cpp
78-
PlanNodeSerdeTest.cpp
83+
OutputBufferManagerTest.cpp
84+
NestedLoopJoinTest.cpp
85+
MixedUnionTest.cpp
86+
VectorHasherTest.cpp
87+
ContainerRowSerdeTest.cpp
88+
ExpandTest.cpp
89+
FunctionResolutionTest.cpp
7990
SpillStatsTest.cpp
80-
WindowFunctionRegistryTest.cpp
81-
PlanNodeToSummaryStringTest.cpp
82-
# group3 (~422s)
91+
AggregateFunctionRegistryTest.cpp
92+
# group4 (~462s)
8393
MergeJoinTest.cpp
84-
OperatorTraceTest.cpp
85-
LocalPartitionTest.cpp
86-
SortBufferTest.cpp
87-
ScaleWriterLocalPartitionTest.cpp
88-
ContainerRowSerdeTest.cpp
94+
HashTableTest.cpp
95+
StreamingAggregationTest.cpp
96+
PartitionedOutputTest.cpp
97+
TopNRowNumberTest.cpp
98+
TopNTest.cpp
8999
EnforceSingleRowTest.cpp
90-
MixedUnionWithTableScanTest.cpp
100+
FilterToExpressionTest.cpp
91101
ColumnStatsCollectorTest.cpp
92-
AggregateFunctionRegistryTest.cpp
93-
# group4 (~422s)
102+
RoundRobinPartitionFunctionTest.cpp
103+
# group5 (~462s)
94104
MultiFragmentTest.cpp
95-
OutputBufferManagerTest.cpp
96-
StreamingAggregationTest.cpp
97-
MixedUnionTest.cpp
98-
VectorHasherTest.cpp
99-
CountingJoinTest.cpp
100-
MergerTest.cpp
101-
ConcatFilesSpillMergeStreamTest.cpp
105+
ExchangeClientTest.cpp
106+
AggregationTest.cpp
107+
MemoryReclaimerTest.cpp
108+
LimitTest.cpp
109+
UnnestTest.cpp
110+
SplitListenerTest.cpp
111+
HilbertIndexTest.cpp
102112
AddressableNonNullValueListTest.cpp
103-
RoundRobinPartitionFunctionTest.cpp
104-
# group5 (~421s)
113+
ArrowStreamTest.cpp
114+
# group6 (~462s)
105115
RowContainerTest.cpp
106-
HashTableTest.cpp
107-
NestedLoopJoinTest.cpp
108-
MemoryReclaimerTest.cpp
116+
SpillTest.cpp
117+
LocalPartitionTest.cpp
118+
HashJoinBridgeTest.cpp
109119
RowNumberTest.cpp
110120
EnforceDistinctTest.cpp
111-
MarkSortedTest.cpp
112121
TraceUtilTest.cpp
113-
FilterToExpressionTest.cpp
122+
PlanNodeSerdeTest.cpp
114123
ScaledScanControllerTest.cpp
115-
# group6 (~421s)
124+
PlanNodeToSummaryStringTest.cpp
125+
# group7 (~461s)
116126
TableWriterTest.cpp
117-
ExchangeClientTest.cpp
118-
OrderByTest.cpp
119-
HashJoinBridgeTest.cpp
120-
FilterProjectTest.cpp
121-
UnnestTest.cpp
122-
StreamingEnforceDistinctTest.cpp
123-
FunctionResolutionTest.cpp
124-
TaskListenerTest.cpp
125-
PlanNodeToStringTest.cpp
126-
# group7 (~419s)
127127
SpillerTest.cpp
128-
SpillTest.cpp
129128
WindowTest.cpp
130-
PartitionedOutputTest.cpp
131-
TopNRowNumberTest.cpp
132-
MarkDistinctTest.cpp
133-
CustomJoinTest.cpp
134-
HilbertIndexTest.cpp
129+
SortBufferTest.cpp
130+
HashPartitionFunctionTest.cpp
131+
SpatialIndexTest.cpp
132+
MarkSortedTest.cpp
133+
ConcatFilesSpillMergeStreamTest.cpp
134+
PlanNodeToStringTest.cpp
135135
)
136136

137137
set(

0 commit comments

Comments
 (0)