Skip to content

Commit cc06911

Browse files
committed
opt: stabilize end-to-end benchmarks
Previously, end-to-end benchmark results were erratic. Automatic collection of index recommendations caused the first run of a query to perform much more poorly than later runs. Also, slow queries which were benchmarked with few iterations could have dramatically different results if they were run with over 6 iterations and generic query plans were selected. Results could look like: ``` BenchmarkEndToEnd/... 6 237211938 ns/op 122903316 B/op 811648 allocs/op BenchmarkEndToEnd/... 6 191910097 ns/op 100521821 B/op 597137 allocs/op BenchmarkEndToEnd/... 7 158093143 ns/op 86201213 B/op 512044 allocs/op BenchmarkEndToEnd/... 8 126410552 ns/op 76577004 B/op 455492 allocs/op BenchmarkEndToEnd/... 12 83743316 ns/op 52329982 B/op 313305 allocs/op ``` These benchmarks are specifically meant to measure the performance of query optimization and building logical plans, so generic query plans and automatic collection of index recommendations have been disabled. This stabilizes the benchmarks. Now results look more like: ``` BenchmarkEndToEnd/... 6 190383688 ns/op 100462772 B/op 598004 allocs/op BenchmarkEndToEnd/... 6 190895507 ns/op 100107917 B/op 596267 allocs/op BenchmarkEndToEnd/... 6 187822812 ns/op 100012398 B/op 595454 allocs/op BenchmarkEndToEnd/... 6 191119153 ns/op 102148360 B/op 606517 allocs/op BenchmarkEndToEnd/... 6 202094812 ns/op 100020434 B/op 595465 allocs/op ``` Release note: None
1 parent 15a7aac commit cc06911

File tree

1 file changed

+2
-0
lines changed

1 file changed

+2
-0
lines changed

pkg/sql/opt/bench/bench_test.go

+2
Original file line numberDiff line numberDiff line change
@@ -1141,6 +1141,8 @@ func BenchmarkEndToEnd(b *testing.B) {
11411141
sr.Exec(b, `SET CLUSTER SETTING sql.stats.automatic_collection.enabled = false`)
11421142
sr.Exec(b, `SET CLUSTER SETTING sql.stats.flush.enabled = false`)
11431143
sr.Exec(b, `SET CLUSTER SETTING sql.metrics.statement_details.enabled = false`)
1144+
sr.Exec(b, `SET CLUSTER SETTING sql.metrics.statement_details.index_recommendation_collection.enabled = false`)
1145+
sr.Exec(b, `SET plan_cache_mode = force_custom_plan`)
11441146
sr.Exec(b, `CREATE DATABASE bench`)
11451147
for _, schema := range schemas {
11461148
sr.Exec(b, schema)

0 commit comments

Comments
 (0)