Skip to content

Commit 1add9c1

Browse files
Update CHANGELOG.md
1 parent b5a5013 commit 1add9c1

1 file changed

Lines changed: 1 addition & 1 deletion

File tree

CHANGELOG.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ Changes in the benchmark methodology or presentation, as well as major news.
55
### 2026-05-11
66
Unified benchmark scripts for different systems by providing a common interface in a set of scripts: `install`, `start`, `check`, `stop`, `load`, `query`, and `data-size`. Make the dataset download scripts common as well. Use a general benchmark runner in `lib/` to ensure different systems get equal treatment. This makes it easier to add more ways of testing, different datasets, and scenarios to the benchmark, and simplifies support of all 88 systems presented. Note: for embedded systems, such as sqlite and the Python duckdb module, wrap them into a Python HTTP server, so that the benchmark can run each query separately.
77

8-
Restart databases before measuring the cold run of each query as requested in [#667](https://github.com/ClickHouse/ClickBench/issues/667) and [#793](https://github.com/ClickHouse/ClickBench/issues/793). This prevents unfair measurements and removes the way for cheating on the benchmark for systems that do excessive in-process caching without flushing it before the cold run. Unify flushing the OS page cache before a cold run, so that all benchmark entries follow the same rules. Notes: for stateless systems (such as query engines on top of Parquet), the restart is a no-op; for systems without durability and in-memory systems, the restart before each query also requires repeated data loading, which time is included in the cold query measurement. A downside and potential problem of this approach is that systems that speculatively prewarm during startup can now get an advantage.
8+
Restart databases before measuring the cold run of each query as requested in [#667](https://github.com/ClickHouse/ClickBench/issues/667) and [#793](https://github.com/ClickHouse/ClickBench/issues/793). This prevents unfair measurements and removes the way for cheating on the benchmark for systems that do excessive in-process caching without flushing it before the cold run. Unify flushing the OS page cache before a cold run, so that all benchmark entries follow the same rules. Notes: for stateless systems (such as query engines on top of Parquet), the restart is a no-op; for systems without durability and in-memory systems, the restart before each query also requires repeated data loading, which time is included in the cold query measurement. A downside and potential problem of this approach is that systems that speculatively prewarm during startup can now get an advantage. Also, this change is not applicable to SaaS (cloud) systems.
99

1010
Introduced a new measurement - QPS and error rate on concurrent workload (10 connections for 10 minutes) to prove the advantage of the refactoring. Currently, the metric is not exposed in the benchmark.
1111

0 commit comments

Comments
 (0)