Benchmark Suite v1.0.5

This release reworks the configuration struct, the result-access API, and the output model, and significantly improves the adaptive benchmarking algorithm. It contains breaking changes — see the migration guide at the bottom before upgrading from v1.0.2.

Highlights

Dual convergence detection. A measurement is now accepted only when it satisfies both an RSE threshold and a mean-convergence criterion. A result that hits a stable RSE but whose mean is still drifting between rounds keeps running, closing a common false-convergence hole.
Welch's t-test tie detection. The previous confidence-interval overlap check has been replaced with a proper unequal-variance Welch's t-test, giving more accurate tie calls when sample sizes differ. Statistically indistinguishable implementations are reported as ties, not wins.
Sliding-window RSE with configurable bounds. RSE is computed over the last k samples, where k scales with total iteration count and is bounded by the new min_k and max_k settings.
Result-object API. run_benchmark() now returns void; results are retrieved through get_test_results() and get_all_results(), which hand back structured objects with .print(), .to_markdown(), and .to_csv().
Non-invasive CMake integration. The library no longer sets INTERFACE compile or link options on its target — your project's own optimization flags are no longer silently altered by adding benchmarksuite as a dependency.

New Features

Dual convergence + mean stability gate

Adaptive sampling now requires the mean to stabilize between rounds (convergence_threshold) in addition to the RSE threshold. Both gates must pass simultaneously before a result is marked converged. If a time or iteration limit is reached first, the best result collected so far is returned with converged = false.

Welch's t-test statistical ties

Tie classification uses an unequal-variance t-test, with overlapping distributions explicitly excluded from win counts and surfaced via position_type::tie.

Configurable RSE window

New min_k (floor, must be > 1) and max_k (ceiling) fields bound the sliding RSE window.

Compiler & OS introspection without CMake defines

Compiler name and version are now captured at compile time via consteval helpers, so the build no longer depends on CMake-injected name defines.

Improved PRNG

The internal xoshiro256+ generator now models the UniformRandomBitGenerator concept (result_type, min(), max()), making it compatible with STL <random> distributions. random_generator specializations inherit from xoshiro_256 directly rather than constructing a fresh engine per call.

Changes

run_benchmark() returns void instead of a metrics reference.
Cache clearing is now a single eviction before the entire run (warm-cache steady state for all subsequent iterations). Per-iteration cache clearing has been removed.
Output is produced from returned result objects rather than printed directly from benchmark_stage.
Default measured_iteration_count changed from 10 to 100.
Default rse_threshold changed from 1.0 to 2.5.
max_time_in_s is now a whole-second uint64_t (was a double of seconds).

Removals

print_results() → use get_test_results("name").print().
generate_markdown() → use get_test_results("name").to_markdown(...).
clear_all_results() removed — benchmark_stage holds results in a static inline member; create a new stage type with a different stage name for an isolated result set.
Per-iteration cache clearing (clear_cpu_cache_between_each_execution) removed.

Migration Guide (from v1.0.2)

1. Rename `stage_config` → `stage_config_data`

// Old
constexpr bnch_swt::stage_config config{ ... };
// New
constexpr bnch_swt::stage_config_data config{ ... };

2. Update configuration field names

v1.0.2 field	v1.0.5 field	Notes
max_execution_count	max_iteration_count	Same purpose
measured_execution_count	measured_iteration_count	Default 10 → 100
desired_rolling_standard_error	rse_threshold	Default 1.0 → 2.5
max_time_seconds (double)	max_time_in_s (uint64_t)	Whole seconds only
clear_cpu_cache_before_all_executions	clear_cpu_caches_before_iterations	Renamed
clear_cpu_cache_between_each_execution	removed	No per-iteration clearing
(none)	convergence_threshold	new — mean stability gate (default 1.0%)
(none)	min_k / max_k	new — RSE window floor/ceiling

// New
constexpr bnch_swt::stage_config_data config{
    .max_iteration_count = 200,
    .measured_iteration_count = 25,
    .benchmark_type = bnch_swt::benchmark_types::cpu,
    .clear_cpu_caches_before_iterations = true,
    .rse_threshold = 1.0,
    .convergence_threshold = 1.0,
    .max_time_in_s = 5
};

3. `run_benchmark` returns `void`

// Old
auto& metrics = bench::run_benchmark<"test", "subject", func>(args...);
// New
bench::run_benchmark<"test", "subject", func>(args...);
auto results = bench::get_test_results("test");

4. Replace `print_results()` / `generate_markdown()`

// Old
bench::print_results();
auto report = bench::generate_markdown("Title", "./results/");
// New
bench::get_test_results("test-name").print();
auto report = bench::get_test_results("test-name").to_markdown(true, true, "./results/");
auto stage_csv = bench::get_all_results().to_csv("./results/");

5. Updated return types

get_test_results() now returns final_test_results (with .sorted_results, .print(), .to_markdown(), .to_csv()) instead of a metrics map.
get_all_results() now returns stage_results_data (with .results, .lib_positions, .to_csv()) instead of a raw results vector.

Compatibility

C++20, GCC 13+ / Clang 16+ / MSVC 2022+
CUDA 11.0+ for GPU benchmarking
Windows, Linux, macOS (CUDA not available on Apple Silicon)

Available on vcpkg as rtc-benchmarksuite.

Full details: GitHub repository

# Benchmark Suite v1.0.5

Highlights

Dual convergence detection. A measurement is now accepted only when it satisfies both an RSE threshold and a mean-convergence criterion. A result that hits a stable RSE but whose mean is still drifting between rounds keeps running, closing a common false-convergence hole.
Welch's t-test tie detection. The previous confidence-interval overlap check has been replaced with a proper unequal-variance Welch's t-test, giving more accurate tie calls when sample sizes differ. Statistically indistinguishable implementations are reported as ties, not wins.
Sliding-window RSE with configurable bounds. RSE is computed over the last k samples, where k scales with total iteration count and is bounded by the new min_k and max_k settings.
Result-object API. run_benchmark() now returns void; results are retrieved through get_test_results() and get_all_results(), which hand back structured objects with .print(), .to_markdown(), and .to_csv().
Non-invasive CMake integration. The library no longer sets INTERFACE compile or link options on its target — your project's own optimization flags are no longer silently altered by adding benchmarksuite as a dependency.

New Features

Dual convergence + mean stability gate

Welch's t-test statistical ties

Tie classification uses an unequal-variance t-test, with overlapping distributions explicitly excluded from win counts and surfaced via position_type::tie.

Configurable RSE window

New min_k (floor, must be > 1) and max_k (ceiling) fields bound the sliding RSE window.

Compiler & OS introspection without CMake defines

Compiler name and version are now captured at compile time via consteval helpers, so the build no longer depends on CMake-injected name defines.

Improved PRNG

Changes

run_benchmark() returns void instead of a metrics reference.
Cache clearing is now a single eviction before the entire run (warm-cache steady state for all subsequent iterations). Per-iteration cache clearing has been removed.
Output is produced from returned result objects rather than printed directly from benchmark_stage.
Default measured_iteration_count changed from 10 to 100.
Default rse_threshold changed from 1.0 to 2.5.
max_time_in_s is now a whole-second uint64_t (was a double of seconds).

Removals

print_results() → use get_test_results("name").print().
generate_markdown() → use get_test_results("name").to_markdown(...).
clear_all_results() removed — benchmark_stage holds results in a static inline member; create a new stage type with a different stage name for an isolated result set.
Per-iteration cache clearing (clear_cpu_cache_between_each_execution) removed.

Migration Guide (from v1.0.2)

1. Rename `stage_config` → `stage_config_data`

// Old
constexpr bnch_swt::stage_config config{ ... };
// New
constexpr bnch_swt::stage_config_data config{ ... };

2. Update configuration field names

v1.0.2 field	v1.0.5 field	Notes
`max_execution_count`	`max_iteration_count`	Same purpose
`measured_execution_count`	`measured_iteration_count`	Default 10 → 100
`desired_rolling_standard_error`	`rse_threshold`	Default 1.0 → 2.5
`max_time_seconds` (double)	`max_time_in_s` (uint64_t)	Whole seconds only
`clear_cpu_cache_before_all_executions`	`clear_cpu_caches_before_iterations`	Renamed
`clear_cpu_cache_between_each_execution`	removed	No per-iteration clearing
(none)	`convergence_threshold`	new — mean stability gate (default 1.0%)
(none)	`min_k` / `max_k`	new — RSE window floor/ceiling

// New
constexpr bnch_swt::stage_config_data config{
    .max_iteration_count = 200,
    .measured_iteration_count = 25,
    .benchmark_type = bnch_swt::benchmark_types::cpu,
    .clear_cpu_caches_before_iterations = true,
    .rse_threshold = 1.0,
    .convergence_threshold = 1.0,
    .max_time_in_s = 5
};

3. `run_benchmark` returns `void`

// Old
auto& metrics = bench::run_benchmark<"test", "subject", func>(args...);
// New
bench::run_benchmark<"test", "subject", func>(args...);
auto results = bench::get_test_results("test");

4. Replace `print_results()` / `generate_markdown()`

// Old
bench::print_results();
auto report = bench::generate_markdown("Title", "./results/");
// New
bench::get_test_results("test-name").print();
auto report = bench::get_test_results("test-name").to_markdown(true, true, "./results/");
auto stage_csv = bench::get_all_results().to_csv("./results/");

5. Updated return types

get_test_results() now returns final_test_results (with .sorted_results, .print(), .to_markdown(), .to_csv()) instead of a metrics map.
get_all_results() now returns stage_results_data (with .results, .lib_positions, .to_csv()) instead of a raw results vector.

Compatibility

C++20, GCC 13+ / Clang 16+ / MSVC 2022+
CUDA 11.0+ for GPU benchmarking
Windows, Linux, macOS (CUDA not available on Apple Silicon)

Available on vcpkg as rtc-benchmarksuite.

Full details: [GitHub repository](https://github.com/RealTimeChris/benchmarksuite)

Release v1.0.5

Benchmark Suite v1.0.5

Highlights

New Features

Dual convergence + mean stability gate

Welch's t-test statistical ties

Configurable RSE window

Compiler & OS introspection without CMake defines

Improved PRNG

Changes

Removals

Migration Guide (from v1.0.2)

1. Rename stage_config → stage_config_data

2. Update configuration field names

3. run_benchmark returns void

4. Replace print_results() / generate_markdown()

5. Updated return types

Compatibility

Highlights

New Features

Dual convergence + mean stability gate

Welch's t-test statistical ties

Configurable RSE window

Compiler & OS introspection without CMake defines

Improved PRNG

Changes

Removals

Migration Guide (from v1.0.2)

1. Rename stage_config → stage_config_data

2. Update configuration field names

3. run_benchmark returns void

4. Replace print_results() / generate_markdown()

5. Updated return types

Compatibility

Uh oh!

1. Rename `stage_config` → `stage_config_data`

3. `run_benchmark` returns `void`

4. Replace `print_results()` / `generate_markdown()`

1. Rename `stage_config` → `stage_config_data`

3. `run_benchmark` returns `void`

4. Replace `print_results()` / `generate_markdown()`