Goal
Compare current downloaded benchmark artifacts with the historic benchmark results stored in the checked-out commit.
Scope
- Load the local checked-out benchmark-history directory as the baseline.
- Aggregate runner samples by median for each test set and benchmark.
- Flag regressions above a configurable threshold, defaulting to 10%.
- Report unmatched cases and runner samples separately.
Acceptance Criteria
Dependencies
Goal
Compare current downloaded benchmark artifacts with the historic benchmark results stored in the checked-out commit.
Scope
Acceptance Criteria
Dependencies