Skip to content

Compare current runner artifacts with checked-out benchmark history #18

Description

@tkilias

Goal

Compare current downloaded benchmark artifacts with the historic benchmark results stored in the checked-out commit.

Scope

  • Load the local checked-out benchmark-history directory as the baseline.
  • Aggregate runner samples by median for each test set and benchmark.
  • Flag regressions above a configurable threshold, defaulting to 10%.
  • Report unmatched cases and runner samples separately.

Acceptance Criteria

  • Comparison uses the history available in the current checkout, not the direct parent commit.
  • Threshold, noise, regression, missing-baseline, and incomplete-candidate scenarios are tested.
  • Unmatched data is reported without being classified as a regression.

Dependencies

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureProduct feature

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions