Skip to content

Add diff tests to lambkin.shepherd #104

@hidmic

Description

@hidmic

Feature description

In order to use benchmarks as performance regression tests, we need a way to assert on a performance regression. To do that, we need a way to summarize benchmark results, compare these benchmark summaries, and fail a benchmark when the comparison suggests a problem.

Implementation considerations

Summaries should be user-defined if at all possible (it really depends on the application which metrics matter and which don't). This ticket will likely require extending RobotFramework libraries in lambkin.shepherd.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions