ci: Automating scheduling benchmarking testing for PRs

[DerekFrank]

Fixes #N/A

Description

New CI to automatically run scheduling benchmark tests for both the existing code and proposed changes whenever a PR modifies a .go file. The new workflow will comment results into PR comments, and also store CPU and MEM profiles into a downloadable artifacts that reviewers and approvers can use to analyze any performance improvements or regressions. The current setup leverages the matrix functionality to make setting up new benchmark CI tests simple in the future.

This is a first iteration of a CI suite for performance testing. There are a couple, sometimes contradictory natural extensions for the future:

• Automatically fail the test if performance regresses and remove performance results from PR comments
• Add Consolidation performance benchmark tests
• Parameterize more components of the go test call
• Refactor the approach to run a single bench test per GitHub action for cleaner replication

CI testing will always be a moving target as the needs of the community develop, this is intended to be a quick and dirty approach to gather feedback and allow the CI performance testing to adapt to the needs of developers.

How was this change tested?

Simulated PRs in my own fork to demonstrate the functionality.

PR that shows benchmarking in action:

• feat: Look! I solved P=NP DerekFrank/karpenter-testing-fork#13

PR that shows how doc and other non .go functionality changes won't get tested:

• docs: Adding Karpenter in the future talk to readme DerekFrank/karpenter-testing-fork#12

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: DerekFrank
Once this PR has been reviewed and has the lgtm label, please assign bwagner5 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

Hi @DerekFrank. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[coveralls]

Pull Request Test Coverage Report for Build 13710467043

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage increased (+0.02%) to 81.633%

---

Totals
Change from base Build 13683176482:	0.02%
Covered Lines:	9596
Relevant Lines:	11755

---

💛 - Coveralls

[rschalo]

Generally looks good. Are we able to run multiple tests and then create a report using ?

[DerekFrank]

Absolutely! I wanted to keep the initial CI relatively lightweight so I avoided any specific analysis of the results. We've got lots of paths to take the general concept of CI performance testing, I tend to err on the side of incremental, fast changes that allow us to see how contributors actually use the CI in practice. If a multi-run and benchstat analysis of the results setup is preferred, I would be happy to add that to the PR or to tackle it in a follow up PR