Continuous Benchmarking using Github Actions #2134
Conversation
Signed-off-by: Pablo Gutiérrez Félix <[email protected]>
Signed-off-by: Pablo Gutiérrez Félix <[email protected]>
Signed-off-by: Pablo Gutiérrez Félix <[email protected]>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
.github/workflows/kem-bench.yml
Outdated
| on: | ||
| workflow_dispatch: | ||
| push: | ||
| branches: [main] |
There was a problem hiding this comment.
I'd like this to be triggered via the commit-to-main.ymll file so that it fits in with the rest of our CI.
|
|
||
| jobs: | ||
| build: | ||
| runs-on: ubuntu-latest |
There was a problem hiding this comment.
Eventually we'll want to have this working on multiple runners. How scalable is the approach taken here?
I walked through the PQCP setup with Matthias and Ry last week; once open-quantum-safe/tsc#180 lands we'll be able to work with a similar setup.
There was a problem hiding this comment.
@SWilson4 Initially, my idea would be to create another matrix for the different runners that we will be using for benchmarking, but of course any ideas are welcome. I am not too familiar working with external runners, but we could follow a similar approach as https://github.com/pq-code-package/mlkem-native, where there are different workflows depending on the runner selected, and they all call an action which executes the benchmarking.
Also, I fixed your other comments. Let me know if you have any other feedback!
|
I think it would be good if the build & configuration information that's currently in the expandable "Latest commit build information" is displayed directly at the top of the page. In the pop-ups that show when you hover over a datapoint, it looks like all the commits have been authored by you. Is that placeholder information? Or should something else be showing here? |
Signed-off-by: Pablo Gutiérrez <[email protected]>
|
@dstebila I have added the build information at the top, let me know if that works: https://pablo-gf.github.io/liboqs/dev/bench/. As for your second comment, that is placeholder information. The idea is that those pop-ups will contain the details of each commit made to the library starting from the first commit after this continuous benchmarking framework is deployed. |
Looks good, thanks! |
Signed-off-by: Pablo Gutiérrez <[email protected]>
|
@SWilson4 I fixed the security warnings that popped-up after my last commit. Let me know if you have any comments or suggestions. As I mentioned at the beginning, to make the entire process work we would also need to create a new gh-pages branch so that the workflows generate and continuously update a Github page with the visualization of the benchmarking results. |
|
Thanks for the updates, @pablo-gf! Are you able to merge this branch into |
This PR aims to provide a new benchmarking approach for liboqs. It uses the Continuous Benchmarking action from Marketplace, like the mlkem-native repository. For speed benchmarking of the current algorithms in the library, 3 new files are included:
scripts/parse_liboqs_speed.py : Retrieves the benchmarking data from speed_kem and speed_sig and outputs it in a json file that matches the format required by the continuous benchmarking action.
workflows/kem-bench.yml: Iterates through the different KEM algorithms executing the speed test and gathering its information using parse_liboqs_speed.py. It then pushes the output json file to a gh-pages branch using the Continuous Benchmarking action.
workflows/sig-bench.yml: Same kem-bench.yml but for signature algorithms.
To complete the benchmarking, it is required to create a new gh-pages branch so that the workflows generate and continuously update a Github page with the visualization of the benchmarking results. I have adapted the html file to include some additional features here . I can include these changes once the new gh-pages branch for liboqs is set up. You can see an example of what the final output should look like here.
Let me know if you have any questions or suggestions!