Move analysis tools (such as plotting) to pyperf

[mdboom]

We should probably be working to move anything that isn't strictly related to automation inside of GitHub Actions upstream to pyperf.

[diegorusso]

Wishlist:

• Profiling information like this one: https://github.com/faster-cpython/benchmarking-public/tree/main/profiling (Both tabular and graphs (histogram/pie))
• Have a way to have the same kind of information that we have here: https://github.com/faster-cpython/benchmarking-public/tree/main/results/bm-20250326-3.14.0a6%2B-ffcb0ed-JIT
• Generate these kinds of graphs: https://github.com/faster-cpython/benchmarking-public/blob/main/msvc-tailcall-vs-default.svg
• Generate these kinds of graphs: https://github.com/faster-cpython/benchmarking-public/blob/main/longitudinal.svg

I'm not sure where all the above belongs to. If something cannot be moved, I would be more than happy also to make bench runner disjoint from GitHub so it can be run manually or on another automation system (Gitlab, Jenkins, cron)

[mdboom]

I'd put these things into 2 categories: Things that are (a) about analyzing 1-2 benchmark runs, and (b) things that are about managing a whole set of benchmark runs. The profiling plots and violin plots are in (a), and could reasonably be moved to pyperf (they are just analogous to tools that already exist there). Things like (b) don't really make sense for pyperf.

If we want to handle things like (b) on things other than GitHub Actions, that really just means reimplementing the workflow descriptions in bench_runner/templates in the format necessary for some other tool. Most of the logic in there already calls out to functionality implemented in Python elsewhere anyway ... we could be more extreme there are also implement the checkout and compilation steps in Python.

Let me propose this as a way forward:

1. Move as many of the individual steps (checkout, compilation, tuning, benchmarking) into a Python script so it's portable between automation systems.
2. Implement the configuration on top of that for another automation system.
3. Move the simple comparative plots to pyperf, and then adjust bench_runner to use those.

I can handle (1) as a first step. It sounds like you are interested in (2). (3) is sort of a "nice to have" and is disjoint and optional from the rest, IIUC.