Make flamegraphs a first-class tool for performance discussions

[moul]

We should make flamegraphs a standard part of how we talk about performance and benchmarking in the project.

First step: add a make profile (or similar) command to generate flamegraphs locally. This will simplify profiling, debugging, and sharing results within the team, while also providing inspiration with copy-pastable commands.

Then: set up CI to generate flamegraphs for master and for PRs, storing them as GitHub artifacts or similar. This gives us shareable links for quick inspection and makes perf insights more accessible in reviews.

Related: #4130 (comment)

[paulogarithm]

should we package a flamegraph implementation in the gno source code ? or do we install flamegraph tools when the make profile command is triggered ?

[moul]

should we package a flamegraph implementation in the gno source code ? or do we install flamegraph tools when the make profile command is triggered ?

If flamegraphs behave as expected, we should focus on exporting a compatible raw output, and then let an external tool handle the conversion to a full flamegraph.

A similar pattern exists in gno mod graph, where we export a minimal core format and let external tooling transform it into something richer: #3588 (comment)

Cc @mvertes — any thoughts or details on this?

from DM: We were thinking of proposing the profile as a flag to gnodev instead of handling it through a make rule, and we’d love to hear your thoughts on this approach.

I think gnodev should focus on orchestrating config and tooling, but not have exclusive features. We should first build a standalone implementation (e.g. ./contribs/profiler or directly in gnovm if the implementation is light) that runs profiling and outputs the necessary report. This gives us something minimal and CI-friendly.

Then, if it makes sense, we can integrate it into gnodev to make live usage smoother — I’m not opposed to that at all, just think the foundation should be standalone and composable.

[mvertes]

The easiest way to work right now with profiling data, and without altering the sources, is to:

1. Identify reproducible scenarios, i.e. interesting go tests, benchmarks or a test cases sufficiently intensive.
2. For those tests, run them with go test -run MY_TEST -cpuprofile MY_TEST-cpu.prof -memprofile MY_TEST-mem.prof
3. Run go tool pprof -http :6666 MY_TEST-cpu.prof, which launches a browser on all the pprof aggregated results (listings, call graphs, flame graphs). What is presented by default is a call graph, you have to navigate through menus to access to flamegraphs. They can be then screen-captured. Maybe other tools exist to generate those flamegraphs offline, I don't know them.

I find it interesting to have the different representations of results all at hand, not only the flame graph alone, to better diagnose and interpret the performance issues, which is exactly what the pprof web interface provides.

For example, in the gnovm/pkg/gnolang/ directory, execute go test -run TestFiles/fib0.gno -cpuprofile fib0-cpu.prof then go tool pprof -http :6666 fib0-cpu.prof. (fib0.gno is maybe not the best candidate).

From that, it is easy to always generate profile data for the specific interesting tests in the CI, through a makefile target, but launch manually later the pprof explorer.

Also keep in mind that pprof data is meaningful only at go runtime level, that is you analyse the gnovm, not the gno program running on top of it.

To analyse gno program performance, we need to:

1. Translate the profiling capture at gnovm level (i.e. tagging all the opcodes, their cumulated exec time, allocs, etc)
2. Combine these with precise gno source locations, as done in the gno debugger,
3. Export produced data in a format compatible with go pprof, so we can reuse their tooling for exploring and visualising.

So this is a different story, but a nice additional project (quite deep technically speaking).