Profile-Guided Optimization (PGO) benchmark results

[zamazan4ik]

Hi!

I evaluate Profile-Guided Optimization (PGO) performance improvements for applications and libraries (including other JSON parsers) in different software domains - all current results can be found here. According to the tests, PGO helps to improve performance in various software domains. I decided to perform PGO benchmarks on the actson-rs library too since some library users may be interested in improving the library's performance. I did some benchmarks and here are the results.

Test environment

• Fedora 39
• Linux kernel 6.7.6
• AMD Ryzen 9 5900x
• 48 Gib RAM
• SSD Samsung 980 Pro 2 Tib
• Compiler - Rustc 1.76
• actson-rs version: main branch on commit 332b8922eb6b3960956902fe80cc031c13f09dd6
• Disabled Turbo boost for improving consistency across runs

Benchmark

For benchmark purposes, I use two benchmarks: via cargo bench and geojson benchmarks.

For cargo bench set of benchmarks, the PGO training phase is done with cargo pgo bench, PGO optimization phase - with cargo pgo optimize bench.

For geojson set of benchmarks, I used run-all.sh script. For the PGO training phase, I used completely the same script but the instrumented binary is built with cargo pgo build, and the number of runs was settled to 1 since there is no need to run more frequently for the training phase in this case.

All PGO-related routines are done with cargo-pgo.

All benchmarks are done on the same machine, with the same hardware/software during runs, with the same background "noise" (as much as I can guarantee, of course).

Results

Firstly, here are the results for cargo bench:

• Release: https://gist.github.com/zamazan4ik/c9d53ede3789ba31ec83a7bf8a0bc30c
• PGO-optimized: https://gist.github.com/zamazan4ik/23066db130df1ce3f950783b4c27b2d4
• (just for reference) PGO instrumented: https://gist.github.com/zamazan4ik/a5bc9bd0ffc85d3cb656f598b33a5d45

Geojson results:

• Release: https://gist.github.com/zamazan4ik/2f392c01f7e7d171c9e89ee7b09099a5
• PGO optimized: https://gist.github.com/zamazan4ik/834e6b2f215608192f95436d34c9f931
• (just for reference) PGO instrumented: https://gist.github.com/zamazan4ik/56a76fc4df6164875e942e34dbfd6058

Raw files (results.json files) are also saved on my machine - I can share them as well if you are interested.

At least in the provided by project benchmarks, there are measurable improvements in many cases. Also, I got the same results for other JSON benchmarks that can be found in the awesome-pgo repo.

Maybe mentioning these results somewhere in the README (or any other user-facing documentation) can be a good idea. Probably you will be interested in integrating building actson-rs with PGO into your applications - who knows :)

Please do not treat the issue as a bug or something like that - it's just a benchmark report.

[michel-kraemer]

Great work! Very useful! Thank you so much! 👍

Since the topic is, as you say, not a bug report, I've converted it to a discussion.