Support profile-guided optimization via Cargo

[michaelwoerister]

Now that PGO is available in stable Rust it would be great to make it easily accessible via cargo. @alexcrichton made a rough proposal for what this could look like here: rust-lang/rust#59913 (comment)

[luser]

Alex's proposal sounds pretty sensible. Having done a lot of work on PGO in the Firefox build I would like to make some suggestions:

1. Require that the crate explicitly specify a binary or binaries to run for the profiling scenario. Finding a good profiling workload is half the problem with PGO. For Firefox, we found that the vast majority of the performance gains were realized from simply launching the browser and then shutting it down after a short while, so that all of the core components that were exercised during startup would be optimized. With MSVC PGO this was enough to produce up to 20% improvements on some benchmarks. I don't think that a crate's testsuite is an appropriate workload to use for most situations.
2. As much as possible, handle the implementation details without requiring manual intervention. Running the llvm-profdata tool, etc, should happen automatically. Ideally C/C++ dependencies compiled via the cc crate should use the correct compiler flags so they are also PGO optimized.
3. Support incremental builds. Compiling nontrivial programs with PGO can take ages—the Windows Firefox MSVC PGO build used to take on the order of 4 hours! Compilation artifacts from both phases of the build should be preserved to make this possible. In the Firefox build system we added special build rules so that object files from each phase of the build were compiled with separate extensions and could coexist in a single object directory. With cargo it might be simple enough to treat the two phases as separate build targets. For Firefox that was the approach we took initially but the Firefox build spends enough time doing non-compilation work that allowing the two phases to share all of their non-compilation build artifacts was a significant time savings for us.
4. Make it straightforward to run benchmarks against the resulting PGO build. With all the lovely tools in the Rust ecosystem for performance testing, ideally one should be able to easily compare performance data for regular optimized builds vs. PGO builds, and to also compare changes in the training workload between PGO builds.
5. Consider supporting a mode of operation where the three phases (build, run profiling scenario, build optimized) can be run independently. We did this work for the Firefox build to enable PGO optimization for our macOS builds which are cross-compiled for Linux. The build-time savings we got from compiling on Linux were well worth the effort to do this, and the machine time in CI needed on actual macOS hardware was much smaller and more tractable. Given Rust's broad range of supported targets this might enable building with PGO even in specialized environments like embedded systems. (Obviously the targets would need runtime support for collecting the profiling data.)

[epage]

In the mean time, see https://crates.io/crates/cargo-pgo