The `--parallel $NUM_JOBS` parameter on Windows with Visual Studio makes builds very slow

[LPGhatguy]

Hello! Today I investigated build performance with cmake-rs and had some interesting findings.

This issue pertains to Windows and Visual Studio 17.13.2. The default CMake generator for my platform is used, Visual Studio 17 2022. My installed CMake version is 3.22.1.

I've been working on bindings to Jolt Physics. Part of that is invoking CMake to build JoltC and Jolt Physics. When I compile the library with CMake directly (i.e. cmake -B build; cmake --build build --config Release), I get a fast parallel build that's complete in a few seconds.

When using cmake-rs to invoke the build, the build takes a lot longer, around a minute and a half.

I was able to isolate this to cmake-rs passing the flag --parallel $NUM_JOBS on Windows. NUM_JOBS is 16 on my machine. Just deleting that block of code from cmake-rs makes the build around 6x faster.

If I build the library from CMake manually with --parallel 16, I do see a slower build, but only around 2x slower than normal. There may be something else going on either with CMake or Cargo that causes the remaining slowdown when the --parallel flag is passed.

Jolt's CMake config passes the /MP flag to the compiler, which enables multi-process parallelism.

Switching the CMake generator to Ninja instead of Visual Studio 17 2022 (the default) also seems to resolve the performance issue. I believe Ninja has a different strategy for parallelization compared with MSBuild.

[LPGhatguy]

I've created a fork of cmake-rs on our organization to work around this for now: https://github.com/SecondHalfGames/cmake-rs/tree/no-parallel

[MHebes]

Do you have these environment variables set on your system? If not, it might be worth a shot. They help by launching more msbuild processes rather than just multiple cl.exe processes. That brings its behavior more in line with Ninja's parallelism.

UseMultiToolTask=true
EnforceProcessCountAcrossBuilds=true

In my experience they can speed up a "Visual Studio ..."-generator build quite a bit, whether building with CMake directly on the CLI, through Visual Studio's Open Folder CMake integration, or (I would hope!) with cmake-rs.

I would image the issue you're seeing is that cargo and msbuild are fighting for cores, so maybe this won't actually help for cmake-rs. Just a guess.

Disclaimer: These flags seem to be experimental, but I've never had a correctness issue with several large codebases and have only seen positives. The only "documentation" I can find is this blog post: https://devblogs.microsoft.com/cppblog/improved-parallelism-in-msbuild/, and this anemic doc page: https://learn.microsoft.com/en-us/visualstudio/msbuild/multitooltask-task?view=vs-2022

[LPGhatguy]

Thanks for the idea, I'll give it a go if I get a chance this week!

I learned that the shaderc-sys crate forces Ninja when running on Windows in their cmake-rs build. I wonder if they hit something similar or if they have other reasons for requiring it.

Do you have these environment variables set on your system?

Not currently, but I'll try 'em! For working day-to-day on CMake projects directly, the build performance has been fine overall.

I would image the issue you're seeing is that cargo and msbuild are fighting for cores, so maybe this won't actually help for cmake-rs. Just a guess.

I see the build slowdown happen even when↑ it's just the build.rs script from joltc-sys running, too. I don't think Cargo is doing anything besides waiting around for CMake.