Performance issue with very large set of ad-hoc tool calls

[svenier]

Transferring from slack thread https://pantsbuild.slack.com/archives/C0D7TNJHL/p1761162382770779

Attempting to build a large graph of 163,375 targets results in pants memory growth and "unbounded" execution time. The source repository is at https://github.com/NVIDIA/Dependency-Patterns/tree/main/scaling and the command to run it is pants run :go.

I was seeing this performance issue with 2.28.0 running on Linux accessing the files via NFS. Benjy was able to reproduce it running on macOS with local storage (not sure which pants version he was using).

Some internal profiling shows lots of time spent waiting on the GIL. We have not done any stack inspections to see if, for example, transitive dependency traversal is doing redundant work for nodes deep in the graph. Our next experiment is going to be to try it against a BuildBarn remote execution farm to see if offloading it makes much difference.

As a point of comparison Bazel is able to do the build in 9-10 minutes.

[benjyw]

To clarify, Bazel completes an apples-to-apples comparison of the same number (~160K) of subprocesses running the same scaling/geomorphy/run_build.sh script? If so then that is more evidence for GIL contention being the issue.

cc @sureshjoshi @cburroughs re prioritizing the upgrade to Python 3.14, to see if that would help.

Meanwhile I will continue to dig at this.

[benjyw]

In the slack thread you mentioned that Bazel does not "fork bomb" - are you saying that it spawns fewer processes? Or that the same number does not cause the machine to crap out?

[sureshjoshi]

Aside: I'll be looking at Py 3.14 next week or the week after, unless someone beats me to it. I want to get this supply chain stuff out of my system first.

The python upgrade is part of some performance improvements I have tabled, to see if we can speed up our CI and startup time a bit more. However, there will be no substantial effect on this issue (a couple percent at best for my other ideas). If we can get free-threading going, that will be suuuuper interesting if we're locking a lot.

[benjyw]

Well, yes, free-threading is what we should strive towards once we're on 3.14. Not sure how involved that would be.

Meanwhile I will see if we can mitigate this in other ways.

[svenier]

Catching up on the questions...

Yes, bazel is running the same DAG with the same work in each directory. In that setup the command line is directly in the genrule call rather than a shell script, but is otherwise identical.

I'm not sure on the mechanics of how bazel doesn't fall over... I'm assuming it limits the concurrency, somehow? Ultimately, it's the same number of md5sum processes being run.

[sureshjoshi]

I haven't had a chance to run this yet, but what about running without the pants daemon? That thing tries to memoize the world, which can't be great. --no-pantsd?

[svenier]

Watching the screen output, it's maybe a little better w/o pantsd. Now it's just python itself that's taking all of the memory:

3341642 svenier   20   0  296.0g   7.2g  35056 S  79.7   0.7  13:21.02 python3.11

Watching things in top, and the output flashing on the screen, it looks like all of the time is spent running shellcheck to determine dependencies. Is there a better Target type I should be using in the BUILD files instead of shell_sources for the .txt files? They don't need to be scanned as shell scripts.

[benjyw]

I'm going to see what it would take to port the adhoc process implementation to Rust and get rid of the GIL and all Python overhead entirely for this case.

[svenier]

I changed all of the text files from shell_sources to files and that seems to have helped, but pantsd still hit the OOM killer while I was away at lunch. I'm trying it now with --no-pantsd. If you think the adhoc implementation is the issue, would writing a real light-weight plugin to do just the exact command I need show some benefit?

[benjyw]

I changed all of the text files from shell_sources to files and that seems to have helped, but pantsd still hit the OOM killer while I was away at lunch. I'm trying it now with --no-pantsd. If you think the adhoc implementation is the issue, would writing a real light-weight plugin to do just the exact command I need show some benefit?

Yeah, the shellcheck invocations would double the number of processes... Can you merge that change to use files() targets (perhaps in a branch), so I can test against that?

[benjyw]

Re the implementation, if porting to rust solves the issue then the plugin would presumably also want to be written in rust so that rules aren't constantly going into and out of Python (and, currently, holding the GIL)

I will contemplate how to do that once we know that it would work...

[svenier]

Yeah, the shellcheck invocations would double the number of processes... Can you merge that change to use files() targets (perhaps in a branch), so I can test against that?

Pushed. See https://github.com/NVIDIA/Dependency-Patterns/tree/shell_to_file/scaling

Re the implementation, if porting to rust solves the issue then the plugin would presumably also want to be written in rust so that rules aren't constantly going into and out of Python (and, currently, holding the GIL)

Is there an easy way to make rust plugins? I'd be willing to try porting the custom ones we've got before we get too much code written.

[svenier]

Using files() and --no-pantsd it finished it just under 7 hours on my farm machine. So WAY better, but not good enough yet.

[benjyw]

Thanks for the update! I am taking a look at what a rust port would look like, will have more in a couple of days hopefully.