Not making full use of parallelism in jobserver mode

[juergbi]

When ninja runs as a jobserver client, it first starts as many commands as tokens it can acquire and then waits for the first pending command to finish. Only then will it again try to acquire tokens and start more commands.

If the jobserver has multiple clients (could be multiple ninja processes or a mix of build systems), it's possible that a token becomes available before the next pending command (of a given ninja process) finishes. In the time between the token becoming available and the next command finishing, the parallelism set by the jobserver is not fully utilized.

@digit-google Is this a known limitation? Are there any plans to improve this? I'm not familiar with the ninja code base, but it may be possible to add the jobserver FIFO to the ppoll() or pselect() call to allow reacting when a jobserver token becomes available.

[juergbi]

Example to trigger this issue:

Makefile:

all: a b

a:
	ninja -f a.ninja -v

b:
	ninja -f b.ninja -v

a.ninja:

rule shortsleep
  command = date; sleep 1

rule longsleep
  command = date; sleep 10

build all: phony along ashort
build along: longsleep
build ashort: shortsleep

b.ninja:

rule shortsleep
  command = date; sleep 1

rule longsleep
  command = date; sleep 10

build all: phony along ashort
build along: longsleep
build ashort: shortsleep

make -j3 results in the following output:

ninja -f a.ninja -v
ninja -f b.ninja -v
ninja: Jobserver mode detected:  -j3 --jobserver-auth=fifo:/tmp/GMfifo3778458
ninja: Jobserver mode detected:  -j3 --jobserver-auth=fifo:/tmp/GMfifo3778458
[1/2] date; sleep 1
Tue Oct 21 15:17:56 CEST 2025
[2/2] date; sleep 10
Tue Oct 21 15:17:56 CEST 2025
[1/2] date; sleep 10
Tue Oct 21 15:17:56 CEST 2025
[2/2] date; sleep 1
Tue Oct 21 15:18:06 CEST 2025

The second sleep 1 is started only once the sleep 10 of the same ninja instance completes, even though it could already be started once the first sleep 1 (of the other ninja instance) completes.

[digit-google]

Thanks for the report. This is indeed a limitation of the current SubprocessSet implementation that cannot react to changes to the jobserver pool FIFO.

Adding an extra file descriptor to the ppoll() / pselect() call would be a first step, but then the SubprocessSet::DoWork() result type may need to change (the function only exists in case of interruption of when a subcommand completes), and callers adjusted to the new scheme. Doable but not completely trivial, there are a few subtleties here.

I don't plan to work on this right now, but PRs are welcome.

[digit-google]

By the way, do you know how GNU Make handles the same situation as a client?

[juergbi]

By the way, do you know how GNU Make handles the same situation as a client?

GNU Make uses pselect() to wait for both, SIGCHLD and the jobserver FIFO, at the same time.

https://cgit.git.savannah.gnu.org/cgit/make.git/tree/src/posixos.c#n468

[juergbi]

Adding an extra file descriptor to the ppoll() / pselect() call would be a first step, but then the SubprocessSet::DoWork() result type may need to change (the function only exists in case of interruption of when a subcommand completes), and callers adjusted to the new scheme. Doable but not completely trivial, there are a few subtleties here.

A possible alternative may be to make TryAcquire() blocking¹, returning an invalid slot only if interrupted by a signal. That might require fewer changes in other parts. TryAcquire() currently has a retry loop in case of EINTR. I don't know whether that's important or whether we could return an invalid slot on EINTR.

Or did you already consider this? Is there a reason the acquire must be non-blocking?

¹ I think we would have to use ppoll() or pselect() to unblock the signals, which may not be ideal as it might require interaction between Jobserver and SubprocessSet, possibly requiring changes in other parts after all.