cmd: propagate a derived worker share to recursive subprocesses#2578
Draft
bendrucker wants to merge 1 commit into
Draft
cmd: propagate a derived worker share to recursive subprocesses#2578bendrucker wants to merge 1 commit into
bendrucker wants to merge 1 commit into
Conversation
toWorkerCommands dropped --max-workers, so under --recursive each subprocess fell back to NumCPU for its per-runner fan-out. Total in-flight work became (directory workers) times NumCPU, and the bound from the per-runner change did nothing in the recursive case that motivated it. Divide the worker budget across the directory workers that run concurrently and pass that share to each subprocess as --max-workers. directories times per-runner checks now stays within the budget in both plain and recursive modes. Claude-Session: https://claude.ai/code/session_01SFtKL5iaFWPVDqdcyvPiCJ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
toWorkerCommandsdropped--max-workers, so under--recursiveeach subprocess fellback to the number of CPUs for its per-runner fan-out. Total in-flight work became
(directory workers) times NumCPU, which means the bound from #2577 did nothing in the
recursive case that motivated this whole effort.
This divides the worker budget across the directory workers that run concurrently and
passes that share to each subprocess as
--max-workers. With a budget of N and N busydirectory workers, each subprocess runs its per-runner checks one at a time, so the
product stays at N. With fewer directories than the budget, each subprocess gets a larger
share. Directories times per-runner checks now stays within the budget in both plain and
recursive modes.
No user-facing flags change.
--max-workerskeeps its meaning and now bounds theper-runner path under
--recursivetoo. This is the gap the benchmark suite cannot reachdirectly (it does not spawn subprocesses), so it is covered by unit tests for the share
math and the propagated command, plus the recursive integration suite.
Stacked on #2577.