Skip to content

Build: add concurrency options for Git and Python builds #12853

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
humitos wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Build: add concurrency options for Git and Python builds #12853

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a25dd60
Build: add concurrency options for Git and Python builds
humitos Mar 18, 2026
692a206
Update readthedocs/vcs_support/backends/git.py
humitos Mar 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions readthedocs/doc_builder/director.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
from readthedocs.projects.constants import BUILD_COMMANDS_OUTPUT_PATH_HTML
from readthedocs.projects.constants import GENERIC
from readthedocs.projects.exceptions import RepositoryError
from readthedocs.projects.models import Feature
from readthedocs.projects.notifications import MESSAGE_PROJECT_SSH_KEY_WITH_WRITE_ACCESS
from readthedocs.projects.signals import after_build
from readthedocs.projects.signals import before_build
Expand Down Expand Up @@ -724,6 +725,21 @@ def build_docs_class(self, builder_class):
success = builder.build()
return success

def _add_parallel_env_var(self, env):
cpus = os.cpu_count()
# These variables are useful for projects that compile libraries during the build.
env.update(
{
# https://cmake.org/cmake/help/latest/envvar/CMAKE_BUILD_PARALLEL_LEVEL.html
"CMAKE_BUILD_PARALLEL_LEVEL": cpus,
# https://www.gnu.org/software/make/manual/html_node/Options_002fRecursion.html
"MAKEFLAGS": f"-j {cpus}",
# Pillow and other libraries use this variable
# https://pillow.readthedocs.io/en/stable/installation/building-from-source.html#build-options
"MAX_CONCURRENCY": cpus,
Copy link
Copy Markdown
Member

@stsewd stsewd Mar 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this is the default

By default, it uses as many CPUs as are present.

Copy link
Copy Markdown
Member Author

@humitos humitos Mar 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I put Pillow as an example, but I found that it was used by other libraries as well while doing the research. We can remove it if we want, tho.

}
)

def _add_git_ssh_command_env_var(self, env):
if settings.ALLOW_PRIVATE_REPOS:
# Set GIT_SSH_COMMAND to use ssh with options that disable host key checking
Expand Down Expand Up @@ -812,6 +828,9 @@ def get_build_env_vars(self):

self._add_git_ssh_command_env_var(env)

if self.data.project.has_feature(Feature.BUILD_IN_PARALLEL):
self._add_parallel_env_var(env)

# Update environment from Project's specific environment variables,
# avoiding to expose private environment variables
# if the version is external (i.e. a PR build).
Expand Down
20 changes: 20 additions & 0 deletions readthedocs/vcs_support/backends/git.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,10 +39,30 @@ def _get_clone_url(self):
return f"{parsed_url.scheme}://$READTHEDOCS_GIT_CLONE_TOKEN@{parsed_url.netloc}{parsed_url.path}"
return self.repo_url

def _set_parallel_configs(self):
"""
Set git configs to enable parallelization.
"""
# https://git-scm.com/docs/git-config#Documentation/git-config.txt-fetchparallel
self.run("git", "config", "--global", "fetch.parallel", "0")
# https://git-scm.com/docs/git-config#Documentation/git-config.txt-checkoutworkers
self.run("git", "config", "--global", "checkout.workers", "0")
# https://git-scm.com/docs/git-config#Documentation/git-config.txt-checkoutthresholdForParallelism
self.run("git", "config", "--global", "checkout.thresholdForParallelism", "100")
Copy link
Copy Markdown
Member

@stsewd stsewd Mar 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this is already the default?

The default is 100.

Copy link
Copy Markdown
Member Author

@humitos humitos Mar 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, yeah, I was testing other numbers locally and forgot to remove it from here.

# https://git-scm.com/docs/git-config#Documentation/git-config.txt-submodulefetchJobs
self.run("git", "config", "--global", "submodule.fetchJobs", "0")
# https://git-scm.com/docs/git-config#Documentation/git-config.txt-packthreads
self.run("git", "config", "--global", "pack.threads", "0")

def update(self):
"""Clone and/or fetch remote repository."""
super().update()

from readthedocs.projects.models import Feature

if self.project.has_feature(Feature.BUILD_IN_PARALLEL):
self._set_parallel_configs()

if self.project.git_checkout_command:
# Run custom checkout step if defined
if isinstance(self.project.git_checkout_command, list):
Expand Down
Loading