Skip to content

docs: parallelize Read the Docs HTML build - #5888

Draft
njzjz-bot wants to merge 4 commits into
deepmodeling:masterfrom
njzjz:docs/sphinx-j2
Draft

docs: parallelize Read the Docs HTML build#5888
njzjz-bot wants to merge 4 commits into
deepmodeling:masterfrom
njzjz:docs/sphinx-j2

Conversation

@njzjz-bot

@njzjz-bot njzjz-bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Override only the Read the Docs HTML build command.
  • Run Sphinx with two workers via -j 2.
  • Preserve the existing HTML builder, language, doctree directory, source directory, and output directory.
  • Keep this change independent from the Linkcode experiment.

Performance measurement

This PR intentionally contains only the Sphinx worker-count change so its Read the Docs preview measures parallelism independently.

Baseline Read the Docs build 33680413:

  • total build: 1889 seconds
  • dependency installation: 303 seconds
  • Sphinx HTML: 1524 seconds
  • Sphinx worker count: 1

Both performance PRs use upstream commit 83e7245646185a2de8f7d6277edc4f64b4f500f0 as their base.

Testing

  • Parsed .readthedocs.yml with PyYAML
  • Ran the configured Sphinx arguments against a Sphinx 9.0.4 smoke project with -j 2
  • ruff check .
  • ruff format .
  • Repository commit hooks, including YAML validation and Prettier

Coding agent: Codex
Codex version: codex-cli 0.144.6
Model: gpt-5.6-sol
Reasoning effort: xhigh

Summary by CodeRabbit

  • Documentation
    • Improved Read the Docs HTML builds with explicit Sphinx settings.
    • Standardized English-language output and build directories.
    • Limited documentation builds to two workers for more predictable performance.

Override the Read the Docs HTML command to use two Sphinx workers while preserving the existing builder, language, doctree, and output settings.

Coding-Agent: Codex
Codex-Version: codex-cli 0.144.6
Model: gpt-5.6-sol
Reasoning-Effort: xhigh
@dosubot dosubot Bot added the Docs label Jul 21, 2026
@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: b0811fff-fac2-4366-bc43-6956763d9821

📥 Commits

Reviewing files that changed from the base of the PR and between 83e7245 and 4cf6179.

📒 Files selected for processing (1)
  • .readthedocs.yml

📝 Walkthrough

Walkthrough

The Read the Docs configuration adds an explicit Sphinx HTML build command using two workers, English language settings, doc as the source, doc/_build/doctrees for doctrees, and $READTHEDOCS_OUTPUT/html for output.

Changes

Read the Docs build

Layer / File(s) Summary
Configure the Sphinx HTML build
.readthedocs.yml
Adds explicit Sphinx options for concurrency, language, doctree storage, source input, and HTML output paths.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: parallelizing the Read the Docs HTML build.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Jul 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.81%. Comparing base (cc908a8) to head (eed8f9c).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #5888      +/-   ##
==========================================
- Coverage   79.03%   78.81%   -0.22%     
==========================================
  Files        1055     1067      +12     
  Lines      122233   123805    +1572     
  Branches     4401     4510     +109     
==========================================
+ Hits        96607    97581     +974     
- Misses      24061    24614     +553     
- Partials     1565     1610      +45     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@njzjz
njzjz marked this pull request as draft July 21, 2026 10:47
njzjz-bot added 3 commits July 25, 2026 09:56
Run the custom Sphinx command from the documentation directory so the source checkout does not shadow the installed package. Add a 45-minute timeout with a forced-kill grace period to turn stalled parallel builds into explicit failures.

Coding-Agent: Codex
Codex-Version: codex-cli 0.144.6
Model: gpt-5.6-sol
Reasoning-Effort: xhigh
dargs.sphinx.DargsDomain advertises parallel_read_safe but does not
implement merge_domaindata, so Sphinx refuses to combine the per-worker
inventories and -j aborts the build. The domain's only state is a flat
targetid -> (docname, objtype) map, so merging is a dict update
restricted to the documents the worker read.

Verified on a minimal two-page project using dargs directives: -j 2
fails with the same NotImplementedError without the shim and succeeds
with it.
@njzjz

njzjz commented Jul 27, 2026

Copy link
Copy Markdown
Member

Rebased onto master and made -j 2 actually work in eed8f9cfc.

The RTD build on this branch was failing at 655 s with:

NotImplementedError: merge_domaindata must be implemented in
<class 'dargs.sphinx.DargsDomain'> to be able to do parallel builds!

dargs.sphinx returns {"parallel_read_safe": True} from its setup, but DargsDomain inherits Domain.merge_domaindata, which raises. Sphinx then tears down the worker pool and the failure surfaces as a secondary KeyError in sphinx/util/parallel.py. doc/conf.py now installs a merge_domaindata on DargsDomain when it is still the base implementation; the domain's only state is a flat targetid -> (docname, objtype) map, so merging is a dict update restricted to the documents that worker read. Worth upstreaming to dargs, at which point the guard makes the shim a no-op.

Verified on a minimal two-page project using .. dargs:: directives: python -m sphinx -j 2 reproduces the exact NotImplementedError without the shim and reports build succeeded with it.

For context on why this matters beyond this PR: the docs/readthedocs.org:deepmd check is red on a large number of open PRs, and every one I sampled is the same 40-minute RTD wall-clock limit — builds 33625117, 33625500, 33626766 all ended at 2409-2411 s with the sphinx step still running. So this PR is the fix for that whole class of failures, not just a speed-up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants