Skip to content

Upgrade docfx to 2.78.5, docs target to net10, and increase docs build perf - #1432

Open
paulirwin wants to merge 3 commits into
apache:masterfrom
paulirwin:feature/docfx-upgrade-perf
Open

Upgrade docfx to 2.78.5, docs target to net10, and increase docs build perf#1432
paulirwin wants to merge 3 commits into
apache:masterfrom
paulirwin:feature/docfx-upgrade-perf

Conversation

@paulirwin

Copy link
Copy Markdown
Contributor
  • You've read the Contributor Guide and Code of Conduct.
  • You've included unit or integration tests for your change, where applicable.
  • You've included inline docs for your change, where applicable.
  • There's an open issue for the PR that you are making. If you'd like to propose a change, please open an issue to discuss the change or find an existing issue.

Upgrade docfx to 2.78.5, docs target to net10, and increase docs build perf

Description

This PR:

  • Upgrades docfx to 2.78.5 (primarily for net10 support, but also to keep it evergreen)
  • Updates the docs plugins to use the docfx 2.78.5 package, which necessitated adding the CancellationToken parameter (with bonus cancellation support too)
  • Updates the docs targets to use the net10.0 target (instead of net8.0)
  • Parallelizes the build to dramatically improve the performance of the docs build.

Performance

The docs build historically has taken several minutes to run. This makes iterating on improving the docs harder. By parallelizing the Powershell build, it now takes a fraction of a time.

Command (from websites/apidocs/):

time pwsh ./docs.ps1 -Clean -LuceneNetVersion 4.8.0-ci

Before (latest master):

331.80 real       359.01 user        41.10 sys

After (this PR):

93.96 real       348.58 user        41.17 sys

Result: roughly 3.5x speed-up (on macOS arm64 M4 Max), reducing it from ~5.5min to ~1.5min. Your mileage will vary based on your OS and hardware, but it should perform better the more cores you have.

Notes

Previously, we had been encountering a file-locking issue with the toc file, that necessitated us specifying parallelism of 1 and sleeping between runs. This contributed to the poor performance. By parallelizing instead at the process level, this conflict seems to be avoided, as I could not reproduce any locking issues after many runs. It also carefully batches them into "waves" to avoid the circular dependency issues, which might have contributed to the issue previously.

Because this uses parallelism in PowerShell, it requires PowerShell 7+, aka PowerShell Core. Windows PowerShell 5.x is no longer supported for building the docs with this change, so a check is added at script launch. PS7 is cross-platform and truly open-source, so it is a better fit for our open-source project anyways.

AI: Coded and tested with the assistance of Claude Code (Opus 5).

@paulirwin
paulirwin requested review from NightOwl888 and a lite review from Copilot August 5, 2026 14:45
@paulirwin paulirwin added the notes:website-or-documentation Documentation or website changes label Aug 5, 2026
@paulirwin
paulirwin marked this pull request as ready for review August 5, 2026 14:46

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Updates the API documentation toolchain to support DocFX 2.78.5 and .NET 10 while significantly reducing doc build time by parallelizing metadata/build steps (and requiring PowerShell 7+ to do so).

Changes:

  • Upgrade DocFX (dotnet tool + plugin packages) to 2.78.5 and update docfx configs to target net10.0.
  • Parallelize DocFX metadata + build execution in docs.ps1, with explicit PS7+ requirement and improved failure aggregation.
  • Update GitHub Actions docs workflow to run the script via pwsh.

Reviewed changes

Copilot reviewed 35 out of 35 changed files in this pull request and generated no comments.

Show a summary per file
File Description
websites/apidocs/docs.ps1 Adds PS7 guard, solution-level restore, and parallel “waves” build orchestration for DocFX.
websites/apidocs/docfx.json Updates multiple docfx metadata entries to TargetFramework: net10.0.
websites/apidocs/docfx.*.json Updates per-project docfx configs to TargetFramework: net10.0.
src/docs/LuceneDocsPlugins/LuceneDocsPlugins.csproj Bumps DocFX plugin dependencies to 2.78.5.
src/docs/LuceneDocsPlugins/*.cs Updates post-processor signatures for cancellation support (DocFX API change).
.config/dotnet-tools.json Upgrades the docfx local tool version to 2.78.5.
.github/workflows/Lucene-Net-Documentation.yml Runs docs build using pwsh instead of Windows PowerShell.
Suppressed comments (6)

websites/apidocs/docs.ps1:1

  • The build wave list is fully hard-coded, which makes it easy for the script to silently drift out of sync when a docfx config is added/removed/renamed in $DocFxJsonMeta. A concrete mitigation is to validate at runtime that the flattened $DocFxBuildWaves set matches the expected project list (e.g., from $DocFxJsonMeta), and throw a clear error listing missing/extra items. Longer-term, consider generating waves from the xref graph to avoid manual maintenance.
    websites/apidocs/docs.ps1:1
  • Get-Content $projFile | ConvertFrom-Json pipes the JSON file line-by-line into ConvertFrom-Json, which will fail for multi-line JSON (the docfx configs shown in this PR are multi-line). Use Get-Content -Raw (or otherwise join the content into a single string) before calling ConvertFrom-Json so the JSON is parsed as one document.
    websites/apidocs/docs.ps1:1
  • Capturing all DocFX output into $output and then converting it via Out-String can be memory-heavy and slow (especially when running multiple projects concurrently). A more scalable approach is to stream each line to Write-Host (or write directly to the log file and only print a tail on failure), which avoids building large in-memory strings and reduces contention when jobs run in parallel.
    websites/apidocs/docs.ps1:1
  • Parameter names in this script are otherwise PascalCase ($StagingPort, $BaseUrl, etc.), but $maximumParallelJobs is camelCase. To keep the public CLI surface consistent while still supporting the psake naming, consider renaming the parameter to MaximumParallelJobs and adding an alias (e.g., [Alias('maximumParallelJobs')]) so existing invocations still work.
    websites/apidocs/docs.ps1:1
  • The TargetFramework=net10.0 used for the one-time restore is now a second source of truth that must be kept in sync with the various docfx configs. To reduce the chance of future drift (and confusing --noRestore failures), consider introducing a single $TargetFramework variable/parameter and using it consistently here (and possibly emitting it in a single place for the docfx configs to reference).
    websites/apidocs/docs.ps1:1
  • The TargetFramework=net10.0 used for the one-time restore is now a second source of truth that must be kept in sync with the various docfx configs. To reduce the chance of future drift (and confusing --noRestore failures), consider introducing a single $TargetFramework variable/parameter and using it consistently here (and possibly emitting it in a single place for the docfx configs to reference).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@paulirwin

Copy link
Copy Markdown
Contributor Author

I forgot one thing to mention in the PR description: this PR also now restores the solution once at the start, then skips restore for each docfx operation. Previously, each docfx run was doing a dotnet restore, which was adding many seconds to the build time unnecessarily. Docfx 2.78.3 added a noRestore flag that allows for skipping this, which contributed to the performance gains. It would not have been possible to get this gain on our prior version of docfx, so that is yet another reason to upgrade.

Add a pull_request trigger to both the documentation and website
workflows, scoped to the paths that affect each build plus the workflow
file itself, so changes to these builds can be validated in CI before
they are merged.

The steps that check out the site repo, copy the built output, and open
a pull request against it are skipped for pull_request events, so a PR
build never publishes anywhere.

The website job's upstream-only guard is relaxed for pull requests;
without this, PR builds from forks would be skipped entirely. Push and
workflow_dispatch on a fork remain blocked as before.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

notes:website-or-documentation Documentation or website changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants