Problems when building very large website with Docusaurus

[AtlanticClaire]

Hello everyone,

I am currently evaluating whether we can use Docusaurus (currently used version: 3.6.3) to publish our very large documentation. This documentation consists of 70,000 and more md files.
I had the following experiences when building this very large website with Docusaurus:

With more than 12.000 md files, I got the following error: FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory (issue described here: #8329). Using NODE_OPTIONS="--max-old-space-size to increase the heap available has not fixed the problem.

I also tried the experimental_faster flags. When I switched these on, I no longer got the error mentioned above. Instead, I got the following error: [ERROR] Error: Unable to build website for locale de. Cause: EMFILE: too many open files, open 'C:\repos\docs.ctrlx\sidebars-apps-plc.js' (issue described here: #8719). I am currently working under Windows and will try to change some limitations under Linux as a next step to get rid of the error.

But of course I wonder whether these attempts can be successful at all and for what size of websites docusaurus is designed. That is why I have the following questions:

1. Question to Docusaurus team: With how many md files has Docusaurus been successfully tested? With how many md files is it possible to successfully build a website?

2. Question to community: Do any of you have experience with building very large websites that have to be built from 70,000 or more individual md files? I would be very grateful for an exchange of experiences.

Thanks in advance!

[slorber]

Building Docusaurus with Webpack consumes a lot more memory than the new Docusaurus Faster build (using Rspack bundler), so the --max-old-space-size is often needed.

For Cause: EMFILE: too many open files, open on Windows, there usually have OS settings you can adjust to allow more open files. I'm not familiar with Windows but you can probably remove this limit too.

It would be great to share which "step" the build is crashing at. You can get more verbose logging with the DOCUSAURUS_PERF_LOGGER=true env variable (see #10556 (comment)) and share it here.

---

We haven't tested Docusaurus at various scales, but we have some community websites that are quite large due to generation large API reference documentation from code.

For example this one has a sitemap reporting 4000 pages: https://xsoar.pan.dev/

I can't tell if Docusaurus supports 12k pages or 70k pages, but if you build that kind of website, maybe a static site generator is not ideal in your case due to long build times and a very inefficient redeploy work when you just want to fix a doc typo.

The question to ask yourself is: why do you have a 70k pages statically generated website in the first place? Why don't you use SSR instead? What prevents you from splitting that site into many smaller websites that you link together?

---

A related example: if you have a Docusaurus website with 1000 pages, but translate it into 30 locales, then you have 30 x 1000 pages Docusaurus websites, and not a single 30k pages website. This makes it possible to deploy the 30 websites in parallel instead of having a very long sequential build.

This splitting strategy is not limited to static site generators, even Vercel/Next.js is using that strategy for their portal: https://vercel.com/blog/how-vercel-adopted-microfrontends

[AtlanticClaire]

@slorber Thank you very much for the detailed answer!

Here is the output with the DOCUSAURUS_PERF_LOGGER=true env variable:

[INFO] Docusaurus version: 3.6.3
Node version: v20.17.0

C:\repos\docs.xxxxx>set DOCUSAURUS_PERF_LOGGER=true

C:\repos\docs.xxxxx>npm run build

> [email protected] build
> docusaurus build --dev

[INFO] Building in dev mode
[PERF] Get locales to build - 273.44 ms - (56mb -> 60mb)
[INFO] Website will be built for all these locales:
- en
- de
[INFO] [en] Creating an optimized production build...
[PERF] Build > en > Load site > Load plugins > Init plugins - 1.27 seconds! - (60mb -> 76mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load classic@default > getDefaultCodeTranslationMessages() - 31.41 ms - (77mb -> 69mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load classic@default - 34.85 ms - (76mb -> 69mb)
[PERF] Build > en > Load site > Load plugins > Load plugins content > Load pages@default > loadContent() - 1.72 seconds! - (76mb -> 331mb)
[ERROR] Sidebars file at "C:\repos\docs.xxxxx\sidebars-apps-plc.js" failed to be loaded.
[ERROR] Loading of version failed for version current
[PERF][KO] Build > en > Load site > Load plugins > Load plugins content > Load docs@docs-apps-plc > loadContent() - 1.72 seconds! - (76mb -> 331mb)
[PERF][KO] Build > en > Load site > Load plugins > Load plugins content > Load docs@docs-apps-plc - 1.72 seconds! - (76mb -> 331mb)
[PERF][KO] Build > en > Load site > Load plugins > Load plugins content - 1.73 seconds! - (76mb -> 331mb)
[PERF][KO] Build > en > Load site > Load plugins - 3.00 seconds! - (60mb -> 331mb)
[PERF][KO] Build > en > Load site - 3.00 seconds! - (60mb -> 331mb)
[PERF][KO] Build > en - 3.00 seconds! - (60mb -> 331mb)
[PERF][KO] Build - 3.00 seconds! - (60mb -> 331mb)

[ERROR] Error: Unable to build website for locale en.
    at tryToBuildLocale (C:\repos\docs.xxxxx\node_modules\@docusaurus\core\lib\commands\build\build.js:78:15)
    at async C:\repos\docs.xxxxx\node_modules\@docusaurus\core\lib\commands\build\build.js:34:9
    at async mapAsyncSequential (C:\repos\docs.xxxxx\node_modules\@docusaurus\utils\lib\jsUtils.js:21:24)
    at async Object.async (C:\repos\docs.xxxxx\node_modules\@docusaurus\logger\lib\perfLogger.js:106:28)
    at async Command.build (C:\repos\docs.xxxxx\node_modules\@docusaurus\core\lib\commands\build\build.js:33:5)
    at async Promise.all (index 0)
    at async runCLI (C:\repos\docs.xxxxx\node_modules\@docusaurus\core\lib\commands\cli.js:56:5)
    at async file:///C:/repos/docs.xxxxx/node_modules/@docusaurus/core/bin/docusaurus.mjs:44:3 {
  [cause]: Error: Docusaurus could not load module at path "C:\repos\docs.xxxxx\sidebars-apps-plc.js"
  Cause: EMFILE: too many open files, open 'C:\repos\docs.xxxxx\sidebars-apps-plc.js'
      at loadFreshModule (C:\repos\docs.xxxxx\node_modules\@docusaurus\utils\lib\moduleUtils.js:36:15)
      at loadSidebarsFileUnsafe (C:\repos\docs.xxxxx\node_modules\@docusaurus\plugin-content-docs\lib\sidebars\index.js:76:54)
      ... 5 lines matching cause stack trace ...
      at async Object.async (C:\repos\docs.xxxxx\node_modules\@docusaurus\logger\lib\perfLogger.js:106:28)
      at async C:\repos\docs.xxxxx\node_modules\@docusaurus\core\lib\server\plugins\plugins.js:39:23
      at async Object.async (C:\repos\docs.xxxxx\node_modules\@docusaurus\logger\lib\perfLogger.js:106:28) {
    [cause]: Error: EMFILE: too many open files, open 'C:\repos\docs.xxxxx\sidebars-apps-plc.js'
        at readFileSync (node:fs:448:20)
        at jiti (C:\repos\docs.xxxxx\node_modules\jiti\dist\jiti.js:1:245281)
        at loadFreshModule (C:\repos\docs.xxxxx\node_modules\@docusaurus\utils\lib\moduleUtils.js:33:16)
        at loadSidebarsFileUnsafe (C:\repos\docs.xxxxx\node_modules\@docusaurus\plugin-content-docs\lib\sidebars\index.js:76:54)
        at async loadSidebars (C:\repos\docs.xxxxx\node_modules\@docusaurus\plugin-content-docs\lib\sidebars\index.js:87:32)
        at async doLoadVersion (C:\repos\docs.xxxxx\node_modules\@docusaurus\plugin-content-docs\lib\index.js:157:34)
        at async loadVersion (C:\repos\docs.xxxxx\node_modules\@docusaurus\plugin-content-docs\lib\index.js:193:28)
        at async Promise.all (index 0)
        at async Object.loadContent (C:\repos\docs.xxxxx\node_modules\@docusaurus\plugin-content-docs\lib\index.js:201:33)
        at async Object.async (C:\repos\docs.xxxxx\node_modules\@docusaurus\logger\lib\perfLogger.js:106:28) {
      errno: -4066,
      code: 'EMFILE',
      syscall: 'open',
      path: 'C:\\repos\\docs.xxxxx\\sidebars-apps-plc.js'
    }
  }
}
[INFO] Docusaurus version: 3.6.3
Node version: v20.17.0

C:\repos\docs.xxxxx>

Unfortunately, I have not been able to change the limits under Windows. That's why I probably will try the build under Linux.

Besides, I have already thought about the splitting strategy you suggested because of the problems you mentioned (long build times and a very inefficient redeploy work). Maybe this is an approach I will try in the future.

[slorber]

So it fails quite early in the build process while loading the docs plugin content. We are not even reaching the bundling phase.

Apart from increasing the limit, I'm not sure what to do 😅 It would be helpful if you provided a runnable repro

[AtlanticClaire]

Hello @slorber, thank you very much for your support! I tried to create a runnable repro for you yesterday. Unfortunately, I did not succeed because the error does not occur when I simply add a large amount of data (e.g. 28,000 md files) to a plain page. It must have something to do with the specific configuration of the project I'm getting the error with, in addition to the amount of data. Unfortunately, I can't share the faulty project publicly because it contains company-specific information. I will keep trying to create a runnable repro with the error and send it to you. However, this will probably take longer because I am currently unable to continue working on the issue.

[slorber]

Alright, that's already good news if Docusaurus can run with 28k files!

Maybe the files in your private project are longer/heavier? Let me know if you find out what cause the crash.