[Bug]: rsdoctor UI /webpack/loaders/overall sometimes just doesn't load for large bundles. loaders timeline also has trouble.

[dgoldstein0]

Version

System:
    OS: Linux 6.8 Ubuntu 22.04.5 LTS 22.04.5 LTS (Jammy Jellyfish)
    CPU: (8) x64 Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
    Memory: 27.27 GB / 61.80 GB
    Container: Yes
    Shell: 5.1.16 - /usr/bin/bash
  npmPackages:
    @rsdoctor/cli: ^1.3.12 => 1.3.16
    @rsdoctor/rspack-plugin: ^1.3.11 => 1.3.13

I'm using Chrome on Windows to view the rsdoctor report - version 144 right now.

Details

I have a very large codebase we're building with rspack. We've seen a lot of variance in our builds (which I now believe is largely due to the filesystem of the remote execution system we're running them on) and in trying to debug where the time was going I pulled out rsdoctor. rsdoctor's web ui seems rather flakey for us - sometimes I can get it to render data, and then sometimes it just won't or takes so long (>30s) that my impatience wins instead. I don't really understand it but I assume it's related to the sheer size of the bundles I'm feeding it - looks like my input is about 11500 files to bundle.

Tried to capture the experience in loom https://www.loom.com/share/5d378599bf31484f9efd3ec73808ae1a

$ du -h .rsdoctor/
55M .rsdoctor/loader
8.0K .rsdoctor/otherReports
20K .rsdoctor/errors
5.1M .rsdoctor/moduleGraph
205M .rsdoctor/moduleCodeMap
192M .rsdoctor/chunkGraph
8.0K .rsdoctor/envinfo
8.0K .rsdoctor/summary
92K .rsdoctor/packageGraph
8.0K .rsdoctor/plugin
8.0K .rsdoctor/resolver
12K .rsdoctor/configs
456M .rsdoctor/

Reproduce link

sorry, I don't have an easy repro I can share

Reproduce Steps

bundle some really large codebase with rspack and rsdoctor enabled - probably in the realm of 500kloc - with builtin:swc-loader & source-map-loader running on all inputs. (apologies, but I can't link to my codebase - it's my company's closed source website).

launch rsdoctor with yarn rsdoctor analyze --profile .rsdoctor/manifest.json --no-open --port 3000, navigate to localhost:3000 (port forwarded if running on a separate machine), enable storage for localhost in the browser if the page doesn't load. navigate around the rsdoctor ui, especially the loader timeline & analysis, and try to see if it'll load in a reasonable time; it probably won't, but seems inconsistent (sometimes it'll load in maybe 30s, other times it takes minutes or never loads)

[yifancong]

You can try this options:

output: {
     reportCodeType: { noCode: true } 
  }

[dgoldstein0]

I don't think it helps. Some of the initial loads might be better but still like 10-30s and on my second attempt to view http://localhost:3000/#/webpack/loaders/analysis right now - I think I've been waiting for 2 minutes so I'm calling this still hung.

[dgoldstein0]

I will say the size of the rsdoctor output is smaller with noCode: true

55M     .rsdoctor/loader
8.0K    .rsdoctor/otherReports
20K     .rsdoctor/errors
5.1M    .rsdoctor/moduleGraph
56K     .rsdoctor/moduleCodeMap
124K    .rsdoctor/chunkGraph
8.0K    .rsdoctor/envinfo
8.0K    .rsdoctor/summary
92K     .rsdoctor/packageGraph
8.0K    .rsdoctor/plugin
8.0K    .rsdoctor/resolver
12K     .rsdoctor/configs
60M     .rsdoctor/

but perhaps it's the size/expense of passing around & parsing the loader analysis data that's more the problem.

[yifancong]

Got it, I will optimize the loader data.

[dgoldstein0]

thanks, I'll look forward to it. once you've merged any optimizations let me know which release they make it into and we'll try them out.

[yifancong]

@dgoldstein0 Please try this version 1.5.3-alpha.0. Can the results of this version be optimized?

[dgoldstein0]

just tried, doesn't seem to help. I had another crash too in yarn rsdoctor analyze (happens in 1.5.2 as well as 1.5.3-alpha.0) and worked around with your earlier suggestion of output: {reportCodeType: { noCode: true }} but once I'm past that /#/webpack/loaders/overall and /#/webpack/loaders/analysis never load.

[dgoldstein0]

On a second attempt it did load the loaders analysis but still very slow. Fwiw looking at the websocket traffic in the browser I see a few very notable leaps in the timeline.

time=12:57:15.278 message= 425["/api/loader/names",null]
time=12:57:15.278 message= 426["/api/loader/filetree",null]
time=12:57:15.313 mesage= 431[{"req":{"api":"/api/routes","body":null},"res":["Overall","Compile.WebpackLoaders","Compile.WebpackPlu...
time=12:57:15.356 message= 432[{"req":{"api":"/api/alerts/overlay","body":null},"res":[{"stack":" ⚠ asset size limit: The following asset(s) ex...
time=12:57:48.586 message= 435[{"req":{"api":"/api/loader/names","body":null},"res":["source-map-loader","builtin:swc-loader","/dev/shm...
time=12:57:48.624 message= 427["/api/graph/layers",null]
time=12:57:58.365 message= 436[{"req":{"api":"/api/loader/filetree","body":null},"res":[{"path":"/dev/shm/bazel-sandbox.2a335a5cc530fb... (length: 13781891)

3rd attempt on that page didn't work.

I was also wondering if the problem could be node gc slowing things down, but node --max-old-space-size=6000 $(yarn bin rsdoctor) analyze --profile .rsdoctor/manifest.json --no-open --port 3000 faired no better than my original yarn rsdoctor analyze --profile .rsdoctor/manifest.json --no-open --port 3000 so I don't think that's the issue.

[yifancong]

I understand the problem. Then I need to reconstruct the loader data structure and display logic, which may take some scheduling time. |