Does it make sense to bring back CommonsChunkPlugin?

[scameron]

Hello,

TL;DR: Would it be possible/feasible to bring back CommonsChunkPlugin as an optional alternative to SplitChunksPlugin ?

I'm working in an enormous code base that SplitChunksPlugin doesn't seem to handle very well. Or, at least, I haven't been able to find a SplitChunks configuration that works well for us, despite literally dozens of hours spent debugging into the plugin and analyzing the internals. No matter what I do, our build output always has:

• Lots of modules duplicated across many chunks
• Too many (relatively) small chunks

In Webpack 3, we had things working pretty well by using CommonsChunkPlugin in combination with some coarse-grained "webpackChunkName" magic comments in dynamic imports that were structured such that we (a) had a relatively small number of well-named chunks, and (b) had a single parent chunk that all other chunks were children of.

Organizing the async chunks like this, of course, made for a lot of duplicated modules. To resolve this we relied on 2 plugins:

1. RemoveParentModulesPlugin. If a module exists in all parents of a chunk, it will be removed from the child chunk.
2. CommonsChunkPlugin. For each functional area, we extracted duplicated modules into a well-named "functional-area-common" chunk. We also, as the final invocation of this plugin, extracted modules duplicated across functional areas or in the main parent chunk into a top-level "common" chunk.

This approach relied on the fact that Webpack executes RemoveParentModulesPlugin before every separate instance of CommonsChunkPlugin. So after a new "functional-area-common" chunk is created as a child of the main parent chunk, the next invocation of RemoveParentModulesPlugin will have a new chance to push the modules up into the parent.

This approach was pretty easy to understand, reason about and control, and it seemed to work well. The loaded chunks were well-named and coarse-grained and the duplication was pretty minimal. Indeed we ended up with a very large main parent chunk (and associated common chunk) but that is fine because that code is needed by all workflows and should be always be loaded. The number of requests at runtime was fairly low.

With Webpack 5 and SplitChunksPlugin, I've become convinced after an enormous effort that there is no way to replicate this behaviour. The new heuristics-based approach is very difficult to reason about in a large code base and even with explicitly named cache groups I can't get close to what we had before. Even with debugging, I honestly can't predict (or even understand in some cases) which configuration changes are going to control what output. Configuration is a lot of guesswork and when you're dealing with 100K modules, there's really no way to guess right.

Additionally, the iterative effect I described above between RemoveParentModulesPlugin and CommonsChunkPlugin is no longer there because all cache groups are handled inside a single SplitChunksPlugin invocation. This means there's only a single RemoveParentModulesPlugin invocation and a single SplitChunksPlugin invocation.

Which bring me to my question. 😄

Would it be possible/feasible to bring back CommonsChunkPlugin as a possible (of course, optional) alternative to SplitChunksPlugin ? I guess it was removed because it was felt that the functionality was superseded by SplitChunks, but that's clearly not the case for all kinds of code bases. Maybe having a choice here would be better?

If you're still with me at this point, thanks for your attention. 😄

[vankop]

Lots of modules duplicated across many chunks
Too many (relatively) small chunks

when using http2 this is not a problem (mostly). you should control this by using https://webpack.js.org/plugins/split-chunks-plugin/#splitchunksmaxasyncrequests https://webpack.js.org/plugins/split-chunks-plugin/#splitchunksmaxinitialrequests

Too many (relatively) small chunks

you can set min size https://webpack.js.org/plugins/split-chunks-plugin/#splitchunksminsize

For long-term caching you can use https://webpack.js.org/plugins/split-chunks-plugin/#splitchunkslayer , module layer could be set in module.rules

[scameron]

Hi @vankop,

Whether or not many small requests matters or not with http2 depends a lot on the number/size of the requests. With smaller or mid-size apps, it won't be a problem at all, as you say. But with really large apps it definitely can make a difference (and it measurably does for our app). The best analysis of this I've seen is in these two articles:

• https://medium.com/@asyncmax/the-right-way-to-bundle-your-assets-for-faster-sites-over-http-2-437c37efe3ff
• https://blog.khanacademy.org/forgo-js-packaging-not-so-fast/

But theory aside, in practice we do observe a difference in our benchmarks.

Regarding the suggestion to change configuration options like maxAsyncRequests and minSize, I've of course played around with every possible setting I could find, but nothing (so far) has come close to what we had before.

The biggest problem of all is just the complexity of the combinations when the optimizer runs against a big code base. There's so much internal magic, it's really hard to know what affects what. I end up just more or less trying semi-random combinations of things and hoping for something good to happen. But with all the possible config options plus if you introduce cache groups and/or layers into the picture, the combinations are essentially endless.

There's something to be said for simplicity... maybe CommonsChunkPlugin wasn't so sophisticated but it was easy to understand and control and debug. I've been fighting with SplitChunks, on an off, for about 8 months now and still haven't produced an output I'm that happy with.

[sokra]

Could you post your webpack 3 CommonsChunkPlugin configuration here?

[scameron]

Here it is. These are meant to run in order with, as I wrote in my original description, the RemoveParentModulesPlugin running before each one.

This config outputs some really massive chunks, especially parent-chunk and common-chunk, so we have another simple plugin that just splits the chunk up into smaller slices that all get loaded together. We typically do a slice for each 4MB of module size.

new CommonsChunkPlugin({
    name: "parent-chunk",
    chunks: ["area1-feature1-chunk", "area1-feature2-chunk"],
    minChunks: 2,
    async: "area1-common-chunk"
}),
new CommonsChunkPlugin({
    name: "parent-chunk",
    chunks: [
        "area2-feature1-chunk",
        "area2-feature2-chunk",
        "area2-feature3-chunk",
        "area2-feature4-chunk",
        "area2-feature5-chunk"
    ],
    minChunks: 2,
    async: "area2-common-chunk"
}),
new CommonsChunkPlugin({
    name: "parent-chunk",
    chunks: [
        "area3-feature1-chunk",
        "area3-feature2-chunk",
        "area3-feature3-chunk",
        "area3-feature4-chunk"
    ],
    minChunks: 2,
    async: "area3-common-chunk"
}),
new CommonsChunkPlugin({
    name: "parent-chunk",
    chunks: Array.from(area4AsyncChunkSet).concat([
        "area4-feature1-chunk",
        "area4-feature2-chunk",
        "area4-feature3-chunk",
        "area4-feature4-chunk"
    ]),
    minChunks: 2,
    async: "area4-common-chunk"
}),
new CommonsChunkPlugin({
    name: "parent-chunk",
    chunks: [
        "area5-feature1-chunk",
        "area5-feature2-chunk",
        "area5-feature3-chunk",
        "area5-feature4-chunk",
        "area5-feature5-chunk",
        "area5-feature6-chunk"
    ],
    minChunks: 5,
    async: "area5-common-chunk"
}),
new CommonsChunkPlugin({
    name: "parent-chunk",
    chunks: [
        "area6-feature1-chunk",
        "area6-feature2-chunk",
        "area6-feature3-chunk"
    ],
    minChunks: 2,
    async: "area6-common-chunk"
}),
new CommonsChunkPlugin({
    name: "parent-chunk",
    chunks: [
        "area7-feature1-chunk",
        "area7-feature2-chunk",
        "area7-feature3-chunk",
        "area7-feature4-chunk",
        "area7-feature5-chunk",
        "area7-feature6-chunk"
    ],
    minChunks: 5,
    async: "area7-common-chunk"
}),
new CommonsChunkPlugin({
    name: "parent-chunk",
    chunks: [
        "area8-feature1-chunk",
        "area8-feature2-chunk",
        "area8-feature3-chunk",
        "area8-feature4-chunk",
        "area8-feature5-chunk"
    ],
    minChunks: 2,
    async: "area8-common-chunk"
}),
new CommonsChunkPlugin({
    name: "parent-chunk",
    chunks: [
        "area9-feature1-chunk"
    ],
    minChunks: function(module) {
        return module.context && module.context.includes(path.join("node_modules", "moment"));
    }
}),
new CommonsChunkPlugin({
    name: "parent-chunk",
    chunks: ["area10-feature1-chunk","area10-feature2-chunk"],
    minChunks: function (module, count) {
        return (
            count>=1&&
            module.context &&
            module.context.includes("specific-context") && !module.context.includes("other-context")
        );
    },
    async: "area10-common-chunk",
}),
new CommonsChunkPlugin({
    name: "parent-chunk",
    minChunks: function (module, count) {
        return (
            count>=1&&
            module.context &&
            module.context.includes("specific-context-2")
        );
    },
    async: "area11-common-chunk",
}),
new CommonsChunkPlugin({
    name: "parent-chunk",
    minChunks: function (module, count) {
        return (
            count >= 2 &&
            module.context &&
            module.context.includes(path.join("shell", "tools", "common")) &&
            !module.resource.endsWith("Tool.ts")
        );
    },
    async: "area12-common-chunk",
}),
new CommonsChunkPlugin({
    name: "parent-chunk",
    chunks: [
        "area13-feature1-chunk",
        "area13-feature2-chunk",
        "area13-feature3-chunk",
        "area13-feature4-chunk",
        "area13-feature5-chunk"
    ],
    minChunks: function (module, count) {
        return (
            count>=2 &&
            module.context &&
            module.context.includes("specific-context-3")
        );
    },
    async: "area13-common-chunk"
}),
new CommonsChunkPlugin({
    name: "parent-chunk",
    chunks: [
        "area14-feature1-chunk",
        "area14-feature2-chunk",
        "area14-feature3-chunk"
    ],
    minChunks: function (module, count) {
        return (
            count>=2 &&
            module.context &&
            module.context.includes("specific-context-4")
        );
    },
    async: "area14-common-chunk"
}),
new CommonsChunkPlugin({
    name: "parent-chunk",
    minChunks: function(module, count){
        if(module.context && (module.context.includes("specific-context-5")
            || module.context.includes("node_modules" + path.sep + "buffer")
            || module.context.includes("node_modules" + path.sep + "base64-js")
            || module.context.includes("node_modules" + path.sep + "ieee754")
            || module.context.includes("node_modules" + path.sep + "isarray")
            || module.context.includes(path.join("node_modules", "node-libs-browser", "mock"))
            || module.context.includes(path.join("node_modules", "webpack", "url"))
            || module.context.includes(path.join("node_modules", "webpack", "buildin"))
            || module.context.includes(path.join("node_modules", "path-browserify"))
         )) {
            return false;
        }
        return count >=2;
    },
    children: true,
    async: "common-chunk"
})