Ideas for dealing with external dependencies

[Charlie-XIAO]

This issue is for tracking ideas for dealing with widget external dependencies.

High-level goals are:

• Widget developers who want to use external dependencies (assumed to have node available) should be able to bundle the external dependencies into the final string of code (this means we need node modules resolution).
• Widget developers should be able to produce a distribution that can be used by widgets users at ease, with minimal (none if possible) restrictions on how they write their widget code.
• Widget users should be able to use the distributed widget without having node or node modules.

[Charlie-XIAO]

Rolldown is a bundler with similar functionalities as rollup, but written in Rust which means we can directly use it into our backend. It is not yet stabilized thus not published to crates.io (rolldown/rolldown#3227), but we can directly depend on its git source. This will only prevent our crates from being published, but it is almost certain that rolldown gets stabilized before Deskulpt does so not a problem.

Widget Developers' Side

Rolldown has built-in support for node modules resolution so external dependencies is not a problem for widget developers who are assumed to have node and thus node_modules. Rolldown also has built-in support for TypeScript/JSX transforms and ESM/CJS interop, so there is zero configuration needed for these. A possible setup of the bundler is:

let bundler_options = BundlerOptions {
    input: Some(vec!["WIDGET_CONFIG.ENTRY".to_string().into()]),
    cwd: Some(WIDGET_CONFIG.DIR),
    format: Some(OutputFormat::Esm),
    // This also has some additional handling of e.g. `process.env.NODE_ENV` so that we will not
    // need plugin-replace as with rollup
    platform: Some(Platform::Browser),
    // This is still experimental
    minify: Some(true),
    // Enable checks so we do not miss warnings
    checks: Some(ChecksOptions { circular_dependency: Some(true) }),
    // Assume that we have `react` and `@emotion/react/jsx-runtime` re-exported, so they can be
    // treated as external (not resolved when bundling)
    external: Some(
        vec![
            "BASE_URL/.scripts/react".to_string(),
            "BASE_URL/.scripts/@emotion/react/jsx-runtime".to_string(),
        ]
        .into(),
    ),
    // Use the automatic JSX runtime with source from `@emotion/react` so it will finally resolve
    // to `@emotion/react/jsx-runtime` which is treated as external
    jsx: Some(Jsx::Enable(JsxOptions {
        runtime: JsxRuntime::Automatic,
        import_source: Some("@emotion/react".to_string()),
        ..Default::default()
    })),
    ..Default::default()
}

let alias_plugin = AliasPlugin {
    entries: vec![
        // Alias both `react/jsx-runtime` and `@emotion/react/jsx-runtime` to the externalized
        // local script so they can share what's used by the Deskulpt runtime; IIRC the JSX
        // runtime of `@emotion/react` is a superset of `react` (it just in addition supports
        // the `css` attribute) so this should be safe
        Alias {
            find: StringOrRegex::String("react/jsx-runtime".to_string()),
            replacement: "BASE_URL/.scripts/@emotion/react/jsx-runtime".to_string(),
        },
        Alias {
            find: StringOrRegex::String("@emotion/react/jsx-runtime".to_string()),
            replacement: "BASE_URL/.scripts/@emotion/react/jsx-runtime".to_string(),
        },
        // Alias `react` to the externalize local script; note that the order matters, i.e.,
        // this aliasing must happen after the `react/jsx-runtime` one, otherwise `react`
        // will be aliased to `BASE_URL/.scripts/react` first and the runtime will be resolved
        // to `BASE_URL/.scripts/react/jsx-runtime` which does not exist
        Alias {
            find: StringOrRegex::String("react".to_string()),
            replacement: "BASE_URL/.scripts/react".to_string(),
        },
    ],
}

let bundler = Bundler::with_plugins(bundler_options, vec![Arc::new(alias_plugin)]);

Widget Developer Packaging

In the previous design of #66 with SWC and rollup, the packaging step is mixed with the normal bundling routing with external dependencies. It may make more sense to keep these as separate processes. The packaging step would be producing a dist/ folder which contains tree-shaked minified external dependencies as from node_modules, such that widget users not having node_modules can use them directly. The difficulty here is that:

• If using advancedChunks in rolldown (similar to manualChunks in rollup), the export signatures cannot be preserved so it would be hard to resolve back from source.
• If specifying the external dependencies as extra entry points, there will be no tree-shaking which leads to extremely large bundles.

Note

It is actually arguable whether tree-shaking should be performed or not. If no tree-shaking, the bundle size might be significantly larger, but users would be able to use whatever is in that external dependency, regardless of whether the widget has used some item or not when distributed. If tree-shaking, the bundle size would be kept minimal, but if users would not be able to use items that the original widget did not use, even if they exist in that external dependency (they would need node in this case).

The following assumes that we still wants tree-shaking. We will definitely ship react, the @emotion/react JSX runtime, and we will likely ship the whole @emotion/react package and @radix-ui/themes if they do not add too much size to the app. This means that we would likely have everything necessary for building UI (except for icons), and external dependencies will mostly be for functionality (e.g. date-fns). I may (unsafely) assume that widgets users, most of the times, would be just tweaking UI appearance rather than functionalities, and if the latter, it somehow makes sense that more efforts is needed because UI is cheaper than functionalities. Also with a sufficiently extensive and reasonably designed plugin system in the future (I hope), widgets will need to rely less and less on external dependencies.

A small benchmark that may or may not be correct: including the whole @radix-ui/themes increases around ~150 kB in size which does not seem too much.

BASELINE: 1098 kB | gzip: 213 kB
--------------------------------

vite v6.0.5 building for production...
✓ 365 modules transformed.
dist/views/canvas.html              0.57 kB │ gzip:   0.34 kB
dist/views/manager.html             0.65 kB │ gzip:   0.36 kB
dist/assets/manager-QtILqZa9.css    0.09 kB │ gzip:   0.09 kB
dist/assets/styles-D15CxL-s.css   690.56 kB │ gzip:  80.94 kB
dist/assets/canvas.js              27.29 kB │ gzip:   9.36 kB
dist/assets/manager.js             62.41 kB │ gzip:  20.66 kB
dist/assets/styles-CZZoRTKl.js    316.13 kB │ gzip: 100.33 kB
✓ built in 3.15s

REACT & JSX-RUNTIME FROM .SCRIPTS: 1099 kB | gzip: 214 kB
---------------------------------------------------------

vite v6.0.5 building for production...
✓ 367 modules transformed.
dist/views/canvas.html              0.72 kB │ gzip:  0.36 kB
dist/views/manager.html             0.80 kB │ gzip:  0.38 kB
dist/assets/manager-QtILqZa9.css    0.09 kB │ gzip:  0.09 kB
dist/assets/styles-D15CxL-s.css   690.56 kB │ gzip: 80.94 kB
dist/.scripts/react.js              9.86 kB │ gzip:  3.67 kB
dist/.scripts/jsx-runtime.js       18.94 kB │ gzip:  7.62 kB
dist/assets/canvas.js              27.35 kB │ gzip:  9.37 kB
dist/assets/manager.js             62.50 kB │ gzip: 20.70 kB
dist/assets/styles-DPJXZ3dX.js    288.09 kB │ gzip: 90.50 kB
✓ built in 2.79s

REACT & JSX-RUNTIME FROM NODE_MODULES: 1100 kB | gzip: 215 kB
-------------------------------------------------------------

vite v6.0.5 building for production...
✓ 369 modules transformed.
dist/views/canvas.html                                                             0.78 kB │ gzip:  0.40 kB
dist/views/manager.html                                                            0.85 kB │ gzip:  0.42 kB
dist/assets/manager-QtILqZa9.css                                                   0.09 kB │ gzip:  0.09 kB
dist/assets/styles-D15CxL-s.css                                                  690.56 kB │ gzip: 80.94 kB
dist/.scripts/react.js                                                             0.10 kB │ gzip:  0.11 kB
dist/.scripts/@emotion/react/jsx-runtime.js                                        1.74 kB │ gzip:  0.88 kB
dist/assets/index-DGRDuEbE.js                                                      8.32 kB │ gzip:  3.19 kB
dist/assets/emotion-use-insertion-effect-with-fallbacks.browser.esm-BS59VCQ9.js   18.21 kB │ gzip:  7.36 kB
dist/assets/canvas.js                                                             27.40 kB │ gzip:  9.41 kB
dist/assets/manager.js                                                            62.52 kB │ gzip: 20.72 kB
dist/assets/styles-1ZgUvcM9.js                                                   289.33 kB │ gzip: 91.04 kB
✓ built in 2.81s

REACT & JSX-RUNTIME & EMOTION FROM NODE_MODULES: 1104 kB | gzip: 217 kB
-----------------------------------------------------------------------

vite v6.0.5 building for production...
✓ 369 modules transformed.
dist/views/canvas.html                                                             0.94 kB │ gzip:  0.44 kB
dist/views/manager.html                                                            1.01 kB │ gzip:  0.45 kB
dist/assets/manager-QtILqZa9.css                                                   0.09 kB │ gzip:  0.09 kB
dist/assets/styles-D15CxL-s.css                                                  690.56 kB │ gzip: 80.94 kB
dist/.scripts/react.js                                                             0.10 kB │ gzip:  0.11 kB
dist/assets/jsx-runtime-CLpGMVip.js                                                0.73 kB │ gzip:  0.46 kB
dist/.scripts/@emotion/react/jsx-runtime.js                                        1.80 kB │ gzip:  0.90 kB
dist/.scripts/@emotion/react.js                                                    4.44 kB │ gzip:  1.96 kB
dist/assets/index-DGRDuEbE.js                                                      8.32 kB │ gzip:  3.19 kB
dist/assets/emotion-use-insertion-effect-with-fallbacks.browser.esm-CjZ5JCMW.js   17.65 kB │ gzip:  7.20 kB
dist/assets/canvas.js                                                             27.48 kB │ gzip:  9.45 kB
dist/assets/manager.js                                                            62.60 kB │ gzip: 20.77 kB
dist/assets/styles-BY8EqReU.js                                                   288.03 kB │ gzip: 90.45 kB
✓ built in 2.85s

REACT & JSX-RUNTIME & EMOTION & RADIX-THEMES FROM NODE_MODULES: 1246 kB | gzip: 255 kB
--------------------------------------------------------------------------------------

vite v6.0.5 building for production...
✓ 369 modules transformed.
dist/views/canvas.html                                                             1.09 kB │ gzip:  0.46 kB
dist/views/manager.html                                                            1.17 kB │ gzip:  0.47 kB
dist/assets/manager-QtILqZa9.css                                                   0.09 kB │ gzip:  0.09 kB
dist/assets/styles-D15CxL-s.css                                                  690.56 kB │ gzip: 80.94 kB
dist/.scripts/react.js                                                             0.10 kB │ gzip:  0.11 kB
dist/assets/index-f8Xf69WF.js                                                      0.48 kB │ gzip:  0.34 kB
dist/assets/jsx-runtime-CLpGMVip.js                                                0.73 kB │ gzip:  0.46 kB
dist/.scripts/@emotion/react/jsx-runtime.js                                        1.84 kB │ gzip:  0.91 kB
dist/.scripts/@radix-ui/themes.js                                                  1.97 kB │ gzip:  1.04 kB
dist/.scripts/@emotion/react.js                                                    4.49 kB │ gzip:  1.97 kB
dist/assets/index-DGRDuEbE.js                                                      8.32 kB │ gzip:  3.19 kB
dist/assets/emotion-use-insertion-effect-with-fallbacks.browser.esm-FMr8odN0.js   17.25 kB │ gzip:  7.02 kB
dist/assets/manager.js                                                            18.83 kB │ gzip:  6.80 kB
dist/assets/canvas.js                                                             27.53 kB │ gzip:  9.46 kB
dist/assets/styles-D_H_tn0E.js                                                   210.89 kB │ gzip: 65.81 kB
dist/assets/tooltip-CWYoolxw.js                                                  260.13 kB │ gzip: 74.96 kB
✓ built in 2.86s

Back to the topic, the goal is to: tree-shake node modules while also keeping the export signatures. One possible way is to turn the imports into dynamic imports. In particular:

import { a } from "a";
import b from "b";
import * as c from "c";

is equivalent to (hopefully):

const { a } = await import("a");
const { default: b } = await import("b");
const c = await import("c");

The difference is that without additional configurations the static imports will be directly resolved, while dynamic imports will result in individual dynamic entries, and it seems that treeshaking can happen with rolldown. So one possible workaround is to convert all external dependency imports into this dynamic form with some AST transform. This is possible because we can assume that all external dependencies (that lives in node_modules) are specified in the dependencies field in package.json so we are able to tell which to convert before actual resolution happens. Also this would be somehow similar to rolldown_plugin_glob_import, which also turns items into (some combinations of) dynamic imports, though from import.meta.glob patterns rather than from certain static imports. This is made possible with an OXC AST visitor and the transform_ast hook for rolldown plugins.

Another problem is that, the names of the dynamic output chunks need to be deterministic by some rules so that they can be correctly located by just looking at widget source code. The information available for each chunk is:

// crates/rolldown_common/src/types/rollup_pre_rendered_chunk.rs
pub struct RollupPreRenderedChunk {
  pub name: ArcStr,
  pub is_entry: bool,
  pub is_dynamic_entry: bool,
  pub facade_module_id: Option<ModuleId>,
  pub module_ids: Vec<ModuleId>,
  pub exports: Vec<Rstr>,
}

From facade_module_id.resource_id() we seem to be able to get the absolute path to the source of the resolved chunk. One possible solution is that, prior to actual bundling, construct a backreference resolution map, mapping resolved paths back to the external dependency names. Something like:

// Let `resolver` be a resolver that is created in the same way as rolldown internally creates
// based on bundler config; see crates/rolldown/src/bundler_builder.rs for reference
let external_deps = vec!["@radix-ui/themes".to_string(), /* ... */];
let mut backreferences = HashMap::new();
for dep in external_deps {
    let resolution = resolver.resolve(None, &dep, ImportKind::Import, false);
    backreferences.insert(resolution.unwrap().path, dep);
}

This would give something like:

{
    "D:\\Projects\\Deskulpt-rolldown\\node_modules\\.pnpm\\@radix-ui+themes@3.1.6_react-dom@19.0.0_react@19.0.0__react@19.0.0\\node_modules\\@radix-ui\\themes\\dist\\esm\\index.js": "@radix-ui/themes",
}

where the keys (hopefully) correspond to the aforementioned facade_module_id.resource_id(). Then for the bundler options, we may add something like this (note it is a draft and many details are not taken into account):

let bundler_options = BundlerOptions {
    // ...
    chunk_filenames: Some(ChunkFilenamesOutputOption::Fn(Arc::new(move |chunk| {
        let chunk = chunk.clone();
        let backreferences = backreferences.clone();
        Box::pin(async move {
            let facade_module_id = chunk.facade_module_id.unwrap();
            let target = facade_module_id.resource_id();
            let item = backreferences.get(target).unwrap();
            Ok(format!("chunks/{item}.js"))
        })
    }))),
    // ...
}

This way the chunks are organized by the original names of the external dependencies.

Widget Users

There shall be a switch between using node_modules or using the bundled external dependencies. Widget users shall choose the latter. All external dependencies, as specified by the dependencies field in package.json, should be aliased to the corresponding files under the packaged (bundled) directory. Since the chunks are properly named and the signatures are preserved, the resolution should succeed.