Improve DX of importing ESM-compiled-to-CJS from native ESM

[nicolo-ribaudo]

What is the problem this feature will solve?

The current ESM-CJS interop has two problems when it comes to files authored as ESM and then compiled to CJS

Problem 1

Due to how ESM works, Node.js needs to statically detect the list of bindings exported form CJS modules. This static analysis is performed by cjs-module-lexer, which recognizes patterns produced by popular tools at the time that this helper library was authored.

This has two main limitations:

• build tools cannot change their generated code, that is usually not considered part of the public API, because it would make Node.js not recognize exports anymore. Example: Use old namespace reexport code pattern for better Node named export sniffing rollup/rollup#4826
• Node.js still does not recognize every pattern generated by popular tools, see for example Unable to detect exports from bundled webpack output cjs-module-lexer#88

In addition to these limitations, cjs-module-lexer has to perform a full lexing pass on the imported CJS file, which has a non-zero cost.

Problem 2

The default export exposed by Node.js for CJS files is always module.exports. This was a good choice at the beginning, to give a way of importing CJS from ESM, but now that Node.js detects named exports it's increasing friction when using the two modules systems together.

Consider this library, authored as ESM and published as CJS:

// as ESM
export default function circ(radius) {
  return radius * PI;
}
export const PI = 3.14;

// compiled to CJS and published
exports.__esModule = true;
exports.default = function circ(radius) {
  return radius * PI;
};
const PI = exports.PI = 3.14;

When importing this library from ESM, the named exports PI "just works":

import { PI } from "lib";

console.log(PI);

However, to use the default export you need to first import the library and then, separately, grab the default export from it:

import _lib from "lib";
const circ = _lib.default;

console.log(circ);

Developers sometimes try to rewrite it to use the "destructuring import" syntax with default as if it was a named import, but it obviously still points to module.exports:

import { default as circ } from "lib";

Unfortunately Node.js cannot change it's behavior and use exports.default as the default export (when exports.__esModule is defined to signal that the CJS file was originally an ES module, as every single other tool does) for backwards compatibility.

What is the feature you are proposing to solve the problem?

Node.js should have some sort of comment/directive at the beginning of the file to let build tools communicate what is the list of exports that a file should expose.

For example, the library example above could be compiled by tools to the following:

"exports:default,PI";

exports.__esModule = true;
exports.default = function circ(radius) {
  return radius * PI;
};
const PI = exports.PI = 3.14;

Then Node.js would know that when imported as ESM this file should have a default export (pointing to exports.default) and a PI export (pointing to exports.PI). Tools would be free to change their output code, and the "exports:..." directive would give an unambiguous and easy-to-parse signal to Node.js about the original intention of the code author.

From a tool author perspective, I have a few opinion about how this should work more in details:

• "exports:foo,bar"; should cause the module to only have foo and bar exports, and not a default export pointing to module.exports. This is so that adding a default export to the module will not be a breaking change.
• the list of exports need a separator (I'm using , just because it looks nice), but , could also appear in an export name. For this reason, all \s and ,s need to be escaped:

  // input
  const foo = 3;
  export { foo as "abc \\ , def", foo };
  
  // CJS output
  "exports:abc \\\\ \\, def,foo"
  const foo = exports["abc \\ ,"] = exports.foo = 3;

• it would be great if there was still a way to say "expose module.exports as the default export", so that tools could start emitting the new directive without breaking changes (and then they would stop enabling that flag when the compiled library is ready for a major release). For example:

  "exports:abc,def"
  // means
  export const abc = exports.abc;
  export const def = exports.def;

  "exports!:abc,def"
  // means
  export const abc = exports.abc;
  export const def = exports.def;
  export default module.exports;

Additionally, I noticed that cjs-module-exports also returns a list of modules re-exported with export * from. This is necessary for Node.js to get the list of re-exported bindings. We would need to have a separate directive for that. For example:

export const foo = 1, bar = 2;
export * from "./dep.js";
export * from "mod";

// becomes

"exports:foo,bar";
"exports*:./dep.js,mod";

What alternatives have you considered?

Instead of a directive this could be a comment, like for linking source maps. However, it is important that this comment/directive must be at the beginning of the file, so that Node.js doesn't need to scan/parse the whole file to find it.

Somebody was already thinking about this in the past (I vaguely remember a discussion about it in a Babel issue with a Node.js collaborator), but I cannot find any references to it.

---

Some people that might be interested in the discussion:
@guybedford, @lukastaegert (Rollup), @kdy1 (SWC), @TheLarkInn (Webpack), @patak-dev (Vite), @evanw (esbuild), @andrewbranch (TypeScript) me (Babel)

[dnalborczyk]

Somebody was already thinking about this in the past (I vaguely remember a discussion about it in a Babel issue with a Node.js collaborator), but I cannot find any references to it.

@nicolo-ribaudo is this possibly what you are looking for?

babel/babel#7294
babel/babel#6455

[lukastaegert]

I really like this.

If we want to make it a directive, though, it should probably start with "use ..." and be limited to the first lines in a module, similar to other directives like "use strict" or "use asm" (acorn has special handling for directives as long as they are only separated by white-space and comments from the module start).

[GeoffreyBooth]

@nodejs/loaders

[ljharb]

This is interesting - i assume i could also use it in normal CJS to ensure that the ESM-exposed exports are what i intend?

[nicolo-ribaudo]

@dnalborczyk Yes, thanks for the links! It looks like one of the main problems was that reading the pragma was slower than just doing export default module.exports;. We are not in the opposite situation, where Node.js is by default doing much more work than just "reading a pragma" for CJS imports :)

---

@lukastaegert

and be limited to the first lines in a module [...] (acorn has special handling for directives as long as they are only separated by white-space and comments from the module start)

Yes, that's exactly the definition of a directive :) I'm not attached to the specific wording, but I proposed it without use just because exports: is probably already unambiguous enough.

---

@ljharb Yes, obviously there would be no way for Node.js to distinguish between a hand-written file and a tool-generated file. However, I would expect it to be much easier to have this directive auto-generated by tools than having to manually keep it in sync with your exports.

[Jarred-Sumner]

I don’t have an opinion on the specifics of how this should work but I like the idea of an explicit way for build tools to specify this behavior. Bun uses __esModule and this is good and bad. It’s bad for node compatibility. It’s good for “I want this code to work”.

[aduh95]

Quoting cjs-module-lexer README:

Detection patterns for this project are frozen. This is because adding any new export detection patterns would result in fragmented backwards-compatibility. Specifically, it would be very difficult to figure out why an ES module named export for CommonJS might work in newer Node.js versions but not older versions. This problem would only be discovered downstream of module authors, with the fix for module authors being to then have to understand which patterns in this project provide full backwards-compatibily. Rather, by fully freezing the detected patterns, if it works in any Node.js version it will work in any other. Build tools can also reliably treat the supported syntax for this project as a part of their output target for ensuring syntax support.

I think this concerns are still valid, if we made a change on how we parse CJS exports, it could land on Node.js v18.20.0 and not any further, which would certainly cause confusion in the ecosystem. I'm not saying this is an absolute show stopper, but we need to address this problem and come up with a plan to avoid ecosystem fragmentation.

[nicolo-ribaudo]

@aduh95 The main problem with changing the detection patterns in cjs-module-lexer is that they are very subtle changes that affect code that is expected to behave identically.

For example, you might have this code that works both in Node.js 16 and Node.js 18:

Object.defineProperty(exports, "foo", {
  get() { return 2 }
});

let's say you decide to rewrite it to

Object.defineProperty(exports, "foo", {
  get: () => 2
});

and you are lucky because Node.js 18 updated their detection patterns to also detect arrow functions in getters: it keeps working, and given that the old and new code are expected to have the same behavior there is nothing that tells you "hey, you are using a new feature that doesn't work in older versions, be careful".

This new marker would be explicit, and not based on which patterns you follow when writing new code, and thus it's discoverability (and incompatibility risk) is similar to any other new feature that does not throw an error when used in older Node.js versions. Think, for example, about introducing new event names, new parameters to existing functions, or new properties to option bags.

In addition, a directive that can explicitly describe the expected shape of the module would never need to be updated to detect different patterns (unless ESM gets new features, in which case the directive can be updated at the same time as when introducing those new features in Node.js). Thus, there won't be need for a "this detection is frozen even though it doesn't support X" anymore just because there won't be a need to change it anymore.

[guybedford]

Note that es-build already does this hinting itself, but it hints to cjs-module-lexer directly.

We even have a test for this pattern in cjs-module-lexer here - https://github.com/nodejs/cjs-module-lexer/blob/main/test/_unit.js#L19:

0 && (module.exports = {a, b, c}) && __exportStar(require('fs'));

Effectively this provides an existing way to achieve the same result as what is asked for in this PR.

I've experimented with string systems a lot in the past for metadata, and I'm all for it fwiw.

[GeoffreyBooth]

Effectively this provides an existing way to achieve the same result as what is asked for in this PR.

With one big benefit, in that it works in maintained all versions of Node (and probably several EOL ones, as far back as whenever this was added). Though there’s no reason not to include both the new directive and the 0 && style. That’s what I would expect most bundlers would do, to target as many old versions of Node as possible.

The proposed directive has two relative benefits that I can see:

1. It’s more human-readable than the 0 && approach, and feels more intentional/standardized as opposed to looking hacky.

2. The proposal instructs Node to return early as soon as the directive is parsed, which should yield performance benefits for large files.

With regard to the second point, about the relative performance improvements:

• We should measure to see just how much gain we’re talking about here. This could be approximated by creating a set of test files where the 0 && version is in the first line, and measure parsing time in current Node and in a version of Node where cjs-module-lexer stops after the first line.

• The “return early” aspect introduces potential incorrectness or inconsistencies. For example, what if the directive lists one set of exports but either a 0 && line or the actual source of the file provide a different set of exports? If we stop parsing as soon as the directive is found, we might end up with exports that don’t actually exist in the file if the directive was written incorrectly (or maliciously). This also introduces the possibility of different exports in different versions of Node.

Does the proposed directive allow defining a module as having no default export? That would introduce the possibility of a user importing the default export in current Node, and then after a Node upgrade their code stops working because the new directive is read instead and it says that a particular module has no default export. This also goes against the error message you get when you try to import a named export that cjs-module-lexer didn’t find, where we provide example code that shows importing the default export and destructuring off of it.

If Node needs to parse the entire file to ensure correctness and consistency, then there wouldn’t be any performance benefit and what remains is the human-readability improvement. But if bundlers output both this new directive and the 0 && one anyway, then Node doesn’t really need to parse the new directive because it already supports the 0 && version; and bundlers are free to output an unused directive if they choose, like a comment or metadata that might be useful to other tools.