Using custom loaders results in a load loop / OOM

[merceyz]

Version

v20.0.0-nightly20230414c94be4125b

Platform

Linux server 5.15.0-69-generic #76-Ubuntu SMP Fri Mar 17 17:19:29 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Subsystem

No response

What steps will reproduce the bug?

touch loader.js
echo "setInterval(() => {}, 1000);" > test.cjs
node --loader ./loader.js test.cjs

How often does it reproduce? Is there a required condition?

No response

What is the expected behavior? Why is that the expected behavior?

Node.js starts up without loading the provided loader over and over again.

What do you see instead?

Node.js spams the console with the following warning while the memory and CPU usage climbs until it's stopped / OOMs.

(node:5652) ExperimentalWarning: Custom ESM Loaders is an experimental feature and might change at any time
(Use `node --trace-warnings ...` to show where the warning was created)

Additional information

Possibly relevant #44710

[aduh95]

@nodejs/loaders

[aduh95]

I'm able to reproduce with the suggested code snippet, but it looks like there's more to it than just combining --require and --loader, here's what I get when running the following at the root of the node repo:

$ out/Release/node --no-warnings --require ./test/fixtures/printA.js --import ./test/fixtures/printB.js --loader ./test/fixtures/empty.js ./test/fixtures/printC.js
A
A
B
C

It looks like the reported bug only happens on REPL. Worth noting that we also another bug that makes the --required script run twice instead of once.

[merceyz]

It looks like the reported bug only happens on REPL

I can also reproduce the issue without the REPL:

touch require.cjs
touch loader.mjs
echo "setInterval(() => {}, 1000);" > test.cjs
node --require ./require.cjs --loader ./loader.mjs test.cjs

[aduh95]

I don't think it has much to do with --require:

$ out/Release/node --loader ./test/fixtures/empty.js
[…never ending output of experimental warning…]

[merceyz]

Indeed, I've updated the issue to reflect that 👍

Worth noting that we also another bug that makes the --required script run twice instead of once

That's expected behaviour in my opinion, once in the main process and once in the Worker for the loaders.

[JakobJingleheimer]

I can't repro at all (admittedly on macOS ARM, but I don't see how this could be OS-related):

At main's current HEAD (e050cec), the OP's

snippet

touch loader.mjs
echo "setInterval(() => {}, 1000);" > test.cjs
node --loader ./loader.mjs test.cjs

runs endlessly (as expected) and memory stays constant (at 20.2 MB, after start-up), with only 1 experimental warning.

I get the same behaviour on v20.0.0-proposal's current HEAD (bb533f7), but with slightly less memory consumption (20.1 MB after start-up).

The code that controls whether the warning is emitted is quite simple and handles multiple instances of the module loader:

node/lib/internal/modules/esm/loader.js

Lines 405 to 420 in b342a1b

	let emittedExperimentalWarning = false;
	/**
	* A loader instance is used as the main entry point for loading ES modules. Currently, this is a singleton; there is
	* only one used for loading the main module and everything in its dependency graph, though separate instances of this
	* class might be instantiated as part of bootstrap for other purposes.
	* @param {boolean} useCustomLoadersIfPresent If the user has provided loaders via the --loader flag, use them.
	* @returns {DefaultModuleLoader | CustomizedModuleLoader}
	*/
	function createModuleLoader(useCustomLoadersIfPresent = true) {
	if (useCustomLoadersIfPresent) {
	const userLoaderPaths = getOptionValue('--experimental-loader');
	if (userLoaderPaths.length > 0) {
	if (!emittedExperimentalWarning) {
	emitExperimentalWarning('Custom ESM Loaders');
	emittedExperimentalWarning = true;
	}

(emittedExperimentalWarning lives at the top level of esm/loader.js)

Am I missing something?

[aduh95]

I think this is happening because the loader is going to try to spawn a worker thread from within the worker thread, which is also going to do the same thing, and that's how we end up with Mr Meeseeks situation until the system is out of memory or the process terminates. I'm working on a fix.