javascript: node_modules should be immutable_input_digests

[tobni]

Is your feature request related to a problem? Please describe.
node_modules folder is currently copied into processes. The amount of IO is significant, with npm and yarn suffering the worst.

Describe the solution you'd like
Instead of merging installation digests with the sources to be acted upon, pass node_modules as immutable inputs.

Describe alternatives you've considered
?

Additional context

[tobni]

I seem to be at an impasse with this.

1. The node resolution algorithm uses the exact name node_modules to determine where to look for packages.
2. JS workspaces need to include their parent workspaces, and insert their own node_modules next to their package.json
3. There is no way (afaict) to inject a "named" immutable digest (i.e. a containing folder within the blarb filepath in tmp).
   This means resolution fails when the immutable digest is directly symlinked to node_modules in the sandbox (because dependencies expect to traverse to a node_modules folder).

15:15:16.33 [ERROR] Completed: Run javascript tests - testprojects/src/js/jest_coverage/src/test/index.test.js:tests - failed (exit code 1).
Partition: @pants/test-jest-coverage-project 

> @pants/test-jest-coverage-project@0.0.1 test
> jest "src/test/index.test.js"


node:internal/modules/cjs/loader:936
  throw err;
  ^

Error: Cannot find module 'import-local'
Require stack:
- /tmp/immutable_inputsDqJ5ap/.tmp5SO7SS/d47a588d6664fc49af9ac49da5957422dd9a0c8025de1dfe3e67e722582e007c/jest/bin/jest.js
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:933:15)
    at Function.Module._load (node:internal/modules/cjs/loader:778:27)
    at Module.require (node:internal/modules/cjs/loader:1005:19)
    at require (node:internal/modules/cjs/helpers:102:18)
    at Object.<anonymous> (/tmp/immutable_inputsDqJ5ap/.tmp5SO7SS/d47a588d6664fc49af9ac49da5957422dd9a0c8025de1dfe3e67e722582e007c/jest/bin/jest.js:9:21)
    at Module._compile (node:internal/modules/cjs/loader:1105:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1159:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/tmp/immutable_inputsDqJ5ap/.tmp5SO7SS/d47a588d6664fc49af9ac49da5957422dd9a0c8025de1dfe3e67e722582e007c/jest/bin/jest.js'
  ]
}

Note that the sandbox "looks" correct in this instance:

➜  pants git:(add/immutable-node-modules) ✗ ls /tmp/pants-sandbox-mVN3rM/testprojects/src/js/jest_coverage -l
total 248
lrwxrwxrwx 1 tobias tobias    103 May 19 14:34 node_modules -> /tmp/immutable_inputs6MxDEl/.tmpceZvoO/d47a588d6664fc49af9ac49da5957422dd9a0c8025de1dfe3e67e722582e007c
-rw-r--r-- 1 tobias tobias    202 May 19 14:34 package.json
-rw-r--r-- 1 tobias tobias 241247 May 19 14:34 package-lock.json
drwxrwxr-x 3 tobias tobias   4096 May 19 14:34 src

however in order for it to behave, d47a588d6664fc49af9ac49da5957422dd9a0c8025de1dfe3e67e722582e007c would have had to be named node_modules.

4. I'm also not allowed to merge immutable digests with mutable digest inputs, because the workspace folders are included in the digest entries (due to the mapping for immutable digests looking like this: {"src/js/jest_coverage": <node-modules-digest>}, I assume? This means I cannot make the immutable digest contain a node_modules folder, as the symlink from the above example cannot be constructed.
   Example of the error:

native_engine.IntrinsicError: Can only merge Directories with no duplicates, but found 2 duplicate entries in testprojects/src/js:

`jest_coverage`: 1.) symlink target=/tmp/immutable_inputsle4b64/.tmpjulx2K/de1ddc7bc46602486d0b5b72bfc3ca5fdbcf4ce6a8f2db534dd0c6884fadc99f:



`jest_coverage`: 2.) dir digest=46bfb66717285b201fc4fd0e5e1ee93ad57aac0c854ba9e51f3f27bb0e9287b2 size=258:

[tobni]

What I'd like to do is construct an immutable digest that allowed me to do this:
node_modules -> /tmp/immutable_inputs6MxDEl/.tmpceZvoO/d47a588d6664fc49af9ac49da5957422dd9a0c8025de1dfe3e67e722582e007c/node_modules.

[stuhood]

I've reopened #17282 to suggest one path to making some of this more automatic.

What I'd like to do is construct an immutable digest that allowed me to do this: node_modules -> /tmp/immutable_inputs6MxDEl/.tmpceZvoO/d47a588d6664fc49af9ac49da5957422dd9a0c8025de1dfe3e67e722582e007c/node_modules.

It would be hacky, but if you add a wrapper (bash) script, you could create some immutable inputs at some temporary path using that layout, and then adjust the structure to match what you needed before running?

I'm not sure how #17282 would interact with that though: it's somewhat implementation dependent, because it's more likely to be a "flattened" tree of symlinks, where any nested node_modules folder would itself be another symlink to another spot.

[tobni]

I'm not sure how #17282 would interact with that though: it's somewhat implementation dependent, because it's more likely to be a "flattened" tree of symlinks, where any nested node_modules folder would itself be another symlink to another spot.

Solving that problem w.r.t to the nodejs module resolution algorithm, I think it'd be close to re-inventing a wheel as far as JS and node_modules is concerned.
Maintaining a flat "node_modules" symlinked w/ deps and hardlinked to a cache folder in its entirety is the key feature of what the pnpm package manager tool does.

[benjyw]

Maybe I'm misunderstanding something, but isn't this exactly what named caches are for?

[tobni]

Nah, append_only_caches/named_caches arent applicable here I dont think? The tools do use caches as well for creating node_modules, but there`s multiple per repo, and they perform "clean" installations when package dependencies change which would wipe the node_modules -> not concurrency safe at all.

[tobni]

@stuhood I think maybe the hack can be done without a bash script: #19069

[sureshjoshi]

Not to derail, but for whatever solution is used here - what would be the plan with pnpm?

Rely on its global content store hardlinks outside of Pants? "shamefully-hoist" it into the Pants cache?

[tobni]

@sureshjoshi currently the pnpm global store is a named cache (so pants "own" the global store), and the tool gets to do whatever w.r.t hardlink/APFS clone/copy into node_modules/.pnpm of a sandbox.

[tobni]

Yea this wasn't easy. I've experimented by trying to construct mutliple SymlinkEntry that points to into the immutable digest that is created in InstalledNodePackage (All first party dependencies + node modules). Lets call it the install.

There's a lot of edge cases in these corners:

1. Cyclical package dependencies (ugh...)
2. Workspaces symlinking "inside the repo" makes the node_modules symlinks invalid when placed in install, so the entire workspace folder have to also be part of the immutable digest to preserve these.
3. Package managers are creating symlinks/node_modules under slightly different conditions in cases where packages depend on different third party versions, or they have no third party deps at all.

1 I have no idea how to solve right now. Maybe it isn't solvable, and falling back to how it currently is could work (since the cyclical dependency is detectable by pants).
2 & 3: These issues mean you have to do a lot of Digest finagling to not have the SymlinkEntry pointing into install accidentally collide with some other digest. E.g 2 causes collisions with transitive source file dependencies from other in-repo packages being attempted to be merged. This is a no-no when the containing directory of those first party packages are symlinks to install.

I think it is doable but will take me some time to get right (and readable). I also think this need to be something you can opt-out of.

Thankfully while testing this out I've found some bugs.

[tobni]

The operation is in essence

1. Find resolve for package:
2. Symlink install/resolve.root/node_modules -> resolve.root/node_modules.
3. For each workspace in resolve that package transitively depends on:
   Symlink install/resolve.root/workspace.root/ -> resolve.root/workspace.root. (workspace.root may contain node_modules.)
4. Symlink install/resolve.root/package.root/node_modules -> resolve.root/package.root/node_modules (This my contain symlinks to workspace.roots).

These SymlinkEntry needs to be merged with any other digests that are to be brought into the sandbox, but cannot collide with them.

install would also benifit from being partitioned per workspace (somehow), so that not unrelated package changes incur re-installation.