Yarn2 package manager misses lots of dependencies

[oheger-bosch]

One of our customers is using Yarn classic and wants to migrate to Yarn 4.x. After changing the version of Yarn in package.json and switching in ORT from Yarn to Yarn2, the number of dependencies reported by the Analyzer went down from 579 to 56. In the dependency trees displayed by the WebApp reports generated for the different versions, the top level of dependencies seem to be identical, but in the case of Yarn2, many nodes here do not have further dependencies.

From my analysis of the issue, it seems that the Yarn2 implementation does not handle virtual nodes in the dependency graph correctly. I had a longer conversation with Gemini about this topic which I will attach below, since it gives a good explanation about what is happening and might help in finding a solution. (Note: I have no idea why the tool has to make me compliments for every question I ask.)

Conversation with AI:

Prompt:
I am using yarn2+. In the following command:

yarn info --all --recursive --manifest --virtuals --json

what is the meaning of --virtuals ?

Reply:
In Yarn 2 and later versions that use the Plug'n'Play (PnP) installation strategy, the --virtuals flag is used with the yarn info command to display information about "virtual packages".

Virtual Packages Explained

Under Yarn's PnP system, the traditional node_modules folder is replaced by a mapping system. When a package has peer dependencies, its own dependency set can change depending on where it is in the dependency tree. To handle this, Yarn creates "virtual packages."

These are not real copies of the package. Instead, they are unique instances that point to the same on-disk location but have different dependency sets encoded. Each virtual package is given a unique path on the filesystem (using a __virtual__ folder) to ensure that Node.js instantiates it with the correct dependencies for that specific context.

In short, the --virtuals flag provides a detailed view of how Yarn is managing these complex peer dependency scenarios by showing each unique instance of a package that has been created to satisfy different dependency requirements within your project.

Prompt:
The above yarn info command outputs a JSON structure that allows reconstructing the dependency graph of the project. In my concrete case, I have a number of packages for which a corresponding "virtual" package exists. These virtual packages do not have any dependencies, but the original packages do have some. Now, in the top-level project node, only virtual packages appear in dependencies. Therefore, the total dependency graph is rather "flat". How can this be explained?

Reply:
This is an excellent and insightful question that gets to the heart of how Yarn's Plug'n'Play (PnP) system works. The "flat" dependency graph you're observing is an artifact of how Yarn represents a complex resolution process. Let's break down why this happens.

The Role of Virtual Packages

First, it's key to understand why virtual packages exist. They are Yarn's solution to a classic and tricky problem: peer dependencies.

A package can have different dependencies depending on the project it's used in. For example, plugin-A might require framework as a peer dependency. If you use plugin-A in two different workspaces, one using framework@v1 and the other framework@v2, plugin-A needs to be instantiated twice, each time with a different version of framework.

To handle this, Yarn PnP creates virtual packages. These are not copies of the code. Instead, they are unique identifiers in the dependency graph that link the same package code to a specific, resolved set of its dependencies. Each unique combination of a package and its resolved peer dependencies gets its own virtual package instance.

Explaining the "Flat" Graph

Now, let's address your specific observations and why the graph appears flat in the yarn info JSON output.

1. Why Virtual Packages are Top-Level Dependencies

The yarn info command shows you the resolved dependency graph. From your project's perspective, its direct dependency isn't the abstract plugin-A; it's a very concrete instance of plugin-A that has been resolved with a specific version of framework. This concrete instance is the virtual package.

Therefore, the output correctly shows your top-level project depending on these virtual packages, as they are the actual, resolved nodes in the dependency tree.

2. Why Virtual Packages Have No Dependencies

This is the key to your question. The JSON object for a virtual package shows no dependencies because a virtual package does not have its own package.json manifest. It's a logical construct created by the Yarn resolver.

Your command uses the --manifest flag, which instructs Yarn to include information from the package's manifest (package.json). Here’s how the information is being presented:

• For "Real" Packages: The output shows the dependencies as they are declared in the dependencies, devDependencies, and peerDependencies fields of their package.json file.
• For "Virtual" Packages: These don't have a manifest. They are pointers. The dependency information isn't part of a JSON file but is encoded within Yarn's central resolution map (.pnp.cjs file). The yarn info command, when displaying the virtual package's node, doesn't attach the resolved dependency list to it in the JSON output, leading it to appear as if it has no dependencies of its own.

In essence, you are seeing a separation of two concepts:

• The Locator (...#virtual:...): A pointer to a specific instance of a package.
• The Manifest (package.json): The definition of a package's properties and its declared dependencies.

The virtual locator has no manifest, so its dependency list appears empty in the output. The real package has a manifest, and its declared dependencies are shown there.

How to Reconstruct the Full Graph

The dependency graph is not actually flat; its depth is simply represented in a different way. To see the full, deep graph, you need to connect the virtual packages to the dependencies of the real packages they represent.

The PnP API provides the tools to do this programmatically. For any given package locator (even a virtual one), you can use pnp.getPackageInformation(locator) to get its packageDependencies. This map contains the actual, resolved dependencies for that specific instance, linking to other locators (which could be virtual themselves).

By traversing these packageDependencies links from one locator to the next, you can reconstruct the entire, deeply-nested dependency graph. The yarn info command is a convenient tool for a high-level view, but its JSON output simplifies the representation by not explicitly linking virtual locators back to their resolved dependency sets.

Prompt:
Where is the .pnp.cjs file located and is its content in a readable form? So, could it be parsed and used to do a mapping from virtual packages to real packages?

Reply:
Of course. You're asking excellent questions about the mechanics of Yarn PnP. Let's get into the details of the .pnp.cjs file.

1. Location of .pnp.cjs

The .pnp.cjs file is located in the root directory of your project.

Yarn generates this single Node.js loader file instead of the traditional node_modules folder. You will typically commit this file directly to your version control system (like Git). Its presence is what signals to Yarn and Node.js how to resolve your project's dependencies.

2. Readability of the File

Yes, the .pnp.cjs file is human-readable, but it's not designed for manual editing. It is a machine-generated JavaScript file (a CommonJS module, hence the .cjs extension) that contains the complete dependency graph of your project.

If you open it, you'll find:

• A shebang line (#!/usr/bin/env node).
• JavaScript code that sets up the Plug'n'Play API.
• A large, embedded JSON-like object that holds all the information about your project's dependency tree. This includes package locations, dependency types (regular, peer), and the relationships between them.

While you can read it to debug or understand its structure, you should never modify it by hand. Any manual changes will be overwritten the next time yarn install is run.

3. Parsing and Mapping Virtual Packages

Yes, the file can be parsed, and it is precisely what you need to map virtual packages back to their "real" counterparts. However, you shouldn't parse it as a plain text file. The correct way to "parse" it is to load it as a module in a Node.js script to access the PnP API it exposes.

Here’s how it works and how you can use it:

• The PnP API: The .pnp.cjs file, when required, provides access to a PnP API that allows you to query the dependency graph programmatically.
• The Data Structure: The core of the mapping is within a data structure inside the .pnp.cjs file called packageRegistryData. This is a map that contains detailed information for every package in your project, including both real and virtual ones.
• Mapping Virtual to Real: A virtual package's entry in this structure will point to the same physical packageLocation (often a .zip file in your .yarn/cache) as the "real" package it's an instance of. The distinction is that the virtual package will have a unique locator and a specific, resolved set of packageDependencies that reflects the peer dependencies it satisfied in that part of the tree.

Example: Programmatic Usage

To explore this, you could create a script (e.g., inspect.js) in your project's root:

// inspect.js
const pnpApi = require('./.pnp.cjs');

// Get information about the top-level package
const topLevelPkg = pnpApi.getPackageInformation(pnpApi.topLevel);

console.log('Top-level package information:', topLevelPkg);

// To find a specific package (replace with a real package name from your project)
const aPackageLocator = { name: 'react', reference: 'npm:18.2.0' }; // Example locator
const packageInfo = pnpApi.getPackageInformation(aPackageLocator);

console.log('\nInformation for "react":', packageInfo);

// You can iterate through packageInfo.packageDependencies to see its resolved dependencies
// which might themselves be virtual locators.

By using the functions provided by the pnpApi (like getPackageInformation, resolveToUnqualified, etc.), you can traverse the entire dependency graph, correctly resolving virtual packages and understanding exactly which dependencies are being used in every context. This is the official and most reliable way to interact with the dependency information managed by Yarn PnP.

Prompt:
My project uses yarn 4.13.0. I have run the command "yarn install". This has created a node_modules folder, but I do not see a .pnp.cjs file. By which command is it created?

Reply:
This is a crucial detail! Your observation is correct and points to a significant difference in how Yarn (especially modern versions like 4.x) handles dependencies by default.

You're seeing a node_modules folder instead of a .pnp.cjs file because your project is likely configured to use the nodeLinker: node-modules setting, which is now the default for new Yarn projects created with Yarn 2+ (often referred to as Yarn Berry).

Here's the breakdown:

Why You're Not Seeing .pnp.cjs

1. Default Linker in Modern Yarn: While Yarn 2+ introduced Plug'n'Play (PnP) as a core feature and a highly efficient way to manage dependencies, it also recognized that node_modules is still the industry standard and sometimes necessary for compatibility. As such, the default linker for new projects in Yarn 2+ is node_modules, not pnp.
2. node_modules Linker: When nodeLinker: node-modules is active (either explicitly in your .yarnrc.yml or implicitly by default), Yarn will generate a node_modules directory structure, similar to npm. In this mode, it does not generate the .pnp.cjs file, because it's not using the PnP linker.
3. yarn install Creates node_modules: When you run yarn install with nodeLinker: node-modules configured, Yarn populates the node_modules directory with your dependencies.

How to Get .pnp.cjs (and Use PnP)

If you want to use the Plug'n'Play (PnP) linker and generate the .pnp.cjs file, you need to explicitly configure your project to do so.

1. Configure nodeLinker: pnp:
   You'll need to create or modify a .yarnrc.yml file in the root of your project and add the following line:

   nodeLinker: pnp

   If you don't have a .yarnrc.yml file, you can often generate one with the PnP configuration by running:

   yarn set version berry # Or yarn set version stable/latest
   yarn config set nodeLinker pnp

2. Run yarn install Again:
   After setting nodeLinker: pnp, run yarn install again:

   yarn install

   This command will then:

   • Delete the node_modules folder (if it exists).
   • Generate the .pnp.cjs file in your project root.
   • Generate a .yarn/cache folder containing the zipped packages.

Once these steps are completed, your project will be using the PnP linker, and you will find the .pnp.cjs file as described previously.

Prompt:
What you said above about virtual packages, does it also apply to the node-modules linker? So, this means for my current case, I have the JSON generated by yarn info, but I have no easy way to resolve the virtual packages, right?

Reply:
That is an excellent and very sharp question. You've hit on a subtle but critical point about how modern Yarn works.

The short answer is: Yes, the concept of virtual packages absolutely applies to the node-modules linker, and you are correct that you have no easy way to resolve them just from the yarn info JSON output alone.

Let's break down why.

1. The Resolver vs. The Linker

The key is to separate two different stages of what Yarn does:

1. The Resolver: This is the brain of Yarn. Its job is to figure out the complete dependency graph, respecting all version constraints and peer dependencies. This resolver is the same powerful engine whether you are using PnP or the node-modules linker. It is at this stage that Yarn identifies the need for virtual instances to satisfy different peer dependency requirements for the same package. The ...#virtual... identifier is the name it gives to these unique instances.

2. The Linker: This is the part that takes the resolved graph from the resolver and writes it to the filesystem so Node.js can find the packages.

   • PnP Linker (pnp): It writes the .pnp.cjs file, which is a direct map of the resolver's output. The virtual instances remain as logical pointers in this map.
   • node-modules Linker (node-modules): It physically creates a node_modules directory. To handle the virtual instances, it creates special directories on disk, often named __virtual__, that contain a package.json and symlinks to the actual package code. This folder structure tricks the Node.js require algorithm into loading the correct dependencies for that specific virtual instance.

2. Your Current Situation

You have correctly diagnosed the problem:

• The yarn info --json command gives you a dump of the Resolver's output. This output contains references to virtual packages (...#virtual...) because the resolver needed to create them.
• The virtual packages in this JSON output appear "flat" or without dependencies because their dependency information isn't stored in a manifest but is part of the resolver's internal state.
• Because you are using the node-modules linker, the crucial map that connects these virtual instances to their actual resolved dependencies was not written to the .pnp.cjs file. That information was used to generate the complex node_modules folder structure and then essentially discarded.

So, you are right. You have the yarn info JSON, which shows you that a virtual package is a dependency, but you do not have a simple, corresponding data structure like .pnp.cjs to look up what that virtual package's own resolved dependencies are.

How Can You Reconstruct the Graph?

Without the PnP API, reconstructing the full graph is much harder and less reliable. Your options are:

1. Infer from the Filesystem (Difficult): You could try to write a script that traverses the node_modules directory. When it finds a path containing /__virtual__/, it would have to parse the symlinks and the surrounding folder structure to try and figure out which packages are being used as the dependencies for that virtual instance. This is complex and brittle.

2. Switch to PnP for Analysis (Recommended): If your goal is purely to analyze the dependency graph, the most effective solution is to temporarily switch your project to the PnP linker.

   • Edit .yarnrc.yml to set nodeLinker: pnp.
   • Run yarn install. This will generate the .pnp.cjs file.
   • Now you can use the PnP API in a script (as described before) to programmatically and accurately traverse the entire, detailed dependency graph.
   • Once your analysis is complete, you can revert the change in .yarnrc.yml and run yarn install again to go back to node_modules.

In summary, your observation is spot-on. In node-modules mode, the yarn info command gives you a glimpse into the sophisticated resolution process, but the final "map" is expressed through the physical folder structure, not a queryable API file.

[myklosbotond]

I think this is the same issue as described under #8058.

I am not sure whether the PnP vs node_modules linker is relevant, or if yarn info --all --recursive --manifest --virtuals --json from Yarn2.kt#L204 is linker-agnostic.

[oheger-bosch]

@myklosbotond thanks for pointing me to this issue. Yes, this seems to be a duplicate then. Maybe, some of the information from the description is still helpful?

[sschuberth]

Maybe, some of the information from the description is still helpful?

Maybe, although the amount of (AI-generated) text there is quite overwhelming.

Anyway, let's close this as a duplicate of #8058 (and #11073), esp. as we already have #11208 in flight which is supposed to fix the issue.