Error with Yarn2+ project: The following references do not actually refer to packages

[mawl]

Describe the bug

Follow-up to #10601:

Instead of creating the dependency graph ORT fails with The following references do not actually refer to packages: 'NPM::lodash:4.17.21'

To Reproduce

Can be reproduced with:

package.json

{
  "packageManager": "yarn@4.9.1+sha512.f95ce356460e05be48d66401c1ae64ef84d163dd689964962c6888a9810865e39097a5e9de748876c2e0bf89b232d583c33982773e9903ae7a76257270986538",
  "name": "yarn4-helloworld",
  "version": "1.0.0",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "description": "",
  "dependencies": {
    "lodash": "^4.17.21"
  }
}

yarn.lock

# This file is generated by running "yarn install" inside your project.
# Manual changes might be lost - proceed with caution!

__metadata:
  version: 8
  cacheKey: 10c0

"lodash@npm:^4.17.21":
  version: 4.17.21
  resolution: "lodash@npm:4.17.21"
  checksum: 10c0/d8cbea072bb08655bb4c989da418994b073a608dffa608b09ac04b43a791b12aeae7cd7ad919aa4c925f33b48490b5cfe6c1f71d827956071dae2e7bb3a6b74c
  languageName: node
  linkType: hard

"yarn4-helloworld@workspace:.":
  version: 0.0.0-use.local
  resolution: "yarn4-helloworld@workspace:."
  dependencies:
    lodash: "npm:^4.17.21"
  languageName: unknown
  linkType: soft

Expected behavior

org.ossreviewtoolkit.model.utils.DependencyGraphBuilder throws no exception.

Console / log output

08:50:02.294 [DefaultDispatcher-worker-3] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'yarn npm info --json lodash@4.17.21' in '/builds/foo/compliance/license-scanning/test/Yarn2/yarn4-helloworld'...
08:50:02.738 [DefaultDispatcher-worker-3] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - {"name":"lodash","dist-tags":{
...
08:50:02.738 [DefaultDispatcher-worker-3] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - 
08:50:02.752 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.plugins.packagemanagers.node.NodePackageManager - Parsing project info from '/builds/foo/compliance/license-scanning/test/Yarn2/yarn4-helloworld/package.json'.
08:50:02.913 [DefaultDispatcher-worker-1] INFO  org.ossreviewtoolkit.utils.ort.OrtProxySelector - Proxy selector was successfully installed.
08:50:02.916 [DefaultDispatcher-worker-1] INFO  org.ossreviewtoolkit.utils.ort.OrtAuthenticator - Authenticator was successfully installed.
08:50:03.013 [DefaultDispatcher-worker-1] DEBUG org.eclipse.jgit.internal.util.ShutdownHook - register org.eclipse.jgit.util.FS$FileStoreAttributes$$Lambda/0x00007f3f4848eba8@12667b21 with shutdown hook
08:50:03.022 [DefaultDispatcher-worker-1] DEBUG org.eclipse.jgit.util.SystemReader - loading config FileBasedConfig[/home/ort/.config/jgit/config]
08:50:03.025 [DefaultDispatcher-worker-1] DEBUG org.eclipse.jgit.util.FS - readpipe [/usr/bin/git, --version],/usr/bin
08:50:03.030 [DefaultDispatcher-worker-1] DEBUG org.eclipse.jgit.util.FS - readpipe may return 'git version 2.34.1'
08:50:03.030 [DefaultDispatcher-worker-1] DEBUG org.eclipse.jgit.util.FS - remaining output:
08:50:03.031 [DefaultDispatcher-worker-1] DEBUG org.eclipse.jgit.util.FS - readpipe [/usr/bin/git, config, --system, --show-origin, --list, -z],/usr/bin
08:50:03.034 [DefaultDispatcher-worker-1] DEBUG org.eclipse.jgit.util.FS - readpipe may return 'file:/etc/gitconfigfilter.lfs.clean'
08:50:03.034 [DefaultDispatcher-worker-1] DEBUG org.eclipse.jgit.util.FS - remaining output:
08:50:03.034 [DefaultDispatcher-worker-1] DEBUG org.eclipse.jgit.util.FS - git-lfs clean -- %ffile:/etc/gitconfigfilter.lfs.smudge
08:50:03.034 [DefaultDispatcher-worker-1] DEBUG org.eclipse.jgit.util.FS - git-lfs smudge -- %ffile:/etc/gitconfigfilter.lfs.process
08:50:03.034 [DefaultDispatcher-worker-1] DEBUG org.eclipse.jgit.util.FS - git-lfs filter-processfile:/etc/gitconfigfilter.lfs.required
08:50:03.037 [DefaultDispatcher-worker-1] DEBUG org.eclipse.jgit.util.FS - true
08:50:03.037 [DefaultDispatcher-worker-1] DEBUG org.eclipse.jgit.util.SystemReader - loading config FileBasedConfig[/etc/gitconfig]
08:50:03.042 [DefaultDispatcher-worker-1] DEBUG org.eclipse.jgit.util.SystemReader - loading config FileBasedConfig[/home/ort/.config/git/config]
08:50:03.043 [DefaultDispatcher-worker-1] DEBUG org.eclipse.jgit.util.SystemReader - loading config UserConfigFile[/home/ort/.gitconfig]
08:50:03.192 [DefaultDispatcher-worker-1] INFO  org.ossreviewtoolkit.analyzer.PackageManager - Yarn 2+ resolved dependencies for path 'test/Yarn2/yarn4-helloworld/package.json' in 2.169353260s.
Exception in thread "main" java.lang.IllegalArgumentException: The following references do not actually refer to packages: 'NPM::lodash:4.17.21'.
	at org.ossreviewtoolkit.model.utils.DependencyGraphBuilder.checkReferences(DependencyGraphBuilder.kt:204)
	at org.ossreviewtoolkit.model.utils.DependencyGraphBuilder.build(DependencyGraphBuilder.kt:177)
	at org.ossreviewtoolkit.model.utils.DependencyGraphBuilder.build$default(DependencyGraphBuilder.kt:176)
	at org.ossreviewtoolkit.plugins.packagemanagers.node.NodePackageManager.createPackageManagerResult(NodePackageManager.kt:94)
	at org.ossreviewtoolkit.analyzer.PackageManager.resolveDependencies(PackageManager.kt:331)
	at org.ossreviewtoolkit.analyzer.PackageManagerRunner$run$3.invokeSuspend(Analyzer.kt:357)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)
	at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:124)
	at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:89)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:586)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:820)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:717)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:704)
08:50:03.198 [ForkJoinPool-1-worker-1] DEBUG org.eclipse.jgit.internal.util.ShutdownHook - Cleanup org.eclipse.jgit.util.FS$FileStoreAttributes$$Lambda/0x00007f3f4848eba8@12667b21 during JVM shutdown

Environment

• ORT version: version 63.0.0-044.sha.

[sschuberth]

I wonder whether this is related to #8058. Can you please check that issue, @mawl?

[mawl]

I wonder whether this is related to #8058. Can you please check that issue, @mawl?

@sschuberth: I have compared the issue you mention to mine.

The difference is:

• we use Yarn v4.9.1 in our project and yarn gets installed via corepack from the version set in packageManager
• so we don't use yarnPath: https://yarnpkg.com/blog/release/4.0#installing-yarn

As I can see, there is no test for Yarn v4.9.1 with corepack in your test suite actually: https://github.com/oss-review-toolkit/ort/tree/d1d0a3d1f7033e418afb4fa001626d2cfd35eca6/plugins/package-managers/node/src/funTest/assets/projects/synthetic/yarn2

Should I add one to reproduce the error?

[mawl]

As I can say the used lodash dependency contains no transitive virtual package dependencies but the error still occurs with ORT v64.0.0 and corepack enabled.

[inderps]

I also face similar issue here

Exception in thread "main" java.lang.IllegalArgumentException: The following references do not actually refer to packages: 'NPM:@ampproject:remapping:2.3.0', 'NPM::acorn-jsx:5.3.2', 'NPM:@nodelib:fs.walk:1.2.8', 'NPM::autoprefixer:10.4.20', 'NPM::parse-json:5.2.0', 'NPM:@eslint:eslintrc:2.1.4', 'NPM:@eslint-community:eslint-utils:4.4.0'.
	at org.ossreviewtoolkit.model.utils.DependencyGraphBuilder.checkReferences(DependencyGraphBuilder.kt:204)
	at org.ossreviewtoolkit.model.utils.DependencyGraphBuilder.build(DependencyGraphBuilder.kt:177)
	at org.ossreviewtoolkit.model.utils.DependencyGraphBuilder.build$default(DependencyGraphBuilder.kt:176)
	at org.ossreviewtoolkit.plugins.packagemanagers.node.NodePackageManager.createPackageManagerResult(NodePackageManager.kt:94)
	at org.ossreviewtoolkit.analyzer.PackageManager.resolveDependencies(PackageManager.kt:331)
	at org.ossreviewtoolkit.analyzer.PackageManagerRunner$run$3.invokeSuspend(Analyzer.kt:357)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)
	at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:124)
	at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:89)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:586)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:820)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:717)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:704)

[sschuberth]

Now with #11208 merged into main (though not contained in a release yet), could the reporters of this issue please check whether they can still reproduce the issue?