Bundle yarn.js with workspaces foreach support

[Darshan808]

Fixes #104

Description

Builds a custom yarn.js (v3.5.0) with @yarnpkg/plugin-workspace-tools baked in, so yarn workspaces foreach works out of the box in downstream projects without any extra plugin setup.

How yarn.js was generated

• Checked out the @yarnpkg/cli/3.5.0 branch from the Yarn Berry repo
• Added @yarnpkg/plugin-workspace-tools to packages/yarnpkg-cli/package.json (dependency + standard bundle profile)
• Built the CLI with yarn workspace @yarnpkg/cli build:cli --no-git-hash
• Replaced jupyter_builder/yarn.js with the generated artifact

Verification

Added a small step in CI in Downstream Tests to check whether yarn workspace foreach <some_command> works or not.

[Darshan808]

Before CI confirms that the yarn workspace foreach command isn't present currently.

[Darshan808]

And After CI confirms that the yarn workspace foreach command works on downstream projects now.

[Darshan808]

@bollwyvl On the @jupyter/jlpm extraction discussion, should that be an npm package or a Python package?

My understanding is that it should be an npm package, since @jupyter/ is already an npm scope and the actual payload (yarn.js) is JavaScript. In that model, a user could simply run:

npm install -g @jupyter/jlpm

and get the jlpm binary without needing Python at all.

What I'm less clear about is the integration with jupyter-builder. We want jlpm to be available automatically when someone installs jupyter-builder via:

pip install jupyter-builder

How would that work if jlpm exists purely as an npm package?

Would we still need a small Python wrapper/package that vendors or downloads yarn.js so that the jlpm CLI becomes available as part of the Python installation? In other words, would pip install jupyter-builder somehow need to bring in the npm package under the hood?

I'm a bit unsure about the packaging story here and how the pieces are expected to fit together if we publish yarn.js as a separate @jupyter/jlpm npm package.

Could you please provide a bit more context on the intended approach so I can look into it further ?

[bollwyvl]

Right in one approach; npm install @jupyter/jlpm would indeed get a well-structured node_modules/@jupyter/jlpm/dist/yarn.js. It could ship a node_modules/.bin/jlpm, and then could then be used for e.g. packageManager in package.json (unless that's a "blessed" list we don't want to mess with, because corepack is crazy). Such a jlpm would be usable from package.json#/scripts/, but would not necessarily be on $PATH itself, depending on which node is presently on $PATH, so might add to confusion.

jupyter-builder (the python package) would vendor this (and the package.json, license files, etc.), and continue to provide the equivalent, and more reliably findable-in-a-Jupyter environment, {sys.prefix}/bin/jlpm.

There might be a compelling reason for jlpm itself to be a wholly standalone PyPI package, such that jupyter-builder contained no .js, but having an even partially relaxed jupyter-builder -> jlpm dependency would introduce a possible "bad dimension of the matrix" situation which jupyter-builder is trying to avoid.

[Darshan808]

Should we track the "seperation of jlpm into a seperate package" in a new issue and merge this one?

[Copilot]

Pull request overview

This PR vendors a custom-built Yarn Berry yarn.js (v3.5.0) with @yarnpkg/plugin-workspace-tools included so downstream projects can use yarn workspaces foreach (via jlpm) without additional plugin configuration, and adds CI checks to validate the bundle and downstream behavior.

Changes:

• Add a new CI workflow that rebuilds Yarn Berry at @yarnpkg/cli/3.5.0, bakes in @yarnpkg/plugin-workspace-tools, smoke-tests workspaces foreach, and SHA-compares the rebuilt yarn.js against the vendored one.
• Add a downstream CI step intended to validate jlpm workspaces foreach in real downstream repos.
• Update downstream artifact upload action version.

Reviewed changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated 1 comment.

File	Description
.github/workflows/verify-yarn-bundle.yml	New workflow to reproducibly rebuild and hash-verify the vendored yarn.js and smoke-test workspaces foreach.
.github/workflows/downstream-tests.yml	Adds a downstream jlpm workspaces foreach step and updates artifact upload action usage.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Darshan808]

Actually it even fails now in jupyterlab (See CI Logs). We'll change this once we fix it upstream.