Build-std: Add builtin dependencies by adamgemmell · Pull Request #16675 · rust-lang/cargo

adamgemmell · 2026-02-25T17:40:41Z

What does this PR try to resolve?

This PR internally introduces:

The builtin source, a way for Cargo to discover builtin packages
Opaque dependencies, the idea that a dependency shouldn't be handled by the main resolve but rather a separate resolve in order to allow things like multiple packages in the same resolve with differing minor versions. The concept has been proposed as a first-class cargo feature, but is only introduced partially here.
(Implicit) builtin dependencies, which are dependencies on builtin packages injected in order to allow packages to migrate to explicit builtin dependencies in the future.

It moves the -Zbuild-std implementation to use them as part of rust-lang/rfcs#3875. While not strictly required for build-std=always which is our immediate goal, I wanted to ensure the implementation moving forward had explicit dependencies in mind.

The main behaviour change in this PR is that host dependencies like build-scripts and proc_macros no longer use build-std. If this is desired we can re-introduce the behaviour in an unstable manner, but it never worked for -Zbuild-std in cross-compile mode.

This PR implements this as a whole by:

Registering the builtin source (lazily, when a dependency is discovered), which discovers packages with a RecursivePathSource.
Creating a default set of opaque implicit builtin dependencies the set of roots defined in std_crates, and injects them to every eligible crate during the resolve.
The builtin source, when queried for a package satisfying one of these dependencies, returns a builtin Summary which has been modified compared to the one discovered by RecursivePathSource. This package has no dependencies, because the dependency queried is opaque. This allows the resolve to succeed as normal without modifying the resolver itself. The intended behaviour, though yet untested, is to make use of all the patching/replacement logic already in the registry query.
Instead of the current -Zbuild-std unit generation step which attaches UnitDeps to each root of the std resolve, the main unit generation loop replaces any builtin Summarys with roots from the separate std resolve. Dependencies of the roots are then added afterwards.

This PR has a very large surface area and will inevitably change some behaviour of -Zbuild-std. In particular we need to be careful about where we expose builtin SourceIds and package specs. The intent here is to avoid disruption as build-std is a very commonly used unstable feature, but some is to be expected and will be fixed in future PRs according to the agreed RFC. The non-build-std route is well tested already and I do not expect to find regressions there as the majority of this code is gated on -Zbuild-std.

How to test and review this PR?

The PR is best read commit-by-commit - all commits are atomic and try to introduce features piecemeal where possible, though somewhat inevitably there's a big "waterfall" enabling commit in the middle. A commit 5bfab35 early in the branch adds a test showing that the unit-graph does not change compared to the old -Zbuild-std behaviour.

The behaviour can be manually tested locally by providing -Zbuild-std as a flag to any cargo command.

davidtwco · 2026-02-26T17:38:40Z

We aren't expecting that this will land prior to those RFCs being approved (unless Cargo want it to), the intent with this is just to get some feedback on the general direction for the implementation, we're happy to adjust as much or as little as required :) @adamgemmell will also be away for a couple of weeks starting next week, so we might not respond to feedback immediately.

epage · 2026-02-26T18:00:05Z

    /// A directory-based registry.
    Directory,
+    /// Package sources distributed with the rust toolchain
+    Builtin,


https://github.com/rust-lang/cargo/blob/master/crates/cargo-util-schemas/src/core/package_id_spec.rs is at least one other place that would need updating

This will impact the unique identifier for the packages from this source in cargo's json output when compiling, cargo metadata, cargo <cmd> -p, etc

I'll modify there too, and add a note to check the stdout in various use cases. The RFCs often make notes on what the output of various commands will be. Note that builtin doesn't actually appear in Units - they're all Path dependencies by that point.

An interesting point on cargo metadata is that we decided that we have an unresolved question regarding if deps of builtins should be shown on output, which will be a little hard here as they're not attached until unit generation.

I've opted to implement pkg spec input/output in a later PR. I've added tests for common output commands like metadata/tree. json output (from build --message-format=json) isn't impacted in this PR - builtins do not exist in the unit graph (see this test which isn't changed in the rest of the branch c1f8964). I plan to address that in a future PR but I'm not sure on the best way to do that at this stage.

epage · 2026-02-26T18:01:56Z

+                    .url
+                    .to_file_path()
+                    .expect("builtin sources should not be remote");
+                Ok(Box::new(PathSource::new(&path, self, gctx)))


Will using a PathSsource directly like this work?

This particularly could have interesting design questions

Moved this to a builtin source

A builtin source now wraps a RecursivePathSource, and there's only one source for all builtin packages. see 4832d04

epage · 2026-02-26T18:10:41Z

+        None
+    } else {
+        Some(builtins.iter())
+    };


I assume this is for implicit builtins. Is there a reason you chose to do this here?

Source replacements happen in dep_cache (see query) so it seemed simpler just to put this here for now. After digging a little more, it might just be the deprecated [replace] section though, in which case I can probably lift this out of the cache to a more appropriate place

This is in build_deps() now (see 1588221), which is a bit more appropriate. It could be moved up to the resolver activate function, but that's already massive, and moving it to PackageRegistry::query would involve modifying Summarys which are intended to be immutable.

epage · 2026-02-26T18:16:02Z

+        }
+    }
+
+    pub fn new_injected_builtin(name: InternedString) -> Dependency {


what do you see as the role of this compared to the other news?

The difference is that opaque is true (making a SourceId was also more complicated in a previous iteration of this patch). public should also be true now that I look closer.

Reworked, see 1588221#diff-3f0dca17873e498a977bdb06abeb78a4f4135fbbf8abf6323332fd0f8a3da1bd

epage · 2026-02-26T18:18:28Z

                )
            })?;

+        if dep.is_opaque() {


I'd like to find a way to ask the source for the opaque variant of the summary. The tricky thing will be that we need to work with both variants.

This pattern works well, see the new Summary::to_opaque_builtin_summary()

Reworked again to make this more like the other constructors: 4832d04

epage · 2026-02-26T18:20:20Z

+        // Injecting builtins earlier (somewhere with access to RustcTargetData) is needed instead of this
+        let home = std::env::var("HOME").expect("HOME is set");
+        let path = format!(
+            "file://{home}/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/"


The URL in source ids is public facing. I think we'll need something more generic and then a new Source

I definitely tried something more generic but found that it had to be valid at various points. Will experiment more with this.

I have hidden the URL in pkgids as per https://rust-lang.github.io/rfcs/3875-build-std-explicit-dependencies.html#cargo-subcommands:~:text=builtin%3A%2F%2F%2E%23core, but it may still leak out in various places. I plan to address this in future PRs that test individual subcommands' behaviour.

epage · 2026-02-26T18:21:35Z

            has_dev_units,
            crate::core::resolver::features::ForceAllTargets::No,
            dry_run,
+            true,


not really a fan of bool constants being used in parameter lists. Makes it a lot harder to figure what what is going on here

I've moved this specific bool, but there's probably scope to improve the passing around of the builtins path and dependencies set

I added this information to ResolveOpts, see 707ac6b

epage · 2026-02-26T18:22:06Z

    keep_previous: Option<Keep<'_>>,
    specs: &[PackageIdSpec],
    register_patches: bool,
+    inject_builtins: bool,


Why do we need to tunnel this through? Not thrilled with this

This bool or the required logic behind it could probably be packaged into the workspace or it's gctx

I added this information to ResolveOpts, see 707ac6b

adamgemmell

Thank you for leaving feedback. My latest set of changes does a bit of tidying and works towards getting the tests passing, but doesn't address these comments yet or otherwise make progress towards removing hacks.

View changes since this review

adamgemmell · 2026-03-18T17:20:22Z

+        None
+    } else {
+        Some(builtins.iter())
+    };


Source replacements happen in dep_cache (see query) so it seemed simpler just to put this here for now. After digging a little more, it might just be the deprecated [replace] section though, in which case I can probably lift this out of the cache to a more appropriate place

adamgemmell · 2026-03-18T17:24:10Z

+        }
+    }
+
+    pub fn new_injected_builtin(name: InternedString) -> Dependency {


The difference is that opaque is true (making a SourceId was also more complicated in a previous iteration of this patch). public should also be true now that I look closer.

adamgemmell · 2026-03-18T17:32:31Z

+        // Injecting builtins earlier (somewhere with access to RustcTargetData) is needed instead of this
+        let home = std::env::var("HOME").expect("HOME is set");
+        let path = format!(
+            "file://{home}/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/"


I definitely tried something more generic but found that it had to be valid at various points. Will experiment more with this.

adamgemmell · 2026-03-19T10:40:20Z

    keep_previous: Option<Keep<'_>>,
    specs: &[PackageIdSpec],
    register_patches: bool,
+    inject_builtins: bool,


This bool or the required logic behind it could probably be packaged into the workspace or it's gctx

rustbot · 2026-08-10T15:50:49Z

This PR was rebased onto a different master commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

This commit is currently an unused API and will be tested in later commits

generation for them The old unit generation used to insert dependencies to every package on the set of roots std was resolved with. The new unit generation replaces dependencies with the relevant root of std, or ignores them if the dependency is not needed. This commit enables tests for the previous set of commits.

This test's assertion was completely broken because of the leading spaces and use of `...` which doesn't work with stderr_does_not_contain. See the comments in this test for additional details of shared dependencies on why the test cannot be fixed right now. It is feasibly possible for Cargo to share the user dep_test dependency and the build-std dep_test, but this is complex as written as dep_test gains an implicit std dependency while the mock-std depends on it. It may be easier to share them with explicit builtin dependencies, but the situation is complex due to -Zforce-unstable-if-unmarked and prone to breakage due to RUSTC_BOOTSTRAP=1.

The ideal behaviour is probably to emit all packages, but currently only the roots of std are emitted. A future PR will change this.

adamgemmell · 2026-08-10T17:12:46Z

@rustbot ready

epage reviewed Feb 26, 2026

View reviewed changes

Comment thread src/cargo/core/source_id.rs Outdated

epage reviewed Feb 26, 2026

View reviewed changes

biscuitrescue mentioned this pull request Mar 3, 2026

fix: ignore implicit std dependencies in unused-crate-dependencies lint #16677

Merged

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-author Status: The marked PR is awaiting some action (such as code changes) from the PR author. label Mar 6, 2026

adamgemmell force-pushed the dev/adagem01/opaque branch from 16dc1ab to ab7d8b0 Compare March 19, 2026 14:20

adamgemmell commented Mar 19, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

davidtwco mentioned this pull request Apr 14, 2026

build-std rust-lang/goals#274

Open

10 tasks

adamgemmell force-pushed the dev/adagem01/opaque branch from ab7d8b0 to 8607d8c Compare May 7, 2026 15:58

rustbot added A-directory-source Area: directory sources (vendoring) A-workspaces Area: workspaces labels May 7, 2026

adamgemmell added 3 commits August 10, 2026 17:18

typo: Fix previously added typo

78e7d26

test(unit-graph): Capture existing behaviour in unit-graph test

5bfab35

test(resolver): Model builtin packages

3b87c7c

adamgemmell force-pushed the dev/adagem01/opaque branch from 9766054 to ee827d1 Compare August 10, 2026 16:19

rustbot added the A-cfg-expr Area: Platform cfg expressions label Aug 10, 2026

adamgemmell added 20 commits August 10, 2026 17:29

feat(source): Allow specifying builtin sources

1bee864

feat(resolver): Allow creating builtin dependencies

e7473c1

test(resolver): Add negative resolver tests

c78fd63

feat(sources): generate opaque summaries to satisfy builtin dependencies

c7483da

This commit is currently an unused API and will be tested in later commits

feat(resolver): Add directive to enable injecting builtins

038ad3a

This commit is currently an unused API and will be tested in later commits

test(resolver): Test for implicit builtin deps

703e41d

feat(resolver): Inject builtin dependencies

984988b

test(resolve): Do not encode builtin packages in the lockfile

f8cc6c0

test(package-diff): Do not emit builtin packages in status messages

6e57738

test(unit-generation): no-std build-std

8b48454

test(build-std): builtins as roots

13f0c62

feat(unit generation): Replace builtin roots with std roots

c640bb9

test(metadata): Do not emit builtins with cargo metadata

a48a9d0

The ideal behaviour is probably to emit all packages, but currently only the roots of std are emitted. A future PR will change this.

fix(metadata): Filter out builtin packages

a288440

test(build-std: Do not emit builtins with cargo tree

420548d

fix(tree): Don't emit builtins

d1027b3

test(vendor): builtins shouldn't be vendored

97a8ed0

fix(vendor): Don't vendor builtins

0b54da3

adamgemmell force-pushed the dev/adagem01/opaque branch from ee827d1 to 0b54da3 Compare August 10, 2026 16:29

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: The marked PR is awaiting some action (such as code changes) from the PR author. labels Aug 10, 2026

Uh oh!

Uh oh!

Conversation

adamgemmell commented Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR try to resolve?

How to test and review this PR?

Uh oh!

davidtwco commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adamgemmell Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adamgemmell Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adamgemmell May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adamgemmell Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adamgemmell Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adamgemmell Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adamgemmell Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adamgemmell Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adamgemmell commented Feb 25, 2026 •

edited

Loading

davidtwco commented Feb 26, 2026 •

edited

Loading

adamgemmell Aug 7, 2026 •

edited

Loading

adamgemmell Aug 7, 2026 •

edited

Loading

adamgemmell May 7, 2026 •

edited

Loading

adamgemmell Aug 7, 2026 •

edited

Loading

adamgemmell Aug 7, 2026 •

edited

Loading

adamgemmell Aug 7, 2026 •

edited

Loading

adamgemmell Aug 7, 2026 •

edited

Loading

adamgemmell Aug 7, 2026 •

edited

Loading

adamgemmell left a comment •

edited by rustbot

Loading