Re-organize build-dir by package + hash, rather than artifact type

[epage]

note: I specify build-dir to clarify which half of #14125 I'm referring to. The files and layout of build-dir does not have compatibility guarantees (source).

Currently, build-dir is laid out like

• target/
  • <target-platform>/?
    • <profile>/
      • incremental/
      • build/
        • <package>-<hash>/
          • build*script-*
      • deps/
        • <package>-<hash>*

Currently,

• cargo clean -p <package> will operate on everything for <package>

In the future, we could have

• GC will track and operate on everything for <package>-<hash> (cargo ./target fills with outdated artifacts as toolchains are updated/changed #5026)
• Change the locking so only overlapping <package>-<hash> that are being built block (More granular locking in cargo_rustc #4282)
• Centrally cache <package>-<hash> artifacts across all projects (Per-user compiled artifact cache #5931)

These could be aided by re-arranging the build-dir to be organized around <package>-<hash>, like

• target/
  • <target-platform>/?
    • <profile>/
      • incremental/
      • build/
        • <package>-<hash>/
          • build*script-*
          • *.d

Side effects

• We'll have to change how we invoke rustc which will increase the length of the command-line
  • Currently, we blindly point rustc at deps/ and rustc finds the files it needs. We'll instead need to point to each individual artifact rustc may need.

Open questions

• Transition plan: while build-dir isn't stable, enough tools rely on the layout that we'd want to setup a transition plan so they can have time to test against the new layout and work to support both
• What do we call the directory? I said build/ as its all encompassing
• Can the old build/ and deps/ content live in the same place?
• How should we handle incremental/?
• Can we share across <profile> at least?
  • <hash> is the -C extra-filename hash and doesn't encompass all of fingerprinting, so we'd need to audit if there are cases that don't change the hash that we'd stil need per-profile
  • Changing of local source is one example, so at least local packages still need to be scoped by profile

[ranger-ross]

These could be aided by re-arranging the build-dir to be organized around <package>-<hash>, like

• target/
  • <target-platform>/?
    • <profile>/
      • incremental/
      • build/
        • <package>-<hash>/
          • build*script-*
          • *.d

I am assuming that final binary would still be located at target/<target-platform>/<profile>/<bin-name> (ie. target/debug/foo).
Is that correct?

Transition plan: while build-dir isn't stable, enough tools rely on the layout that we'd want to setup a transition plan so they can have time to test against the new layout and work to support both

I think we should check how many of the most popular tools rely on the layout. Depending the impact we can adjust how aggressive we want to be with the migration.

Regarding strategies, I have 2 ideas.

1. Simply write the files to the new and old layout directories. This of course with the additional overheads of doubling the disk writes, storage and the complexity of cleaning up multiple directories. But this provides the best backwards compatibility story.
2. Only write the files to the new layout and create symlinks in the previous layout. This would help mitigate the overhead of the disk writes/storage.

What do we call the directory? I said build/ as its all encompassing

I do not have a strong opinion on this. I was thinking perhaps packages or crates since its a list of the built packages. But I think build also makes sense here too 😄 Maybe build might be better to leave the door open for future possibilities of adding other artifacts to this directory?

[epage]

I am assuming that final binary would still be located at target/// (ie. target/debug/foo).
Is that correct?

This does not touch final artifacts. I'd recommend reading up on the following note

note: I specify build-dir to clarify which half of #14125 I'm referring to.

I think we should check how many of the most popular tools rely on the layout. Depending the impact we can adjust how aggressive we want to be with the migration.

I know there is at least

• https://crates.io/crates/cargo-cache
• https://crates.io/crates/cargo-sweep

There might be some other tools that do weirder stuff, like inspecting debug files or rlibs.

Regarding strategies, I have 2 ideas.

Writing to both or symlinks won't work for the above two tools.

A common approach we take is to have a feature be opt-in and then transition it to opt-out. A question in this is if we'd want to still support the old layout, for which we'd do this through a config, or if we'll only support the new layout, for which we use an env variable and after a sufficient time we remove the opt-out.

[ranger-ross]

Okay, I finally had some time to read up on #14125 some other related threads.

Would it would be better to focus on making progress on separating target-build-dir/target-artifact-dir out as proposed here before attempting re-organize the layout? Doing it all at once would lead to less fragmentation of "build layouts". But adding to the scope would be more work and make it harder to land #14125.

I am leaning towards doing this re-organization after separating the build/artifact dirs.

[epage]

I've wondered about doing the build-dir change first, like you said. It would make the scope of the change clear and it would help to communicate out what has compatibility guarantees.