trim-paths: Remap rules for different dependency kinds

[weihanglo]

^{This is part of #12137}

As of 6982b44, the remap of -Ztrim-paths is relatively simple. It follows what is described in RFC 3127:

For the the current package (where the current working directory is in), from the the absolute path of the package root to empty string. For other packages, from the absolute path of the package root to [package name]-[package version].

In Cargo, that is:

cargo/src/cargo/core/compiler/mod.rs

Lines 1222 to 1230 in 6982b44

	// Remapped to path relative to workspace root:
	//
	// * path dependencies under workspace root directory
	//
	// Remapped to `<pkg>-<version>`
	//
	// * registry dependencies
	// * git dependencies
	// * path dependencies outside workspace root directory

However, this is not ideal for debugger. Several questions emerge in rust-lang/rust#111540 (comment).

• Impossible to manually configure remap for debugger — People need to configure one remap rules per per dependency. That is not possible for people to do it manually for each <pkg>-<version>/ back to absolute paths. It would require a dedicated tool to restore all the paths.
• The same issue also affects path dependency remap. For path dependencies outside the current workspace root, we remap paths to <pkg>-<version> as well.
• There is no way to distinguish relative path from different Cargo workspace — For path dependencies inside a workspace Cargo currently remaps to paths relative to workspace root. Imagine you have multiple Cargo workspaces and there is one non-Rust root project compiles them together. Now all your debuginfo files are messed up since they were remapped to relative path like src/some/file.rs.

Possible options

Registry and git dependencies

Updated as suggested in #13171 (comment)

In rust-lang/rust#111540 (comment), @michaelwoerister brought up a smarter remap strategy. We could remap paths from $CARGO_HOME/registry/src and $CARGO_HOME/git/checkouts to a empty string for registry and git dependencies respectively, so that there is a finite set of rule (two exactly) to configure in debugger for each platform. That is to say, cargo could unconditionally pass these to the compiler:

• --remap-path-prefix=$CARGO_HOME/.cargo/registry/src=
• --remap-path-prefix=$CARGO_HOME/.cargo/git/checkouts=

(or conditionally if we want to reduce number of args passed in to rustc)

That is, example

/home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/serde-1.0.193/src/lib.rs
-> index.crates.io-6f17d22bba15001f/serde-1.0.193/src/lib.rs

/home/user/.cargo/git/checkouts/cargo-new-release-57ee03ad5db79cc6/589ded5/src/lib.rs
-> cargo-new-release-57ee03ad5db79cc6/589ded5/src/lib.rs

To restore source paths for debuggers, you only need to set two paths. For example in gdb, set

directory /home/user/.cargo/registry/src
directory /home/user/.cargo/git/checkouts

Path dependencies

For path dependencies, I wonder if we can always remap to relative path to workspace root, regardless whether the package is inside or outside the workspace. Developers should be responsible for locals dependencies, and if they want, they are able to have their own remap rules via --remap-path-prefix.

Other sources

Vendor and local-registry dependencies

They are both sort of local dependencies. I would suggest treating them as path dependencies. We can always apply new rules to them later on.

Patched dependencies

Just apply all the suggested remap rule above, if not sufficient, people can write their own rule via --remap-path-prefix.

[epage]

We should make sure we keep the comment from https://github.com/rust-lang/cargo/blob/master/src/cargo/util/workspace.rs#L97 in mind

/// The path that we pass to rustc is actually fairly important because it will
/// show up in error messages (important for readability), debug information
/// (important for caching), etc. As a result we need to be pretty careful how we
/// actually invoke rustc.
///
/// In general users don't expect `cargo build` to cause rebuilds if you change
/// directories. That could be if you just change directories in the package or
/// if you literally move the whole package wholesale to a new directory. As a
/// result we mostly don't factor in `cwd` to this calculation. Instead we try to
/// track the workspace as much as possible and we update the current directory
/// of rustc/rustdoc where appropriate.
///
/// The first returned value here is the argument to pass to rustc, and the
/// second is the cwd that rustc should operate in.

I think we're fine but wanted to make sure we keep this in mind

[epage]

In <rust-lang/rust#111540 (comment), @michaelwoerister brought up a smarter remap strategy. We could remap paths from $CARGO_HOME/registry/src and $CARGO_HOME/git/checkouts to a fixed string for registry and git dependencies respectively, so that there is a finite set of rule (two exactly) to configure in debugger. That is to say, cargo could unconditionally pass these to the compiler:

• --remap-path-prefix=$CARGO_HOME/.cargo/registry/src=registry@

• --remap-path-prefix=$CARGO_HOME/.cargo/git/checkouts=git@

For registry, the path prefix would be things like index.crates.io-6f17d22bba15001f/winnow-0.5.26

For git, the path prefix would be things like winnow-110da4674123f04f/

What are the stability guarantees for those paths?

[epage]

Vendor and local-registry dependencies

They are both sort of local dependencies. I would suggest treating them as path dependencies. We can always apply new rules to them later on.

I wonder what code path these would go down and what the "for free" option is. If its registries, I expect the do-nothing would be good enough.

[weihanglo]

For registry, the path prefix would be things like index.crates.io-6f17d22bba15001f/winnow-0.5.26

For git, the path prefix would be things like winnow-110da4674123f04f/

What are the stability guarantees for those paths?

They should be fairly stable on each platform across different Rust versions. See here and here.

Granted, we should have a test to verify short_name is stable.

Vendor and local-registry dependencies
They are both sort of local dependencies. I would suggest treating them as path dependencies. We can always apply new rules to them later on.

I wonder what code path these would go down and what the "for free" option is. If its registries, I expect the do-nothing would be good enough.

cargo/src/cargo/core/compiler/mod.rs

Line 1220 in 6982b44

let is_local = unit.pkg.package_id().source_id().is_path();

I was wrong with local-registry. Unpacked sources go into the same ~/.cargo/registry/src directory for local registry, so no special treatment needed.

People might want to remap vendor sources, as they might do release builds with them. However, from the above code path cargo only knows the pkg is from a Registry source, since in fact it is a ReplacedSource. We have no way to tell whether a package is from a vendor source.

I still think it's better to remap to relative paths for vendor sources, as they are designed to be used as local files. If people can get free debugging experience without configuring remap restore, then why not?

[michaelwoerister]

Thanks for picking this up, @weihanglo!

• --remap-path-prefix=$CARGO_HOME/.cargo/registry/src=registry@
• --remap-path-prefix=$CARGO_HOME/.cargo/git/checkouts=git@

Please note that the registry@ and git@ prefixes would complicate things again. Remapped paths would then look like:

registry@index.crates.io-6f17d22bba15001f\regex-1.10.2\src\lib.rs

instead of just

index.crates.io-6f17d22bba15001f\regex-1.10.2\src\lib.rs

A debugger would look for a file registry@index.crates.io-6f17d22bba15001f\regex-1.10.2\src\lib.rs in its source directories, but would fail finding a directory called registry@index.crates.io-6f17d22bba15001f because it's actually called index.crates.io-6f17d22bba15001f.

[michaelwoerister]

They should be fairly stable on each platform across different Rust versions. See here and here.

Granted, we should have a test to verify short_name is stable.

Stability across platforms and across time would indeed be desirable. The test case linked above excludes a few platforms. The compiler has an example of a hasher the gives the same result on different kinds of platforms.

[weihanglo]

Please note that the registry@ and git@ prefixes would complicate things again.

Hmm… Maybe I understand the original concern wrong 🤔 ?

I would expect remapped paths to be like

registry@/index.crates.io-6f17d22bba15001f\regex-1.10.2\src\lib.rs

So that we can restore the prefix-map by searching a specific prefix like registry@. For example,

# in gdb
set substitute-path registry@ /home/user/.cargo/registry/src
# in lldb
settings set target.source-map git@ /home/user/.cargo/git/checkouts

That makes the substitution rules down to 2.

[weihanglo]

The compiler has an example of a hasher the gives the same result on different kinds of platforms.

Good reference! Thank you! I guess cargo might want to adopt some techniques there to have a stable hash across platforms (though it may break things on some platforms).

Despite that, is having a cross-platform stable hash in debuginfo a hard requirement? I assume things like DWARF are platform/architecture dependant. We cannot use DWARF generated on aarch64 to debug on x86_64. But again, I might be wrong 😞.

[michaelwoerister]

That makes the substitution rules down to 2.

Yes, that looks correct to me. The only change I'm suggesting is to omit the registry@ and git@ prefixes, so that one doesn't need the substitution feature. So instead of doing this in GDB:

set substitute-path registry@ /home/user/.cargo/registry/src

we would do this:

directory /home/user/.cargo/registry/src

or add -d /home/user/.cargo/registry/src to GDB's command line.

That should have the same effect, but path substitution is less widely supported than specifying a source path. For example, I was not able to find an equivalent feature in WinDbg or the MSVC debugger. And when using any kind of source server, having something that does not match the file system path will likely also lead to complications. It's quite possible that all of these issues could be worked around somehow, but why make things more complicated than necessary? I'd like trim-paths to be the default (at least for release builds), and I think that will be easier if we remove as much friction as possible.

Despite that, is having a cross-platform stable hash in debuginfo a hard requirement?

I'm not 100% sure if it has to be a hard requirement -- but it might also be something where we can prevent headaches down the line. The hash value basically represents a URL/URI, right? Registries in general are not platform specific, so there is no reason why an identifier for a registry should be platform specific.

And there are quite a few cross platform scenarios involving debuginfo. Analyzing crashdumps from platform X on platform Y is pretty common, or cross-building from one platform to another. E.g. most 32-bits builds probably happen on 64-bit hosts, right?

I think it might be worth biting the bullet now and aligning these "short-names" on all platforms.

[weihanglo]

Registries in general are not platform specific, so there is no reason why an identifier for a registry should be platform specific.

Totally agree on this. We might want to make hashes cross-platform stable. Just that would invalidate source caches on some platforms.

Do you think this is a major blocker on -Ztrim-paths? I am seeing that this could be a follow-up even after -Ztrim-paths stabilized, if we don't get stable hashes in time.