Non-reproducible builds when depending on local crates

[shkoo]

It looks like meta fingerprints are not particularly stable when it comes to depending on local crates (i.e. with a dependency specifying a "path", a "[patch.crates-io]" section or a "__CARGO_TESTS_ONLY_SRC_ROOT" environment variable); different absolute pathnames seem to produce different build fingerprints. This makes it difficult to reproduce builds when running as a different user or on a different operating system or distribution.

--remap-path-prefix doesn't seem to help at all here.

To reproduce this problem:

for i in a b
do
    mkdir $i
    (   
        cd $i
        TOPDIR=`pwd`
        cargo new --lib repro
        cat > repro/src/lib.rs <<EOF 
pub fn foo() -> usize { 
    ext_crate::add(1, 2) 
} 
EOF 
        cp -r ../ext_crate ext_crate
        cat > repro/Cargo.toml <<EOF 
[package] 
name = "repro" 
version = "0.1.0" 
edition = "2021" 
 
[dependencies] 
ext_crate = { version = "0.1.0", path="../ext_crate" } 
 
EOF 
        (   
            cd repro
            RUSTFLAGS="--remap-path-prefix=${TOPDIR}=/stablepath" cargo build -vv --release --target-dir=build
        )
    )
done
diff -ur [ab]/repro/build

My expected behavior here would be that those two builds would be identical and reproducible.

These are the versions I am using:

$ cargo --version --verbose
cargo 1.63.0-nightly (4d92f07f3 2022-06-09)
release: 1.63.0-nightly
commit-hash: 4d92f07f34ba7fb7d7f207564942508f46c225d3
commit-date: 2022-06-09
host: x86_64-unknown-linux-gnu
libgit2: 1.4.2 (sys:0.14.2 vendored)
libcurl: 7.83.1-DEV (sys:0.4.55+curl-7.83.1 vendored ssl:OpenSSL/1.1.1n)
os: Ubuntu 20.04 (focal) [64-bit]
$ rustc --version --verbose
rustc 1.63.0-nightly (546c826f0 2022-06-12)
binary: rustc
commit-hash: 546c826f0ccaab36e897860205281f490db274e6
commit-date: 2022-06-12
host: x86_64-unknown-linux-gnu
release: 1.63.0-nightly
LLVM version: 14.0.5

[eggyal]

@rustbot label +A-reproducibility

[ijackson]

I think this bug may be responsible for the nonreproducibility of Hippotat in Debian:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1028254
https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/hippotat.html

[rwmacleod]

This bug may also be the same problem that is seen in the Yocto builds:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14875

[ijackson]

This might be related to #5505

[workingjubilee]

The script here had several minor errors due to being copied from a terminal probably and bash being actually astonishingly whitespace sensitive, which caused all the heredocs to not be recognized upon copy-pasting this back into a script. I had to fix it to make it work, like so:

#!/bin/bash
cargo new --lib ext_crate
for i in a b
do
    mkdir $i
    (   
        cd $i
        TOPDIR=`pwd`
        cargo new --lib repro
        cat > repro/src/lib.rs <<EOF
pub fn foo() -> usize { 
    ext_crate::add(1, 2) as usize
} 
EOF
        cp -r ../ext_crate ext_crate
        cat > repro/Cargo.toml <<EOF
[package] 
name = "repro" 
version = "0.1.0" 
edition = "2021" 
 
[dependencies] 
ext_crate = { version = "0.1.0", path="../ext_crate" } 
 
EOF
        (   
            cd repro
            RUSTFLAGS="--remap-path-prefix=${TOPDIR}=/stablepath" cargo build -vv --release --target-dir=build
        )
    )
done
diff -ur [ab]/repro/build

In the future, I would suggest you upload shell scripts directly rather than copy-pasting from the terminal (or whatever the original source was) to GitHub if you have the slightest doubt over whether a single space or newline from the script might be altered. With a shebang, too, that properly denotes it's bash or what.

Please? So that it's, er, easier to reproduce the reproducer that reproduces the non-reproducibility?

[workingjubilee]

I guess this is as close to an MCVE as we're gonna get.

[bjorn3]

This is an issue somewhere inside cargo. It passes different -Cmetadata arguments for both builds of ext_crate due to computing a different dep_hash, which then causes a different build output. I don't know know why it computes a different dep_hash though.

[workingjubilee]

Thanks for clarifying! I suppose we can get this moved to rust-lang/cargo then?

[workingjubilee]

@rustbot transfer cargo