`rustup` installation broken after upgrading from macOS 13.6 to 14.2

[str4d]

Problem

I installed Rust using rustup some time last year (or maybe earlier, I forget) onto a Macbook Air M1 running macOS 13. A few weeks ago I upgraded from 13.6 to 14.2, and since then my Rust installation has been subtly broken in ways I keep discovering.

Steps

1. Install Rust via rustup on macOS 13.6.
2. Upgrade to macOS 14.2.
3. Try to do things that require interacting with environment variables.

Possible Solution(s)

No response

Notes

I first noticed problems when I started trying to cross-compile for iOS targets. See rust-lang/rust#114276 (comment) for more information (as that issue seems to be the same as mine, so I posted a comment there initially).

I just now noticed that the rustup overrides are not working (neither manual not rust-toolchain.toml), which is why I now think it's a rustup problem rather than a rustc or cargo problem:

$ rustup --version
rustup 1.26.0 (5af9b9484 2023-04-05)
info: This is the version for the rustup toolchain manager, not the rustc compiler.
info: The currently active `rustc` version is `rustc 1.75.0 (82e1608df 2023-12-21)`

$ cargo --version
cargo 1.75.0

$ cd rage/
$ rustup --version
rustup 1.26.0 (5af9b9484 2023-04-05)
info: This is the version for the rustup toolchain manager, not the rustc compiler.
info: The currently active `rustc` version is `rustc 1.65.0 (897e37553 2022-11-02)`

$ cargo --version
cargo 1.75.0

Rustup version

rustup 1.26.0 (5af9b9484 2023-04-05)

Installed toolchains

rustup show
Default host: aarch64-apple-darwin
rustup home:  /Users/str4d/.rustup

installed toolchains
--------------------

stable-aarch64-apple-darwin (default)
beta-aarch64-apple-darwin
nightly-2022-11-03-aarch64-apple-darwin
nightly-2022-12-26-aarch64-apple-darwin
nightly-2023-03-10-aarch64-apple-darwin
nightly-2023-05-03-aarch64-apple-darwin
nightly-2023-06-09-aarch64-apple-darwin
nightly-aarch64-apple-darwin
1.56.0-aarch64-apple-darwin
1.56.1-aarch64-apple-darwin
1.59.0-aarch64-apple-darwin
1.60.0-aarch64-apple-darwin
1.65.0-aarch64-apple-darwin
1.67.1-aarch64-apple-darwin
1.69.0-aarch64-apple-darwin

active toolchain
----------------

1.65.0-aarch64-apple-darwin (overridden by '/Users/str4d/dev/rust/age/rage/rust-toolchain.toml')
rustc 1.65.0 (897e37553 2022-11-02)

[str4d]

As the breakages aren't currently blocking for me, I'm happy to leave my rustup installation as-is and do more investigation of it if the rustup devs think it would be helpful and can tell me what to look for.

[str4d]

I just tried running cargo +1.65.0 --version to force the override, and got this:

$ cargo +1.65.0 --version
error: no such command: `+1.65.0`

	Cargo does not handle `+toolchain` directives.
	Did you mean to invoke `cargo` through `rustup` instead?

That got me thinking:

$ where cargo
/opt/homebrew/bin/cargo
/Users/str4d/.cargo/bin/cargo

$ echo $PATH
/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/Apple/usr/bin:/Applications/Wireshark.app/Contents/MacOS:/opt/homebrew/opt/python@3.9/libexec/bin:/Users/str4d/go/bin:/Users/str4d/bin:/Users/str4d/.cargo/bin

$ PATH=~/.cargo/bin:$PATH cargo --version
cargo 1.65.0 (4bc8f24d3 2022-10-20)

So this might be the same root cause as rust-lang/rust#114276 (comment) - the upgrade breaks however rustup is configuring the PATH variable, causing it to be looked up last instead of first.

[str4d]

I initially guessed that rustup might be configuring my PATH here:

$ cat ~/.profile
export PATH="$HOME/bin:$PATH"
export PATH="$HOME/go/bin:$PATH"
. "$HOME/.cargo/env"

However, there are two problems with that guess:

• My PATH ends with ...:/Users/str4d/go/bin:/Users/str4d/bin:/Users/str4d/.cargo/bin, but ~/.cargo/env says that it prepends itself, and that happens after the two prior exports, so that would in theory produce ...:/Users/str4d/.cargo/bin:/Users/str4d/go/bin:/Users/str4d/bin
• These three elements are at the end of my PATH instead of the beginning, which indicates that if rustup is indeed relying on ~/.profile then there is some wider OS issue causing the PATH created by ~/.profile to be appended to the global PATH, which seems odd...

[str4d]

Aha, it's (ironically) path_helper that is causing this! It is taking the PATH settings from ~/.profile and appending them to the default system path. https://gist.github.com/Linerre/f11ad4a6a934dcf01ee8415c9457e7b2 is a great write-up of the problem, and per the ordering that it gives for Zsh, the solution appears to be to put the rustup environment loader in ~/.zprofile instead. Moving my .profile contents there immediately fixed the Go and local binary paths (by causing them to appear twice), but rustup's is still at the end due to I assume caching. I'll restart my laptop to confirm...

[str4d]

Hmm, restarting did not help. Now it seems I have a different problem:

• path_helper is finding the content of ~/.zprofile and appending it to then (login-level?) PATH.
• Zsh is then running ~/.zprofile again on top of that PATH in the shell.
• The Go and local binary paths get prepended to the path_helper path (appearing twice) because they use simple export PATHs. The rustup path is not prepended because it uses a script that checks whether its path already exists in PATH (because it does, because path_helper appended it).

I will have to do more reading to figure out what the correct fix is here, and if there is something rustup can do to better ensure it behaves correctly on macOS.

[str4d]

Looking at how rustup-init works, it should be writing to ~/.profile (because it always writes to the POSIX location) and ~/.zshenv (because macOS defaults to Zsh since 10.15 Catalina):

rustup/src/cli/self_update/shell.rs

Lines 124 to 128 in 33ad3a4

	fn update_rcs(&self) -> Vec<PathBuf> {
	// Write to .profile even if it doesn't exist. It's the only rc in the
	// POSIX spec so it should always be set up.
	self.rcfiles()
	}

rustup/src/cli/self_update/shell.rs

Lines 193 to 206 in 33ad3a4

	fn update_rcs(&self) -> Vec<PathBuf> {
	// zsh can change $ZDOTDIR both _before_ AND _during_ reading .zshenv,
	// so we: write to $ZDOTDIR/.zshenv if-exists ($ZDOTDIR changes before)
	// OR write to $HOME/.zshenv if it exists (change-during)
	// if neither exist, we create it ourselves, but using the same logic,
	// because we must still respond to whether $ZDOTDIR is set or unset.
	// In any case we only write once.
	self.rcfiles()
	.into_iter()
	.filter(|env| env.is_file())
	.chain(self.rcfiles())
	.take(1)
	.collect()
	}

And indeed I found the . "$HOME/.cargo/env" line in my ~/.profile (as mentioned above, when I tried moving it to ~/.zprofile), and my ~/.zshenv. Incidentally, I also found my Go and local bin exports in those same two places, so maybe when I set up this computer I just searched for what Rustup changed and added those exports myself to the same places? I don't recall.

I separately note that my Homebrew installation added itself only to ~/.zprofile, and whatever it is doing, it is managing to get itself prepended to my PATH instead of appended (which is part of why my Rust installation started breaking: alongside that upgrade from macOS 13.6 to 14.2, I installed a piece of software via brew that required Rust, so brew installed the stable Rust toolchain directly, and that now overrides my Rustup installation). What is really interesting is that path_helper does append it, but then in my PATH Homebrew is only prepended and does not appear twice:

$ /usr/libexec/path_helper
PATH="/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/Apple/usr/bin:/Applications/Wireshark.app/Contents/MacOS:/opt/homebrew/opt/python@3.9/libexec/bin:/Users/str4d/go/bin:/Users/str4d/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/Users/str4d/.cargo/bin"; export PATH;
MANPATH="/usr/share/man:/usr/local/share/man:/Applications/Wireshark.app/Contents/Resources/share/man:/opt/homebrew/share/man:"; export MANPATH;
$ echo $PATH
/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/Apple/usr/bin:/Applications/Wireshark.app/Contents/MacOS:/opt/homebrew/opt/python@3.9/libexec/bin:/Users/str4d/go/bin:/Users/str4d/bin:/Users/str4d/.cargo/bin