Implicit install breakage falls below usual project standards

[tmandry]

Hi folks, I just learned about the breaking change in #3985 as it was released today. I can understand why this change might be desirable, but I have concerns that the change was released without the kind of messaging or feedback mechanism that Rust users have come to expect from the project.

Rustup is an incredibly useful and valuable tool for Rust, forming the entry point of the vast majority of users' experience with Rust. Of course, that kind of importance means we must be careful. The kind of experience that Rust tries to prevent with its stability guarantee is "this used to work and it doesn't anymore". Each of these is frustrating to users and they are usually experienced "in anger". Each time we create them, we are taking out a loan against Rust's reputation with the expectation that Rust will get significantly better as a result of the change.

I do see that there was care to add an error message prompting the user with the exact command to run - that is in line with what users expect from Rust, and is highly appreciated. At the same time, there were two steps not taken that I have considered standard for any breaking change on the lang team:

• A final comment period run with @rfcbot that indicates a supermajority of the team is on board with the change. This gets published in places like This Week in Rust, giving users the opportunity to voice their opinion and team members to raise an objection on their behalf.
• For any change with significant disruption (i.e. any nontrivial number of broken crates in crater), starting with a future compatibility warning that gives users time to update ahead of the breaking change.

Additionally, some changes are accompanied with a dedicated blog post ahead of the change explaining the rationale for the change with instructions to users. Going through this process is more work, but I would argue it is not worth making a disruptive change without taking these steps.

Finally, I suspect this situation is arising because we don't have a project-wide standard for how breaking changes are handled. I have seen a need for this in other recent situations too; this just happens to be the one I'm commenting on. In parallel with this conversation, I would ask @rust-lang/leadership-council to adopt such a standard – even a basic one we can iterate on is better than none. I think it rises to the level of the leadership council because of the reputational risk each breaking change creates for Rust, and we should not expect individual teams to have to decide for themselves how to navigate that risk without guidance.

[oli-obk]

some changes are accompanied with a blog post explaining the rationale for the change with instructions to users.

This part was done:

https://blog.rust-lang.org/2025/03/02/Rustup-1.28.0.html

[tmandry]

What I meant was more like a dedicated blog post ahead of the change (I edited the text to clarify). There was a blog post announcing the release and describing the change, but it didn't include rationale.

[str4d]

some changes are accompanied with a blog post explaining the rationale for the change with instructions to users.

This part was done:

https://blog.rust-lang.org/2025/03/02/Rustup-1.28.0.html

The blog post contains instructions to users (albeit without any mention of how to update CI systems specifically, which is the primary instantaneous breaking surface), but does not explain the rationale for the change.

[tmandry]

I do agree with most of what's suggested in #4210 as a remedy, with a caveat that bugfixes that are technically breaking changes are allowed.

I'm not sure if this falls into the category of "bugfix" though; if I could sum up the rationale I saw in #3635 it amounted to something like, "the previous maintainers had one philosophy (prioritize convenience) and the current maintainers have a different philosophy (prioritize predictable behavior), so we're changing the behavior".

[chriskrycho]

Concur re: the distinction about bug-fixes-that-are-breaking being allowed (I’ll be the first to say that SemVer is hard!), and that it’s hard for me to frame this as a bug-fix.

Re: blog post—as @tmandry notes (and as I suggested on #4210), it would be really good to get feedback on this before rolling it out, and other teams have started doing that for these kinds of things. But I’ll also second @tmandry in noting that this is likely a thing we need to solve for at a project guidance level. From my POV this not at all a “you screwed up, boooo!” attitude for rustup, but rather an “ah, this exposes a gap we’ve had as a project in lacking clarity about how to approach things like this!”

[sunshowers]

I really appreciate everything the rustup team has been doing here, and on balance I support the change. Thank you for making it happen!

I don't want to add to the pile on, but I wanted to gently suggest maybe putting this behavior behind an edition boundary. I hate suggesting this because the 2024 edition is now out, but I think this might benefit from being behind an opt-in flag in rust-toolchain.toml and then becoming the default for the 2027 edition.

[ChrisDenton]

Just on a procedural note, call for beta testing/feedback happened here https://internals.rust-lang.org/t/seeking-beta-testers-for-rustup-v1-28-0/ and this was cross-posted to e.g. reddit, etc.

[weihanglo]

• A final comment period run with @rfcbot that indicates a supermajority of the team is on board with the change. This gets published in places like This Week in Rust, giving users the opportunity to voice their opinion and team members to raise an objection on their behalf.

FWIW, FCP doesn't always go to TWiR. There is a call-for-testing section though it needs an existing-RFC to trigger the automation. Otherwise you can only open PR to TWiR for including the change.

[tmandry]

@ChrisDenton I appreciate that the team tried to solicit feedback. I haven't found beta testing to be very helpful for this kind of feedback unfortunately, though it looks like it may have surfaced bugs.

@weihanglo I see.. there's a section with issues tagged final-comment-period on rust-lang/rust, but I guess issues from this repo aren't included there?

[Rigidity]

This breaks CI workflows that rely on installing Rustup and assuming the rust-toolchain.toml file will be respected automatically. That said, the dtolnay/rust-toolchain workflow skips installing Rustup if it's already installed, and the GitHub runners probably have an older version still. Therefore, unless you're using a runner without it preinstalled the fallout will be minimized until GitHub updates all of their actions to the new version.

[weihanglo]

@tmandry yes, and issues in rust-lang/cargo aren't included either (and that's totally okay because the cargo team use FCP as a general polling tool).

For any change with significant disruption (i.e. any nontrivial number of broken crates in crater), starting with a future compatibility warning that gives users time to update ahead of the breaking change.

Note that the breakage is mostly in CI. Only a few people check non-fatal warnings in successful CI logs. The mechanism helps, but doesn't really help much IMO.

To me, one of the critical issues is lacking of opt-in testing process for rustup. For other Rust tools we rely on rustup to do that, but rustup itself cannot.

[Rigidity]

Personally, widespread breaking changes like this should be a major version bump, and specified as part of the URL passed into cURL. So for instance, you would go from sh.rustup.rs to sh2.rustup.rs for this version, making updates explicit

[iliana]

On the topic of a call for feedback, I would like to echo the ask to post to blog.rust-lang.org (either the main blog or the "Inside Rust" blog) in the future ahead of major behavior changes in rustup, or provide some other way that allows me to see upcoming rustup changes in my RSS reader. As someone who is mostly an end-user of the project I do not find myself lurking the Rust Internals forum (and there doesn't seem to be any categorization that would allow me to get something other than a firehose in my RSS reader), and I don't read Reddit. I know that the methods for getting word out to end-users is somewhat fragmented but it seems so surprising to not see this behavioral change announced ahead of release on blog.rust-lang.org.

(There is also no better way to get the word out than to have rustup itself loudly warn users that the implicit behavior is changing over a period of a few months!)

I think this flag day didn't impact many, many more users in CI is because rustup is cached on GitHub Actions runners, and rustup appears to self-update only after installing the requested toolchain. The next time GitHub updates their image there will be more breakage.

To me, one of the critical issues is lacking of opt-in testing process for rustup. For other Rust tools we rely on rustup to do that, but rustup itself cannot.

Another problem that compounds this issue is that it's difficult to lock to a particular rustup release (to mitigate against behavior changes in CI in the same way rust-toolchain.toml and Cargo.lock do) without rolling the installation process yourself. While investigating CI issues we started having this morning I found there wasn't a way to specify the version of rustup using rustup-init.sh. I can file a separate issue and contribute that change if that sounds useful?

[djc]

We've been discussing and I am inclined to figure out a way to revert this release or to roll out a 1.28.1 with this behavior change rolled back. That would buy us some time to figure out a better migration strategy.