Annotate "Compiling X" status lines with active features

[pnkfelix]

Problem

Developers of systems that invoke cargo build multiple times are sometimes surprised to see the same dependencies show up repeatedly. "Why isn't Rust caching these" I have heard many times.

There are many reasons this can happen, but one is especially insidious: different feature-sets for the given dependency, which leads to the previous build artifact not being reusable for the current context.

Proposed Solution

My suggestion is that we actually make Compiling X include (in addition to its current output), the feature set that it is being built with.

Once you know that this is the problem, you can investigate it via cargo tree and such.

By including that information, people will be able to more immediately see why a crate is being rebuilt.

Concretely, today:

      Compiling serde_json v1.0.149
[...]
      Compiling serde_json v1.0.149

With this proposal:

      Compiling serde_json v1.0.149  [features: default, preserve_order, std]
[...]
      Compiling serde_json v1.0.149  [features: arbitrary_precision, default, std]

Even just having it as an opt-in via an env var would be enough to allow quick investigations. E.g. something like CARGO_TERM_COMPILING_FEATURE_SET, analogous to flags like CARGO_TERM_PROGRESS_WHEN. (I suggest an env var specifically because I would like to have access to this in wrappers around cargo rather than be forced to achieve it via flags like --message-format.)

Notes

No response

[epage]

This can get very nosiy, particularly because we likely can't just list the top-level features but would need to list all. We'd likely need to do that anyways to accomplish the purpose of this. I worry if this could even fulfill its purpose then.

We are working on structured logging for builds with one cargo report subcommand focused on showing rebuild reasons. It doesn't help yet when independent cache entries are used but that is something we should improve.

[pnkfelix]

Are you saying even with an opt-in flag that you do not think this would be useful?

(I have found a way to get an approximate effect with a wrapper script that requests json and reprocesses it into human readable form. I just figured we might benefit from a knob to allow quicker investigations.)

[weihanglo]

Would the use case is for people to scroll back to look at the status message and figure out "Why isn't Rust caching these" by themselves, or do they want it to be answered by a command like cargo report rebuilds? While not perfect like Ed has said, it is available as a nightly feature today. See Unstable: Build analysis.

If the cargo report command isn't helpful, you might want to check raw logs directly and see if there is anything useful for people to look at. I've try using some LLM agents to analyze those structured logs and they did find rebuild cause in a few minutes.

[epage]

Are you saying even with an opt-in flag that you do not think this would be useful?

I am having a hard time seeing how it could meet the purpose for which you are suggesting

(I have found a way to get an approximate effect with a wrapper script that requests json and reprocesses it into human readable form. I just figured we might benefit from a knob to allow quicker investigations.)

Might be worth sharing that script and including real world examples and talk through how users can reasonably leverage this information.

[weihanglo]

Might be worth sharing that script and including real world examples and talk through how users can reasonably leverage this information.

That information is already in structured logs.

cargo/src/cargo/util/log_message.rs

Lines 85 to 87 in 3c4678d

	/// Enabled features.
	#[serde(default, skip_serializing_if = "Vec::is_empty")]
	features: Vec<String>,

[pnkfelix]

I'll look into making the script sharable, its not in that kind of state at the moment.

The real world example is a monorepo with multiple distinct artifacts (e.g. binaries, dll's) and their dependencies are such that they have third party crates that independently resolve to different feature sets under the "most obvious" / naive way to do things.

What I'm trying to say is that the way that cargo's output is generated right now, the naive user doesn't have any hint as to why they're seeing what seems like redundant build of the same intermediate crate even when the intermediate build directory is shared between the artifacts.

There are semi-standard workarounds for this situation, for example hakari. But as it stands, there's not enough signal for someone to understand that they might need to adopt one of those solutions.

While I'm here: After reflecting on things, I realized that I probably optimized my proposal above to be "what's the simplest thing to implement that would make this problem more immediately apparent." Emphasis on "simplest"

But there are other options. For example: Perhaps cargo could dynamically detect "even though I cannot find a compiled artifact here suitable for me to reuse, there is at least one previously complied artifact sitting in the build_dir with the same crate name and version, but which solely differs in its feature set" and then emit a message for just those crates that have a matching name+version, and spelling out what feature set(s) for which cargo already had done previous builds that are sitting around in the build directory.

That might address the "noisiness" issue, and alert the developer, via a pretty high signal-to-noise ratio, that other configurations of their overall system might allow them to get better artifact reuse? (But it also seems a bit more complicated to implement?)

[weihanglo]

But there are other options. For example: Perhaps cargo could dynamically detect "even though I cannot find a compiled artifact here suitable for me to reuse, there is at least one previously complied artifact sitting in the build_dir with the same crate name and version, but which solely differs in its feature set" and then emit a message for just those crates that have a matching name+version, and spelling out what feature set(s) for which cargo already had done previous builds that are sitting around in the build directory.

While Cargo message is not yet that smart, the information of "spelling out what feature set(s) for which cargo already had done previous builds" is already in structured log as aforementioned. The Cargo timing report is also now backed by it, so whatever you can get in timing report, you tool should have those available.

I would love to learn about your experience on those logs, and why or why not it isn't suitable for your workflow.