Make `cargo --list` print description for custom subcommands

[dtolnay]

Problem

Currently cargo --list prints descriptions only for the builtin subcommands. For third-party subcommands like cargo expand it does not print a description.

$ cargo --list
Installed Commands:
    …
    doc                  Build a package's documentation
    expand
    fetch                Fetch dependencies of a package from the network
    …

Proposed Solution

First of all, I don't want cargo --list to run any of the binaries (in order to, for example, run --help on them and parse the output, or some other handshake to cause the binary to print out a description of itself). Arbitrary binaries in the PATH are not necessarily safe to run. An arbitrary binary might completely ignore --help or any other handshake being passed by Cargo, and immediately do unwanted stuff instead, which would be unexpected and undesirable to the person running cargo --list.

Instead, Cargo subcommands should be able to embed a description into their compiled binary, which can then be directly retrieved from the binary by Cargo in a way that does not involve running it.

The way this is exposed to subcommands can be as simple as:

// subcommand's main.rs

cargo_subcommand_metadata::description!("…");

fn main() {
    /* … */
}

The underlying implementation would need to be platform-specific based on the file format used for executables on the platform. On platforms that use ELF executables, I would propose that this uses a Note. On those platforms the macro shown above would expand to:

#[used]
#[link_section = ".note.cargo.subcommand"]
static ELF_NOTE: ElfNote = {
    …
}

This note can then be retrieved efficiently from the ELF executable during cargo --list by using the object crate to parse the executable's program header.

From testing on my machine with the cargo-expand crate, cargo build --release produces a 13 MB executable, and retrieving the subcommand description ELF note from it using the object crate takes 0.03 milliseconds, so this should be plenty fast enough for cargo --list.

We do not necessarily need to implement support for a large number of platforms immediately up front. Supporting just ELF would already benefit a large fraction of Cargo users.

Notes

Yes, exactly.

[dtolnay]

As proof of concept, I have released a version of the cargo-expand crate containing a suitable ELF note (dtolnay/cargo-expand#142) and created a draft PR #10663 which makes cargo --list retrieve the subcommand description from that ELF note.

[epage]

Currently, cargo hard codes the descriptions for first-party cargo external subcommands like cargo-clippy and cargo-fmt. This would also allow us to remove those hard coded descriptions and scale up better for adding more first-part external subcommands. I've been toying with the idea of splitting some of cargo's built-in subcommands out as external subcommands. This has the downside increasing the distribution size though it means we could possibly remove them in the minimal rustup profile. On the other hand, modularization like this could help with a more approachable code base that is easier to review, dogfooding of cargo's APIs, and faster compile/test times for whichever section of cargo a developer is in. This is blocked on our need to scale up our processes via workspaces which is blocked on the fact that we are part of the rust-lang/rust workspace and can't have our own.

[ehuss]

The Cargo Team discussed this a bit, and here are some questions or thoughts from that discussion:

• There were some concerns that the proposed solution seemed to be ELF-specific. Although this could be extended for other formats, like Windows, it seems risky to try to support that across all of Cargo's platforms. Additionally, some people use shell scripts or other kinds of executables that can't embed metadata.
• Would it be possible to use the description from Cargo.toml instead? I think there was some concern that what people put in description might be slightly different from what they would use for cargo --list, but I suspect in most cases it should be sufficient. If so, then one option that may be simpler is to embed the metadata in cargo's "install" JSON tracker.

[timmyjose]

I was thinking that I'd missed something in the documentation, but I see that this issue is still open. Hopefully it should be resolved sometime in the near future.