panic when attempting to `cargo doc -p` a dev-dependency

[Nemo157]

Problem

Panic occurs when attempting to build the docs of a crate added as a dev-dependency.

Steps

> cargo new foo && cd foo && cargo add --dev serde && RUST_BACKTRACE=1 cargo doc -p serde
     Created binary (application) `foo` package
    Updating crates.io index
      Adding serde v1.0.144 to dev-dependencies.
             Features:
             + std
             - alloc
             - derive
             - rc
             - serde_derive
             - unstable
thread 'main' panicked at 'activated_features for invalid package: features did not find PackageId { name: "serde", version: "1.0.144", source: "registry `crates-io`" } NormalOrDevOrArtifactTarget(None)', src/tools/cargo/src/cargo/core/resolver/features.rs:318:14
stack backtrace:
   0: rust_begin_unwind
             at /rustc/c07a8b4e09f356c7468b69c50cac7fc5b5000b8a/library/std/src/panicking.rs:584:5
   1: core::panicking::panic_fmt
             at /rustc/c07a8b4e09f356c7468b69c50cac7fc5b5000b8a/library/core/src/panicking.rs:142:14
   2: core::result::unwrap_failed
             at /rustc/c07a8b4e09f356c7468b69c50cac7fc5b5000b8a/library/core/src/result.rs:1785:5
   3: cargo::ops::cargo_compile::generate_targets
   4: cargo::ops::cargo_compile::create_bcx
   5: cargo::ops::cargo_compile::compile_ws
   6: cargo::ops::cargo_compile::compile
   7: cargo::ops::cargo_doc::doc
   8: cargo::commands::doc::exec
   9: cargo::cli::main
  10: cargo::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Possible Solution(s)

No response

Notes

No response

Version

cargo 1.65.0-nightly (6da726708 2022-08-23)
release: 1.65.0-nightly
commit-hash: 6da726708a4406f31f996d813790818dce837161
commit-date: 2022-08-23
host: x86_64-unknown-linux-gnu
libgit2: 1.4.2 (sys:0.14.2 vendored)
libcurl: 7.83.1-DEV (sys:0.4.55+curl-7.83.1 vendored ssl:OpenSSL/1.1.1q)
os: Arch Linux [64-bit]

[Rustin170506]

@rustbot claim
I can reproduce it. I will take a look and try to fix it.

[Rustin170506]

I debugged it. I found the cause of the panic is we do not contain the package information in

cargo/src/cargo/core/resolver/features.rs

Line 55 in 13ae438

activated_features: ActivateMap,

[Rustin170506]

Do we really support this feature?
Because I found the reason for this information missing is we set opts.decouple_dev_deps to true.

If we set the opts.decouple_dev_deps to false, it works well.

Also, if you have the same dep in both dep and dev-dep. It also works.

[package]
name = "foo"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
serde = "1.0.145"

[dev-dependencies]
serde = "1.0.145"

➜  foo git:(master) ✗ RUST_BACKTRACE=1 cargo doc -p serde
 Documenting serde v1.0.145
    Finished dev [unoptimized + debuginfo] target(s) in 3.08s

[Nemo157]

Ah, that reminds me of something I should have checked. This only panics on edition = "2021". Also cargo +1.51 doc -p serde doesn't seem to have Edition 2021 gated and successfully creates the docs 🤔. Every version after that which supports Edition 2021 panics.

And going a little deeper, it is definitely resolver = "2" that makes it panic, on 1.51+ (the versions that support it).

[weihanglo]

You're right. That's the reason this issue is labelled as A-features2 Area: issues specifically related to the v2 feature resolver . My impression is that the new feature resolver only enables features for dev-dependencies if they are actually needed to build tests/benches/examples.

I guess cargo doc -p <build-denpendency> also panics with the same reason.

[Rustin170506]

My impression is that the new feature resolver only enables features for dev-dependencies if they are actually needed to build tests/benches/examples.

Yes. That's why I asked if we support this feature in feature2. For now, it looks like we do not want to enable dev-dep by design.

[Nemo157]

IMO it's an important usecase, when you are developing offline and need to check the API documentation of your dev-dependencies. There are workarounds like temporarily adding them to your dependencies, or adding an example and using cargo doc --examples (though this builds docs for all dev-dependencies, not just whichever one you are interested in), but it seems like it should just work.

[Rustin170506]

@weihanglo How about when we find out that this package is a development dep package and we set it to yes?

cargo/src/cargo/ops/cargo_compile.rs

Line 383 in b2ee005

let has_dev_units = if filter.need_dev_deps(build_config.mode) || need_reverse_dependencies {

But I don't know how to determine if this pkg is a dev-dep in here.

[weihanglo]

@hi-rustin, I feel like it is a larger issue than we thought.

As far as I know, with feature resolver v2, Cargo runs the feature resolution in two passes:

• The first pass activates features without decoupling dev-dependencies, build-dependencies, and targets specified dependencies. This happens during the dependency resolution.
• The second pass narrows down the activated features set. This happens after the dependency resolution. This happens when explicitly enabling “resolver = 2" or in Edition 2021.

As a result, the issue you found starts happening after the second pass strips down “unnecessary” features. Currently, Cargo determines whether to unify feature based on the followings:

• For dev deps, CompileFilter::need_dev_deps does the job. However it only takes CompileMode into account.
• For target specified deps, you cannot activate their features unless you're building for those targets. Almost all code path set force_all_targets to ForceAllTargets::No, except cargo metadata and cargo tree.
• For build deps (host deps), they are always decoupled from normal deps, unless they are proc macros.

The problem is not only cargo doc but all cargo commands related to feature resolution. At least all build commands would panic by doing so. When a user specifies a package with -p|--package, it brings some design questions, such like:

• Do we expect cargo build -p <dev-dep|build-dep> to work?
• Do we expect cargo doc -p <dev-dep|build-dep> to work?
• Do we expect cargo test -p <dev-dep|build-dep> to work?
  (Except they also requires dev-deps, which is already handled here)
• What does it mean to cargo fix -p <dev-dep|build-dep> if they are non-local deps?
• Cargo might tell user to pass --target when executing cargo test -p <target-specified-dep>.
• cargo doc -p <target-specified-dep> might also want to suggest --target if possible.
• Should we treat a certain command special in terms of feature unification?
  (I don't think so as it might break something but Z-scrape-examples Nightly: rustdoc scrape examples already did that to some extent AFAIK)

---

Here is a demo Cargo.toml that you can tweak resolver to see the difference between v1 and v2.

• 🤯 cargo check -p cfg-if
• 🤯 cargo check -p autocfg
• ✅ cargo check -p serde_derive
• 🤯 cargo check -p empty-library
• ✅ cargo check -p empty-library --target wasm32-wasi (if it can cross-compile)

[package]
name = "issue11105"
version = "0.1.0"
edition = "2021"
resolver = "2"

[dev-dependencies]
cfg-if = "1"

[build-dependencies]
autocfg = "1"
serde_derive = "1"

[target.wasm32-wasi.dependencies]
empty-library = "1"

[Rustin170506]

Yes. I am also considering do we really need to support this. Also, can this behavior change be considered an incompatible change? Otherwise, I think we'd better be able to handle it, at least not panic.