`cargo vendor` loses some nonempty directories

[dtolnay]

Problem

For example, this one: https://docs.rs/crate/bat/0.25.0/source/src/syntax_mapping/builtins/macos/

Steps

Confirm this directory exists in the published crate, as shown in the docs.rs link above.

$ curl -L https://crates.io/api/v1/crates/bat/0.25.0/download -o bat-0.25.0.crate

$ tar tf bat-0.25.0.crate bat-0.25.0/src/syntax_mapping/builtins/macos
bat-0.25.0/src/syntax_mapping/builtins/macos/.gitkeep

Now let's try vendoring.

# Cargo.toml
[package]
name = "repro-cargo"
edition = "2021"

[dependencies]
bat = "=0.25.0"

$ cargo vendor

$ ls vendor/bat/src/syntax_mapping/builtins/
bsd-family  common  linux  README.md  unix-family

"macos" is not there.

In bat-0.25.0's case, this actually breaks the crate (when built on macos from vendored sources).

error: failed to run custom build command for `bat v0.25.0`

Caused by:
  process didn't exit successfully: `target/debug/build/bat-126fec914d2804b6/build-script-main` (exit status: 1)
  --- stdout
  cargo:rerun-if-changed=build/
  cargo:rerun-if-changed=src/syntax_mapping/builtins/

  --- stderr
  Error: IO error for operation on src/syntax_mapping/builtins/macos: No such file or directory (os error 2)

  Caused by:
      No such file or directory (os error 2)

Possible Solution(s)

No response

Notes

No response

Version

cargo 1.86.0-nightly (088d49608 2025-01-10)
release: 1.86.0-nightly
commit-hash: 088d496082726091024f1689c124a0c3dccbd775
commit-date: 2025-01-10
host: x86_64-unknown-linux-gnu
libgit2: 1.9.0 (sys:0.20.0 vendored)
libcurl: 8.9.0-DEV (sys:0.4.74+curl-8.9.0 vendored ssl:OpenSSL/1.1.1w)
ssl: OpenSSL 1.1.1w  11 Sep 2023
os: Ubuntu 24.4.0 (noble) [64-bit]

[weihanglo]

It is because of this:

If the package is not in a git repository, all “hidden” files starting with a dot will be skipped

And cargo vendor reuses PathSource::list_files which applies the same rules, hence the bug.

[ehuss]

Thanks for the report!

We have a number of cargo vendor issues related to it not replicating the contents of the .crate file because it essentially re-computes the list of files. Because the .crate file is no longer in a git repository, that logic changes (in this case, skipping hidden fiels). #14034 (comment) contains some notes about this (there are several other issues as well, though I don't see any mentioning this particular case).

I don't recall if there are any updates since #12509 (comment) about trying to change that.

[ehuss]

In the short term, if I had to guess, adding the hidden files explicitly to the package.include list might help (but it is too late for me to test that).

[dtolnay]

if I had to guess, adding the hidden files explicitly to the package.include list might help

From what I think is an equivalent experiment¹, I think it doesn't. Here is what I tried:

• Based on a GitHub search for /include =.*\.git/ path:Cargo.toml I identified this crate:
  https://docs.rs/crate/mithril-stm/0.3.34/source/Cargo.toml.orig#11
  https://docs.rs/crate/mithril-stm/0.3.34/source/.gitignore

• One of my own crates as a control, containing .gitignore but no package.include (nor exclude):
  https://docs.rs/crate/anyhow/1.0.95/source/Cargo.toml.orig
  https://docs.rs/crate/anyhow/1.0.95/source/.gitignore

• # Cargo.toml
  [package]
  name = "repro-cargo"
  edition = "2021"
  
  [dependencies]
  anyhow = "=1.0.95"
  mithril-stm = "=0.3.34"

• cargo vendor

• ls vendor/mithril-stm vendor/anyhow -a

  vendor/anyhow:
  .   build     .cargo-checksum.json  Cargo.toml      LICENSE-MIT  rust-toolchain.toml  tests
  ..  build.rs  Cargo.lock            LICENSE-APACHE  README.md    src
  
  vendor/mithril-stm:
  .  ..  benches  build.rs  .cargo-checksum.json  Cargo.lock  Cargo.toml  examples  README.md  src  tests

Neither crate ended up vendoring .gitignore from the published crate.

Footnotes

1. .gitignore follows the same logic as .gitkeep? ↩

[ehuss]

.gitignore is specially handled here where we explicitly exclude it from the vendor output due to bad interactions with git otherwise.

Other filenames like .gitkeep or .foo should work.