Fails to connect through trusted SSL proxy

[koxu1996]

Problem Description / User Story

Let us suppose you want to use an SSL proxy, which generates certificates signed by a custom CA, and this CA is trusted by your OS.

The following curl command confirms the setup is done correctly:

curl -v --proxy https://localhost:8002 https://index.crates.io/config.json

However, when calling cargo it MIGHT surprisingly fail, for example:

error: failed to get `proc-macro2` as a dependency of package `my-app v0.1.0 (/home/andrew/Downloads/tmp/my-app2)`

Caused by:
  download of config.json failed

Caused by:
  failed to download from `https://index.crates.io/config.json`

Caused by:
  [60] SSL peer certificate or SSH remote key was not OK (SSL certificate problem: unable to get local issuer certificate)

Code 60 is a well known case and the solution is to simply add the CA certificate to the trust store.

Wait... you already have CA trusted and curl worked. So why Cargo fails?

Solving this mystery involved diffing Wireshark packets, hooking into libcurl calls, and doing bunch of other thing I would prefer to avoid, but finally I found the answer!

Who is Affected?

This is a little spoiler, but if your cargo has libcurl vendored then you are affected!

NOTE: I am talking about Linux, not sure about other systems.

You can check it with ldd $(which cargo) | grep libcurl - if the output is empty, then it means libcurl was vendored, instead of being dynamically linked.

However, if you see output like:

        libcurl.so.4 => /nix/store/ssrb11xyyfsccjmy98mhqn6yb6dnsq6d-curl-8.12.1/lib/libcurl.so.4 (0x00007f02e6801000)

then you are NOT affected.

Reproducing the Issue

Before jumping to the answer, I would like to give you a simple way of reproducing the issue.

Getting Cargo

If your cargo is dynamically linked with libcurl, then you need to get a static one; otherwise skip this section.

It can be done by compiling sources and making sure curl-sys will not be able to find the system's libcurl library via pkg-config:

git clone https://github.com/rust-lang/cargo.git && cd ./cargo
git checkout 0.85.0
export LIBCURL_NO_PKG_CONFIG=""
cargo build

Binary will be located at ./target/debug/cargo.

Running Proxy

To run MITM proxy we will use mitmdump tool, but firstly let us prepare CA certificates:

mkdir ./certs
cat <<EOF > ./certs/mitmproxy-ca.pem
-----BEGIN RSA PRIVATE KEY-----
MIIEogIBAAKCAQEAqi3o0aq+KpJry0/mcPfjXmXl1aqpEYRRaBB45t8FfzqYqpB5
81CJeYHau4Xn04OJGRL3lkrKzqVIyoR3yMS0qtXj4oCpAgert8uIZKEWWSUreHJI
oSFS7auTALO5HBDn9PYrzpCgdi2zK3X2+pMiHre+Z0FQ3h4WHtKivz/qIxqym2+P
b0cNxPqw3SbeBbaqZw3CJKvBMNa8/hRDuGUlSQlVRBqcYEj6xHumOL1vX2/hmNr0
m7t1FIxEgYLWpksPxcz1zO7Weu9T6hxmtMlwr9XqQPvAVnrjNwSQVlnUsTnzMnAF
tvl6NFOzLhydK0kER/63vmiatT1acy9iHCSsvQIDAQABAoIBAEhroBv1ttakF/Ze
CrmBbENkRrIuXaOENSZ9eA3707mIhKZ1NEh8ufw47A4IVpcW71V50l4ldvgIs5jn
VZhdSWs3+bIOwZ2JCST+mw0D1mBaZozryYe0iDd3X+gZE4njkeqXhh3yQy73lLtG
iLOqwIbk98S2MYVMtWophjOwLeIAzPaRWuAaFtS8ZlLE8CHB1jkchh8ExdeSzlrc
nCnfsE4MUbYJK73g56r9CIwlLX56YwLIo/qUhyEjFib8cMBx4/6hA3YLehnTViI5
BfiuGsCtX6xGYl1ByGAUbj+dKzVVz6j3naUR3CBcw6Ft2F0jbi9lt9GgIPSToaRd
wL44yEsCgYEA35Id9MIzCx9RNNJQ7rfTpXVgTlexnB6qgpYeQNG5gtwjTpG6STao
lKOyO2J08Ki0D9rgL/kmdQYkY8SsqIqYDnYWZ5SuItM2icPmkPuzR4UjS2yRcntJ
ygkrZCSq1UVM9yP0hJqnOsa6N6E4nvqFVGjD/b6uDDj7FQp0sf2IJ1cCgYEAwt0z
gH4/L0zkxGBkOahY0IziZlWnlbfUtM2S4IQwftX+SLpTkWaiTG/SSLP3EhZNgoG3
ri0q8XSXBNx7DXa+yE30ioLHPP1kCiB9gzSteCsmpAvz6A3npWjAlVztT8LvHmgf
cRPhjevGtUC8j75LQSV0cVDZbyx0g/I6b9RbZAsCgYAsI8IMT8caHA3BWwClkN+V
XC4uaXvkcd2COLcyXvW8U1AZNAZObB8ucX8/4DlsRjJ+qtz2O7+O/S50ECFyiSJq
QIA95QnYUFEFNAsydq83oJ6ORpoUuyifXd4WQHMF9k+yv4OIJqx7mz6quwbdxGJM
fv6q+1mLijhwOAdAGOknswKBgHW2M9dJ7BdBpuvton2l8UjB4j2ZPmvVqZsyl6gi
kjz8PehnDfZGfoIYBno+pGB7soTEoVWJGWjUWinomjjO436SgMdvKrXwB4+KgudQ
Wf2+a1ArnEIwrB8igTt48C3/fnzqvxuKu/UEWXvCSGW4uTL4UjfldAORYi0WlTBa
on5zAoGARRU0VdMFjECzc5uwpAwHhnpgrQKfqGzbmxSYLMH9klKPcyTorl1RlLKP
l+6sAkJoK+h5+i7SpyUlFdxn7mMLAKwI2qWWlTDvTKPpu4skkzDInTKsbDpsfP0G
m/L4VzPT9zqM2TPtcl2nHsRGwTVzWN2vATmZUL9vAEzuUWqoLdk=
-----END RSA PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
MIIDNTCCAh2gAwIBAgIUOy81lQozE/jEyfVYs/m0zBEuM6MwDQYJKoZIhvcNAQEL
BQAwKDESMBAGA1UEAwwJbWl0bXByb3h5MRIwEAYDVQQKDAltaXRtcHJveHkwHhcN
MjUwMzIzMTAyNzA3WhcNMzUwMzIzMTAyNzA3WjAoMRIwEAYDVQQDDAltaXRtcHJv
eHkxEjAQBgNVBAoMCW1pdG1wcm94eTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC
AQoCggEBAKot6NGqviqSa8tP5nD3415l5dWqqRGEUWgQeObfBX86mKqQefNQiXmB
2ruF59ODiRkS95ZKys6lSMqEd8jEtKrV4+KAqQIHq7fLiGShFlklK3hySKEhUu2r
kwCzuRwQ5/T2K86QoHYtsyt19vqTIh63vmdBUN4eFh7Sor8/6iMasptvj29HDcT6
sN0m3gW2qmcNwiSrwTDWvP4UQ7hlJUkJVUQanGBI+sR7pji9b19v4Zja9Ju7dRSM
RIGC1qZLD8XM9czu1nrvU+ocZrTJcK/V6kD7wFZ64zcEkFZZ1LE58zJwBbb5ejRT
sy4cnStJBEf+t75omrU9WnMvYhwkrL0CAwEAAaNXMFUwDwYDVR0TAQH/BAUwAwEB
/zATBgNVHSUEDDAKBggrBgEFBQcDATAOBgNVHQ8BAf8EBAMCAQYwHQYDVR0OBBYE
FMRaGr65L8Q0qEWtirGGO2uVF1P4MA0GCSqGSIb3DQEBCwUAA4IBAQCVZuc5yxtd
bppEHyG6XHtx+CuXlQIHjK1S9SknazdmJCkMB/mU5u/p3lkB3KErNA9lrRhXsYQf
X/bNWO+N9m3R+D7ed0jYtCixJEEmFvaDuHqzdbsEh+9Nd9eYKq0BnxaDDXppR9ih
8IbzXso8jb2K6C83TgEpvM0ZzoX7f6RboQLHtwytunR8GMJ93+O39QWwYR0MFJxB
h3RN1Ol/XKw4bIp8cHZ1L6UyxynZl6TsZjcP2DzOry9qws6xFx4ruLGkhxszNrty
afXF55bspcO/YVjDcqgP97Nvmm3BgeD3j4fpksAi83tSotQPmfwntPc9/yDpWzT2
WPGHnvoShcUB
-----END CERTIFICATE-----
EOF
cat <<EOF > ./certs/mitmproxy-ca-cert.cer
-----BEGIN CERTIFICATE-----
MIIDNTCCAh2gAwIBAgIUOy81lQozE/jEyfVYs/m0zBEuM6MwDQYJKoZIhvcNAQEL
BQAwKDESMBAGA1UEAwwJbWl0bXByb3h5MRIwEAYDVQQKDAltaXRtcHJveHkwHhcN
MjUwMzIzMTAyNzA3WhcNMzUwMzIzMTAyNzA3WjAoMRIwEAYDVQQDDAltaXRtcHJv
eHkxEjAQBgNVBAoMCW1pdG1wcm94eTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC
AQoCggEBAKot6NGqviqSa8tP5nD3415l5dWqqRGEUWgQeObfBX86mKqQefNQiXmB
2ruF59ODiRkS95ZKys6lSMqEd8jEtKrV4+KAqQIHq7fLiGShFlklK3hySKEhUu2r
kwCzuRwQ5/T2K86QoHYtsyt19vqTIh63vmdBUN4eFh7Sor8/6iMasptvj29HDcT6
sN0m3gW2qmcNwiSrwTDWvP4UQ7hlJUkJVUQanGBI+sR7pji9b19v4Zja9Ju7dRSM
RIGC1qZLD8XM9czu1nrvU+ocZrTJcK/V6kD7wFZ64zcEkFZZ1LE58zJwBbb5ejRT
sy4cnStJBEf+t75omrU9WnMvYhwkrL0CAwEAAaNXMFUwDwYDVR0TAQH/BAUwAwEB
/zATBgNVHSUEDDAKBggrBgEFBQcDATAOBgNVHQ8BAf8EBAMCAQYwHQYDVR0OBBYE
FMRaGr65L8Q0qEWtirGGO2uVF1P4MA0GCSqGSIb3DQEBCwUAA4IBAQCVZuc5yxtd
bppEHyG6XHtx+CuXlQIHjK1S9SknazdmJCkMB/mU5u/p3lkB3KErNA9lrRhXsYQf
X/bNWO+N9m3R+D7ed0jYtCixJEEmFvaDuHqzdbsEh+9Nd9eYKq0BnxaDDXppR9ih
8IbzXso8jb2K6C83TgEpvM0ZzoX7f6RboQLHtwytunR8GMJ93+O39QWwYR0MFJxB
h3RN1Ol/XKw4bIp8cHZ1L6UyxynZl6TsZjcP2DzOry9qws6xFx4ruLGkhxszNrty
afXF55bspcO/YVjDcqgP97Nvmm3BgeD3j4fpksAi83tSotQPmfwntPc9/yDpWzT2
WPGHnvoShcUB
-----END CERTIFICATE-----
EOF

CAUTION: Remember the location of ./certs/mitmproxy-ca-cert.cer as it will be needed in the next section.

Now we run the proxy (including predefined certificates):

docker run \
  --detach \
  --name mitmproxy \
  --rm \
  --publish 0.0.0.0:8002:8000 \
  --volume $(pwd)/certs:/home/mitmproxy/.mitmproxy \
  mitmproxy/mitmproxy:11.1.3 \
  mitmdump \
  --set mode=regular \
  --set listen_port=8000 \
  --set block_global=false

It will be running in the background on port 8002.

At this point you should be able to use the proxy, but certificates are not trusted yet. All below calls should succeed:

curl --proxy http://localhost:8002 http://index.crates.io/config.json
curl --proxy http://localhost:8002 --insecure https://index.crates.io/config.json
curl --proxy https://localhost:8002 --proxy-insecure http://index.crates.io/config.json
curl --proxy https://localhost:8002 --proxy-insecure --insecure https://index.crates.io/config.json

TIP: Use --verbose to see more details about the connection.

Trusting CA

To avoid using --(proxy-)insecure flags we need to add the CA certificate to the system's trust store.

Remember the location of ./certs/mitmproxy-ca-cert.cer? Now it is time to use it.

Instructions depend on the system, but on Arch Linux / Fedora you can trust the certificate with:

sudo trust anchor --store `./certs/mitmproxy-ca-cert.cer`

TIP: On Debian / Ubuntu you probably have to use update-ca-certificates.

At this point you should be able to use SSL proxy with HTTPS URLs:

curl --proxy https://localhost:8002 https://index.crates.io/config.json

Let Cargo Fail

Create small Rust project:

cargo new --bin my_app && cd ./my_app

Enable the use of our SSL proxy:

export CARGO_HTTP_PROXY="https://localhost:8002"

Try adding dependency:

cargo add proc-macro2@1.0.94

Boom! It fails with [60] SSL peer certificate or SSH remote key was not OK (SSL certificate problem: unable to get local issuer certificate).

Why Cargo Fails? Why Curl Works?

Now you should be curious why cargo fails even though curl works just fine.

The answer is simple: system curl has extra CA configuration!

• On Arch Linux, package is configured with --with-ca-bundle, pointing to the system trust store. Therefore updating the trust store is enough to make system curl aware of the custom CA.

• On NixOS, package is configured with --with-ca-fallback, which allows using the built-in CA store of the SSL library.

NOTE: I have not tested this with other Linux distributions, but I guess the same applies.

When libcurl is vendored in Cargo, it has no extra CA configuration that would allow discovering certificates from the system.

Workarounds / Solutions

1. Use cargo with dynamically linked libcurl - use system package (might work) or build from sources making sure that pkgconfig is able to find the library; do NOT use rustup!

2. Wait until CARGO_HTTP_PROXY_CAINFO is supported - cargo#15374 - then use it along with CARGO_HTTP_CAINFO to provide certificate path.

---

Extra Notes

Fallback of Proxy CA

curl does an interesting thing: when --proxy-capath is not set, then --capath value is used for it.

This behavior was introduced as part of curl#1257, but work has stalled.

Compiling Cargo on NixOS

On NixOS, to compile cargo from sources with dynamically linked libcurl, you have to make sure the following command succeeds:

pkg-config --modversion libcurl

In my case I had to:

• add curl.dev package, and
• update pkg-config path: export PKG_CONFIG_PATH=/nix/store/4nra0djq6vb9j7x37qwr56ac18lbbvrv-curl-8.12.1-dev/lib/pkgconfig:$PKG_CONFIG_PATH.

That allowed curl-sys to finally use the system's libcurl library instead of vendoring it from source.

NOTE: It is possible to use cargo package (it does NOT use vendored libcurl, which fixes the problem), but it conflicts with the rustup.

[ehuss]

it has no extra CA configuration that would allow discovering certificates from the system.

Cargo uses openssl-probe to try to find the certificates on the system. Are you specifically running on Nix, and is it placing the certificates in an unpredictable location, or not in one of the known locations?

[koxu1996]

@ehuss On my NixOS, when building cargo from the source openssl-probe is enabled. I added debug logs and ssl_configure does the following:

self.cainfo("/etc/pki/tls/certs/ca-bundle.crt");
self.capath("/etc/pki/tls/certs");

It correctly detects and sets the CA info/path, thus cargo works with non-SSL proxy:

$ export CARGO_HTTP_PROXY="http://localhost:8002"
$ cargo fetch
  Downloaded unicode-ident v1.0.18
  Downloaded proc-macro2 v1.0.94
  Downloaded 2 crates (100.1 KB) in 0.29s

However, no information about proxy CA was provided, thus cargo fails with SSL proxy:

$ export CARGO_HTTP_PROXY="https://localhost:8002"
$ cargo fetch
error: failed to get `proc-macro2` as a dependency of package `my-app v0.1.0 (/home/andrew/Downloads/tmp/my-app2)`

Caused by:
  download of config.json failed

Caused by:
  failed to download from `https://index.crates.io/config.json`

Caused by:
  [60] SSL peer certificate or SSH remote key was not OK (SSL certificate problem: unable to get local issuer certificate)

The following change in curl-rust would fix it:

diff --git a/src/easy/handler.rs b/src/easy/handler.rs
index e944b87..ab68b77 100644
--- a/src/easy/handler.rs
+++ b/src/easy/handler.rs
@@ -691,9 +691,11 @@ impl<H: Handler> Easy2<H> {

         if let Some(ref path) = probe.cert_file {
             let _ = self.cainfo(path);
+            let _ = self.proxy_cainfo(&format!("{}", path.display()));
         }
         if let Some(ref path) = probe.cert_dir {
             let _ = self.capath(path);
+            let _ = self.proxy_capath(path);
         }
     }

NOTE: curl does similar thing - see Fallback of Proxy CA section.

However, I still have a strong feeling that option to configure proxy certs directly in cargo should be possible - cargo#15374. Disabling verification entirely too - cargo#13460.

If I am not mistaken, cargo provided by rustup uses vendored libcurl - almost all developers are affected.

[weihanglo]

@koxu1996 Would you mind open an issue against curl crate proposing the fallback proxy CA solution?

[joshtriplett]

We talked about this in today's @rust-lang/cargo meeting, and we talked about doing this in cargo rather than in curl-rust.

[koxu1996]

@weihanglo I could have opened a PR, but I was not entirely convinced that curl-rust should include such a proxy CA fallback.

@joshtriplett Implementing the fallback directly in cargo makes sense, but I still believe there is no harm in exposing CARGO_HTTP_PROXY_CAINFO. I can come up with valid use cases if that is a requirement 😁.

[joshtriplett]

On Wed, Apr 16, 2025 at 02:14:02AM -0700, koxu1996 wrote: @joshtriplett Implementing the fallback directly in `cargo` makes sense, but I still believe there is no harm in exposing `CARGO_HTTP_PROXY_CAINFO`. I can come up with valid use cases if that is a requirement 😁.

Please do. Nobody on the team was *opposed* to it, we just wanted to make sure it was independently needed even after having the fallback in Cargo.