Handling incomplete certificate chains in Node TLS

[pimterry]

What is the problem this feature will solve?

Servers should return a complete certificate chain, which can be validated up to a trusted root.

Sadly, some don't, and instead return a chain that references an intermediate cert signed by a trusted root, but doesn't actually include the intermediate. There's also possible cases where an intermediate cert expires, and the authority has reissued a new intermediate with the same key, but the chain only contains the old intermediate.

There's a test site for this here: https://incomplete-chain.badssl.com/. You can open this in your browser just fine, but in Node:

> require('https').request('https://incomplete-chain.badssl.com/')
...
Uncaught Error: unable to verify the first certificate
    at TLSSocket.onConnectSecure (node:_tls_wrap:1679:34)
    at TLSSocket.emit (node:events:518:28)
    at TLSSocket.emit (node:domain:552:15)
    at TLSSocket._finishInit (node:_tls_wrap:1078:8)
    at ssl.onhandshakedone (node:_tls_wrap:864:12)
    at TLSWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
  code: 'UNABLE_TO_VERIFY_LEAF_SIGNATURE',

Incomplete chains like this are bad behaviour, but it's also more common than you'd think, because it works in most places. More specifically: all modern browsers (Chrome, Edge, Safari, FF) and Mac/Windows OS libraries (Secure Transport & schannel) all seem to handle this automatically.

These missing intermediates are generally handled with one a few different approaches:

• Caching intermediate certs seen elsewhere, so you can validate any subsequent certificates that reference this intermediate, even if they don't include it.
• Reading Authority Information Access (AIA) metadata from the certificates that are provided, to dynamically fetch missing intermediate certs when required.
• Preloading commonly used intermediates directly - effectively starting with a complete cache.

AFAICT Secure Transport (macOS), schannel (Windows), Chrome, Edge & Safari all use AIA fetching to handle this, while Firefox uses intermediate preloading (more details: https://wiki.mozilla.org/Security/CryptoEngineering/Intermediate_Preloading).

Python's similar discussion about this may be interesting: python/cpython#62817. There is an OpenSSL issue about this but zero activity: openssl/openssl#27016.

Currently Node has no good way to solve this problem. It's not handled automatically, but it's also not really very practical to handle manually either unless you disable TLS validation entirely and do it all yourself in userspace (not a good idea).

What is the feature you are proposing to solve the problem?

Assuming people are on board with trying to offer a solution here, there's a few initial questions:

• Is there interest in Node trying to handle this transparently for users, so everything that succeeds in a browsers succeeds in Node? Or should we just offer an API to make it easy to handle in userspace?
• How do we feel about AIA vs intermediate preloading? Firefox has gone for the latter, Python seems to be leaning that way too, but Chrome et al or the OS implementations are all on the AIA train (AFAICT).

From a quick exploration of the options, to me it looks like there's no easy way to support AIA within OpenSSL today, so AIA would imply connecting, failing, doing AIA fetching if it might help, and then connecting again. That could be done transparently in Node, or in userspace if we exposed enough cert info in errors for users to fetch & retry this themselves.

It is possible to include extra intermediate certificates with OpenSSL, by using X509_STORE_CTX_set0_untrusted to add certificates that can be used to build a chain, but which aren't actually trusted in themselves (but we don't currently use or expose this to JS anywhere). That could be used to implement preloading, caching, or the connection-retry AIA fetching approach.

Would love to hear thoughts from @nodejs/crypto.

What alternatives have you considered?

No response

[tniessen]

I typically turn to curl for debugging these issues, which does not (natively) support AIA on Linux (curl/curl#13776). I am also not sure if curl supports specifying additional untrusted certificates since --cacert appears to implicitly trust the certificate. If curl does not natively support either mechanism, then my instinct would be that Node.js should not enable either mechanism by default either.

Of course, we could still expose APIs to add untrusted certificates to OpenSSL's verification contexts. That should make it straightforward for users to follow Firefox's approach, albeit with the added difficulty of obtaining the list of intermediate certificates in the first place.

[bnoordhuis]

How do we feel about AIA vs intermediate preloading?

The former isn't really a solution because there's no guarantee node can make the outbound connection or that fetching succeeds. If the choice is between always failing reliably, or sometimes succeeding and sometimes not, failing reliably is the better choice.

Preloading: there are like a gazillion intermediate certificates out there. Maintaining that list is likely a lot of effort.

[pimterry]

If the choice is between always failing reliably, or sometimes succeeding and sometimes not, failing reliably is the better choice.

This is a good argument, I think that puts any kind of automatic AIA to bed for me as a built-in feature at least (it'd still be interesting to explore the API changes required to allow userland to support it).

Similar arguments would apply against any kind of automatic intermediate caching too imo.

Of course, we could still expose APIs to add untrusted certificates to OpenSSL's verification contexts. That should make it straightforward for users to follow Firefox's approach, albeit with the added difficulty of obtaining the list of intermediate certificates in the first place.

Preloading: there are like a gazillion intermediate certificates out there. Maintaining that list is likely a lot of effort.

I've talked to a contact at Mozilla about their solution. The intermediates they use are exported directly from the Common CA Database, and are provided by the CAs themselves. AFAICT that should cover all current intermediates for all root store trusted CAs. In effect this would cover all of the public web, everything except org-internal CAs etc.

That preload list is available from Mozilla's APIs directly at https://firefox.settings.services.mozilla.com/v1/buckets/security-state/collections/intermediates/records (1677 records right now), and each cert can be fetched in full with https://firefox-settings-attachments.cdn.mozilla.net/${record.attachment.location}. Looks like the full dataset is about 3MB (as raw PEM, uncompressed). It'd also be possible to extract the same certs from CCADB directly, as they're doing. CCADB's licensing is here and (AFAICT) would just require attribution.

I also found a blog post from 2020 when Mozilla first introduced this, discussing mechanisms and the results they saw in practice: 'unknown issuer' handshake failures dropping from ~2.2% of TLS handshakes to below 1%.

With some extra APIs on our side, users could manually build this list themselves, and add those certs as untrusted intermediates to their contexts everywhere to get similar results. Alternatively, we could do that automatically in Node, in much the same way that we use & update from Mozilla's root CA list. This issue definitely comes up at intervals and causes users problems that rapidly lead towards misuse of NODE_TLS_REJECT_UNAUTHORIZED=0 and similar (see #16336 for an example discussion) so there certainly is some benefit to eliminating that issue for everybody. All else being equal, having Node work more reliably out-of-the-box with real-world TLS seems like a good goal.

The downsides though are the extra binary weight, and the hassle of another external dependency to manage & update. Any thoughts on the tradeoffs?

[bnoordhuis]

Using CCADB (edit: at compile time) - either directly or through mozilla.com - seems like a good way forward, it just needs someone to implement it. We'll want to check the source data into git, like we do with certdata.txt

CCADB's license is CDLA-2.0 Permissive, which is acceptable, I think?

NODE_TLS_REJECT_UNAUTHORIZED=0

I really regret adding that...

Node didn't verify certificates at all back then. When I fixed that, I added the environment variable as an opt-out for existing users, but it really took a life of its own.