Implement AIA fetching on Linux, for sites with broken cert chains

[sideshowbarker]

This implements TLS AIA-fetching support on Linux — by doing the following:

1. Install a cert-verification callback that, on a failure, records URLs from the AIA extension of any cert whose issuer is missing.
2. Make RequestServer fetch each missing intermediate on a fresh handle.
3. Parse each such intermediate, cache it, and then retry the request.

Otherwise, without this change, some sites reachable in other browsers (which already do this) are unreachable in Ladybird — because we fail to load those whose responses have missing intermediate CA certs, even when a missing intermediate is pointed at by an AIA extension. Fixes #4560.

Note

The callback is OpenSSL-specific, so it’s only installed on Linux — where curl verifies with OpenSSL. On macOS, curl verifies with Apple’s SecTrust — which fetches missing intermediates itself.

Important

On retry the fetched intermediates are only ever offered to chain-building as untrusted certs. The completed chain must still validate to a locally-trusted root; thus, this doesn’t introduce new/specious trust.

[InvalidUsernameException]

There is also a second, much older issue about AIA fetching: #4560

[shannonbooth]

I don't believe Firefox implements AIA FWIW, it preloads a bunch of disclosed intermediate certificates, and I suppose there are downsides to doing it. Not sure what Safari does, I didn't research that much. This isn't a problem on MacOS since curl uses SecTrust (I guess Safari probably also just relies on this?). Probably on windows we can use their cert stuff. I personally much prefer being able to defer to the platform like how it works today with MacOS, but the issue is supporting Linux. So I'm not sure if we want to have this be platform specific, or if we want intermediate cert caching (which I suppose has other advantages like pages loading faster but then we have to deal with things like being revoked, maintaining the distribution of that list, etc)

All this to say I think there are definitely big tradeoffs with whatever approach we take

[sideshowbarker]

I don't believe Firefox implements AIA FWIW, it preloads a bunch of disclosed intermediate certificates

Right

Not sure what Safari does, I didn't research that much

Safari just relies on what the macOS system networking does. And as far as I can see that essentially does AIA fetching.

I personally much prefer being able to defer to the platform like how it works today with MacOS, but the issue is supporting Linux.

Exactly

So I'm not sure if we want to have this be platform specific, or if we want intermediate cert caching (which I suppose has other advantages like pages loading faster but then we have to deal with things like being revoked, maintaining the distribution of that list, etc)

All this to say I think there are definitely big tradeoffs with whatever approach we take

I personally don’t have any strong opinion about it one way or the other. But if we do want something that’s not platform-specific, then that’s what this PR does. And it’s a relatively-modest amount of actual code (less then 500 lines).

That said, it does seems to me that doing something that’s platform-specific is likely to be net more complicated — and more fragile — than maintaining the implementation for it all ourselves. That’s also essentially what Chrome does, as far as I can see — though their implementation is more complicated. (Not because it absolutely needs to be, but instead because that’s generally true for anything like this that’s implemented in Chrome.)

[shannonbooth]

Yeah, I'm mostly just thinking about complexity. Does it make any sense to only enable this for platforms where we can't rely on system store?

[sideshowbarker]

Does it make any sense to only enable this for platforms where we can't rely on system store?

Yes — and after the updates I pushed today, that’s what this now does.