trusted_leaf_cert_file does not work as documented

[julianbrost]

I played around with TLS client certificates and found the behavior of trusted_leaf_cert_file really confusing and to be inconsistent with Caddy's documentation.

Example

First, generate two self-signed certificates to use for our tests:

openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -keyout key1.pem -out crt1.pem -sha256 -days 42 -subj '/CN=subj1' -nodes
openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -keyout key2.pem -out crt2.pem -sha256 -days 42 -subj '/CN=subj2' -nodes

Create the following Caddyfile:

{
	admin off
	local_certs
	skip_install_trust
	http_port 8000
}

localhost:8001 {
	tls {
		client_auth {
			trusted_ca_cert_file crt1.pem
		}
	}
}

localhost:8002 {
	tls {
		client_auth {
			trusted_leaf_cert_file crt2.pem
		}
	}
}

localhost:8003 {
	tls {
		client_auth {
			mode require
			trusted_leaf_cert_file crt2.pem
		}
	}
}

localhost:8004 {
	tls {
		client_auth {
			trusted_ca_cert_file crt1.pem
			trusted_leaf_cert_file crt2.pem
		}
	}
}

Note that in all cases where mode it is not explicitly set, it defaults to require_and_verify as at least one CA or leaf certificate was specified.

And start Caddy:

caddy run --config Caddyfile

Expected behavior

The documentation states the following:

Multiple trusted_* directives may be used to specify multiple CA or leaf certificates. Client certificates which are not listed as one of the leaf certificates or signed by any of the specified CAs will be rejected according to the mode.

Therefore, I'd expect crt1 to be accepted on each port that has it listed as a trusted CA and crt2 on each port that has it listed as a trusted leaf. In addition, when more require is set, I'd expect every certificate to be accepted.

Actual behavior

trusted_ca_cert_file crt1.pem

Only the specified crt1 works, as expected:

$ curl -ki --cert crt1.pem --key key1.pem https://localhost:8001 
HTTP/2 200 
server: Caddy
content-length: 0
date: Mon, 10 Jan 2022 20:25:25 GMT

$ curl -ki --cert crt2.pem --key key2.pem https://localhost:8001 
curl: (56) OpenSSL SSL_read: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate, errno 0

trusted_leaf_cert_file crt2.pem

Neither certificate works, I'd expect crt2 to work:

$ curl -ki --cert crt1.pem --key key1.pem https://localhost:8002
curl: (56) OpenSSL SSL_read: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate, errno 0
$ curl -ki --cert crt2.pem --key key2.pem https://localhost:8002
curl: (56) OpenSSL SSL_read: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate, errno 0

mode require + trusted_leaf_cert_file crt2.pem

Only crt2 works, even though the configuration explicitly states to require but not verify a certificate:

$ curl -ki --cert crt1.pem --key key1.pem https://localhost:8003
curl: (56) OpenSSL SSL_read: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate, errno 0
$ curl -ki --cert crt2.pem --key key2.pem https://localhost:8003
HTTP/2 200 
server: Caddy
content-length: 0
date: Mon, 10 Jan 2022 20:25:48 GMT

trusted_ca_cert_file crt1.pem + trusted_leaf_cert_file crt2.pem

Again, neither certificate works, I'd expect both to work:

$ curl -ki --cert crt1.pem --key key1.pem https://localhost:8004
curl: (56) OpenSSL SSL_read: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate, errno 0
$ curl -ki --cert crt2.pem --key key2.pem https://localhost:8004
curl: (56) OpenSSL SSL_read: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate, errno 0

Reading code and docs

For trusted_ca_cert_file, Caddy populates a x509.CertPool and sets it to the ClientCAs attribute of tls.Config:

caddy/modules/caddytls/connpolicy.go

Lines 361 to 379 in c48fadc

	// enforce CA verification by adding CA certs to the ClientCAs pool
	if len(clientauth.TrustedCACerts) > 0 || len(clientauth.TrustedCACertPEMFiles) > 0 {
	caPool := x509.NewCertPool()
	for _, clientCAString := range clientauth.TrustedCACerts {
	clientCA, err := decodeBase64DERCert(clientCAString)
	if err != nil {
	return fmt.Errorf("parsing certificate: %v", err)
	}
	caPool.AddCert(clientCA)
	}
	for _, pemFile := range clientauth.TrustedCACertPEMFiles {
	pemContents, err := os.ReadFile(pemFile)
	if err != nil {
	return fmt.Errorf("reading %s: %v", pemFile, err)
	}
	caPool.AppendCertsFromPEM(pemContents)
	}
	cfg.ClientCAs = caPool
	}

For trusted_leaf_cert_file, Caddy sets the VerifyPeerCertificate attribute of tls.Config to a custom function checking the certificates against the list of specified leaf certificates:

caddy/modules/caddytls/connpolicy.go

Lines 381 to 394 in c48fadc

	// enforce leaf verification by writing our own verify function
	if len(clientauth.TrustedLeafCerts) > 0 {
	clientauth.trustedLeafCerts = []*x509.Certificate{}
	for _, clientCertString := range clientauth.TrustedLeafCerts {
	clientCert, err := decodeBase64DERCert(clientCertString)
	if err != nil {
	return fmt.Errorf("parsing certificate: %v", err)
	}
	clientauth.trustedLeafCerts = append(clientauth.trustedLeafCerts, clientCert)
	}
	// if a custom verification function already exists, wrap it
	clientauth.existingVerifyPeerCert = cfg.VerifyPeerCertificate
	cfg.VerifyPeerCertificate = clientauth.verifyPeerCertificate
	}

According to the documentation, VerifyPeerCertificate is checked in addition to any other default verification:

VerifyPeerCertificate, if not nil, is called after normal certificate verification by either a TLS client or server. It receives the raw ASN.1 certificates provided by the peer and also any verified chains that normal processing found. If it returns a non-nil error, the handshake is aborted and that error results.
If normal verification fails then the handshake will abort before considering this callback. If normal verification is disabled by setting InsecureSkipVerify, or (for a server) when ClientAuth is RequestClientCert or RequireAnyClientCert, then this callback will be considered but the verifiedChains argument will always be nil.

So if I understand correctly, when setting only trusted_leaf_cert_file but neither specifying trusted_ca_cert_file nor mode require, it is impossible to use any certificate to connect to the server.

[julianbrost]

For completeness: I tested using version 2.4.6-1 of the Arch Linux package. But given that I could explain the behavior using the current master code, that shouldn't be relevant.

[mholt]

Hey, thanks for the detailed issue complete with investigation. You might be onto something, I or someone else will look at this soon when we have a chance!

[julianbrost]

I'm just wondering what's the best way going forward here. Given that Caddy currently checks certificates for more constraints than documented, simply changing the implementation to match the documentation might put existing configurations at risk.

Also, I don't fully understand the purpose of trusted_leaf_cert_file: given that Go also accepts leaf certificates in ClientCAs¹, even if they have X509v3 Basic Constraints CA:FALSE set. So for my config, I just went with using trusted_ca_cert_file instead. The only difference I see between trusted_ca_cert_file and trusted_leaf_cert_file is that you could put a certificate that's technically a CA in the latter but only accept that certificate itself and not any child certificates. But I'm not sure if that case is of any practical relevance.

__
¹ I think most CA options would better be called trust anchors as most of them happily accept leaf certificates as well.

[francislavoie]

Should we deprecate trusted_leaf_cert_file for the next release, and maybe link to this issue in the warning message? Maybe we could collect information about legitimate usecases of this option from users if there any, and outright remove it in a later version if there's no feedback from users saying they need it.

[mholt]

you could put a certificate that's technically a CA in the latter but only accept that certificate itself and not any child certificates.

We have had people request this, which is why I implemented it. I am surprised that you can put leaf certs in ClientCAs though. 🤔

@francislavoie Maybe, that's definitely an option.

[julianbrost]

We have had people request this, which is why I implemented it.

What have they asked for? Allow specifying trusted self-signed certificates in addition/as an alternative to a CA?

I am surprised that you can put leaf certs in ClientCAs though.

One thing I noticed that might make a difference: if you do this, the CNs of these certificates will be leaked in the TLS handshake, openssl s_client will show then after "Acceptable client certificate CA names".

[francislavoie]

I realized @Gr33nbl00d might have some experience with this. What do you make of this issue?

I forgot that we had an open PR that touches leaf cert validation.

[Gr33nbl00d]

I dont really get what you are trying there?

I'm just wondering what's the best way going forward here. Given that Caddy currently checks certificates for more constraints than documented, simply changing the implementation to match the documentation might put existing configurations at risk.

Also, I don't fully understand the purpose of trusted_leaf_cert_file: given that Go also accepts leaf certificates in ClientCAs¹, even if they have X509v3 Basic Constraints CA:FALSE set. So for my config, I just went with using trusted_ca_cert_file instead. The only difference I see between trusted_ca_cert_file and trusted_leaf_cert_file is that you could put a certificate that's technically a CA in the latter but only accept that certificate itself and not any child certificates. But I'm not sure if that case is of any practical relevance.

__ ¹ I think most CA options would better be called trust anchors as most of them happily accept leaf certificates as well.

You are right, the go crypto implementation is not fully RFC compliant in multiple cases this is one of these cases, i think they simply do not check for the defined cert usage, i am not even sure they check that the client cert has been defined for client authentication usage. I think you can also use a server cert as client cert.

Like you wrote we should see it more as trust anchors. But i think it is not such a big deal.
Would be possible to check by caddy during config initialization to report if a CA cert is used which is actually not a CA cert,
however i am not sure this is so important as it is a clear misconfiguration.

[Gr33nbl00d]

We have had people request this, which is why I implemented it.

What have they asked for? Allow specifying trusted self-signed certificates in addition/as an alternative to a CA?

I am surprised that you can put leaf certs in ClientCAs though.

One thing I noticed that might make a difference: if you do this, the CNs of these certificates will be leaked in the TLS handshake, openssl s_client will show then after "Acceptable client certificate CA names".

The use case would be really important, why someone wanted this.

I was not completly aware of the documentation of this parameter and simply read the code while implementing my CRL feature.

I thought that it is an additional check so an "AND" not an "OR" :)
Actually for that purpose it would work:
If i want to allow certain ports only to specific leaf certificates not to all leaf certs signed by the ca.
Like a whitelist, but if this was not the target, at least that is what the documentation says it will not work like expected.

I did not fully analyse this but asuming your analysis was correct and this code was implemented as an alternative way to ca certs verification,
It would mean that this could be impossible to fix without implementing the whole handshake on our own.

Funny enough i even had the idea of doing so at some point because there are multiple issues within the handshake :)
But i think if even possible we should stick with the default implementation (which hopefully gets enhanced at some point in time) ;)

[julianbrost]

So what I was trying to do in the first place is to somewhat force a SSH authorized_keys like trust model onto TLS. Since I won't have more than like 5 certificates in total, I decided it was easier to just use self-signed certificates and deploy 5 certificates everywhere rather than bothering to maintain a CA (bonus: this allow me to trivially revoke certificates by simply redeploying my config with one remove, whereas otherwise I'd have to bother with maintaining a CRL and figuring out how to use it with every piece of software).

Sure, that's not how X.509 was intended but at least the documentation looks like someone had something similar in mind. I wouldn't mind if the leaf cert options were documented as additional restrictions.

Actually for that purpose it would work:
If i want to allow certain ports only to specific leaf certificates not to all leaf certs signed by the ca.
Like a whitelist, but if this was not the target, at least that is what the documentation says it will not work like expected.

Wouldn't you get very similar behavior by just leaving out the CA certificate assuming that you validate all leaf certificates against the CA before configuring them? Should only make a difference in two scenarios: you want additional revocation checks for the certificate (looks like this might be the purpose of your PR) or you have a client that relies on the list of CAs sent by the server when requesting a client certificate.

It would mean that this could be impossible to fix without implementing the whole handshake on our own.

I think the "only" thing that has to reimplemented would be certificate verification. mode require, no CA but leaf certificates works right now, so the verification function would just have to additionally check against a trust store which should be fairly easy in Go: https://pkg.go.dev/crypto/x509#Certificate.Verify

[Gr33nbl00d]

Wouldn't you get very similar behavior by just leaving out the CA certificate assuming that you validate all leaf certificates against the CA before configuring them? Should only make a difference in two scenarios: you want additional revocation checks for the certificate (looks like this might be the purpose of your PR) or you have a client that relies on the list of CAs sent by the server when requesting a client certificate.

It would be similar as long as the client trusts the server cert directly. However if the the server cert was signed by a ca cert And the client trusts the rootca which signed the ca cert it would need the still the ca cert from the server in order to build a trust chain correctly. So there is a slight difference :)

I think the "only" thing that has to reimplemented would be certificate verification. mode require, no CA but leaf certificates works right now, so the verification function would just have to additionally check against a trust store which should be fairly easy in Go: https://pkg.go.dev/crypto/x509#Certificate.Verify

Not sure without diving deeper into this topic. I am not sure if the chain is build correctly in that case.
I mean the chain building is anyway broken in go crypto and does not work for all possibilites the RFC defines.
The certificates in the chain refer to each other so you can identify the parent (ca cert of the client cert).
In best case it is done via serial number but this is not a must. (See RFC 5280 section 4.2.1.1)

It could also be that it is only connected by issuer name and if you now have multiple CA certificates from the same ca you can hopefully identify the right one by the algorithm but if both certs have the same algorithm you have to try both certificates and if the validation succeeds you know you have the right one. I guess this is a rare case and i guess it is not supported by go anyway and rarely used in real world. In most cases serialnumber is set.