p11-kit-proxy.so module not being loaded?

[apteryks]

Hi,

I've packaged pkcs11-provider on GNU Guix, which is built with p11-kit support, so the default pkcs11 module should be p11-kit-proxy.so, as the following check suggests must be the case:

strings  /gnu/store/zdqi552p2y27f5418zkkgrjzmpqlf73b-pkcs11-provider-1.0/lib/ossl-modules/pkcs11.so | grep proxy
/gnu/store/nzqyc5gbw9gh0fhrcdxl1fr8qcv9hs3v-p11-kit-0.25.5/lib/p11-kit-proxy.so

I'm using the following snippet of openssl configuration, which I'm appending to the global openssl.cnf config file:

[openssl_init]
providers = provider_sect

[provider_sect]
default = default_sect
pkcs11 = pkcs11_sect

[default_sect]
activate = 1

[pkcs11_sect]
module = /gnu/store/zdqi552p2y27f5418zkkgrjzmpqlf73b-pkcs11-provider-1.0/lib/ossl-modules/pkcs11.so
activate = 1

With this, I'd expect openssl to be able to find the p11-kit trusted certificates, but i doesn't, as this test in a container where /etc is mostly empty shows:

$ guix shell --container --network openssl strace
[env]$ openssl s_client -connect example.org:443 -brief
depth=1 C = US, O = DigiCert Inc, CN = DigiCert Global G3 TLS ECC SHA384 2020 CA1
verify error:num=20:unable to get local issuer certificate
CONNECTION ESTABLISHED
Protocol version: TLSv1.3
Ciphersuite: TLS_AES_256_GCM_SHA384
Peer certificate: C = US, ST = California, L = Los Angeles, O = Internet Corporation for Assigned Names and Numbers, CN = *.example.org
Hash used: SHA256
Signature type: ECDSA
Verification error: unable to get local issuer certificate
Server Temp Key: X25519, 253 bits

The unable to get local issuer certificate is the message that suggests the system trust store is not being made available. strace also shows that p11-kit is not being loaded:

[env]$ strace -e file -s200 -f openssl s_client -connect example.org:443 -brief
execve("/gnu/store/xsb8ia221bkxvhlg89xlrjl954x68wn4-profile/bin/openssl", ["openssl", "s_client", "-connect", "example.org:443", "-brief"], 0x7ffc796e0470 /* 13 vars */) = 0
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
readlinkat(AT_FDCWD, "/proc/self/exe", "/gnu/store/xy2vphn3r3zxzcc040fk3d14rqi4l42g-openssl-3.0.8/bin/openssl", 4096) = 69
openat(AT_FDCWD, "/gnu/store/xy2vphn3r3zxzcc040fk3d14rqi4l42g-openssl-3.0.8/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/gnu/store/xy2vphn3r3zxzcc040fk3d14rqi4l42g-openssl-3.0.8/lib/libssl.so.3", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/gnu/store/xy2vphn3r3zxzcc040fk3d14rqi4l42g-openssl-3.0.8/lib/libcrypto.so.3", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/gnu/store/d69awcc5wahh71amx0dmgaimsdvvp2bg-gcc-11.4.0-lib/lib/libgcc_s.so.1", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/gnu/store/hw6g2kjayxnqi8rwpnmpraalxi0djkxc-glibc-2.39/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/gnu/store/xy2vphn3r3zxzcc040fk3d14rqi4l42g-openssl-3.0.8/share/openssl/openssl.cnf", O_RDONLY) = 3
openat(AT_FDCWD, "/gnu/store/wqsqv5vhs1adclnfzjgnf5pm9afg1ss8-pkcs11-provider-1.0/lib/ossl-modules/pkcs11.so", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/gnu/store/xy2vphn3r3zxzcc040fk3d14rqi4l42g-openssl-3.0.8/share/openssl/ct_log_list.cnf", O_RDONLY) = 3
openat(AT_FDCWD, "/gnu/store/xy2vphn3r3zxzcc040fk3d14rqi4l42g-openssl-3.0.8/share/openssl/cert.pem", O_RDONLY) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/etc/nsswitch.conf", {st_mode=S_IFREG|0444, st_size=132, ...}, 0) = 0
newfstatat(AT_FDCWD, "/", {st_mode=S_IFDIR|0755, st_size=180, ...}, 0) = 0
openat(AT_FDCWD, "/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 3
newfstatat(AT_FDCWD, "/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=61, ...}, 0) = 0
openat(AT_FDCWD, "/etc/host.conf", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/gnu/store/xy2vphn3r3zxzcc040fk3d14rqi4l42g-openssl-3.0.8/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/gnu/store/hw6g2kjayxnqi8rwpnmpraalxi0djkxc-glibc-2.39/lib/glibc-hwcaps/x86-64-v4/libnss_mdns_minimal.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/gnu/store/hw6g2kjayxnqi8rwpnmpraalxi0djkxc-glibc-2.39/lib/glibc-hwcaps/x86-64-v4/", 0x7ffd324f2f20, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/gnu/store/hw6g2kjayxnqi8rwpnmpraalxi0djkxc-glibc-2.39/lib/glibc-hwcaps/x86-64-v3/libnss_mdns_minimal.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/gnu/store/hw6g2kjayxnqi8rwpnmpraalxi0djkxc-glibc-2.39/lib/glibc-hwcaps/x86-64-v3/", 0x7ffd324f2f20, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/gnu/store/hw6g2kjayxnqi8rwpnmpraalxi0djkxc-glibc-2.39/lib/glibc-hwcaps/x86-64-v2/libnss_mdns_minimal.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/gnu/store/hw6g2kjayxnqi8rwpnmpraalxi0djkxc-glibc-2.39/lib/glibc-hwcaps/x86-64-v2/", 0x7ffd324f2f20, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/gnu/store/hw6g2kjayxnqi8rwpnmpraalxi0djkxc-glibc-2.39/lib/libnss_mdns_minimal.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/gnu/store/hw6g2kjayxnqi8rwpnmpraalxi0djkxc-glibc-2.39/lib/", {st_mode=S_IFDIR|0555, st_size=1148, ...}, 0) = 0
openat(AT_FDCWD, "/etc/gai.conf", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/gnu/store/xy2vphn3r3zxzcc040fk3d14rqi4l42g-openssl-3.0.8/share/openssl/certs/a6570d26.0", 0x7ffd324f40a0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/gnu/store/xy2vphn3r3zxzcc040fk3d14rqi4l42g-openssl-3.0.8/share/openssl/certs", {st_mode=S_IFDIR|0555, st_size=0, ...}, 0) = 0
openat(AT_FDCWD, "/gnu/store/xy2vphn3r3zxzcc040fk3d14rqi4l42g-openssl-3.0.8/share/openssl/certs", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
openat(AT_FDCWD, "/etc/localtime", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/gnu/store/xy2vphn3r3zxzcc040fk3d14rqi4l42g-openssl-3.0.8/share/openssl/certs/dd8e9d41.0", 0x7ffd324f40a0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/gnu/store/xy2vphn3r3zxzcc040fk3d14rqi4l42g-openssl-3.0.8/share/openssl/certs", {st_mode=S_IFDIR|0555, st_size=0, ...}, 0) = 0
openat(AT_FDCWD, "/gnu/store/xy2vphn3r3zxzcc040fk3d14rqi4l42g-openssl-3.0.8/share/openssl/certs", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
newfstatat(AT_FDCWD, "/gnu/store/xy2vphn3r3zxzcc040fk3d14rqi4l42g-openssl-3.0.8/share/openssl/certs/dd8e9d41.0", 0x7ffd324f40a0, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/gnu/store/xy2vphn3r3zxzcc040fk3d14rqi4l42g-openssl-3.0.8/share/openssl/certs", {st_mode=S_IFDIR|0555, st_size=0, ...}, 0) = 0
openat(AT_FDCWD, "/gnu/store/xy2vphn3r3zxzcc040fk3d14rqi4l42g-openssl-3.0.8/share/openssl/certs", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
depth=1 C = US, O = DigiCert Inc, CN = DigiCert Global G3 TLS ECC SHA384 2020 CA1
verify error:num=20:unable to get local issuer certificate
CONNECTION ESTABLISHED
Protocol version: TLSv1.3
Ciphersuite: TLS_AES_256_GCM_SHA384
Peer certificate: C = US, ST = California, L = Los Angeles, O = Internet Corporation for Assigned Names and Numbers, CN = *.example.org
Hash used: SHA256
Signature type: ECDSA
Verification error: unable to get local issuer certificate
Server Temp Key: X25519, 253 bits
+++ exited with 0 +++

How is p11-kit-proxy.so supposed to be loaded? Is it perhaps dlopen? That often fails on Guix due to to relying on the library being on the RUNPATH (Guix not following the file hierarchy standard (FHS)).

[apteryks]

Looks like it is dlopen'd in

pkcs11-provider/src/interface.c

Line 304 in 61bd6af

mctx->dlhandle = dlopen(mctx->path, P11PROV_DLOPEN_FLAGS);

, and that there should be some debug output available. I'll try to enable the debug output and see. Even if dlopen is used, my pkcs11 module should be provided as its complete absolute file name, so that shouldn't be an issue I think.

[apteryks]

Here's the debug.log content seen when invoking the earlier openssl command in a shell where export PKCS11_PROVIDER_DEBUG=file:/tmp/debug.log was run earlier:

[../pkcs11-provider-1.0/src/provider.c:1494] OSSL_provider_init(): Provided config params:
[../pkcs11-provider-1.0/src/provider.c:1504] OSSL_provider_init():   pkcs11-module-path: [none]
[../pkcs11-provider-1.0/src/provider.c:1504] OSSL_provider_init():   pkcs11-module-init-args: [none]
[../pkcs11-provider-1.0/src/provider.c:1504] OSSL_provider_init():   pkcs11-module-token-pin: [****]
[../pkcs11-provider-1.0/src/provider.c:1504] OSSL_provider_init():   pkcs11-module-allow-export: [none]
[../pkcs11-provider-1.0/src/provider.c:1504] OSSL_provider_init():   pkcs11-module-login-behavior: [none]
[../pkcs11-provider-1.0/src/provider.c:1504] OSSL_provider_init():   pkcs11-module-load-behavior: [none]
[../pkcs11-provider-1.0/src/provider.c:1504] OSSL_provider_init():   pkcs11-module-cache-pins: [none]
[../pkcs11-provider-1.0/src/provider.c:1504] OSSL_provider_init():   pkcs11-module-cache-keys: [none]
[../pkcs11-provider-1.0/src/provider.c:1504] OSSL_provider_init():   pkcs11-module-quirks: [none]
[../pkcs11-provider-1.0/src/provider.c:1504] OSSL_provider_init():   pkcs11-module-cache-sessions: [none]
[../pkcs11-provider-1.0/src/provider.c:1504] OSSL_provider_init():   pkcs11-module-encode-provider-uri-to-pem: [none]
[../pkcs11-provider-1.0/src/provider.c:1504] OSSL_provider_init():   pkcs11-module-block-operations: [none]
[../pkcs11-provider-1.0/src/provider.c:1504] OSSL_provider_init():   pkcs11-module-assume-fips: [none]
[../pkcs11-provider-1.0/src/provider.c:1523] OSSL_provider_init(): PIN not available
[../pkcs11-provider-1.0/src/provider.c:1537] OSSL_provider_init(): Export allowed
[../pkcs11-provider-1.0/src/provider.c:1556] OSSL_provider_init(): Login behavior: auto
[../pkcs11-provider-1.0/src/provider.c:1573] OSSL_provider_init(): PINs will not be cached
[../pkcs11-provider-1.0/src/provider.c:1590] OSSL_provider_init(): Key caching: in session object
[../pkcs11-provider-1.0/src/provider.c:1645] OSSL_provider_init(): No quirks
[../pkcs11-provider-1.0/src/provider.c:1664] OSSL_provider_init(): Cache Sessions: 5
[../pkcs11-provider-1.0/src/provider.c:1672] OSSL_provider_init(): Assuming FIPS token is not used
[../pkcs11-provider-1.0/src/provider.c:1681] OSSL_provider_init(): PK11-URI will not be written instead of PrivateKeyInfo
[../pkcs11-provider-1.0/src/provider.c:1723] OSSL_provider_init(): Blocked Operations: None
[../pkcs11-provider-1.0/src/provider.c:1746] OSSL_provider_init(): Load behavior: default

I'm surprised that it prints pkcs11-module-path: [none]; it should have been p11-kit-proxy.so according to the documentation/code.

[apteryks]

Hm, even with export PKCS11_PROVIDER_MODULE=/gnu/store/zdqi552p2y27f5418zkkgrjzmpqlf73b-pkcs11-provider-1.0/lib/ossl-modules/pkcs11.so, I see pkcs11-module-path: [none]. What am I missing?

[apteryks]

I thought GDB could help, but it seems it can't access the process memory:

(gdb) i b
Num     Type           Disp Enb Address            What
1       breakpoint     keep y   0x000000000001f07b in p11prov_module_new at ../pkcs11-provider-1.0/src/interface.c:214
2       breakpoint     keep y   0x000000000002f866 in OSSL_provider_init at ../pkcs11-provider-1.0/src/provider.c:1450
(gdb) c
Continuing.
Warning:
Cannot insert breakpoint 1.
Cannot access memory at address 0x1f07b
Cannot insert breakpoint 2.
Cannot access memory at address 0x2f866
Command aborted.

[apteryks]

I should probably add that I have two different copies of openssl that are being used here: one bootstrap variant called openssl-bootstrap, which I've linked pkcs11-provider to, and another openssl that has access to pkcs11-provider in its build so that it can reference it in the openssl.cnf file. Could that cause problems?

[apteryks]

I've just built pkcs11-provider as part of building openssl (I've used muon and samurai to do so), so there's now a single openssl library involved:

$ ./pre-inst-env guix size openssl
store item                                                       total    self
/gnu/store/zvlp3n8iwa1svxmwv4q22pv1pb1c9pjq-glibc-2.39              40.5    38.8  43.1%
/gnu/store/zzpbp6rr43smwxzvzd4qd317z5j7qblj-gcc-11.4.0-lib          75.6    35.0  38.9%
/gnu/store/6mk6l3czlh4kngix8rmvxvy190csj4p1-openssl-3.0.8           90.2     8.5   9.5%
/gnu/store/c2h68fw7av3jdji79v60di5padnysv6z-p11-kit-0.25.5          81.6     4.3   4.8%
/gnu/store/87z5k84hxbqs87plgwsl2v6a4j7m3k7h-bash-static-5.1.16       1.7     1.7   1.9%
/gnu/store/3jhfhxdf6v5ms10x5zmnl166dh3yhbr1-bash-minimal-5.1.16     41.5     1.0   1.1%
/gnu/store/4ss7i4n3kbswmbz1s0w1jdjdx54j5wjf-nss-certs-3.101.3        0.3     0.3   0.4%
/gnu/store/b801mrqqcsnhbr34544mlfyanzg3skfx-libtasn1-4.19.0         75.8     0.2   0.3%
/gnu/store/s6iqwc5sqjrk76kzslqc1n1wlcvfyqkw-libffi-3.4.4            75.8     0.2   0.2%
/gnu/store/s80acxbk1mv5nfb8br9260c4i42q8y3x-nss-certs                0.3     0.0   0.0%
total: 90.2 MiB

$ find /gnu/store/6mk6l3czlh4kngix8rmvxvy190csj4p1-openssl-3.0.8 -name 'pkcs11.so'
/gnu/store/6mk6l3czlh4kngix8rmvxvy190csj4p1-openssl-3.0.8/lib/ossl-modules/pkcs11.so

$ tail -n15 /gnu/store/6mk6l3czlh4kngix8rmvxvy190csj4p1-openssl-3.0.8/share/openssl/openssl.cnf

[openssl_init]
providers = provider_sect

[provider_sect]
default = default_sect
pkcs11 = pkcs11_sect

[default_sect]
activate = 1

[pkcs11_sect]
module = /gnu/store/6mk6l3czlh4kngix8rmvxvy190csj4p1-openssl-3.0.8/lib/ossl-modules/pkcs11.so
pkcs11-module-path = /gnu/store/c2h68fw7av3jdji79v60di5padnysv6z-p11-kit-0.25.5/lib/p11-kit-proxy.so
activate = 1

I'm out of ideas for now!

[apteryks]

Another thing I've tried is to add pkcs11-module-load-behavior = early to the openssl.cnf configuration file. Now, I see that the p11-kit-trust.so module is indeed loaded:

 strace -e file openssl s_client -connect example.org:443 |& grep proxy
openat(AT_FDCWD, "/gnu/store/9vq4pwblpilnvgyiax885ldrnbanvzz4-p11-kit-0.25.5/lib/p11-kit-proxy.so", O_RDONLY|O_CLOEXEC) = 3

Yet my connect test still can't validate the remote certificates. Should OpenSSL have access to the trust store to authenticate peers now that the proxy driver is loaded? I know that my p11-kit setup works because I use it successfully with gnutls, simply by building it with the --with-default-trust-store-pkcs11=pkcs11: configuration flag.

[apteryks]

I've now tried with OpenSSL 3.5 instead of 3.0.8: same result. I'm still seeing the verify error:num=20:unable to get local issuer certificate error in the TLS connection test.

[Jakuje]

With this, I'd expect openssl to be able to find the p11-kit trusted certificates, but i doesn't, as this test in a container where /etc is mostly empty shows:

The p11-kit-proxy is not used for finding trusted CAs -- it is used for loading multiple pkcs11 modules.

Actually, one of the modules in the p11-kit proxy is a trust module, which provides the list of trusted CAs, but the pkcs11-provider is not the tool to provide them to OpenSSL. OpenSSL gets them through different means as it does not have native support for pkcs11.

[apteryks]

I understand that. I want pkcs11-provider to extend OpenSSL so that it's able to load the system certificates via p11-kit, which is designed to do that, among other things (via its trust module, as you mentioned).

That should be a valid use case for pkcs11-provider, unless I'm mistaken?

[Jakuje]

The primary use case of the pkcs11-provider is to offload the cryptographic operations to external modules, generally hardware, for security, speed or other reasons. Around this all what is implemented in the pkcs11-provider revolves.

Providing the trust module to OpenSSL was never a main use case as this is solved differently in operating systems. For example in Fedora we have ca-certificates package, that provides directory structure (among other things), that OpenSSL can natively consume, which will always be more performant than going through two more external modules (pkcs11-provider, p11-kit):

https://src.fedoraproject.org/rpms/ca-certificates/blob/rawhide/f/README.openssl

It might be possible to bend the provider to do that, but as far as I know, it was never attempted before. It will likely need to implement some new APIs that would provide this information to OpenSSL.

[simo5]

While we would love to be able to expose certificates from a token as a trusted store, this is untested and OpenSSL almost certainly does not have all the needed internal glue to make it work.
This is something tat should be first discussed upstream in OPenSSL to ensure providers can provide a full trust store, and then it would require implementing the necessary facility upstream and in pkcs11-provider afterwards.

[apteryks]

I'm curious why something is missing; from my naive understanding:

• pkcs11-provider enables OpenSSL to read secrets or certificates from devices via p11-kit modules, via the standardized PKCS#11 API.
• p11-kit provides a PKCS#11 compatible trust module (/lib/pkcs11/p11-kit-trust.so) that is used to expose certificates stored on disk.

So already, technically, via these two components configured correctly, OpenSSL should have access to the p11-kit-trust.so exposed objects?

Perhaps then, the only missing bit is some logic/configuration for OpenSSL that would tell it: please retrieve your trust certificates from this specific PKCS#11 URI. For example, GnuTLS has this configuration option to use the global PKCS#11 URI:

./configure --with-default-trust-store-pkcs11=pkcs11:

[simo5]

It is possible that this already works to some extent, by configuring OpenSSL appropriately, assuming the certificate verification code can use pkcs#11 URIs in its configuration.
But I am not aware of such support yet.

However having certificates is only part of the story, another part is figuring out if they are trusted and for what. I do not know if OpenSSL attaches any metadata to their cert bundles, but the PKCS#11 3.2 spec has a whole set of new objects to deal with trust in a more fine grained manner that we do not support yet in pkcs11-provider.