[API Proposal]: Flexible OpenSSL provider support

[JustAnotherSwCoderAndTester]

Background and motivation

.NET 9 added support for OpenSSL 3.x providers:

• Add an easier way of opening named keys via OpenSSL #55356
• OpenSSL provider support  #89167

This is helpful but I learned the hard way, it has some limitations. A provider might implement decoders/encoders which allows to load/store additional file types. However, the basic file handling and finding appropriate decoders/encoders is done as part of the default provider. The current API only allows to load a single specific provider.

To give a concrete example of what currently cannot be done with the .NET API: The tpm2 provider supports direct TPM keys (via the handle URI prefix) but also TSS2 files (a key file encrypted with an TPM key, useful since a TPM has limit key storage). Details can be found here:
https://github.com/tpm2-software/tpm2-openssl/blob/master/docs/keys.md#loading-a-private-key
However, TSS2 files do not work since this would require loading the default provider along with tpm2.

Thus, I would suggest to provide an overload for the method OpenKeyFromProvider which supports a list of providers and to give full flexibility also one for the propertyQuery. The last one helps OpenSSL to choose the right provider for individual operations.

Implementation would be straight-forward and end up in:
https://github.com/dotnet/runtime/blob/5f97ddfff8074355964e6d13a690c236dd818050/src/native/libs/System.Security.Cryptography.Native/pal_evp_pkey.c
The single OSSL_PROVIDER_load can be replaced with a loop of several calls. And OSSL_STORE_open_ex has already the necessary property query parameter (currently a NULL is passed).

I tested this and directly in C# using the appropriate DllImports.

What also speaks for this API extension: OpenSSL engines are deprecated. However, for TPM based TSS key files there is no migration path. With OpenSSL 1.x and the tpm2tss engine one could use SafeEvpPKeyHandle.OpenPrivateKeyFromEngine in .NET. As explained above this is not possible wich a switch to providers at the moment.

API Proposal

namespace System.Security.Cryptography
{
    public partial class SafeEvpPKeyHandle
    {
        // existing OpenSSL Providers API (new plugin model for OSSL 3.0+)
        public static SafeEvpPKeyHandle OpenKeyFromProvider(string providerName, string keyUri) => throw null;

        // more flexible overload which supports loading multiple providers and a provider query string
        public static SafeEvpPKeyHandle OpenKeyFromProvider(IEnumerable<string> providerList, string keyUri) => throw null;
        public static SafeEvpPKeyHandle OpenKeyFromProvider(IEnumerable<string> providerList, string keyUri, string propertyQuery) => throw null;
    }
}

API Usage

byte[] data = ...;

// For TPM settings refer to provider documentation you're using, i.e. https://github.com/tpm2-software/tpm2-openssl/tree/master
// specific values are just example
string[] providers = { "default", "tpm2" };
const string propQuery = "?provider!=tpm2";

using (SafeEvpPKeyHandle priKeyHandle = SafeEvpPKeyHandle.OpenKeyFromProvider(providers, "file:/path/to/key.tss", propQuery))
using (ECDsa ecdsaPri = new ECDsaOpenSsl(priKeyHandle))
{
    byte[] signature = ecdsaPri.SignData(data, HashAlgorithmName.SHA256);
    // do stuff with signature
    // note: tpm2 does not allow Verify operations, public key needs to be exported and re-imported into new instance
}

Alternative Designs

I currently have not use case that requires loading more than 2 providers and one being the default one. So one option could be to add a boolean flag as a third parameter to the existing API function, stating whether the default provider should be loaded and setting it to false (so that the current behaviour does not change). But why not give full flexibility and avoid further change/extension for future use cases?

Risks

I have designed the new functionality as an extension to the existing one without any breaking change.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-security, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

[bartonjs]

Does the ienumerable with a single common query parameter map to anything in normal OSSL usage? It feels a lot like "I don't know what I want, so try everything"

[JustAnotherSwCoderAndTester]

Well, what is normal OSSL usage? Most common probably is OSSL without any provider. But details may vary: some might need legacy provider, some use FIPS and as explained regarding the tpm2 provider: it dependds whether the current capabilities of the .NET API are sufficient.

Is the API suggestion too general/broad for you? As said, for my use case I would be fine with a flag to say also load the default provider.

One could also address this from a different perspective: usually you have an appropriate openssl config file for your system and this includes which providers to load. However, the .NET LoadKeyFromProvider creates a new OSSL library context and this means even the default openssl config is not respected. One could look for the default config file (if there is one) and use OSSL_LIB_CTX_load_config to apply it to the context:
https://docs.openssl.org/3.4/man3/OSSL_LIB_CTX/
Then one one would have to specify a provider at all, not even the config file if we include a mechanism to respect OPENSSL_CONF environment variable. This moves all the details away from .NET

[bartonjs]

Ah, so this is less "try all of these providers", but "load all of these providers into the same LIB_CTX first".

Since the underlying OpenSSL API is "load this URI from this LIB_CTX", I guess it is fairly canonical.

@krwq Do you forsee any problems, based on your experience with provider loading?

[JustAnotherSwCoderAndTester]

Ah, so this is less "try all of these providers", but "load all of these providers into the same LIB_CTX first".

Yes, exactly!

And the implementation CryptoNative_LoadKeyFromProvider creates its own library context, which means loading the provider(s) should not affect anything else within the .NET application that uses OSSL.

[JustAnotherSwCoderAndTester]

@bartonjs @krwq is there any update on this or any information that is required to proceed?

[JustAnotherSwCoderAndTester]

@bartonjs @krwq Could you try to bring this further in the API Review Process? Would be important for us regarding .NET 10

[Nextra]

For whatever it is worth, I will +1 this proposal.

Using persisted keys via handles works, but wastes precious storage space inside the TPM. Using TSS2 PEM files is the better option, but impossible with how .NET currently implements/exposes the API.

[krwq]

I find the list of providers API reasonable although I think 99% of the time the second one will be default but more flexible API doesn't seem to hurt - there is still artificial restriction we've added which returns a single key - do you think we should consider also returning list of EVP_PKEYs? (IMO one sounds more reasonable as I can't think of scenario with multiple).

For property query - would that be only OSSL_STORE_open_ex or would we need to pass that in to all APIs using EVP_PKEY? If just OSSL_STORE_open_ex then I'm ok with this but if we need to now review entire code to find all places which maybe have prop query in one of the overloads this sounds a bit like a whack-a-mole as something will very likely get missed (especially this is not trivial to test).

Could you provide me some examples (other than PEM-TSS) how I could test this? I found property query a bit tricky as most of the time default (NULL) does the right thing already.

[JustAnotherSwCoderAndTester]

Yes, I agree most of the time the second provider will be default.

Regarding multiple EVP_PKEY: I also can't think of any scenario. Maybe one could combine private and public key and return both? For me it is fine if this restriction still applies. The next thing is: if we allow to return several keys, we probably should return metadata like the key type (OSSL_STORE_INFO_get_type). Furthermore, the public key is someething that does not have to be protected, so we would one add the complexity to protect it via tpm/pkcs#11 or something like that?

The property query is only necesarry for OSSL_STORE_open_ex . You have to create a new openssl context with OSSL_LIB_CTX_new and then pass this context reference and the property query string to OSSL_STORE_open_ex . This means loading the providers and setting the property query will only affect this single operation and nothing else by design. I agree that eversthing else would be complicated to test.

I have been looking around in the internet but you hardly find property query usage for openssl. pkc#11 provider is another example where it works the same way as with tpm. You find some examples for the FIPS provider but this is more related to algorithm selection and I guess not helpful for this scenario where we only load keys.