WinHttp with SSL enabled is not binding valid certificates to https requests

[anonKangaroo]

Describe the bug

On Windows 10 the S3Crt client will not send requests when SSL is enabled. The issue lies where the system is not processing certificates specifically in this case self signed ones in the local windows certification store. In effect I am struggling to load a custom cert similar to this github issue #1863.

Expected Behavior

I expected to be able to configure the tls configuration options of the S3Crt client and be able to send a request to the endpoint via https and my own self signed certification. The reproduction step more or less demonstrates what I am trying to do there might be more issues with putObject and others but right now this simple example does not work.

Current Behavior

For the first few warns they time out from my testing this is from the initialization of Aws::S3Crt::ClientConfiguration object which pings for EC2 metadata or something, this is not the subject of the bug. Search for WinHttpSyncHttpClient [47408] Send request failed: A security error occurred. This is from the request being sent. When I looked into the lower level calls this is from 'WinHttp' returned 'ERROR_WINHTTP_SECURE_FAILURE' error, which is about how the certificate provided is not good for SSL communication.

aws_sdk_2023-07-24-19.log

However, you can look in the logs and see earlier on we have the lines

[DEBUG] 2023-07-24 19:32:35.152 tls-handler [47408] static: assuming certificate is in a system store, loading now.
[INFO] 2023-07-24 19:32:35.152 pki-utils [47408] static: loading certificate at windows cert manager path 'LocalMachine\MY\7F680C430C91C55AE8CCCCE63D1043A4E61919E4'.
[DEBUG] 2023-07-24 19:32:35.152 pki-utils [47408] static: determined registry value for lookup as 131072.
[DEBUG] 2023-07-24 19:32:35.154 HttpClientFactory [47408] Initializing Http Static State

The certificate is a self signed one from OpenSSL which has been installed in the windows system store. pki-utils does check if it is in the system store so the cert does exist.

Therefore as we have a network error for the certificate and debugging into WinHttpSyncHttpClient showed there was no

 WinHttpSetOption( hRequest, 
                            WINHTTP_OPTION_CLIENT_CERT_CONTEXT,
                            (LPVOID) pCertContext, 
                            sizeof(CERT_CONTEXT) );

as seen https://learn.microsoft.com/en-us/windows/win32/winhttp/ssl-in-winhttp, then perhaps the cert is not being added to handle.

Reproduction Steps

#pragma once 
#include <iostream>
#include <aws/core/Aws.h>
#include <aws/core/utils/logging/DefaultLogSystem.h>
#include <aws/core/utils/logging/AWSLogging.h>
#include <aws/core/utils/memory/stl/AWSStringStream.h>
#include <aws/core/utils/logging/CRTLogSystem.h>
#include <aws/s3-crt/S3CrtClient.h>
#include <aws/s3-crt/model/CreateBucketRequest.h>
#include <aws/s3-crt/model/BucketLocationConstraint.h>
#include <aws/s3-crt/model/DeleteBucketRequest.h>
#include <aws/s3-crt/model/PutObjectRequest.h>
#include <aws/s3-crt/model/GetObjectRequest.h>
#include <aws/s3-crt/model/DeleteObjectRequest.h>
#include <aws/core/utils/UUID.h>

int main(int argc, char* argv[])
{
	const char *systemStoreCertPath; 
	const char *exampleCert = "CurrentUser\\MY\\A11F8A9B5DF5B98BA3508FBCA575D09570E0D2C6"; 
	systemStoreCertPath = exampleCert; 
	Aws::SDKOptions options;
	options.loggingOptions.logLevel = Aws::Utils::Logging::LogLevel::Trace;

#ifdef _WIN32
	options.ioOptions.tlsConnectionOptions_create_fn = [systemStoreCertPath]() {

		Aws::Crt::Io::TlsContextOptions tlsCtxOptions;
		tlsCtxOptions = tlsCtxOptions.InitClientWithMtlsSystemPath(systemStoreCertPath);
		Aws::Crt::Io::TlsContext tlsContext(tlsCtxOptions, Aws::Crt::Io::TlsMode::CLIENT);
		return Aws::MakeShared<Aws::Crt::Io::TlsConnectionOptions>("connection", tlsContext.NewConnectionOptions());
   };
#endif

	Aws::InitAPI(options);
	//make your SDK calls in here.
	{
		Aws::S3Crt::ClientConfiguration config;
		config.region = Aws::Region::US_EAST_1;
		config.endpointOverride = Aws::String(("YOUR IP HERE" + ":" + "PORT HERE").c_str());
		config.verifySSL = true; config.scheme = Aws::Http::Scheme::HTTPS;

		Aws::S3Crt::S3CrtClient s3CrtClient(
			Aws::Auth::AWSCredentials("USERNAME", "PASSWORD"),
			config,
			Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never,
			false
		);

		Aws::S3Crt::Model::ListBucketsOutcome outcome = s3CrtClient.ListBuckets();

		if (outcome.IsSuccess()) {
			std::cout << "All buckets under my account:" << std::endl;

			for (auto const& bucket : outcome.GetResult().GetBuckets())
			{
				std::cout << "  * " << bucket.GetName() << std::endl;
			}
			std::cout << std::endl;

		}
		else {
			std::cout << "ListBuckets error:\n" << outcome.GetError() << std::endl << std::endl;
		}
	}
	Aws::ShutdownAPI(options);
}

Provide the necessary values. You can find relevant cert from the windows store by opening PowerShell and running

Set-Location Cert:\CurrentUser\My\
Get-ChildItem

Which should show some valid certs else cd up a level and then run Get-ChildItem and then cd into a non empty store.

Possible Solution

In WinHttpSyncHttpClient.cpp add the following for either Windows or if it has winhttp client enabled for the build of the sdk.

WinHttpSyncHttpClient::WinHttpSyncHttpClient(const ClientConfiguration& config){

    //code

    m_tlsOtptions = config.tlsConnectionOptions;
   
    //code
}

void* WinHttpSyncHttpClient::OpenRequest(const std::shared_ptr<HttpRequest>& request, void* connection, const Aws::StringStream& ss) const
{
    //code 

    // WinHttpOpenRequest uses a connection handle to create a request handle
    HINTERNET hHttpRequest = WinHttpOpenRequest(connection, StringUtils::ToWString(HttpMethodMapper::GetNameForHttpMethod(request->GetMethod())).c_str(),
        wss.c_str(), nullptr, nullptr, accept, requestFlags);

    //more code 

    if (m_verifySSL) //SSL on 
    {
#ifdef _WIN32 
       //https://learn.microsoft.com/en-us/windows/win32/winhttp/ssl-in-winhttp
       //alloc store and context like in above here not sure how you all want to handle that 
        if (!WinHttpSetOption( hHttpRequest, 
                            WINHTTP_OPTION_CLIENT_CERT_CONTEXT,
                            (LPVOID) pCertContext, 
                            sizeof(CERT_CONTEXT)){
            AWS_LOGSTREAM_FATAL(GetLogTag(), "Failed to sign with cert.");
         
    }
    CertFreeCertificateContext( pCertContext );
#endif 
    // more code 
}

Additional Information/Context

This was built from the package manager vcpkg build. I also built from source with both curl enabled and disabled, and the issue was similar. Perhaps I built with curl I followed the instructions found from here: but debugging into the program showed the sdk was still using the winhttp client instead so I gave up more or less on that route. Although I should let it be noted that using the command line cURL I was able to hit the endpoint if I used a correct cert. from the windows certificate store.

AWS CPP SDK version used

1.11.65 (vcpkg build)

Compiler and Version used

Microsoft Visual Studio 2017 Version 15.9.54

Operating System and version

Windows 10 Pro 22H2

[jmklix]

Thanks for the detailed write up. Looking into the possible cause of this.

[drejx]

Hi, just asking a related question. It appears that setting the TlsConnectionOptions in Aws::SDKOptions does not apply globally, but only for S3. So for example, it doesn't affect any https calls through Aws::Http::CreateHttpRequest() etc.

Taken from the code snippet of this issue:

Aws::SDKOptions options;

options.ioOptions.tlsConnectionOptions_create_fn = [systemStoreCertPath]() {
	Aws::Crt::Io::TlsContextOptions tlsCtxOptions;
	tlsCtxOptions = tlsCtxOptions.InitClientWithMtlsSystemPath(systemStoreCertPath);
	Aws::Crt::Io::TlsContext tlsContext(tlsCtxOptions, Aws::Crt::Io::TlsMode::CLIENT);
	return Aws::MakeShared<Aws::Crt::Io::TlsConnectionOptions>("connection", tlsContext.NewConnectionOptions());
};

Aws::InitAPI(options);

• tlsConnectionOptions_create_fn ends up calling Aws::SetDefaultTlsConnectionOptions(tlsConnectionOptions)
• Which can be retrieved via Aws::GetDefaultTlsConnectionOptions()
• But only S3CrtClient actually calls Aws::GetDefaultTlsConnectionOptions()

So overriding the cert path must be done "manually" on every https call like so? :

Aws::Client::ClientConfiguration config;
config.caFile = Aws::String("/etc/ssl/certs/ca-certificates.crt");
std::shared_ptr<Aws::Http::HttpClient> httpClient = Aws::Http::CreateHttpClient(config);

std::shared_ptr<Aws::Http::HttpRequest> request = 
    Aws::Http::CreateHttpRequest(Aws::String("https://aws.amazon.com/"), Aws::Http::HttpMethod::HTTP_GET, Aws::Utils::Stream::DefaultResponseStreamFactoryMethod);		
std::shared_ptr<Aws::Http::HttpResponse> response = httpClient->MakeRequest(request);	

In my testing on aws c++ sdk v1.10.8 that's what it seems like, but would like to validate I'm not missing some other way to do it globally :)

Thanks!