S3 upload fails after enabling FIPS in OpenSSL

[parsley72]

Describe the bug

We've had the SDK working with our app fine, but we've recently added FIPS support to OpenSSL. When we run our app uploads to S3 failed.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

We expect the S3 upload to work as before.

Current Behavior

Uploads to S3 fail with the error:
InvalidDigest, Unable to parse
ExceptionName: InvalidDigest Message: The Content-MD5 you specified was invalid.

Reproduction Steps

Our code looks like this:

bool upload_image_to_s3(const char* bucket_name, const char* source_file_name, const char* s3_name) {
       Aws::S3::Model::PutObjectRequest object_request;
        const std::shared_ptr<Aws::IOStream> input_data = Aws::MakeShared<Aws::FStream>(
            "PutObjectInputStream", source_file_name, std::ios_base::in | std::ios_base::binary);
        object_request.SetBucket(bucket_name);
        object_request.SetKey(s3_name);
        object_request.SetContentType("image/jpeg");
        object_request.SetBody(input_data);

        auto put_object_outcome = _pS3Client->PutObject(object_request);
        if (!put_object_outcome.IsSuccess()) {
            auto error = put_object_outcome.GetError();
            lgr_warn("MotorolaWebRequests::upload_image_to_s3: false. %s, %s", error.GetExceptionName().c_str(),
                error.GetMessage().c_str());
            return false;
        }

Possible Solution

We fixed this in our app by calculating the MD5 ourselves then adding it to the PutObjectRequest:

        std::string strMyMD5;
        dfcCalcHashMD5File_Base64(source_file_name, strMyMD5);
        object_request.SetContentMD5(strMyMD5);

Additional Information/Context

No response

AWS CPP SDK version used

1.11.404

Compiler and Version used

gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0

Operating System and version

Ubuntu 22

[DmitriyMusatkin]

Openssl in fips mode does not support MD5 and CPP SDK enabled content-md5 calculation by default on puts. You can consider using one of the additional checksums (crc32, crc32c, sha1, sha256), which will prevent sdk from generating md5.

Im guessing the reason it was crashing with stripped openssl was due to md5 symbol being stripped out completely and cpp sdk relying on it.

[parsley72]

Which suggests that somewhere in this SDK or its dependencies it's using the legacy OpenSSL functions to generate MD5. I used the newer ones to implement this on the app side so there's no problem with MD5 in FIPS.

[DmitriyMusatkin]

I am assuming you are referring to openssl 3 md5 interface, which allows you to jump through some hoops to reenable md5 in fips mode. CPP SDK targets openssl 1.1.1 as a more common ground for crypto and we dont have too much openssl version specific code. In general, afaik using md5 in any way breaks your fips compliance and it is not something we would want to support in sdk. But we should probably tweak the messaging here to make it more clear whats failing.

[amberkushwaha]

bool upload_image_to_s3(const char* bucket_name, const char* source_file_name, const char* s3_name) {
       Aws::S3::Model::PutObjectRequest object_request;
        const std::shared_ptr<Aws::IOStream> input_data = Aws::MakeShared<Aws::FStream>(
            "PutObjectInputStream", source_file_name, std::ios_base::in | std::ios_base::binary);
        object_request.SetBucket(bucket_name);
        object_request.SetKey(s3_name);
        object_request.SetContentType("image/jpeg");
        object_request.SetBody(input_data);

        auto put_object_outcome = _pS3Client->PutObject(object_request);
        if (!put_object_outcome.IsSuccess()) {
            auto error = put_object_outcome.GetError();
            lgr_warn("MotorolaWebRequests::upload_image_to_s3: false. %s, %s", error.GetExceptionName().c_str(),
                error.GetMessage().c_str());
            return false;
        }

this code os been influenced by the main circuit file format. also remember to contribute this repository in prior file importance.but its not supported.

[csi-amolpawar]

The below fatal error is observed with latest version 1.11.530 on RHEL 9 FIPS enabled environment

s2n_init() failed: 469762137 (FIPS mode is not supported for the libcrypto)
Fatal error condition occurred in /home/user/s3demo/.build/third_party/aws/src/aws_sdk_cpp/crt/aws-crt-cpp/crt/aws-c-io/source/s2n/s2n_tls_channel_handler.c:233: 0 && "s2n_init() failed"

for both read and upload s3 object.

Note: The same code is working when FIPS disabled.

[lordgamez]

The below fatal error is observed with latest version 1.11.530 on RHEL 9 FIPS enabled environment

s2n_init() failed: 469762137 (FIPS mode is not supported for the libcrypto)
Fatal error condition occurred in /home/user/s3demo/.build/third_party/aws/src/aws_sdk_cpp/crt/aws-crt-cpp/crt/aws-c-io/source/s2n/s2n_tls_channel_handler.c:233: 0 && "s2n_init() failed"

for both read and upload s3 object.

Note: The same code is working when FIPS disabled.

Hi, I had the same issue and I found that the root cause for me was that the s2n-tls library (which is used as a submodule in AWS SDK) only allows OpenSSL version 3.0 to be used if FIPS is enabled, anything above 3.1 fails. The latest release of AWS SDK uses version 1.5.13 of s2n-tls, but fortunately this restriction is lifted and FIPS support for OpenSSL is added in the latest 1.5.15 version of s2n-tls as written in the release notes: https://github.com/aws/s2n-tls/releases/tag/v1.5.15

You can also check the last commit which removes the mentioned restriction: https://github.com/aws/s2n-tls/pull/5191/files

After upgrading the submodule to v1.5.15 the issue was solved. I was using OpenSSL 3.3.2 with the separate FIPS validated 3.0.9 version FIPS provider.

[csi-amolpawar]

@lordgamez Thank you for info. will check it and update here. thanks

[DmitriyMusatkin]

1.11.536 and up include the latest s2n

[csi-amolpawar]

@DmitriyMusatkin Thanks it is working with 1.11.537, not more aborting/crashing.

When content md5 is enabled explicitly for upload object, it returning error message Unable to parse ExceptionName: InvalidDigest Message: The Content-MD5 you specified was invalid which is quite irrelevant on the FIPS enabled env.

m_request.SetContentMD5(hash::Base64Encode(hash::CalculateMD5(*data)))