Using BC FIPS in AWS Lambda breaks AWS SDK clients

[mgrundie-r7]

I've recently done some work to use BC FIPS providers in our JAVA lambdas deployed in GovCloud. Local testing with LocalStack worked as expected but now that I've deployed the changes, every lambda is experiencing issues like below.

BC FIPS & Java VERSIONS

  <properties>
    <maven.compiler.source>17</maven.compiler.source>
    <maven.compiler.target>17</maven.compiler.target>
    <bctls-fips.version>2.0.19</bctls-fips.version>
    <bc-fips.version>2.0.0</bc-fips.version>
    <bcutil-fips.version>2.0.3</bcutil-fips.version>
  </properties>

  <dependencies>
    <dependency>
      <groupId>org.bouncycastle</groupId>
      <artifactId>bctls-fips</artifactId>
      <version>${bctls-fips.version}</version>
    </dependency>

    <dependency>
      <groupId>org.bouncycastle</groupId>
      <artifactId>bc-fips</artifactId>
      <version>${bc-fips.version}</version>
    </dependency>

    <dependency>
      <groupId>org.bouncycastle</groupId>
      <artifactId>bcutil-fips</artifactId>
      <version>${bcutil-fips.version}</version>
    </dependency>

How I package the BC Jars:
Use maven assembly plugin to include the unmodified jars in 'lib' folder in the root of the jar. AWS lambda environment loads these on to the class path

How I configure and start BC FIPS

The below is in a static block in the handler class. AWS lambda environment starts up the jvm then loads your app jar. The handler class is the main entry point to your app.

static {
     /*
     * LocalStack lambda environment was using SunX509, which is not available in FIPS mode,
     * not sure if this would be a problem in production
     */
     Security.setProperty("ssl.KeyManagerFactory.algorithm", "PKIX");
     Security.setProperty("ssl.TrustManagerFactory.algorithm", "PKIX");

     log.debug("Removing default security providers");

     for (var provider : Security.getProviders()) {

          if (!provider.getName().equals("SUN")) {

          log.trace("Removing provider {}", provider.getName());
          Security.removeProvider(provider.getName());
          }
     }

     CryptoServicesRegistrar.setApprovedOnlyMode(true);
     Security.insertProviderAt(new BouncyCastleFipsProvider(), 1);
     Security.insertProviderAt(new BouncyCastleJsseProvider("fips:BCFIPS"), 2);
}

Problem

I see org.bouncycastle.jsse.provider.ProvTlsClient notifyConnectionClosed spammed in the logs thousands times without any further detail and the lambda fails.

Full Logs

INIT_START Runtime Version: java:17.v11 Runtime Version ARN: arn:aws-us-gov:lambda:us-gov-west-1::runtime:12345REMOVED
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
{"timestamp":"2025-01-16T19:52:19.140+0000UTC","instant":{"epochSecond":1737057139,"nanoOfSecond":140000000},"thread":"main","level":"INFO","loggerName":"com.REMOVED.fips.FipsModeConfiguration","message":"FIPS mode is enabled. Configuring security providers.","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":1,"threadPriority":5}
Jan 16, 2025 7:52:21 PM org.bouncycastle.jsse.provider.PropertyUtils getStringSecurityProperty
INFO: Found string security property [jdk.tls.disabledAlgorithms]: SSLv3, RC4, MD5withRSA, DH keySize < 1024, EC keySize < 224, DES40_CBC, RC4_40, 3DES_EDE_CBC
Jan 16, 2025 7:52:21 PM org.bouncycastle.jsse.provider.PropertyUtils getStringSecurityProperty
INFO: Found string security property [jdk.certpath.disabledAlgorithms]: MD2, MD5, SHA1 jdkCA & usage TLSServer, RSA keySize < 1024, DSA keySize < 1024, EC keySize < 224, SHA1 usage SignedJAR & denyAfter 2019-01-01
Jan 16, 2025 7:52:21 PM org.bouncycastle.jsse.provider.DisabledAlgorithmConstraints create
WARNING: Ignoring unsupported entry in 'jdk.certpath.disabledAlgorithms': SHA1 jdkCA & usage TLSServer
Jan 16, 2025 7:52:21 PM org.bouncycastle.jsse.provider.DisabledAlgorithmConstraints create
WARNING: Ignoring unsupported entry in 'jdk.certpath.disabledAlgorithms': SHA1 usage SignedJAR & denyAfter 2019-01-01
Jan 16, 2025 7:52:21 PM org.bouncycastle.jsse.provider.PropertyUtils getBooleanSecurityProperty
INFO: Found boolean security property [keystore.type.compat]: true
Jan 16, 2025 7:52:21 PM org.bouncycastle.jsse.provider.PropertyUtils getStringSystemProperty
INFO: Found string system property [javax.net.ssl.trustStore]: /etc/pki/java/cacerts
START RequestId: 98e06ef0-abf2-4072-902f-1eb41e141663 Version: $LATEST
Jan 16, 2025 7:52:23 PM org.bouncycastle.jsse.provider.PropertyUtils getStringSystemProperty
INFO: Found string system property [javax.net.ssl.trustStore]: /etc/pki/java/cacerts
Jan 16, 2025 7:52:23 PM org.bouncycastle.jsse.provider.PropertyUtils getStringSystemProperty
INFO: Found string system property [javax.net.ssl.trustStore]: /etc/pki/java/cacerts
Jan 16, 2025 7:52:24 PM org.bouncycastle.jsse.provider.ProvTlsClient notifyConnectionClosed
INFO: [client #1 @60eb4439] disconnected from dynamodb.us-gov-west-1.amazonaws.com:443
Jan 16, 2025 7:52:24 PM org.bouncycastle.jsse.provider.ProvTlsClient notifyConnectionClosed
INFO: [client #2 @2a581202] disconnected from dynamodb.us-gov-west-1.amazonaws.com:443
Jan 16, 2025 7:52:24 PM org.bouncycastle.jsse.provider.ProvTlsClient notifyConnectionClosed
INFO: [client #3 @77f6da57] disconnected from dynamodb.us-gov-west-1.amazonaws.com:443
Jan 16, 2025 7:52:24 PM org.bouncycastle.jsse.provider.ProvTlsClient notifyConnectionClosed
INFO: [client #4 @26a7a21c] disconnected from dynamodb.us-gov-west-1.amazonaws.com:443
Jan 16, 2025 7:52:24 PM org.bouncycastle.jsse.provider.ProvTlsClient notifyConnectionClosed
INFO: [client #5 @f0eb394] disconnected from dynamodb.us-gov-west-1.amazonaws.com:443
Jan 16, 2025 7:52:24 PM org.bouncycastle.jsse.provider.ProvTlsClient notifyConnectionClosed
INFO: [client #6 @3ee2bce3] disconnected from dynamodb.us-gov-west-1.amazonaws.com:443
Jan 16, 2025 7:52:24 PM org.bouncycastle.jsse.provider.ProvTlsClient notifyConnectionClosed
INFO: [client #7 @74a42a08] disconnected from dynamodb.us-gov-west-1.amazonaws.com:443

... TRUNCATED CLIENT # went up to 6000+ 

INFO: [client #6094 @1bfddfd8] disconnected from dynamodb.us-gov-west-1.amazonaws.com:443

Additional notes
The above errors are using the aws sdk dynamodb client but In our other lambdas its happening with s3/sns/sqs/ etc. I don't think we have any lambdas the connect to non-aws services to say whether or not this is limited to the aws sdk

[mgrundie-r7]

I was setting the approved mode in code via the static block as show in my original post. I found that setting it via a VM arg resolved the problem

-Dorg.bouncycastle.fips.approved_only=true

You can do this in AWS lambdas via an environment variable

JAVA_TOOL_OPTIONS: "-Dorg.bouncycastle.fips.approved_only=true"