Make test-suite runnable under FIPS JVM

CHANGELOG.md

@@ -9,6 +9,54 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Add temporal routing processors for time-based document routing ([#18920](https://github.com/opensearch-project/OpenSearch/issues/18920))
 - The dynamic mapping parameter supports false_allow_templates ([#19065](https://github.com/opensearch-project/OpenSearch/pull/19065))
 
+- [Feature Request] Enhance Terms lookup query to support query clause instead of docId ([#18195](https://github.com/opensearch-project/OpenSearch/issues/18195))
+- Add hierarchical routing processors for ingest and search pipelines ([#18826](https://github.com/opensearch-project/OpenSearch/pull/18826))
+- Add ACL-aware routing processors for ingest and search pipelines ([#18834](https://github.com/opensearch-project/OpenSearch/pull/18834))
+- Add support for Warm Indices Write Block on Flood Watermark breach ([#18375](https://github.com/opensearch-project/OpenSearch/pull/18375))
+- FS stats for warm nodes based on addressable space ([#18767](https://github.com/opensearch-project/OpenSearch/pull/18767))
+- Add support for custom index name resolver from cluster plugin ([#18593](https://github.com/opensearch-project/OpenSearch/pull/18593))
+- Rename WorkloadGroupTestUtil to WorkloadManagementTestUtil ([#18709](https://github.com/opensearch-project/OpenSearch/pull/18709))
+- Disallow resize for Warm Index, add Parameterized ITs for close in remote store ([#18686](https://github.com/opensearch-project/OpenSearch/pull/18686))
+- Ability to run Code Coverage with Gradle and produce the jacoco reports locally ([#18509](https://github.com/opensearch-project/OpenSearch/issues/18509))
+- Extend BooleanQuery must_not rewrite to numeric must, term, and terms queries ([#18498](https://github.com/opensearch-project/OpenSearch/pull/18498))
+- [Workload Management] Update logging and Javadoc, rename QueryGroup to WorkloadGroup ([#18711](https://github.com/opensearch-project/OpenSearch/issues/18711))
+- Add NodeResourceUsageStats to ClusterInfo ([#18480](https://github.com/opensearch-project/OpenSearch/issues/18472))
+- Introduce SecureHttpTransportParameters experimental API (to complement SecureTransportParameters counterpart) ([#18572](https://github.com/opensearch-project/OpenSearch/issues/18572))
+- Create equivalents of JSM's AccessController in the java agent ([#18346](https://github.com/opensearch-project/OpenSearch/issues/18346))
+- [WLM] Add WLM mode validation for workload group CRUD requests ([#18652](https://github.com/opensearch-project/OpenSearch/issues/18652))
+- Introduced a new cluster-level API to fetch remote store metadata (segments and translogs) for each shard of an index. ([#18257](https://github.com/opensearch-project/OpenSearch/pull/18257))
+- Add last index request timestamp columns to the `_cat/indices` API. ([10766](https://github.com/opensearch-project/OpenSearch/issues/10766))
+- Introduce a new pull-based ingestion plugin for file-based indexing (for local testing) ([#18591](https://github.com/opensearch-project/OpenSearch/pull/18591))
+- Add support for search pipeline in search and msearch template ([#18564](https://github.com/opensearch-project/OpenSearch/pull/18564))
+- [Workload Management] Modify logging message in WorkloadGroupService ([#18712](https://github.com/opensearch-project/OpenSearch/pull/18712))
+- Add BooleanQuery rewrite moving constant-scoring must clauses to filter clauses ([#18510](https://github.com/opensearch-project/OpenSearch/issues/18510))
+- Add functionality for plugins to inject QueryCollectorContext during QueryPhase ([#18637](https://github.com/opensearch-project/OpenSearch/pull/18637))
+- Add support for non-timing info in profiler ([#18460](https://github.com/opensearch-project/OpenSearch/issues/18460))
+- [Rule-based auto tagging] Bug fix and improvements ([#18726](https://github.com/opensearch-project/OpenSearch/pull/18726))
+- Extend Approximation Framework to other numeric types ([#18530](https://github.com/opensearch-project/OpenSearch/issues/18530))
+- Add Semantic Version field type mapper and extensive unit tests([#18454](https://github.com/opensearch-project/OpenSearch/pull/18454))
+- Pass index settings to system ingest processor factories. ([#18708](https://github.com/opensearch-project/OpenSearch/pull/18708))
+- Add fetch phase profiling. ([#18664](https://github.com/opensearch-project/OpenSearch/pull/18664))
+- Include named queries from rescore contexts in matched_queries array ([#18697](https://github.com/opensearch-project/OpenSearch/pull/18697))
+- Add the configurable limit on rule cardinality ([#18663](https://github.com/opensearch-project/OpenSearch/pull/18663))
+- Disable approximation framework when dealing with multiple sorts ([#18763](https://github.com/opensearch-project/OpenSearch/pull/18763))
+- [Experimental] Start in "clusterless" mode if a clusterless ClusterPlugin is loaded ([#18479](https://github.com/opensearch-project/OpenSearch/pull/18479))
+- [Star-Tree] Add star-tree search related stats ([#18707](https://github.com/opensearch-project/OpenSearch/pull/18707))
+- Add support for plugins to profile information ([#18656](https://github.com/opensearch-project/OpenSearch/pull/18656))
+- Add support for Combined Fields query ([#18724](https://github.com/opensearch-project/OpenSearch/pull/18724))
+- Make GRPC transport extensible to allow plugins to register and expose their own GRPC services ([#18516](https://github.com/opensearch-project/OpenSearch/pull/18516))
+- Added approximation support for range queries with now in date field ([#18511](https://github.com/opensearch-project/OpenSearch/pull/18511))
+- Upgrade to protobufs 0.6.0 and clean up deprecated TermQueryProtoUtils code ([#18880](https://github.com/opensearch-project/OpenSearch/pull/18880))
+- Expand fetch phase profiling to multi-shard queries ([#18887](https://github.com/opensearch-project/OpenSearch/pull/18887))
+- Prevent shard initialization failure due to streaming consumer errors ([#18877](https://github.com/opensearch-project/OpenSearch/pull/18877))
+- APIs for stream transport and new stream-based search api action ([#18722](https://github.com/opensearch-project/OpenSearch/pull/18722))
+- Added the core process for warming merged segments in remote-store enabled domains ([#18683](https://github.com/opensearch-project/OpenSearch/pull/18683))
+- Streaming aggregation ([#18874](https://github.com/opensearch-project/OpenSearch/pull/18874))
+- Optimize Composite Aggregations by removing unnecessary object allocations ([#18531](https://github.com/opensearch-project/OpenSearch/pull/18531))
+- [Star-Tree] Add search support for ip field type ([#18671](https://github.com/opensearch-project/OpenSearch/pull/18671))
+- [Derived Source] Add integration of derived source feature across various paths like get/search/recovery ([#18565](https://github.com/opensearch-project/OpenSearch/pull/18565))
+- Add build-tooling to run in the FIPS environment ([#18921](https://github.com/opensearch-project/OpenSearch/pull/18921))
+- Make test-suite runnable under FIPS compliance support ([#18491](https://github.com/opensearch-project/OpenSearch/pull/18491))
 
 ### Changed
 - Add CompletionStage variants to methods in the Client Interface and default to ActionListener impl ([#18998](https://github.com/opensearch-project/OpenSearch/pull/18998))

build.gradle

@@ -419,7 +419,11 @@ gradle.projectsEvaluated {
           jvmArgs += ["-javaagent:" + project(':libs:agent-sm:agent').jar.archiveFile.get()]
         }
         if (BuildParams.inFipsJvm) {
-          task.jvmArgs += ["-Dorg.bouncycastle.fips.approved_only=true"]
+          def fipsSecurityFile = project.rootProject.file('distribution/src/config/fips_java.security')
+          task.jvmArgs += [
+              "-Dorg.bouncycastle.fips.approved_only=true",
+              "-Djava.security.properties=${fipsSecurityFile}"
+          ]
         }
       }
     }

buildSrc/src/main/groovy/org/opensearch/gradle/test/StandaloneRestTestPlugin.groovy

@@ -31,6 +31,8 @@
 package org.opensearch.gradle.test
 
 import groovy.transform.CompileStatic
+import org.gradle.api.artifacts.VersionCatalog
+import org.gradle.api.artifacts.VersionCatalogsExtension
 import org.opensearch.gradle.OpenSearchJavaPlugin
 import org.opensearch.gradle.ExportOpenSearchBuildResourcesTask
 import org.opensearch.gradle.RepositoriesSetupPlugin
@@ -92,6 +94,10 @@ class StandaloneRestTestPlugin implements Plugin<Project> {
         // create a compileOnly configuration as others might expect it
         project.configurations.create("compileOnly")
         project.dependencies.add('testImplementation', project.project(':test:framework'))
+        if (BuildParams.inFipsJvm) {
+            VersionCatalog libs = project.extensions.getByType(VersionCatalogsExtension).named("libs")
+            project.dependencies.add('testImplementation', libs.findBundle("bouncycastle").get())
+        }
 
         EclipseModel eclipse = project.extensions.getByType(EclipseModel)
         eclipse.classpath.sourceSets = [testSourceSet]

buildSrc/src/main/java/org/opensearch/gradle/test/rest/RestTestUtil.java

@@ -102,6 +102,18 @@ static void setupDependencies(Project project, SourceSet sourceSet) {
                 );
         }
 
+        if (BuildParams.isInFipsJvm()) {
+            project.getDependencies()
+                .add(
+                    sourceSet.getImplementationConfigurationName(),
+                    "org.bouncycastle:bc-fips:" + VersionProperties.getVersions().get("bouncycastle_jce")
+                );
+            project.getDependencies()
+                .add(
+                    sourceSet.getImplementationConfigurationName(),
+                    "org.bouncycastle:bctls-fips:" + VersionProperties.getVersions().get("bouncycastle_tls")
+                );
+        }
     }
 
 }

client/rest-high-level/build.gradle

@@ -36,6 +36,7 @@ apply plugin: 'opensearch.build'
 apply plugin: 'opensearch.rest-test'
 apply plugin: 'opensearch.publish'
 apply plugin: 'opensearch.rest-resources'
+apply from: "$rootDir/gradle/fips.gradle"
 
 base {
   group = 'org.opensearch.client'
@@ -66,6 +67,7 @@ dependencies {
   testImplementation "junit:junit:${versions.junit}"
   //this is needed to make RestHighLevelClientTests#testApiNamingConventions work from IDEs
   testImplementation project(":rest-api-spec")
+  testFipsRuntimeOnly libs.bundles.bouncycastle
 }
 
 tasks.named('forbiddenApisMain').configure {

client/rest/src/test/java/org/opensearch/client/RestClientBuilderIntegTests.java

@@ -56,22 +56,22 @@
 import java.security.PrivilegedAction;
 import java.security.SecureRandom;
 
+import static org.hamcrest.MatcherAssert.assertThat;
 import static org.hamcrest.Matchers.instanceOf;
 import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertThat;
 import static org.junit.Assert.fail;
 
 /**
  * Integration test to validate the builder builds a client with the correct configuration
  */
-public class RestClientBuilderIntegTests extends RestClientTestCase {
+public class RestClientBuilderIntegTests extends RestClientTestCase implements RestClientFipsAwareTestCase {
 
     private static HttpsServer httpsServer;
 
     @BeforeClass
     public static void startHttpServer() throws Exception {
         httpsServer = HttpsServer.create(new InetSocketAddress(InetAddress.getLoopbackAddress(), 0), 0);
-        httpsServer.setHttpsConfigurator(new HttpsConfigurator(getSslContext(true)));
+        httpsServer.setHttpsConfigurator(new HttpsConfigurator(new RestClientBuilderIntegTests().getSslContext(true)));
         httpsServer.createContext("/", new ResponseHandler());
         httpsServer.start();
     }
@@ -91,7 +91,6 @@ public static void stopHttpServers() throws IOException {
     }
 
     public void testBuilderUsesDefaultSSLContext() throws Exception {
-        assumeFalse("https://github.com/elastic/elasticsearch/issues/49094", inFipsJvm());
         final SSLContext defaultSSLContext = SSLContext.getDefault();
         try {
             try (RestClient client = buildRestClient()) {
@@ -118,24 +117,23 @@ private RestClient buildRestClient() {
         return RestClient.builder(new HttpHost("https", address.getHostString(), address.getPort())).build();
     }
 
-    private static SSLContext getSslContext(boolean server) throws Exception {
+    @Override
+    public SSLContext getSslContext(boolean server, String keyStoreType, SecureRandom secureRandom, String fileExtension) throws Exception {
         SSLContext sslContext;
         char[] password = "password".toCharArray();
-        SecureRandom secureRandom = SecureRandom.getInstanceStrong();
-        String fileExtension = ".jks";
 
         try (
             InputStream trustStoreFile = RestClientBuilderIntegTests.class.getResourceAsStream("/test_truststore" + fileExtension);
             InputStream keyStoreFile = RestClientBuilderIntegTests.class.getResourceAsStream("/testks" + fileExtension)
         ) {
-            KeyStore keyStore = KeyStore.getInstance("JKS");
+            KeyStore keyStore = KeyStore.getInstance(keyStoreType);
             keyStore.load(keyStoreFile, password);
-            KeyManagerFactory kmf = KeyManagerFactory.getInstance("PKIX");
+            KeyManagerFactory kmf = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
             kmf.init(keyStore, password);
 
-            KeyStore trustStore = KeyStore.getInstance("JKS");
+            KeyStore trustStore = KeyStore.getInstance(keyStoreType);
             trustStore.load(trustStoreFile, password);
-            TrustManagerFactory tmf = TrustManagerFactory.getInstance("PKIX");
+            TrustManagerFactory tmf = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
             tmf.init(trustStore);
 
             SSLContextBuilder sslContextBuilder = SSLContextBuilder.create().setProtocol(getProtocol()).setSecureRandom(secureRandom);

client/rest/src/test/java/org/opensearch/client/RestClientFipsAwareTestCase.java

@@ -0,0 +1,28 @@
+/*
+ * SPDX-License-Identifier: Apache-2.0
+ *
+ * The OpenSearch Contributors require contributions made to
+ * this file be licensed under the Apache-2.0 license or a
+ * compatible open source license.
+ */
+
+package org.opensearch.client;
+
+import javax.net.ssl.SSLContext;
+
+import java.security.SecureRandom;
+
+import static org.opensearch.client.RestClientTestCase.inFipsJvm;
+
+public interface RestClientFipsAwareTestCase {
+
+    default SSLContext getSslContext(boolean server) throws Exception {
+        String keyStoreType = inFipsJvm() ? "BCFKS" : "JKS";
+        String fileExtension = inFipsJvm() ? ".bcfks" : ".jks";
+        SecureRandom secureRandom = inFipsJvm() ? SecureRandom.getInstance("DEFAULT", "BCFIPS") : new SecureRandom();
+
+        return getSslContext(server, keyStoreType, secureRandom, fileExtension);
+    }
+
+    SSLContext getSslContext(boolean server, String keyStoreType, SecureRandom secureRandom, String fileExtension) throws Exception;
+}

client/sniffer/build.gradle

@@ -29,6 +29,7 @@
  */
 apply plugin: 'opensearch.build'
 apply plugin: 'opensearch.publish'
+apply from: "$rootDir/gradle/fips.gradle"
 
 java {
   targetCompatibility = JavaVersion.VERSION_11
@@ -47,6 +48,9 @@ dependencies {
   api "commons-codec:commons-codec:${versions.commonscodec}"
   api "commons-logging:commons-logging:${versions.commonslogging}"
   api "com.fasterxml.jackson.core:jackson-core:${versions.jackson}"
+  fipsRuntimeOnly "org.bouncycastle:bc-fips:${versions.bouncycastle_jce}"
+  fipsRuntimeOnly "org.bouncycastle:bctls-fips:${versions.bouncycastle_tls}"
+  fipsRuntimeOnly "org.bouncycastle:bcutil-fips:${versions.bouncycastle_util}"
 
   testImplementation project(":client:test")
   testImplementation "com.carrotsearch.randomizedtesting:randomizedtesting-runner:${versions.randomizedrunner}"
@@ -57,6 +61,10 @@ dependencies {
   testImplementation "net.bytebuddy:byte-buddy-agent:${versions.bytebuddy}"
 }
 
+tasks.named("dependencyLicenses").configure {
+  mapping from: /bc.*/, to: 'bouncycastle'
+}
+
 tasks.named('forbiddenApisMain').configure {
   //client does not depend on server, so only jdk signatures should be checked
   replaceSignatureFiles 'jdk-signatures'

client/sniffer/licenses/bc-fips-2.0.0.jar.sha1

@@ -0,0 +1 @@
+ee9ac432cf08f9a9ebee35d7cf8a45f94959a7ab

client/sniffer/licenses/bctls-fips-2.0.20.jar.sha1

@@ -0,0 +1 @@
+1138f7896e0d1bb0d924bc868ed2dfda4f69470e

client/sniffer/licenses/bcutil-fips-2.0.3.jar.sha1

@@ -0,0 +1 @@
+a1857cd639295b10cc90e6d31ecbc523cdafcc19

client/sniffer/licenses/bouncycastle-LICENSE.txt

@@ -0,0 +1,14 @@
+Copyright (c) 2000 - 2023 The Legion of the Bouncy Castle Inc. (https://www.bouncycastle.org)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
+documentation files (the "Software"), to deal in the Software without restriction, including without limitation
+the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software,
+and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the
+Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
+WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
+OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

client/sniffer/licenses/bouncycastle-NOTICE.txt

@@ -0,0 +1 @@
+

distribution/docker/build.gradle

@@ -295,3 +295,10 @@ subprojects { Project subProject ->
 tasks.named("composeUp").configure {
   dependsOn preProcessFixture
 }
+
+dockerCompose {
+  useComposeFiles = ['docker-compose.yml']
+  if (BuildParams.inFipsJvm) {
+    environment.put("KEYSTORE_PASSWORD", "notarealpasswordphrase")
+  }
+}

distribution/docker/docker-compose.yml

@@ -16,6 +16,7 @@ services:
        - cluster.routing.allocation.disk.watermark.high=1b
        - cluster.routing.allocation.disk.watermark.flood_stage=1b
        - node.store.allow_mmap=false
+       - "KEYSTORE_PASSWORD=${KEYSTORE_PASSWORD}"
     volumes:
        - ./build/repo:/tmp/opensearch-repo
        - ./build/logs/1:/usr/share/opensearch/logs
@@ -40,6 +41,7 @@ services:
        - cluster.routing.allocation.disk.watermark.high=1b
        - cluster.routing.allocation.disk.watermark.flood_stage=1b
        - node.store.allow_mmap=false
+       - "KEYSTORE_PASSWORD=${KEYSTORE_PASSWORD}"
     volumes:
        - ./build/repo:/tmp/opensearch-repo
        - ./build/logs/2:/usr/share/opensearch/logs

distribution/tools/keystore-cli/src/test/java/org/opensearch/bootstrap/BootstrapTests.java

@@ -71,6 +71,7 @@ public void setupEnv() throws IOException {
     }
 
     public void testLoadSecureSettings() throws Exception {
+        assumeFalse("Can't use empty password in a FIPS JVM", inFipsJvm());
         final Path configPath = env.configDir();
         final SecureString seed;
         try (KeyStoreWrapper keyStoreWrapper = KeyStoreWrapper.create()) {

distribution/tools/keystore-cli/src/test/java/org/opensearch/tools/cli/keystore/AddFileKeyStoreCommandTests.java

@@ -78,6 +78,7 @@ private void addFile(KeyStoreWrapper keystore, String setting, Path file, String
     }
 
     public void testMissingCreateWithEmptyPasswordWhenPrompted() throws Exception {
+        assumeFalse("Can't use empty password in a FIPS JVM", inFipsJvm());
         String password = "";
         Path file1 = createRandomFile();
         terminal.addTextInput("y");
@@ -86,14 +87,16 @@ public void testMissingCreateWithEmptyPasswordWhenPrompted() throws Exception {
     }
 
     public void testMissingCreateWithEmptyPasswordWithoutPromptIfForced() throws Exception {
+        assumeFalse("Can't use empty password in a FIPS JVM", inFipsJvm());
         String password = "";
         Path file1 = createRandomFile();
         execute("-f", "foo", file1.toString());
         assertSecureFile("foo", file1, password);
     }
 
     public void testMissingNoCreate() throws Exception {
-        terminal.addSecretInput(randomFrom("", "keystorepassword"));
+        var password = inFipsJvm() ? "keystorepassword" : randomFrom("", "keystorepassword");
+        terminal.addSecretInput(password);
         terminal.addTextInput("n"); // explicit no
         execute("foo");
         assertNull(KeyStoreWrapper.load(env.configDir()));
@@ -221,6 +224,7 @@ public void testIncorrectPassword() throws Exception {
     }
 
     public void testAddToUnprotectedKeystore() throws Exception {
+        assumeFalse("Can't use empty password in a FIPS JVM", inFipsJvm());
         String password = "";
         Path file = createRandomFile();
         KeyStoreWrapper keystore = createKeystore(password);