Skip to content

MultipleOpenSeachReplicas.md

Latest commit

 

History

History
175 lines (119 loc) · 7.7 KB

MultipleOpenSeachReplicas.md

File metadata and controls

175 lines (119 loc) · 7.7 KB
Raw

Deploying Multiple OpenSearch Replicas

Malcom-Helm allows for single OpenSearch node deployments or (with singleNode mode disabled) you can specify Multiple OpenSearch replicas. With the addition of the OpenSearch Security Plugin (as of Malcolm v25.06.0) all OpenSearch replica communications must now be encrypted with a common shared TLS certificate. For a multiple replica deployment this certificate must be externally created and provided to the Malcolm-Helm platform via a Kubernetes Secret prior to install. The following steps explain how to generate a TLS certificate and a Secret to hold the certificate files.

Generating a Certificate Authority and related key

Use the OpenSSL tool to generate an RSA key file named ca.key:

openssl genrsa -out ca.key 2048

Use that key to generate a new certificate:

openssl req -x509 -new -nodes \
  -key ca.key -sha256 -days 3650 \
  -subj "/CN=opensearch/OU=ca/O=Malcolm/ST=ID/C=US" \
  -out ca.crt

This should leave you with two files that we will store in a Kubernetes Secret:

$ ls -al
-rw-r--r--  1 user user 1269 Dec 14 09:27 ca.crt
-rw-------  1 user user 1704 Dec 14 09:26 ca.key

Create a Kubernetes Secret from the certificate files

You will need to know the Kubernetes namespace (if any) for your Malcom-Helm deployment so that the Secret object will be available to Malcolm-Helm at run-time. Create the Kubernetes namespace if it doesn't already exist. For this example we will use the "existingca" namespace:

kubectl create ns existingca

You should see a message that the new namespace was created successfully

$ kubectl create ns existingca
namespace/existingca created

Next create the Secret object in the appropriate namespace.

In the command below the new Secret object is named "opensearch-ca-secret". Make note of the this name as it will be used in values.yaml modifications or command line options later. The certificate and key files created in the previous step are specified for storage in the Kubernetes Secret

kubectl create secret generic opensearch-ca-secret \
  --from-file=ca.crt=./ca.crt \
  --from-file=ca.key=./ca.key \
  -n existingca

You should see a message that the Secret was created successfully

$ kubectl create secret generic opensearch-ca-secret
--from-file=ca.crt=./ca.crt
--from-file=ca.key=./ca.key
-n existingca
secret/opensearch-ca-secret created

Verify the Secret object contains the ca.crt and ca.key files:

kubectl describe secret opensearch-ca-secret -n existingca

You should see the name and namespace match what was provided above as well as a Data section listing the two Certificate Authority files.

$ kubectl describe secret opensearch-ca-secret -n existingca

  Name:         opensearch-ca-secret    
  Namespace:    existingca    
  Labels:       <none>    
  Annotations:  <none>    
  
  Type:  Opaque
  
  Data
  ====
  ca.crt:  1269 bytes
  ca.key:  1704 bytes

With the certificate information stored in the Kubernetes Secret for the appropriate namespace we are ready to customize Malcolm-Helm.

Provide the Secret object name to Malcolm-Helm

When installing Malcolm-Helm from source code you can edit the Helm Chart values.yaml "existingCASecretName" value directly:

opensearch:
singleNode: false
replicas: 3
...
existingCASecretName: "opensearch-ca-secret"

When installing from the Malcolm-Helm repository you can either provide the new Secret object name via helm install command line options. e.g.:

helm install malcolm \
  --set "opensearch.existingCASecretName=opensearch-ca-secret" \
  --set "opensearch.singleNode=false" \
  --namespace "existingca" \    
  ...

or create a values.yaml file with the secret name specified:

opensearch:    
  singleNode: false    
  replicas: 3    
  ...    
  existingCASecretName: "opensearch-ca-secret"

and pass the new values.yaml file to Helm via command line:

helm install malcolm malcolm/malcolm \
  --values ./values.yaml \
  --namespace "existingca" \

Verify Malcolm is using the provided Certificate Authority files

The OpenSearch pod logs will contain entries verifying the Secret files were recognized and used rather than generating a new certificate internally. First find the names of the running OpenSearch pods with kubectl. The pods will be named "opensearch-" with a number starting from 0.

kubectl get pods -n existingca

The number of pods should match the number of replicas set in values.yaml.

$ kubectl get pods -n existingca
...
opensearch-0 1/1 Running 0 68s
opensearch-1 1/1 Running 0 61s
opensearch-2 1/1 Running 0 55s

Messages about existing certificate files will contain the string "CA certificate". You should see Malcolm skipped generating a new certificate by searching the pod logs. Search the first OpenSearch pod (opensearch-0) logs to verify the Secret contents were used.

kubectl logs opensearch-0 -n existingca | grep "CA certificate"

You should see that Malcolm found the provided certificate.

$ kubectl logs opensearch-0 -n existingca | grep "CA certificate"
Defaulted container "opensearch-container" out of: opensearch-container, opensearch-dirinit-container (init)
CA certificate and key already exist at /usr/share/opensearch/config/certs. Skipping generation.

Each pod replica should log the same "Skipping generation" message as all OpenSeach pods now contain the same Certificate Authority files provided in the Kubernetes Secret. Below we search the second pod (opensearch-1) logs to verify the existing certificate was used.

kubectl logs opensearch-1 -n existingca | grep "CA certificate"

Again the logs show the certificate files were found and Malcolm skipped generating a new one.

$ kubectl logs opensearch-1 -n existingca | grep "CA certificate"
Defaulted container "opensearch-container" out of: opensearch-container, opensearch-dirinit-container (init)
CA certificate and key already exist at /usr/share/opensearch/config/certs. Skipping generation.

How this works in Malcolm-Helm

When the opensearch existingCASecretName value is specified Malcolm-Helm mounts the Secret's files into every OpenSearch pod at: /usr/share/opensearch/config/certs/secretmap. Malcolm has a mechanism that copies the Secret's contents up one directory to: /usr/share/opensearch/config/certs. With the custom Certificate Authority files in the proper place, each OpenSearch pod uses the provided certificate to generate server and client certificates for that pod. Since all of the OpenSearch pods leveraged the same Certificate Authority files (provided by you) trusted encryption channels can be established between the Opensearch pods satisfying the encryption requirements of the OpenSearch Security Plugin.