Server certs support (#100)

* Added support to configure server certificates for TLS
* Improved logic to set and log pod active labels as part of readiness check
* Updated names and file locations for readiness check related files to align with long-term broker file system strategy
* Renamed internal script setup-config-sync to startup-broker
* Added more default exposed ports, unified port names
* Added automation coverage for TLS configuration
* Fixed reference to Helm hub
* Updated chart version to 2.4

Closes  issues #87, #92, #94, #98

Author: bczoma

.github/workflows/build-test.yml

@@ -1,6 +1,6 @@
 name: build
 
-# Controls when the action will run. 
+# Controls when the action will run.
 on:
   # pull_request:
 
@@ -53,7 +53,10 @@ jobs:
       run: |
         REPO=$(echo ${{ secrets.BROKER_DOCKER_IMAGE_REF }} | cut -d ":" -f 1)
         TAG=$(echo ${{ secrets.BROKER_DOCKER_IMAGE_REF }} | cut -d ":" -f 2)
-        helm install my-release pubsubplus --set solace.size=dev,solace.redundancy=true,solace.usernameAdminPassword=admin,image.repository=$REPO,image.tag=$TAG
+        openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=*"
+        kubectl create secret tls test-tls --key="tls.key" --cert="tls.crt"
+        
+        helm install my-release pubsubplus --set solace.size=dev,solace.redundancy=true,tls.enabled=true,tls.serverCertificatesSecret=test-tls,solace.usernameAdminPassword=admin,image.repository=$REPO,image.tag=$TAG
         kubectl get statefulset,svc,pods,pvc,pv --show-labels
         echo "Waiting for broker to become active"
         sleep 40; kubectl describe nodes
@@ -62,19 +65,15 @@ jobs:
         until kubectl get pods --show-labels | grep pubsubplus-2 | grep -m 1 -E '1/1'; do sleep 10; done
         until kubectl get pods --show-labels | grep pubsubplus- | grep -m 1 -E 'active=true'; do sleep 10; done
         kubectl get statefulset,svc,pods,pvc,pv --show-labels
-        if [[ `kubectl get po --show-labels | grep -c "1/1"` -ne 3 ]]; then echo "Some pods are not ready!"; kubectl get po --show-labels; exit 1; fi
-        #
-        kubectl logs my-release-pubsubplus-0 | grep [.]sh
-        kubectl logs my-release-pubsubplus-1 | grep [.]sh
-        kubectl logs my-release-pubsubplus-2 | grep [.]sh
-        #
+        bash -c 'if [[ `kubectl get po --show-labels | grep -c "1/1"` -ne 3 ]]; then echo "Some pods are not ready!"; kubectl get po --show-labels; exit 1; fi'
         export url="$(kubectl get statefulset,svc,pods,pvc,pv --show-labels | grep LoadBalancer | awk '{print $4}')"; echo $url
         curl -O https://sftp.solace.com/download/SDKPERF_C_LINUX64
         tar -xvf SDKPERF_C_LINUX64
-        pubSubTools/sdkperf_c -cip=$url -mn=100000 -mr=0 -ptl=t1 -stl=t1 | grep "Total Messages"
+        pubSubTools/sdkperf_c -cip=tcp://$url:55555 -mn=10000 -mr=0 -ptl=t1 -stl=t1 | grep "Total Messages"
+        pubSubTools/sdkperf_c -cip=tcps://$url:55443 -mn=10000 -mr=0 -ptl=t1 -stl=t1 | grep "Total Messages"
         sleep 30
-        curl -sS -u admin:admin http://$url:8080/SEMP -d "<rpc><show><redundancy></redundancy></show></rpc>"
-        curl -sS -u admin:admin http://$url:8080/SEMP -d "<rpc><show><config-sync></config-sync></show></rpc>"
+        curl -k -sS -u admin:admin https://$url:1943/SEMP -d "<rpc><show><redundancy></redundancy></show></rpc>"
+        curl -k -sS -u admin:admin https://$url:1943/SEMP -d "<rpc><show><config-sync></config-sync></show></rpc>"
         if [[ -z `curl -sS -u admin:admin http://$url:8080/SEMP -d "<rpc><show><config-sync></config-sync></show></rpc>" | grep "<oper-status>Up</oper-status>"` ]] ; then echo "config-sync not up!"; exit 1; fi
         helm list
         helm delete $(helm list | grep deployed | awk '{print $1}')
@@ -125,4 +124,4 @@ jobs:
       if: ${{ always() }}
       run: |
         gcloud container clusters delete $TESTCLUSTERNAME --quiet --zone us-east4-a
-        sleep 20; gcloud compute disks list | grep gha-test | sed 1d $rpt | while read -r a b c; do gcloud compute disks delete $a --zone $b --quiet || echo "Run into issues with deleting $a, skipping"; done
+        gcloud compute disks list | grep gha-test | sed 1d $rpt | while read -r a b c; do gcloud compute disks delete $a --zone $b --quiet; done

README.md

@@ -6,11 +6,11 @@ The [Solace PubSub+ Platform](https://solace.com/products/platform/)'s [software
 
 ## Overview
 
-This document provides a quick getting started guide to install a software event broker in various configurations onto a [Kubernetes](https://kubernetes.io/docs/home/) cluster. The recommended software event broker version is 9.4 or later.
+This project is a best practice template intended for development and demo purposes. The tested and recommended Solace PubSub+ Software Event Broker version is 9.8.
 
-*Detailed* *documentation* is provided in the [Solace PubSub+ Software Event Broker on Kubernetes Documentation](docs/PubSubPlusK8SDeployment.md).
+This document provides a quick getting started guide to install a software event broker in various configurations onto a [Kubernetes](https://kubernetes.io/docs/home/) cluster.
 
-This quick start is intended mainly for development and demo purposes. Consult the [Deployment Considerations](https://github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/blob/master/docs/PubSubPlusK8SDeployment.md#pubsub-event-broker-deployment-considerations) section of the Documentation when planning your deployment.
+Detailed documentation is provided in the [Solace PubSub+ Software Event Broker on Kubernetes Documentation](docs/PubSubPlusK8SDeployment.md). Consult the [Deployment Coonsiderations](https://github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/blob/master/docs/PubSubPlusK8SDeployment.md#pubsub-event-broker-deployment-considerations) section of the Documentation when planning your deployment.
 
 This document is applicable to any platform supporting Kubernetes, with specific hints on how to set up a simple MiniKube deployment on a Linux-based machine. To view examples of other Kubernetes platforms see:
 
@@ -26,7 +26,7 @@ Solace PubSub+ Software Event Broker can be deployed in either a three-node High
 
 We recommend using the Helm tool for convenience. An [alternative method](/docs/PubSubPlusK8SDeployment.md#alternative-deployment-with-generating-templates-for-the-kubernetes-kubectl-tool) using generated templates is also provided.
 
-In this quick start we go through the steps to set up a PubSub+ Software Event Broker using [Solace PubSub+ Helm charts](//hub.helm.sh/charts/solace).
+In this quick start we go through the steps to set up a PubSub+ Software Event Broker using [Solace PubSub+ Helm charts](//artifacthub.io/packages/search?ts_query_web=solace).
 
 There are three Helm chart variants available with default small-size configurations:
 1.	`pubsubplus-dev` - recommended PubSub+ Software Event Broker for Developers (standalone) - no guaranteed performance

docs/PubSubPlusK8SDeployment.md

@@ -18,9 +18,12 @@ Contents:
       - [Creating a new storage class](#creating-a-new-storage-class)
       - [Using an existing PVC (Persistent Volume Claim)](#using-an-existing-pvc-persistent-volume-claim-)
       - [Using a pre-created provider-specific volume](#using-a-pre-created-provider-specific-volume)
-    + [Exposing the PubSub+ Event Broker Services](#exposing-the-pubsub-event-broker-services)
+    + [Exposing the PubSub+ Event Broker Services](#exposing-the-pubsub-software-event-broker-services)
       - [Using pod label "active" to identify the active event broker node](#using-pod-label-active-to-identify-the-active-event-broker-node)
-    + [The PubSub+ Docker image](#the-pubsub-docker-image)
+    + [Enabling use of TLS to access broker services](#enabling-use-of-tls-to-access-broker-services)
+      - [Setting up TLS](#setting-up-tls)
+      - [Rotating the server key](#rotating-the-server-key)
+    + [The PubSub+ Docker image](#the-pubsub-software-event-broker-docker-image)
       - [Using a public registry](#using-a-public-registry)
       - [Using private registries](#using-private-registries)
       - [Using ImagePullSecrets for signed images](#using-imagepullsecrets-for-signed-images)
@@ -242,6 +245,48 @@ This label is set by the `readiness_check.sh` script in `pubsubplus/templates/so
 - the Kubernetes service account associated with the Solace pod must have sufficient rights to patch the pod's label when the active event broker is service ready
 - the Solace pods must be able to communicate with the Kubernetes API at `kubernetes.default.svc.cluster.local` at port $KUBERNETES_SERVICE_PORT. You can find out the address and port by [SSH into the pod](#ssh-access-to-individual-message-brokers).
 
+### Enabling use of TLS to access broker services
+
+#### Setting up TLS
+
+Default deployment does not have TLS over TCP enabled to access broker services. Although the exposed `service.ports` include ports for secured TCP, only the insecure ports can be used by default.
+
+To enable accessing services over TLS a server key and certificate must be configured on the broker.
+
+It is assumed that a provider out of scope of this document will be used to create a server key and certificate for the event broker, that meet the [requirements described in the Solace Documentation](https://docs.solace.com/Configuring-and-Managing/Managing-Server-Certs.htm). If the server key is password protected it shall be transformed to an unencrypted key, e.g.:  `openssl rsa -in encryedprivate.key -out unencryed.key`.
+
+The server key and certificate must be packaged in a Kubernetes secret, for example by [creating a TLS secret](https://kubernetes.io/docs/concepts/configuration/secret/#tls-secrets). Example:
+```
+kubectl create secret tls <my-tls-secret> --key="<my-server-key-file>" --cert="<my-certificate-file>"
+```
+
+This secret name and related parameters shall be specified when deploying the PubSub+ Helm chart:
+```
+tls:
+  enabled: true    # set to false by default
+  serverCertificatesSecret: <my-tls-secret> # replace by the actual name
+  certFilename:    # optional, default if not provided: tls.crt 
+  certKeyFilename: # optional, default if not provided: tls.key
+```
+
+Note: ensure filenames are matching the files reported from running `kubectl describe secret <my-tls-secret>`.
+
+Here is an example new deployment with TLS enabled using default `certFilename` and `certKeyFilename`:
+```
+helm install my-release solacecharts/pubsubplus \
+--set tls.enabled=true,tls.serverCertificatesSecret=<my-tls-secret>
+```
+
+Important: it is not possible to update an existing deployment to enable TLS that has been created without TLS enabled, by a simply using the [modify deployment](#modifying-or-upgrading-a-deployment) procedure. In this case, for the first time, certificates need to be [manually loaded and set up](//docs.solace.com/Configuring-and-Managing/Managing-Server-Certs.htm) on each broker node. After that it is possible to use `helm upgrade` with a secret specified.
+
+#### Rotating the server key
+
+In the event the server key or certificate need to be rotated a new Kubernetes secret must be created, which may require deleting and recreating the old secret if using the same name.
+
+Next, if using the same secret name, the broker Pods need to be restarted, one at a time waiting to reach `1/1` availability before continuing on the next one: starting with the Monitor (ordinal -2), followed by the node in backup role with `active=false` label, and finally the third node. If using a new secret name, the [modify deployment](#modifying-or-upgrading-a-deployment) procedure can be used and an automatic rolling update will follow these steps restarting the nodes one at a time.
+
+Note: a pod restart will result in provisioning the server certificate from the secret again so it will revert back from any other server certificate that may have been provisioned on the broker through other mechanism.
+
 ### The PubSub+ Software Event Broker Docker image
 
 The `image.repository` and `image.tag` parameters combined specify the PubSub+ Software Event Broker Docker image to be used for the deployment. They can either point to an image in a public or a private Docker container registry. 
@@ -318,7 +363,7 @@ If other settings control `fsGroup` and `runAsUser`, e.g: when using a [PodSecur
 
 #### Securing Helm v2
 
-Using current Helm v2, Helm's server-side component Tiller must be installed in your Kubernetes environment with rights granted to manage deployments. By default, Tiller is deployed in a permissive configuration. There are best practices to secure Helm and Tiller, and they need to be applied carefully if strict security is required; for example, in a production environment.
+Using Helm v2, Helm's server-side component Tiller must be installed in your Kubernetes environment with rights granted to manage deployments. By default, Tiller is deployed in a permissive configuration. There are best practices to secure Helm and Tiller, and they need to be applied carefully if strict security is required; for example, in a production environment.
 
 [Securing your Helm Installation](//v2.helm.sh/docs/using_helm/#securing-your-helm-installation ) provides an overview of the Tiller-related security issues and recommended best practices.
 
@@ -329,6 +374,9 @@ Particularly, the [Role-based Access Control section of the Helm documentation](
 Services require [pod label "active"](#using-pod-label-active-to-identify-the-active-event-broker-node) of the serving event broker.
 * In a controlled environment it may be necessary to add a [NetworkPolicy](//kubernetes.io/docs/concepts/services-networking/network-policies/ ) to enable [required communication](#using-pod-label-active-to-identify-the-active-event-broker-node).
 
+#### Securing TLS server key and certificate
+
+Using secrets for TLS server keys and certificates follows Kubernetes recommendations, however, particularly in a production environment, additional steps are required to ensure only authorized access to these secrets following Kubernetes industry best practices, including setting tight RBAC permissions and fixing possible security holes.
 
 ## Deployment Prerequisites
 

pubsubplus/Chart.yaml

@@ -1,7 +1,7 @@
 apiVersion: v1
 description: Deploy Solace PubSub+ Event Broker Singleton or HA redundancy group onto a Kubernetes Cluster
 name: pubsubplus
-version: 2.3.0
+version: 2.4.0
 icon: https://solaceproducts.github.io/pubsubplus-kubernetes-quickstart/images/PubSubPlus.png
 maintainers:
   - name: Solace Community Forum

pubsubplus/README.md

@@ -91,6 +91,10 @@ For more ways to override default chart values, refer to [Customizing the Helm C
 | `securityContext.runAsUser`    | Specifies `runAsUser` in pod security context                                                           | set to default PubSub+ appuser id 1000001 |
 | `serviceAccount.create`        | `true` will create a service account dedicated to the deployment in the namespace                       | `true` |
 | `serviceAccount.name`          | Refer to https://helm.sh/docs/topics/chart_best_practices/rbac/#using-rbac-resources                    | Undefined |
+| `tls.enabled`                  | Enable to use TLS to access exposed broker services                                                     | `false` (not enabled) |
+| `tls.serverCertificatesSecret` | Name of the Kubernetes Secret that contains the certificates - required if TLS is enabled               | Undefined |
+| `tls.certFilename`             | Name of the Certificate file in the `serverCertificatesSecret`                                          | `tls.crt` |
+| `tls.certKeyFilename`          | Name of the Key file in the `serverCertificatesSecret`                                                  | `tls.key` |
 | `service.type`                 | How to expose the service: options include ClusterIP, NodePort, LoadBalancer                            | `LoadBalancer`                                          |
 | `service.annotations`                 | service.annotations allows to add provider-specific service annotations                          | Undefined  |
 | `service.ports`                | Define PubSub+ service ports exposed. servicePorts are external, mapping to cluster-local pod containerPorts | initial set of frequently used ports, refer to values.yaml |

pubsubplus/templates/NOTES.txt

@@ -6,6 +6,13 @@ Watch progress by running:
 
 For troubleshooting, refer to ***TroubleShooting.md***
 
+== TLS support ==
+{{- if not .Values.tls.enabled }}
+TLS has not been enabled for this deployment.
+{{- else }}
+TLS is enabled, using secret {{ .Values.tls.serverCertificatesSecret }} for server certificates configuration.
+{{- end }}
+
 == Admin credentials and access ==
 {{- if not .Values.solace.usernameAdminPassword }}
 *********************************************************************

pubsubplus/templates/service-discovery.yaml

@@ -13,18 +13,18 @@ metadata:
 spec:
   ports:
   - port: 8080
-    name: semp
+    name: tcp-semp
   - port: 8741
-    name: ha-mate-link
+    name: tcp-ha-mate-link
   - port: 8300
-    name: ha-conf-sync0
+    name: tcp-ha-conf-sync0
   - port: 8301
-    name: ha-conf-sync1
+    name: tcp-ha-conf-sync1
   - port: 8302
-    name: ha-conf-sync2
+    name: tcp-ha-conf-sync2
   clusterIP: None
   selector:
     app.kubernetes.io/name: {{ template "solace.name" . }}
     app.kubernetes.io/instance: {{ .Release.Name }}
   publishNotReadyAddresses: true
-{{- end }}
+{{- end }}