Kubernetes QS v3.2.0 Release (#117)

* Support for PodDisruptionBudget for Broker nodes in High Availability mode - Resolves #107
* Readiness check reliability improvements
* Config-sync improvements in High Availability mode
* Readiness check clearer logging

Author: LewisKSaint

.github/workflows/build-test.yml

@@ -18,7 +18,6 @@ jobs:
         echo "TESTCLUSTERNAME=k8s-gha-test-$(date +%s)" >> $GITHUB_ENV
         echo "TESTRUNBRANCH=${GITHUB_REF##*/}" >> $GITHUB_ENV
         #
-        sudo gem update --system 3.0.6
         sudo gem install yaml-lint
         sudo snap install kubectl --classic
         kubectl version --client
@@ -30,7 +29,7 @@ jobs:
       uses: actions/checkout@v2
 
     - name: Set up Cloud SDK
-      uses: google-github-actions/setup-gcloud@v0
+      uses: google-github-actions/setup-gcloud@v0.6.0
       with:
         project_id: ${{ secrets.GCP_PROJECT_ID }}
         service_account_key: ${{ secrets.GCP_SA_KEY }}
@@ -49,10 +48,13 @@ jobs:
 
     - name: Setup K8s env in GKE
       run: |
+        gcloud components install gke-gcloud-auth-plugin --quiet
+        gcloud components update
+        export USE_GKE_GCLOUD_AUTH_PLUGIN=True
         mkdir gke_test; pushd gke_test
         wget https://raw.githubusercontent.com/SolaceProducts/solace-gke-quickstart/master/scripts/create_cluster.sh
         chmod +x create_cluster.sh
-        ./create_cluster.sh -z us-east4-a,us-east4-b,us-east4-c -c $TESTCLUSTERNAME -m e2-standard-2
+        ./create_cluster.sh -z us-east4-a,us-east4-b,us-east4-c -c $TESTCLUSTERNAME -i ubuntu_containerd -m e2-standard-4
         gcloud container clusters get-credentials $TESTCLUSTERNAME --zone us-east4-a --project capable-stream-180018
         popd
         kubectl get statefulset,svc,pods,pvc,pv
@@ -74,7 +76,7 @@ jobs:
         TAG=latest
         openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=*"
         kubectl create secret tls test-tls --key="tls.key" --cert="tls.crt"
-        helm install my-release pubsubplus --set solace.size=dev,solace.redundancy=true,solace.podModifierEnabled=true,tls.enabled=true,tls.serverCertificatesSecret=test-tls,solace.usernameAdminPassword=admin,image.repository=$REPO,image.tag=$TAG
+        helm install my-release pubsubplus --set solace.size=dev,solace.redundancy=true,solace.podDisruptionBudgetForHA=true,solace.podModifierEnabled=true,tls.enabled=true,tls.serverCertificatesSecret=test-tls,solace.usernameAdminPassword=admin,image.repository=$REPO,image.tag=$TAG
         kubectl get statefulset,svc,pods,pvc,pv --show-labels
         echo "Waiting for broker to become active"
         sleep 40; kubectl describe nodes
@@ -101,8 +103,11 @@ jobs:
     - name: Upgrade HA broker and test
       run: |
         REPO=solace/solace-pubsub-standard
-        UPGRADETAG=$(wget -q https://registry.hub.docker.com/v1/repositories/solace/solace-pubsub-standard/tags -O -  | sed -e 's/[][]//g' -e 's/"//g' -e 's/ //g' | tr '}' '\n'  | awk -F: '{print $3}' | sort -t "." -k1,1n -k2,2n -k3,3n | tail -1)
-        helm upgrade my-release pubsubplus --set solace.size=dev,solace.redundancy=true,solace.podModifierEnabled=true,tls.enabled=true,tls.serverCertificatesSecret=test-tls,solace.usernameAdminPassword=admin,image.repository=$REPO,image.tag=$UPGRADETAG,storage.useStorageGroup=true
+        # grab a tag from Docker Hub that has the same SHA as "latest", so upgrade is easy
+        DOCKERHUBRESULTS=`curl --silent "https://hub.docker.com/v2/repositories/$REPO/tags?page_size=1000" | jq -r '.results[] | "\(.digest) \(.name)"' | sort`
+        SHA=`echo "$DOCKERHUBRESULTS" | grep latest | awk '{print $1;}'`
+        UPGRADETAG=`echo "$DOCKERHUBRESULTS" | grep $SHA | head -n 1 | awk '{print $2;}'`
+        helm upgrade my-release pubsubplus --set solace.size=dev,solace.redundancy=true,solace.podDisruptionBudgetForHA=true,solace.podModifierEnabled=true,tls.enabled=true,tls.serverCertificatesSecret=test-tls,solace.usernameAdminPassword=admin,image.repository=$REPO,image.tag=$UPGRADETAG,storage.useStorageGroup=true
         kubectl get statefulset,svc,pods,pvc,pv --show-labels
         echo "Waiting for broker to become active after upgrade"
         statefulset_name=$(kubectl get statefulset | grep pubsubplus |  awk '{print $1}')

README.md

@@ -6,7 +6,7 @@ The [Solace PubSub+ Platform](https://solace.com/products/platform/)'s [software
 
 ## Overview
 
-This project is a best practice template intended for development and demo purposes. The tested and recommended Solace PubSub+ Software Event Broker version is 9.10.
+This project is a best practice template intended for development and demo purposes. The tested and recommended Solace PubSub+ Software Event Broker version is 10.0.
 
 This document provides a quick getting started guide to install a software event broker in various configurations onto a [Kubernetes](https://kubernetes.io/docs/home/) cluster.
 

docs/PubSubPlusK8SDeployment.md

@@ -14,13 +14,15 @@ Contents:
     + [Deployment scaling](#deployment-scaling)
       - [Simplified vertical scaling](#simplified-vertical-scaling)
       - [Comprehensive vertical scaling](#comprehensive-vertical-scaling)
+      - [Enabling a Disruption Budget for HA deployment](#enabling-a-disruption-budget-for-ha-deployment)
       - [Reducing resource requirements of Monitoring Nodes in an HA deployment](#reducing-resource-requirements-of-monitoring-nodes-in-an-ha-deployment)
     + [Disk Storage](#disk-storage)
       - [Allocating smaller storage to Monitor pods in an HA deployment](#allocating-smaller-storage-to-monitor-pods-in-an-ha-deployment)
       - [Using the default or an existing storage class](#using-the-default-or-an-existing-storage-class)
       - [Creating a new storage class](#creating-a-new-storage-class)
       - [Using an existing PVC (Persistent Volume Claim)](#using-an-existing-pvc-persistent-volume-claim-)
       - [Using a pre-created provider-specific volume](#using-a-pre-created-provider-specific-volume)
+      - [Tested storage environments and providers](#tested-storage-environments-and-providers)
     + [Exposing the PubSub+ Event Broker Services](#exposing-the-pubsub-software-event-broker-services)
       - [Specifying Service Type](#specifying-service-type)
       - [Using Ingress to access event broker services](#using-ingress-to-access-event-broker-services)
@@ -89,6 +91,10 @@ There are two deployment options described in this document:
 * The recommended option is to use the [Kubernetes Helm tool](https://github.com/helm/helm/blob/master/README.md), which can also manage your deployment's lifecycle, including upgrade and delete.
 * Another option is to generate a set of templates with customized values from the PubSub+ Helm chart and then use the Kubernetes native `kubectl` tool to deploy. The deployment will use the authorizations of the requesting user. However, in this case, Helm will not be able to manage your Kubernetes rollouts lifecycle.
 
+It is also important to know that Helm is a templating tool that helps package PubSub+ Software Event Broker deployment into charts.
+It is most useful when first setting up broker nodes on the Kubernetes cluster. It can handle the install-update-delete lifecycle for the broker nodes deployed to the cluster.
+It can not be used to scale-up, scale down or apply custom configuration to an already deployed PubSub+ Software Event Broker.
+
 The next sections will provide details on the PubSub+ Helm chart, dependencies and customization options, followed by [deployment prerequisites](#deployment-prerequisites) and the actual [deployment steps](#deployment-steps).
 
 ## PubSub+ Software Event Broker Deployment Considerations
@@ -135,6 +141,17 @@ Note: beyond CPU and memory requirements, required storage size (see next sectio
 
 Also note, that specifying maxConnections, maxQueueMessages and maxSpoolUsage on initial deployment will overwrite the broker’s default values. On the other hand, doing the same using Helm upgrade on an existing deployment will not overwrite these values on brokers configuration, but it can be used to prepare (first step) for a manual scale up through CLI where these parameters can be actually changed (second step).
 
+#### Enabling a Disruption Budget for HA deployment
+
+One of the important parameters available to configure PubSub+ Software Event Broker HA is the [`podDisruptionBudget`](https://kubernetes.io/docs/tasks/run-application/configure-pdb/).
+This helps you control and limit the disruption to your application when its pods need to be rescheduled for upgrades, maintenance or any other reason.
+This is only available when we have the PubSub+ Software Event Broker deployed in [high-availability (HA) mode](//docs.solace.com/Overviews/SW-Broker-Redundancy-and-Fault-Tolerance.htm), that is, `solace.redundancy=true`.
+
+In an HA deployment with Primary, Backup and Monitor nodes, we require a minimum of 2 nodes to reach a quorum. The pod disruption budget defaults to a minimum of two nodes when enabled.
+
+To enable this functionality you have to set  `solace.podDisruptionBudgetForHA=true` and `solace.redundancy=true`.
+
+
 #### Reducing resource requirements of Monitoring Nodes in an HA deployment
 
 The Kubernetes StatefulSet which controls the pods that make up a PubSub+ broker [deployment in an HA redundancy group](#deployment-scaling) does not distinguish between PubSub+ HA node types: it assigns the same CPU and memory resources to pods hosting worker and monitoring node types, even though monitoring nodes have minimal resource requirements.
@@ -259,6 +276,12 @@ Another example is using [hostPath](//kubernetes.io/docs/concepts/storage/volume
       # this field is optional
       type: Directory
 ```
+#### Tested storage environments and providers
+
+The PubSub+ Software Event Broker has been tested to work with the following, Portworx, Ceph, Cinder (Openstack), vSphere storage for Kubernetes as documented [here](https://docs.solace.com/Cloud/Deployment-Considerations/resource-requirements-k8s.htm#supported-storage-solutions).
+However, note that for [EKS](https://docs.solace.com/Cloud/Deployment-Considerations/installing-ps-cloud-k8s-eks-specific-req.htm) and [GKE](https://docs.solace.com/Cloud/Deployment-Considerations/installing-ps-cloud-k8s-gke-specific-req.htm#storage-class), `xfs` produced the best results during tests.
+[AKS](https://docs.solace.com/Cloud/Deployment-Considerations/installing-ps-cloud-k8s-aks-specific-req.htm) users can opt for `Local Redundant Storage (LRS)` redundancy. This is because they produce the best results
+when compared with the other types available on Azure.
 
 ### Exposing the PubSub+ Software Event Broker Services
 

pubsubplus/Chart.yaml

@@ -1,7 +1,7 @@
 apiVersion: v2
 description: Deploy Solace PubSub+ Event Broker Singleton or HA redundancy group onto a Kubernetes Cluster
 name: pubsubplus
-version: 3.1.0
+version: 3.2.0
 icon: https://solaceproducts.github.io/pubsubplus-kubernetes-quickstart/images/PubSubPlus.png
 kubeVersion: '>= 1.10.0-0'
 maintainers:

pubsubplus/README.md

@@ -81,6 +81,7 @@ For more ways to override default chart values, refer to [Customizing the Helm C
 | `nameOverride`                 | Kubernetes objects will be named as `<release-name>-nameOverride`                                       | Undefined, default naming is `<release-name>-<chart-name>` |
 | `fullnameOverride`             | Kubernetes objects will be named as `fullnameOverride`                                                  | Undefined, default naming is `<release-name>-<chart-name>` |
 | `solace.redundancy`            | `false` will create a single-node non-HA deployment; `true` will create an HA deployment with Primary, Backup and Monitor nodes | `false` |
+| `solace.podDisruptionBudgetForHA`  | `true` will set up a [Pod disruption budget](https://kubernetes.io/docs/tasks/run-application/configure-pdb/) for the PubSub+ broker in HA deployment, `false` does not set up a pod disruption budget. HA deployment with Primary, Backup and Monitor nodes requires a minimum of 2 nodes to reach a quorum, the pod disruption budget is defaulted to `2` minimum nodes when enabled.                                       | `false`                |
 | `solace.size`                  | Event broker simple vertical scaling by number of client connections. **Ignored** if `solace.systemScaling` is set. Options: `dev` (requires minimum resources but no guaranteed performance), `prod100`, `prod1k`, `prod10k`, `prod100k`, `prod200k`. | `prod100` |
 | `solace.systemScaling.*`       | Event broker fine-grained vertical scaling definition. If defined, all sub-settings must be provided and these settings will **override** `solace.size`. For scaling documentation, look for "system scaling" at [docs.solace.com](https://docs.solace.com/Search.htm?q=system%20scaling). Use the [online calculator](https://docs.solace.com/Assistance-Tools/Resource-Calculator/pubsubplus-resource-calculator.html) to determine CPU, Memory and Storage requirements for "Container (messaging)" type. </br> `maxConnections`: max supported number of client connections </br> `maxQueueMessages`: max number of queue messages, in millions of messages </br> `maxSpoolUsage`: max Spool Usage, in MB. Also ensure adequate storage.size parameter, use the calculator </br> `cpu`: CPUs in cores </br> `memory`: host Virtual Memory, in MiB | Undefined |
 | `solace.podModifierEnabled`    | Enables modifying (reducing) CPU and memory resources for Monitoring nodes in an HA deployment. Also requires the ["solace-pod-modifier" Kubernetes admission plugin](https://github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/blob/master/solace-pod-modifier-admission-plugin/README.md#how-to-use) deployed to work.  | Undefined, meaning not enabled. |

pubsubplus/templates/podDisruptionBudget.yaml

@@ -0,0 +1,24 @@
+# PodDisruptionBudget for  Statefulsets
+{{- if .Values.solace.podDisruptionBudgetForHA }}
+{{- if .Values.solace.redundancy }}
+{{ if $.Capabilities.APIVersions.Has "policy/v1/PodDisruptionBudget" -}}
+apiVersion: policy/v1
+{{- else -}}
+apiVersion: policy/v1beta1
+{{- end }}
+kind: PodDisruptionBudget
+metadata:
+  name: {{ template "solace.fullname" . }}
+  labels:
+    app.kubernetes.io/name: {{ template "solace.name" . }}
+    app.kubernetes.io/instance: {{ .Release.Name }}
+    app.kubernetes.io/managed-by: {{ .Release.Service }}
+    helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version }}
+spec:
+  minAvailable: 2
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: {{ template "solace.name" . }}
+      app.kubernetes.io/instance: {{ .Release.Name }}
+{{- end }}
+{{- end }}