Fleet k8s deployment support with Terraform

k8s/.header.md

@@ -0,0 +1,286 @@
+## Fleet deployment with Terraform
+
+This deployment guide has been tested with
+- k3s 
+  - istio ingress
+  - nginx ingress
+
+### Usage
+
+#### 1. Create namespace
+
+This terraform will not auto-provision a namespace. You can add one with `kubectl create namespace <name>` or by creating a YAML file containing a service and applying it to your cluster. 
+
+#### 2. Create Secrets
+
+If you have a requirement to pull container images from a Private registry via `image_pull_secrets`, you can configure them using the instructions below. Additionally, you can instruct the terraform desployment to add the image_pull_secret via `module.fleet.image_pull_secrets`.
+
+```
+kubectl create secret docker-registry <secret_name> \
+  --docker-server=<your_private_registry_url> \
+  --docker-username=<your_user_name> \
+  --docker-password=<your_password> \
+  --docker-email=<your_email> \
+  --dry-run=client -o yaml
+```
+
+The output that is generated by the above command will generate a file that looks like the below format, you can copy and paste to save to a file or redirect the output directly into a file.
+
+```yaml
+apiVersion: v1
+kind: Secret
+metadata:
+  name: <secret_name>
+  namespace: <namespace_name>
+type: kubernetes.io/dockerconfigjson
+data:
+  .dockerconfigjson: <base64_encoded_config>
+```
+
+In order for the deployment to go through successfully, you'll need to create some secrets so Fleet knows how to authenticate against things like MySQL and Redis. While we'll only cover creating simple secrets, the deployment via terraform also supports instructing Fleet to pull the `valuesFrom:` an external secret store such as AWS Secrets Manager.
+
+```yaml
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: redis-password
+  namespace: <namespace_name>
+type: kubernetes.io/basic-auth
+stringData:
+  password: <redis-password-here>
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: mysql-password
+  namespace: <namespace_name>
+type: kubernetes.io/basic-auth
+stringData:
+  password: <mysql-password-here>
+```
+
+If you use Fleet's TLS capabilities, TLS connections to the MySQL server, or AWS access secret keys, additional secrets and keys are needed. The name of each `Secret` must match the value of `secret_name` for each section in `module.fleet` located in `main.tf`. The key of each secret must match the related key value from the values file. For example, to configure Fleet's TLS, you would use a secret like the one below.
+
+```yaml
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: fleet
+  namespace: <namespace_name>
+type: kubernetes.io/tls
+data:
+  tls.crt: |
+    <base64-encoded-tls-cert-here>
+  tls.key: |
+    <base64-encoded-tls-key-here>
+```
+
+If you have a Fleet premium license you would create a secret like the one below.
+
+```yaml
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: fleet-license
+  namespace: <namespace_name>
+data:
+  license: <fleet-license-here>
+```
+
+Once all of your secrets are configured, use `kubectl apply -f <secret_file_name.yaml> --namespace <your_namespace>` to configure them in the cluster.
+
+#### 3. Further Configuration
+
+To configure how Fleet runs, such as specifying the number of Fleet instances to deploy or changing the logger plugin for Fleet, edit the `module.fleet` located in `main.tf` file to your desired settings.
+
+##### nginx ingress
+
+Assuming no other ingress controllers are deployed to your environment, you can deploy nginx ingress components by executing `kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.7.1/deploy/static/provider/baremetal/deploy.yaml`
+
+You will need an nginx ingress `Service`, similar to the one below. To deploy run `kubectl apply -f <path_to_service_file>`.
+
+```yaml
+apiVersion: v1
+kind: Service
+metadata:
+  name: ingress-nginx-controller-loadbalancer
+  namespace: ingress-nginx
+spec:
+  selector:
+    app.kubernetes.io/component: controller
+    app.kubernetes.io/instance: ingress-nginx
+    app.kubernetes.io/name: ingress-nginx
+  ports:
+    - name: http
+      port: 80
+      protocol: TCP
+      targetPort: 80
+    - name: https
+      port: 443
+      protocol: TCP
+      targetPort: 443
+  type: LoadBalancer
+```
+
+The below will be in preparation for deployment through the `terraform apply` in step 5.
+
+In `main.tf` make sure the following variable `fleet.tls.enabled = false`, otherwise the Fleet terraform deployment will fail.
+In `main.tf` make sure the following map is configured with the correct values.
+  - `ingress.enabled` must be `true`, if you'd like Fleet to deploy the nginx ingress for you.
+  - `ingress.class_name` needs to be set to `nginx`
+  - `ingress.hosts.name` must have a matching entry in `ingress.tls.hosts`
+    - example: `ingress.hosts.name = fleet.example.com` and `ingress.tls.hosts = fleet.example.com`
+  - Last, `ingress.tls.secret_name` must be a valid secret name in your current namespace.*
+    - Note: The TLS must contain a valid certificate that matches the hostnames provided for `ingress.hosts.name` and `ingress.tls.hosts`.
+
+```
+...
+    ingress = {
+        enabled = true
+        class_name = "nginx"
+        annotations = {}
+        labels = {}
+        hosts = [{
+            name = "fleet.example.com"
+            paths = [{
+                path = "/"
+                path_type = "ImplementationSpecific"
+            }]
+        }]
+        tls = {
+            secret_name = "fleet-tls"
+            hosts = [
+                "fleet.example.com"
+            ]
+        }
+    }
+...
+```
+
+##### istio ingress
+
+There are different ways to deploy istio to your cluster. We will cover the helm deployment in the [official istio documentation](https://istio.io/latest/docs/setup/install/helm/).
+
+Assuming no other ingress controllers are deployed to your environment, you can deploy istio components by executing the following commands.
+
+```sh
+helm repo add istio https://istio-release.storage.googleapis.com/charts
+helm repo update
+kubectl create namespace istio-system
+helm install istio-base istio/base -n istio-system
+helm install istiod istio/istiod -n istio-system
+kubectl create namespace istio-ingress
+helm install istio-ingress istio/gateway -n istio-ingress --wait
+```
+
+In `main.tf` make sure the following variable `fleet.tls.enabled = false`, otherwise the Fleet terraform deployment will fail.
+
+You will need to create a TLS secret specifically for use by the istio ingress gateway like the example below or re-use Fleet secret created in step 2. You can apply the secret with the following command `kubectl apply -f <path_to_secret_yaml_file>`.
+
+
+```yaml
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: fleet
+  namespace: <namespace_name>
+type: kubernetes.io/tls
+data:
+  tls.crt: |
+    <base64-encoded-tls-cert-here>
+  tls.key: |
+    <base64-encoded-tls-cert-here>
+```
+
+After the secret has been created, you can create your istio ingress `Gateway` and istio `Virtual Service`. In the examble below you should make reference, in the `Gateway` and `VirtualService` to hostname (example: `fleet.example.com`) covered by your TLS certificate stored in the TLS secret created above (example: `fleet`).
+
+```yaml
+---
+apiVersion: networking.istio.io/v1
+kind: Gateway
+metadata:
+  name: istio-gateway-fleet
+  namespace: <namespace_name>
+spec:
+  selector:
+    istio: ingress # use istio default ingress gateway
+  servers:
+  - port:
+      number: 443
+      name: https
+      protocol: HTTPS
+    tls:
+      mode: SIMPLE
+      credentialName: fleet # must be the same as secret
+    hosts:
+    - 'fleet.example.com'
+---
+apiVersion: networking.istio.io/v1
+kind: VirtualService
+metadata:
+  name: fleet-vs
+  namespace: <namespace_name>
+spec:
+  hosts:
+  - "fleet.example.com"
+  gateways:
+  - istio-gateway-fleet
+  http:
+  - match:
+    - uri:
+        prefix: "/"
+    route:
+    - destination:
+        port:
+          number: 8080
+        host: fleet
+```
+
+#### 4. Setup provider.tf
+
+Setup your `provider.tf` with the correct credentials, whether it's for a self-hosted or managed service k8s deployment. The following [link to the kubernetes provider terraform docs](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/guides/getting-started.html) has examples documented for AWS EKS, GCP GKE, and Azure.
+
+```
+provider "kubernetes" {
+  # config_path = "/path/to/kubeconfig"
+  config_path = ""
+}
+```
+
+#### 5. Deploy Fleet
+
+From the location where your `main.tf` resides, execute the following commands.
+
+```sh
+terraform init
+terraform plan
+terraform apply
+```
+
+### Upgrade Fleet
+
+Fleet should not be running when an upgrade is initiated because database migrations need to take place first. After `main.tf` has been updated to increment the version of the Fleet `image_tag`, the following commands can be executed to upgrade Fleet while bringing Fleet down so migrations can run.
+
+```sh
+terraform init
+terraform apply -replace=module.fleet.kubernetes_deployment.fleet
+```
+
+### Remove Fleet
+
+#### 1. Tear down Fleet
+
+From the location where your `main.tf` resides, execute the following commands.
+
+```sh
+terraform init
+terraform destroy
+```
+
+#### 2. Remove all secrets
+
+Using the file configured for your secrets, use `kubectl delete -f <secret_file_name.yaml> --namespace <your_namespace>` to remove the secrets.

k8s/.terraform-docs.yml

@@ -0,0 +1 @@
+header-from: .header.md